voice-cat/docs/voice.md

# Voice & Media

Real-time audio runs over **UDP**, secured per [security.md](security.md). The control
channel (TCP/TLS) handles *signaling* — announcing streams, channel membership, talk state
— while UDP carries only the encoded audio frames. This split keeps media latency low and
independent of TCP head-of-line blocking.

## 1. The multi-stream model

A **user** publishes one or more **streams**. Each stream is an independent audio source
with its own encoder, its own `stream_id` (unique per user) and `ssrc` (media-plane id
assigned by the server), and is independently mutable/mutable at the receiver.

```
        User "Alex"                              Receiver "Sam"
   ┌────────────────────┐                   ┌──────────────────────────┐
   │ mic       → enc ───┼──ssrc 1001──▶     │ jitter(1001)→dec→┐        │
   │ desktop   → enc ───┼──ssrc 1002──▶     │ jitter(1002)→dec→┤        │
   │ 2nd mic   → enc ───┼──ssrc 1003──▶     │ jitter(1003)→dec→┴─mix──▶ out
   └────────────────────┘                   └──────────────────────────┘
```

Stream **kinds** (v1): `MIC`, `SCREEN_AUDIO` (system/desktop audio for listening together),
`AUX_DEVICE` (a second capture device). Receivers can set, per incoming stream: **gain**,
**mute**, and **noise reduction** (see §10) — so Sam can turn down Alex's desktop audio
while keeping the mic, *and* independently apply noise suppression to a third user who has a
loud fan. The mixer sums all active streams from all users in the channel into the local
playback device. All of these receiver-side controls are **local to the listener** and carry
no protocol traffic.

`SCREEN_AUDIO` capture is platform-specific and covered in §9 — it is supported on Windows,
macOS, **and iOS** (via a ReplayKit broadcast extension).

## 2. Voice frame format (UDP payload, inside the media AEAD)

A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.

The header is **20 bytes** (protocol v2; v1 was 14 bytes with a u16 seq — see note below).

```
 0      1      2      3      4      5      6      7      8 ............ 15
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
│ type │flags │  codec      │     ssrc (u32)            │  seq (u64) ──▶ │
├──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴───────────────┤
│  ◀── seq (u64) ──┤       timestamp (u32 @48k)         │  payload ...   │
└──────────────────┴────────────────────────────────────┴───────────────┘
   bytes [8..15] = seq (u64)         [16..19] = timestamp (u32)

type    u8   1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
flags   u8   bit0 marker (start of talkspurt) · bit1 FEC-present
             bit2 DTX/comfort-noise · bit3 last-frame-before-stop
codec   u16  0 = OPUS  (room for future codecs)
ssrc    u32  media-plane stream id. Client sends its own ssrc; the server
             validates it against the bound session and relays unchanged.
seq     u64  full monotonic send counter. This IS the AEAD nonce counter, so the
             receiver derives the nonce directly from it — no rollover guessing.
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
payload      one Opus packet (the encoder's output for one frame)
```

> **Why u64 (protocol v2).** v1 carried only the low 16 bits of the counter and the
> receiver zero-extended them to rebuild the AEAD nonce. After 65,536 frames the seq
> wrapped, the reconstructed nonce diverged from the sealing nonce, and **every frame
> failed authentication permanently** (no rollover counter). v2 puts the full 64-bit
> counter on the wire so the nonce is always exact. A v2 server and a v1 client cannot
> interoperate; the `Hello` handshake rejects on `proto_version` mismatch.

This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's
full machinery. The **server relays the payload unmodified** — it only reads the header to
route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once
assigned at `StreamAnnounce`). No server-side decode.

### Why client-sends-ssrc is safe

The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks
that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are
dropped. So identity is anchored by the session binding + transport encryption, not by
trusting the header.

## 3. Per-channel audio configuration

Opus is configured **per channel** and pushed to clients in `JoinChannelResult.audio` /
`StreamAnnounceResult.effective_audio`. All members of a channel encode with mutually
decodable parameters.

```proto
message AudioConfig {
  uint32 codec = 1;               // 0 = OPUS
  ChannelMode mode = 2;           // MONO / STEREO
  uint32 sample_rate = 3;         // 8000/12000/16000/24000/48000 (48000 recommended)
  uint32 bitrate_bps = 4;         // e.g. 24000 (speech) … 128000 (music/stereo)
  uint32 frame_ms = 5;            // 2.5/5/10/20/40/60 (20 default)
  OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY
  bool   fec = 7;                 // in-band forward error correction
  uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness
  bool   dtx = 9;                 // discontinuous transmission (silence suppression)
  uint32 complexity = 10;         // 0..10 encoder complexity
}
```

Guidance baked into defaults / docs:

- **Sample rate: always run Opus at 48 kHz internally.** Opus resamples internally anyway;
  48 kHz avoids surprises, and the whole audio stack (capture, `vc_stream_feed_pcm`, mixing,
  playback) runs at 48 kHz. The per-channel `sample_rate` field is **channel-authoritative**
  (not a client request) and does *not* change the codec/PCM clock — it caps the encoder's
  audio bandwidth via `OPUS_SET_MAX_BANDWIDTH` (8000 → narrowband ~4 kHz, 16000 → wideband
  ~8 kHz, 24000 → super-wideband ~12 kHz, 48000 → full ~20 kHz). This lets a low-bitrate room
  shed out-of-band content while every endpoint keeps a single 48 kHz clock. Default **48000**
  (full band). See `OpusEncoder::init` and `vc_client::opus_params_from_audio_config`.
- **Frame size: 20 ms default.** Smaller (10 ms) lowers latency at the cost of more
  per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at
  the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms.
  The capture engine runs on a fixed 48 kHz / 20 ms clock (960-sample frames), so the send
  path **reframes** each captured/fed block to the channel's `frame_ms` before encoding
  (accumulating two 960-frames for a 40 ms channel, splitting each into two 480-frames for a
  10 ms channel, etc.). This keeps the hardware/`vc_stream_feed_pcm` contract a single 48 kHz
  clock regardless of the channel's window — see `vc_client::on_capture_frame`.
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 24–32 kbps, DTX on, FEC on.
  Music/screen-audio channels → `STEREO`, `AUDIO`, 96–128 kbps, DTX off, FEC optional.
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
  monitoring use cases.

## 4. Packet-loss resilience (Opus 1.6)

Layered, all configurable per channel:

1. **In-band FEC** — the encoder embeds a low-bitrate copy of the current frame in the
   *next* packet (`OPUS_SET_INBAND_FEC`, redundancy scaled by `expected_packet_loss`). On a
   loss, the receiver decodes that copy out of the next already-buffered packet with
   `opus_decode(..., decode_fec=1)` — costing one frame of latency on recovery. Gated on the
   per-stream `fec` flag; if the next packet carries no redundancy libopus yields PLC output,
   so it is at worst a no-op relative to (2).
2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an
   unrecovered loss; always on, free. The terminal fallback when neither DRED nor FEC applies.
3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise
   updates; cuts bandwidth and is bandwidth-friendly on busy channels.
4. **DRED (Deep REDundancy, per-channel toggle)** — Opus 1.6's ML redundancy: the encoder
   embeds 20 ms of acoustic features in every packet (`bool dred` in `AudioConfig`, off by
   default). When a packet is lost, the receiver peeks at the next already-buffered packet,
   parses its DRED extension (`opus_dred_parse`), and reconstructs the lost frame with
   `opus_decoder_dred_decode` — producing significantly better audio than PLC comfort noise
   for single-frame gaps. Heavier CPU on the encoder (~5–10 % at 24 kbps); minimal overhead
   on the decoder (parse is a fast header check on non-DRED packets).

When a frame is lost, `AudioEngine::on_playback` tries these recovery paths in quality order,
falling through on failure: **DRED → in-band FEC → PLC**. DRED and FEC both need the next
packet already buffered (one frame of look-ahead); when it has not arrived yet, recovery falls
straight through to PLC.

## 5. Jitter buffer

Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth playout**
(`core/src/audio/audio_engine.cpp` — `JitterBuffer` + `AudioEngine::on_playback`).

- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
- **The playout clock is always bounded against the stream's *leading edge* (newest buffered
  frame), never re-synced to the oldest.** The clock free-runs at the playback hardware rate,
  while the sender omits VAD/PTT/DTX silence from its timestamps, so the two diverge across gaps
  and late joins. Two corrections keep latency bounded:
  - **(Re)seed to the leading edge** on first frame, on a talkspurt `marker`, or when the clock
    has run past the newest frame (starved after silence). No artificial prebuffer — latency
    starts as low as possible; buffered frames still play oldest-first.
  - **Frame-skip catch-up:** when the backlog grows past `target + hysteresis` (clock drift,
    bursty arrival, reordering), fast-forward the clock to leave `target` buffered and drop the
    now-stale frames. This is the downward force that prevents latency from ratcheting upward.
- `target` is the adaptive jitter estimate (EWMA of inter-arrival gap vs. the per-frame gap),
  floored; silence gaps and reordered stragglers are rejected as outliers so they don't inflate
  it. The late-drop window tracks `target` (floored/capped at 500 ms).
- Late frames past the playout point are dropped; gaps are filled by DRED (if the next frame
  arrived) or PLC.
- The `marker` flag (start of talkspurt) — set by the sender on the first frame after a
  transmission gap — lets the buffer reseed cleanly after silence/DTX without accumulating drift.
- Diagnostics per stream: `packets_lost`, `duplicates`, `underruns`, `target_depth_ms`.

```
 incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer
                                     ▲
                             jitter estimate feeds depth
```

## 6. UDP keepalive & NAT

- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to
  hold NAT bindings and measure media-path RTT/loss independent of TCP. The frame is
  plaintext (14-byte header, no payload, no AEAD) — the server identifies the sender by
  its already-verified UDP endpoint (established during the `UdpBinding` handshake). On
  receipt the server bumps the sender's `last_seen` (so media activity defers the TCP
  reaper independently of control-channel traffic) and echoes the frame back so the
  client can measure media-path RTT.
- If the media path dies but TCP is alive, the client surfaces a "voice disconnected"
  state and attempts UDP re-binding (re-derive media keys + fresh `UdpBinding`) without dropping
  the control session.
- No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the **server** is reachable
  (public IP or port-forward); **clients** sit behind NAT and initiate, so their bindings
  are created by their outbound first packet.

## 7. Talk-state signaling

"Who is talking" can be derived two ways; we use both:

- **Implicit:** presence of recent voice frames for an ssrc → that stream is "active". The
  receiver drives talk indicators from the jitter buffer, so they're accurate and need no
  extra messages.
- **Explicit (optional):** `StreamStateUpdate` on TCP for coarse UI state (muted, hold) and
  for users not currently subscribed to the media. Server-side mute/deafen is authoritative
  and always signaled on TCP.

## 8. Capture/playback pipeline (inside the core)

```
 mic device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
        → Opus encode → frame header → AEAD → UDP send

 screen audio ─(WASAPI loopback, 48k, mono or stereo per channel mode)→ Opus encode
        → frame header → AEAD → UDP send

 UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
         → per-stream recv-side NS (optional, per user) → per-stream gain/mute
         → mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
           48k, stereo) → device
```

- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
  Playback is genuinely stereo end-to-end. **Mic capture** is mono by default; **stereo mic
  capture** is supported via `vc_set_capture_channels(stream_id, 2)` — when enabled, the
  capture device opens in stereo (interleaved L/R). All native clients expose this as a
  per-user toggle (iOS in Settings; the Windows and macOS desktop clients via a "Stereo
  microphone" checkbox in Audio settings — a live toggle there restarts the capture device via
  `vc_audio_restart` so it takes effect immediately). Whether stereo actually reaches the wire
  depends on the **channel's** Opus mode, which decides the encoder's channel count — the mic's
  channel count and the channel's mode are independent knobs:
  - **stereo mic + stereo channel** → real interleaved L/R is encoded directly (no upmix).
  - **mono mic + stereo channel** → the mono frame is upmixed L=R so the Opus bitstream is
    still spec-correct stereo.
  - **stereo mic + mono channel** → the interleaved L/R is folded to mono before the mono
    encoder. (Handing interleaved pairs straight to a mono `opus_encode` would make it read 2×
    the samples it should — wrong pitch / garbage — so the fold keeps the toggle safe on any
    channel.)
  **Screen-audio (`SCREEN_AUDIO`) loopback** captures
  in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
  downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
  genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
- **iOS audio — one path, always external.** On iOS the core **never opens a miniaudio device**:
  a single `AVAudioEngine` (`IOSAudioEngine`) drives *both* directions, and the core runs fully
  external for the whole connection. This is the single most important property of the iOS audio
  stack — there is no second (miniaudio) path to switch to, so a preset/route change cannot leave
  one direction dropped. The single ordering rule is: `vc_set_external_playback(1)` is set **once
  at connect** (before the session is activated or any remote stream arrives), and every MIC
  stream is started with `vc_stream_desc.external_feed=1`.
  - **core → speaker:** the core's mixer-timer thread decodes+mixes on a ~20 ms cadence and
    delivers the FINAL mixed PCM via `vc_set_mixed_output_sink`; an `AVAudioSourceNode` pulls it
    from a lock-free ring and renders it. This runs the whole time we are connected, so remote
    audio plays even before the user joins voice (kills the "can't hear anyone" race).
  - **mic → core:** when the mic is active a tap on the engine's input node converts to 48 kHz
    int16 (`vc_set_capture_channels` decides mono/stereo) and calls `vc_stream_feed_pcm`.
- **iOS routing** is still driven from Swift via `AVAudioSession` by the `IOSAudioRouter`
  singleton — miniaudio never touches `AVAudioSession` on iOS. Input port selection
  (`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom),
  polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic
  processing mode (Standard vs `.measurement` Raw), Bluetooth mode (`.allowBluetoothHFP` HFP voice
  vs `.allowBluetoothA2DP` stereo output vs neither), and stereo capture (`.stereo` polar pattern
  + `setPreferredInput` + `setInputDataSource` → `vc_set_capture_channels`) are all set from
  Swift. Any preset / route / interruption change funnels through one deterministic, Swift-only
  rebuild: `IOSAudioEngine` stops, `IOSAudioRouter.applyConfiguration()` re-applies the
  `AVAudioSession`, the graph is rebuilt against the new route, and the engine restarts. No
  `vc_audio_restart`/`vc_audio_suspend` dance is needed for routing (the core has no hardware
  devices to reopen) — this is the spirit of TeamTalk5's "close then re-init sound devices", but
  entirely inside the Swift engine.
- **iOS voice processing (AEC/NS/AGC) — native VPIO.** Real iOS echo cancellation, noise
  suppression and AGC come ONLY from Apple's **Voice-Processing I/O audio unit (VPIO)**, which
  `inputNode.setVoiceProcessingEnabled(true)` enables; for it to cancel echo it must own BOTH the
  mic capture and the playback — which the unified engine already does. VPIO forces **mono**, so
  it is engaged only when the active config wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`:
  mono + standard + non-A2DP + the user's master toggle). iOS exposes no per-stage VPIO control,
  so the Advanced UI offers exactly two switches: a master **Voice Processing** (AEC + NS bundled)
  and **AGC** (`isVoiceProcessingAGCEnabled`).
- **iOS presets** (`IOSAudioRouter.AudioPreset`): **Voice Chat** (VPIO mono, system output incl.
  HFP/wired), **Stereo Mic** (internal stereo built-in mic regardless of output, A2DP-capable, no
  VPIO), **Mono Mic** (internal mono built-in mic regardless of output, A2DP-capable, no VPIO),
  and **Advanced** (every knob manual). A2DP output requires an internal-mic preset (the Bluetooth
  device is output-only); the Stereo/Mono Mic presets fall back to the built-in speaker when no
  external output is connected (`applyA2dpSpeakerFallback`).
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
  VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
  (confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
  `abseil-cpp` dependency, Linux-tested only —
  [gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)).
  v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS,
  or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real
  `webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor`
  interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or
  upstream Windows support matures.
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.

## 10. Noise reduction — two-sided

Noise reduction can be applied **at the sender, at the listener, or both** — they are
independent.

- **Sender-side** (the talker's choice): the publishing client runs noise suppression on its
  mic before the input gain and the VAD/PTT gate, controlled by that user's own settings
  (`vc_set_input_noise_reduction`). This cleans the signal for *everyone* in one pass and helps
  bitrate/VAD. MIC stream only.
- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding
  each stream and *before* mixing, the listener can enable an **additional** NS pass on a
  **specific** sender's stream (`vc_set_remote_stream(..., noise_reduction)`). So even if Alex
  chose not to denoise his mic, Sam can locally suppress Alex's background noise without
  affecting how anyone else hears Alex.

**Backend: RNNoise** (vendored in [`third_party/rnnoise/`](../third_party/rnnoise), BSD-3 + CC0).
The original plan was WebRTC's APM, but `webrtc-audio-processing` has no working Windows/MSVC
build (see §8). RNNoise is a small, dependency-free C library — a hybrid DSP/RNN speech denoiser
that runs ~60× faster than real time. Both NR paths share one `ApmProcessor` implementation
(`RnnoiseProcessor`, `core/src/audio/apm_processor.cpp`), selected by `ApmProcessor::create()`
when the core is built with `VOICECAT_HAS_NS` (a no-op `ApmPassthrough` otherwise). Allocation
happens at construction; `process_capture()` runs lock-free on the RT thread (architecture.md §3).

RNNoise is a **mono, 48 kHz, 480-sample (10 ms)** denoiser. Our engine clock is fixed at 48 kHz
and every Opus frame size (480/960/1920/2880) is a multiple of 480, so frames are processed as
whole 480-sample chunks with no resampling. Because it's mono-only:
- **Send-side:** a stereo mic is downmixed to mono **only when NR is enabled** — with NR off a
  stereo mic keeps full stereo (we never collapse mic quality unless asked).
- **Receive-side:** NR is skipped on stereo streams (a stereo stream is a screen-audio share,
  not voice).

Implementation: a per-`ssrc` NS instance (`RemoteStream::recv_ns`) on the receive path,
instantiated lazily only for streams the listener has flagged; the send-side instance
(`vc_client::mic_ns_`) is built once with the MIC stream and gated by an atomic flag so toggling
never allocates on the capture callback. State lives entirely on the local machine; toggling
either is a local UI action with **no protocol message** and no effect on other users. Because
each receive stream is decoded independently before the mixer (voice.md §1), per-user receive
NS is a clean drop-in on that per-stream stage.

All three receive-side controls (gain, mute, NR) are queryable via `vc_get_remote_stream` —
the counterpart to `vc_set_remote_stream` — so a UI can reopen its per-stream mix controls at
the listener's actual current settings (defaults: gain 1.0, unmuted, NR off). Like the setter,
it carries no protocol traffic.

## 11. Input activation — VAD and PTT (client-configurable)

Whether the mic transmits is decided locally by the **input gate**, and the client supports
**both** modes, switchable per client (`vc_set_input_mode`):

- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free
  energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no
  external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC
  build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold
  (default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms,
  matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping
  word tails. DTX naturally complements this — when the gate is closed nothing (or only
  comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the
  originally-planned APM, not just a deferred VAD.
- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI
  exposes a configurable keybind; the core just receives gate open/close.

Gating applies to the **MIC stream only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
(gating a desktop-audio share on the user's own voice activity would silently drop shared
music/video audio whenever the user isn't talking, which defeats the feature).

This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
needs **no protocol support**; remote talk indicators are still derived from the presence of
received frames (§7), so they work identically under VAD or PTT.

## 9. System / screen audio capture (`SCREEN_AUDIO`)

"Listen together" needs to capture the audio another app is playing. The capture mechanism
differs per OS, but it always feeds the **same** Opus-encode → media-AEAD → UDP path as a
normal stream; only the *source* is platform-specific.

| Platform | Mechanism | Notes |
|----------|-----------|-------|
| **Windows** | **WASAPI loopback** (whole-device, via miniaudio) **or WASAPI process loopback** (`AUDIOCLIENT_ACTIVATION_PARAMS`, Win10 2004+) for per-app / self-exclude | **Implemented.** Default *entire desktop* uses miniaudio's whole-device loopback in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when mono — so a stereo channel gets genuine stereo end-to-end (no downmix). It inherently captures this app's own incoming voice mix (self-echo). The **per-app modes and the "exclude VoiceCat's own audio" option** instead drive `ProcessLoopbackCapture` (process-specific INCLUDE/EXCLUDE) through the external-feed mixer (`vc_stream_feed_pcm`, `external_feed=1`), which avoids self-echo and supports true "everything except". See below. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). **Supports per-app audio selection** — see below. |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). ReplayKit only ever delivers the *mixed* system stream as `.audioApp`, so **per-app filtering / VoiceOver exclusion is not possible on iOS** (it has no per-app granularity, unlike ScreenCaptureKit). |

### macOS detail — per-app audio selection

ScreenCaptureKit filters audio at the **application** level, so before sharing starts the user
picks a scope in `ScreenSharePickerSheet` (`clients/apple/macOS/VoiceCatMac/Sheets/`):

- **Everything** — whole display, the original behaviour (`SCContentFilter(display:excludingWindows:)`).
- **Only selected apps** — capture just the ticked apps (`init(display:including:exceptingWindows:)`).
- **All except selected apps** — capture everything but the ticked apps
  (`init(display:excludingApplications:exceptingWindows:)`).

A dedicated **"Exclude screen reader (VoiceOver) audio"** toggle merges the screen-reader
process(es) into the exclude set (`ScreenAudioCapture.screenReaderBundleIDs` — VoiceOver plus
the speech-synthesis daemon that actually renders the spoken audio). The chosen
`ScreenAudioSelection` is passed into `ScreenAudioCapture`, which builds the matching
`SCContentFilter`. iOS/ReplayKit has no equivalent control (see the table note above).

### Windows detail — per-app audio selection and self-echo

`AppAudioPickerDialog` (`clients/windows/VoiceCat.App/Forms/`) offers the same shape as macOS:

- **Entire desktop** — whole-device miniaudio loopback handled by the core (default path).
- **Only selected apps** — one `ProcessLoopbackCapture` in **INCLUDE** mode per ticked app,
  mixed by `ProcessAudioMixer` and fed via `vc_stream_feed_pcm`.
- **All apps except selected** — a **single** `ProcessLoopbackCapture` in **EXCLUDE** mode of
  the chosen process tree. WASAPI's `AUDIOCLIENT_PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE`
  captures the whole render mix minus that tree *dynamically* (apps launched after sharing
  starts are included automatically). The activation params take a **single** target PID, so
  exclude is restricted to **one** app — the picker enforces single-selection in this mode.

An **"Exclude VoiceCat's own audio (prevents echo)"** checkbox (default on, enabled for
*entire desktop*) routes the desktop capture through the same EXCLUDE path targeting
VoiceCat's **own** process id (`Environment.ProcessId`) — i.e. "entire desktop except this
app" — which removes the self-echo loop the whole-device path otherwise has. The per-app
INCLUDE modes already never capture this app's tree, so they have no self-echo to remove.

### iOS detail

The extension **captures**, the host app **sends**. Unlike a self-connecting extension, this
keeps a **single session** — the screen-audio share appears as a second stream of the *same*
user (exactly like macOS/Windows), and no credentials are ever persisted to disk.

- The user starts a broadcast from Control Center's screen-record button; we surface it via
  `RPSystemBroadcastPickerView` from inside the app (`VoiceControlsView`) for one-tap start.
- The **Broadcast Upload Extension** (`clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift`)
  receives `RPSampleBufferType.audioApp` (system/app audio), `.audioMic`, and `.video`. We
  consume **`.audioApp`** only and drop video + mic — video is what blows the **~50 MB**
  extension memory budget, so an audio-only consumer stays comfortably inside it. The extension
  does **not** link `libvoicecat`.
- The extension converts each chunk to the core's canonical format (48 kHz int16 stereo, via
  `AVAudioConverter`) and writes it into a lock-free single-producer/single-consumer ring in a
  shared **App Group** mmap'd file (`clients/apple/iOS/Shared/BroadcastAudioRing.swift`). It
  posts Darwin notifications on start/stop so the host reacts promptly.
- The **host app** owns the stream: its `BroadcastAudioPump` announces the `SCREEN_AUDIO`
  stream over the control channel (`StreamAnnounce`), drains the ring, and calls
  `vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the Opus
  encode + AEAD + send path. It downmixes to mono when the channel's effective config is mono.
- Mic + voice also run in the host app. When the broadcast stops (`broadcastFinished`), the
  extension clears the ring's active flag (and posts a Darwin notification); the host stops
  feeding and emits `StreamStop`. The host must be alive to relay — always true while in a
  call (the app declares the `audio` background mode).