Files
voice-cat/docs/voice.md

378 lines
26 KiB
Markdown
Raw Normal View History

# Voice & Media
Real-time audio runs over **UDP**, secured per [security.md](security.md). The control
channel (TCP/TLS) handles *signaling* — announcing streams, channel membership, talk state
— while UDP carries only the encoded audio frames. This split keeps media latency low and
independent of TCP head-of-line blocking.
## 1. The multi-stream model
A **user** publishes one or more **streams**. Each stream is an independent audio source
with its own encoder, its own `stream_id` (unique per user) and `ssrc` (media-plane id
assigned by the server), and is independently mutable/mutable at the receiver.
```
User "Alex" Receiver "Sam"
┌────────────────────┐ ┌──────────────────────────┐
│ mic → enc ───┼──ssrc 1001──▶ │ jitter(1001)→dec→┐ │
│ desktop → enc ───┼──ssrc 1002──▶ │ jitter(1002)→dec→┤ │
│ 2nd mic → enc ───┼──ssrc 1003──▶ │ jitter(1003)→dec→┴─mix──▶ out
└────────────────────┘ └──────────────────────────┘
```
Stream **kinds** (v1): `MIC`, `SCREEN_AUDIO` (system/desktop audio for listening together),
`AUX_DEVICE` (a second capture device). Receivers can set, per incoming stream: **gain**,
**mute**, and **noise reduction** (see §10) — so Sam can turn down Alex's desktop audio
while keeping the mic, *and* independently apply noise suppression to a third user who has a
loud fan. The mixer sums all active streams from all users in the channel into the local
playback device. All of these receiver-side controls are **local to the listener** and carry
no protocol traffic.
`SCREEN_AUDIO` capture is platform-specific and covered in §9 — it is supported on Windows,
macOS, **and iOS** (via a ReplayKit broadcast extension).
## 2. Voice frame format (UDP payload, inside the media AEAD)
A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.
The header is **20 bytes** (protocol v2; v1 was 14 bytes with a u16 seq — see note below).
```
0 1 2 3 4 5 6 7 8 ............ 15
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
│ type │flags │ codec │ ssrc (u32) │ seq (u64) ──▶ │
├──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴───────────────┤
│ ◀── seq (u64) ──┤ timestamp (u32 @48k) │ payload ... │
└──────────────────┴────────────────────────────────────┴───────────────┘
bytes [8..15] = seq (u64) [16..19] = timestamp (u32)
type u8 1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present
bit2 DTX/comfort-noise · bit3 last-frame-before-stop
codec u16 0 = OPUS (room for future codecs)
ssrc u32 media-plane stream id. Client sends its own ssrc; the server
validates it against the bound session and relays unchanged.
seq u64 full monotonic send counter. This IS the AEAD nonce counter, so the
receiver derives the nonce directly from it — no rollover guessing.
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
payload one Opus packet (the encoder's output for one frame)
```
> **Why u64 (protocol v2).** v1 carried only the low 16 bits of the counter and the
> receiver zero-extended them to rebuild the AEAD nonce. After 65,536 frames the seq
> wrapped, the reconstructed nonce diverged from the sealing nonce, and **every frame
> failed authentication permanently** (no rollover counter). v2 puts the full 64-bit
> counter on the wire so the nonce is always exact. A v2 server and a v1 client cannot
> interoperate; the `Hello` handshake rejects on `proto_version` mismatch.
This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's
full machinery. The **server relays the payload unmodified** — it only reads the header to
route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once
assigned at `StreamAnnounce`). No server-side decode.
### Why client-sends-ssrc is safe
The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks
that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are
dropped. So identity is anchored by the session binding + transport encryption, not by
trusting the header.
## 3. Per-channel audio configuration
Opus is configured **per channel** and pushed to clients in `JoinChannelResult.audio` /
`StreamAnnounceResult.effective_audio`. All members of a channel encode with mutually
decodable parameters.
```proto
message AudioConfig {
uint32 codec = 1; // 0 = OPUS
ChannelMode mode = 2; // MONO / STEREO
uint32 sample_rate = 3; // 8000/12000/16000/24000/48000 (48000 recommended)
uint32 bitrate_bps = 4; // e.g. 24000 (speech) … 128000 (music/stereo)
uint32 frame_ms = 5; // 2.5/5/10/20/40/60 (20 default)
OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY
bool fec = 7; // in-band forward error correction
uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness
bool dtx = 9; // discontinuous transmission (silence suppression)
uint32 complexity = 10; // 0..10 encoder complexity
}
```
Guidance baked into defaults / docs:
- **Sample rate: always run Opus at 48 kHz internally.** Opus resamples internally anyway;
48 kHz avoids surprises, and the whole audio stack (capture, `vc_stream_feed_pcm`, mixing,
playback) runs at 48 kHz. The per-channel `sample_rate` field is **channel-authoritative**
(not a client request) and does *not* change the codec/PCM clock — it caps the encoder's
audio bandwidth via `OPUS_SET_MAX_BANDWIDTH` (8000 → narrowband ~4 kHz, 16000 → wideband
~8 kHz, 24000 → super-wideband ~12 kHz, 48000 → full ~20 kHz). This lets a low-bitrate room
shed out-of-band content while every endpoint keeps a single 48 kHz clock. Default **48000**
(full band). See `OpusEncoder::init` and `vc_client::opus_params_from_audio_config`.
- **Frame size: 20 ms default.** Smaller (10 ms) lowers latency at the cost of more
per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at
the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms.
The capture engine runs on a fixed 48 kHz / 20 ms clock (960-sample frames), so the send
path **reframes** each captured/fed block to the channel's `frame_ms` before encoding
(accumulating two 960-frames for a 40 ms channel, splitting each into two 480-frames for a
10 ms channel, etc.). This keeps the hardware/`vc_stream_feed_pcm` contract a single 48 kHz
clock regardless of the channel's window — see `vc_client::on_capture_frame`.
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 2432 kbps, DTX on, FEC on.
Music/screen-audio channels → `STEREO`, `AUDIO`, 96128 kbps, DTX off, FEC optional.
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
monitoring use cases.
## 4. Packet-loss resilience (Opus 1.6)
Layered, all configurable per channel:
1. **In-band FEC** — Opus embeds a low-bitrate copy of the previous frame; the decoder
recovers a lost packet from the *next* one (costs one frame of latency on recovery).
Tuned by `expected_packet_loss`.
2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an
unrecovered loss; always on, free.
3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise
updates; cuts bandwidth and is bandwidth-friendly on busy channels.
4. **DRED (Deep REDundancy, per-channel toggle)** — Opus 1.6's ML redundancy: the encoder
embeds 20 ms of acoustic features in every packet (`bool dred` in `AudioConfig`, off by
default). When a packet is lost, the receiver peeks at the next already-buffered packet,
parses its DRED extension (`opus_dred_parse`), and reconstructs the lost frame with
`opus_decoder_dred_decode` — producing significantly better audio than PLC comfort noise
for single-frame gaps. Falls back silently to PLC if the next packet has not arrived yet
or if the sender did not embed DRED. Heavier CPU on the encoder (~510 % at 24 kbps);
minimal overhead on the decoder (parse is a fast header check on non-DRED packets).
## 5. Jitter buffer
Each receiver keeps an **adaptive jitter buffer per ssrc**.
- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
- Target depth adapts to observed network jitter between a configurable **min/max latency**
(channel-level "stability vs latency" knob). A "low-latency" channel runs a shallow
buffer; a "stable" channel runs deeper.
- Late frames past the playout point are dropped; gaps are filled by FEC (if the next frame
arrived) or PLC.
- The `marker` flag (start of talkspurt) lets the buffer resynchronize cleanly after
silence/DTX without accumulating drift.
```
incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer
jitter estimate feeds depth
```
## 6. UDP keepalive & NAT
- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to
fix(net): broadcast LEFT on disconnect, add keepalive/reaper, cap PLC hiss Three reported bugs traced to one root cause plus two missing designed features: 1. Stale users + eternal PLC hiss (root cause): ConnSession::close() silently erased dropped users without broadcasting UserEvent::LEFT, so peers never learned the user left and their audio engines never called remove_stream — Opus PLC synthesized comfort noise forever. Fix: broadcast_left() helper + close() broadcasts LEFT before erasing. 2. PLC cap (defense-in-depth): on_playback now caps pure PLC at ~2s, then emits digital silence so a stale stream can never hiss forever even if remove_stream is skipped. Resets automatically on fresh packets. 3. No timeout / no ping: client never sent Ping, server had no last_seen / reaper, so half-open connections (NAT timeout, wifi loss, sleep) left ghost users forever. Fix: client Ping every 15s with RTT measurement, ConnSession::last_seen bumped on every inbound TCP/UDP frame, steady_timer reaper sweeps every 15s and drops sessions older than 45s (configurable via server::Config). 4. UDP KEEPALIVE: client sends plaintext kFrameKeepalive every 5s; server bumps last_seen + echoes back. Keeps NAT bindings alive and lets media activity defer the reaper independently of TCP. 5. Graceful client disconnect: vc_disconnect() sends Disconnect{code=0} via a flag-based io-thread exit (no double-close race); server handles client-sent Disconnect with immediate close() + LEFT broadcast. 3 new tests: disconnect_left, plc_cap, reaper_timeout. 21/21 ctest green. Docs: protocol.md §6/§7, voice.md §6, architecture.md §5, PROGRESS.md.
2026-06-18 01:18:33 +02:00
hold NAT bindings and measure media-path RTT/loss independent of TCP. The frame is
plaintext (14-byte header, no payload, no AEAD) — the server identifies the sender by
its already-verified UDP endpoint (established during the `UdpBinding` handshake). On
receipt the server bumps the sender's `last_seen` (so media activity defers the TCP
reaper independently of control-channel traffic) and echoes the frame back so the
client can measure media-path RTT.
- If the media path dies but TCP is alive, the client surfaces a "voice disconnected"
state and attempts UDP re-binding (re-derive media keys + fresh `UdpBinding`) without dropping
the control session.
- No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the **server** is reachable
(public IP or port-forward); **clients** sit behind NAT and initiate, so their bindings
are created by their outbound first packet.
## 7. Talk-state signaling
"Who is talking" can be derived two ways; we use both:
- **Implicit:** presence of recent voice frames for an ssrc → that stream is "active". The
receiver drives talk indicators from the jitter buffer, so they're accurate and need no
extra messages.
- **Explicit (optional):** `StreamStateUpdate` on TCP for coarse UI state (muted, hold) and
for users not currently subscribed to the media. Server-side mute/deafen is authoritative
and always signaled on TCP.
## 8. Capture/playback pipeline (inside the core)
```
mic device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
→ Opus encode → frame header → AEAD → UDP send
screen audio ─(WASAPI loopback, 48k, mono or stereo per channel mode)→ Opus encode
→ frame header → AEAD → UDP send
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
→ mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
48k, stereo) → device
```
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
feat(ios): audio overhaul, Join/Leave Voice, channel-id sync fix, stereo mic capture Three iOS client problems fixed plus a new core stereo-mic capture ABI: 1. Channel-id sync bug (mic button permanently dimmed): SessionState never synced currentChannelId from the self user's channelId on connect, so the mic button (gated on currentChannelId == 0) stayed dimmed. Added syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522); called from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult. Added applyServerMuteState() + serverMuted/serverDeafened to VoiceState. 2. Join/Leave Voice button: replaced icon-only mic toggle with explicit text button (parity with macOS). Mute/deafen disable when not in voice. 3. IOSAudioRouter.swift (new): full AVAudioSession routing layer — input port selection, built-in mic orientation/polar patterns, Bluetooth HFP/A2DP/Off modes, Standard/Raw mic processing, stereo capture, AirPlay, UserDefaults persistence. AudioSessionManager delegates to it. 4. Core stereo-mic capture (append-only ABI): vc_set_capture_channels() lets the core open the mic device in stereo (2-ch interleaved). LocalStream gains capture_channels; ensure_audio_running reads it; audio_engine.cpp capture_accum_ + on_capture updated to channel-aware accumulation. Test test_stereo_mic_capture (headless, L!=R stereo round-trip). Swift wrapper VoiceCatClient.setCaptureChannels. 5. Settings UI rework: AVAudioSession-derived input/output tree replaces miniaudio device picker. 6. iOS deployment target raised to 18.0 (Package.swift + project.pbxproj). swift-tools-version 6.0 with swiftLanguageModes .v5. Docs: tech-stack.md, architecture.md, voice.md, roadmap.md, building.md updated; stale 'vc_audio_suspend/resume deferred' claims corrected. Verified: ctest --preset dev 21/21 green; swift test 6/6 green; xcodebuild -target VoiceCatiOS -sdk iphonesimulator BUILD SUCCEEDED.
2026-06-19 13:17:52 +02:00
Playback is genuinely stereo end-to-end. **Mic capture** is mono by default; **stereo mic
fix(ios): stereo mic + A2DP output, add vc_audio_restart ABI Diagnosed by comparing against TeamTalk5 (Client/iTeamTalk), which achieves stereo mic + A2DP output. Five fixes: 1. configureStereoCapture now calls setPreferredInput + setInputDataSource (mirroring TeamTalk5's SoundDevicesModel). Previously omitted based on incorrect diagnosis that setPreferredInput collapsed A2DP — the real culprit was setPreferredInputNumberOfChannels(2), which neither project uses. 2. New C ABI: vc_audio_restart (full stop + re-init, unlike suspend/resume which only stop/start). Swift wrapper added. The withAudioSuspend wrapper that used it was removed after on-device testing showed it killed all audio (including VoiceOver) when switching presets — the core's set_capture_channels handles engine restart internally. 3. Bluetooth options: Voice Chat preset now includes BOTH .allowBluetoothHFP AND .allowBluetoothA2DP (matching TeamTalk5's UtilSound.swift:228). Previously HFP-only blocked A2DP headphones. 4. Capture channels now reset when switching stereo→mono via selectCaptureChannels/applyPreset. AudioSessionManager tracks activeMicStreamId (set by SessionState on join/leave voice). 5. Docs synced: voice.md, tech-stack.md, architecture.md, PROGRESS.md. Removed stale setPreferredInputNumberOfChannels(2) references. Verified: ctest --preset dev 21/21 green, iOS client builds. Stereo mic + A2DP output still needs on-device debugging — the core recipe is correct but iOS 26 route behavior requires hands-on testing with a debugger.
2026-06-19 16:58:21 +02:00
capture** is supported via `vc_set_capture_channels(stream_id, 2)` — when enabled, the
capture device opens in stereo (interleaved L/R) and the encoder receives real stereo PCM
(no upmix). A mono mic frame on a stereo channel is upmixed L=R before encoding so the Opus
bitstream is still spec-correct stereo. **Screen-audio (`SCREEN_AUDIO`) loopback** captures
in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
feat(ios): audio overhaul, Join/Leave Voice, channel-id sync fix, stereo mic capture Three iOS client problems fixed plus a new core stereo-mic capture ABI: 1. Channel-id sync bug (mic button permanently dimmed): SessionState never synced currentChannelId from the self user's channelId on connect, so the mic button (gated on currentChannelId == 0) stayed dimmed. Added syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522); called from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult. Added applyServerMuteState() + serverMuted/serverDeafened to VoiceState. 2. Join/Leave Voice button: replaced icon-only mic toggle with explicit text button (parity with macOS). Mute/deafen disable when not in voice. 3. IOSAudioRouter.swift (new): full AVAudioSession routing layer — input port selection, built-in mic orientation/polar patterns, Bluetooth HFP/A2DP/Off modes, Standard/Raw mic processing, stereo capture, AirPlay, UserDefaults persistence. AudioSessionManager delegates to it. 4. Core stereo-mic capture (append-only ABI): vc_set_capture_channels() lets the core open the mic device in stereo (2-ch interleaved). LocalStream gains capture_channels; ensure_audio_running reads it; audio_engine.cpp capture_accum_ + on_capture updated to channel-aware accumulation. Test test_stereo_mic_capture (headless, L!=R stereo round-trip). Swift wrapper VoiceCatClient.setCaptureChannels. 5. Settings UI rework: AVAudioSession-derived input/output tree replaces miniaudio device picker. 6. iOS deployment target raised to 18.0 (Package.swift + project.pbxproj). swift-tools-version 6.0 with swiftLanguageModes .v5. Docs: tech-stack.md, architecture.md, voice.md, roadmap.md, building.md updated; stale 'vc_audio_suspend/resume deferred' claims corrected. Verified: ctest --preset dev 21/21 green; swift test 6/6 green; xcodebuild -target VoiceCatiOS -sdk iphonesimulator BUILD SUCCEEDED.
2026-06-19 13:17:52 +02:00
- **iOS mic capture:** all iOS audio routing is driven from Swift via `AVAudioSession` by the
`IOSAudioRouter` singleton *before* the core (miniaudio) opens its device — miniaudio does
NOT touch `AVAudioSession` on iOS. Input port selection (`availableInputs`), built-in mic
orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns
(`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic processing mode
(`.voiceChat` = Standard with AEC/AGC/HPF, or `.measurement` = Raw/Studio with all processing
fix(ios): stereo mic + A2DP output, add vc_audio_restart ABI Diagnosed by comparing against TeamTalk5 (Client/iTeamTalk), which achieves stereo mic + A2DP output. Five fixes: 1. configureStereoCapture now calls setPreferredInput + setInputDataSource (mirroring TeamTalk5's SoundDevicesModel). Previously omitted based on incorrect diagnosis that setPreferredInput collapsed A2DP — the real culprit was setPreferredInputNumberOfChannels(2), which neither project uses. 2. New C ABI: vc_audio_restart (full stop + re-init, unlike suspend/resume which only stop/start). Swift wrapper added. The withAudioSuspend wrapper that used it was removed after on-device testing showed it killed all audio (including VoiceOver) when switching presets — the core's set_capture_channels handles engine restart internally. 3. Bluetooth options: Voice Chat preset now includes BOTH .allowBluetoothHFP AND .allowBluetoothA2DP (matching TeamTalk5's UtilSound.swift:228). Previously HFP-only blocked A2DP headphones. 4. Capture channels now reset when switching stereo→mono via selectCaptureChannels/applyPreset. AudioSessionManager tracks activeMicStreamId (set by SessionState on join/leave voice). 5. Docs synced: voice.md, tech-stack.md, architecture.md, PROGRESS.md. Removed stale setPreferredInputNumberOfChannels(2) references. Verified: ctest --preset dev 21/21 green, iOS client builds. Stereo mic + A2DP output still needs on-device debugging — the core recipe is correct but iOS 26 route behavior requires hands-on testing with a debugger.
2026-06-19 16:58:21 +02:00
off), Bluetooth mode (`.allowBluetoothHFP` HFP voice vs `.allowBluetoothA2DP` stereo output
vs neither), and stereo capture (`.stereo` polar pattern + `setPreferredInput` +
`setInputDataSource``vc_set_capture_channels`) are all set from Swift. The core then
opens whatever route AVAudioSession has established. When the user changes audio settings
mid-session, `IOSAudioRouter` suspends the core's devices (`vc_audio_suspend`), reconfigures
`AVAudioSession`, then restarts the devices (`vc_audio_restart`) so they reopen against the
new route — mirroring TeamTalk5's `closeSoundDevices`/`initSoundInputDevice`/
`initSoundOutputDevice` pattern.
- **iOS voice processing (AEC/NS/AGC) — native VPIO path.** Real iOS echo cancellation, noise
suppression and AGC are provided ONLY by Apple's **Voice-Processing I/O audio unit (VPIO)**,
*not* by the `AVAudioSession` mode alone. The core uses miniaudio's plain `RemoteIO` audio
units, which never engage VPIO — so `.voiceChat` mode by itself yields no AEC. For VPIO to
cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
played-back signal from the mic), so on the AEC presets (Voice Chat / Bluetooth Headset HFP /
Wired Headset) the Swift layer runs a native `AVAudioEngine` with
`inputNode.setVoiceProcessingEnabled(true)` and the core runs in **external mode**:
- **Mic:** the MIC stream is started with `vc_stream_desc.external_feed=1`; the VPIO input tap
feeds processed mic PCM via `vc_stream_feed_pcm`. The core skips its hardware capture device
(`AudioParams.external_capture`).
- **Playback:** `vc_set_external_playback(1)` makes the core skip its hardware playback device;
a mixer-timer thread drives decode+mix on a ~20 ms cadence and delivers the FINAL mixed PCM
via `vc_set_mixed_output_sink`. The Swift engine renders that through the VPIO output, so
VPIO has its echo-cancellation reference signal.
The Stereo Mic / Studio / A2DP presets keep the miniaudio path (they want raw / stereo /
no-AEC routing that VPIO can't provide — VPIO forces mono).
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
`abseil-cpp` dependency, Linux-tested only —
[gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)).
v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS,
or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real
`webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor`
interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or
upstream Windows support matures.
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
## 10. Noise reduction — two-sided
Noise reduction can be applied **at the sender, at the listener, or both** — they are
independent.
- **Sender-side** (the talker's choice): the publishing client runs APM noise suppression on
its mic before encoding, controlled by that user's own settings. This cleans the signal for
*everyone* and saves bitrate.
- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding
each stream and *before* mixing, the listener can enable an **additional** NS pass on a
**specific** sender's stream. So even if Alex chose not to denoise his mic, Sam can locally
suppress Alex's background noise without affecting how anyone else hears Alex.
Implementation: a per-`ssrc` APM NS instance on the receive path, instantiated lazily only
for streams the listener has flagged. State lives entirely on the listener's machine; toggling
it is a local UI action with **no protocol message** and no effect on other listeners. Because
each receive stream is decoded independently before the mixer (voice.md §1), per-user receive
NS is a clean drop-in on that per-stream stage.
All three receive-side controls (gain, mute, NR) are queryable via `vc_get_remote_stream`
the counterpart to `vc_set_remote_stream` — so a UI can reopen its per-stream mix controls at
the listener's actual current settings (defaults: gain 1.0, unmuted, NR off). Like the setter,
it carries no protocol traffic.
## 11. Input activation — VAD and PTT (client-configurable)
Whether the mic transmits is decided locally by the **input gate**, and the client supports
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
**both** modes, switchable per client (`vc_set_input_mode`):
- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free
energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no
external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC
build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold
(default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms,
matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping
word tails. DTX naturally complements this — when the gate is closed nothing (or only
comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the
originally-planned APM, not just a deferred VAD.
- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI
exposes a configurable keybind; the core just receives gate open/close.
Gating applies to the **MIC stream only**`SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
(gating a desktop-audio share on the user's own voice activity would silently drop shared
music/video audio whenever the user isn't talking, which defeats the feature).
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
needs **no protocol support**; remote talk indicators are still derived from the presence of
received frames (§7), so they work identically under VAD or PTT.
## 9. System / screen audio capture (`SCREEN_AUDIO`)
"Listen together" needs to capture the audio another app is playing. The capture mechanism
differs per OS, but it always feeds the **same** Opus-encode → media-AEAD → UDP path as a
normal stream; only the *source* is platform-specific.
| Platform | Mechanism | Notes |
|----------|-----------|-------|
| **Windows** | **WASAPI loopback** (whole-device, via miniaudio) **or WASAPI process loopback** (`AUDIOCLIENT_ACTIVATION_PARAMS`, Win10 2004+) for per-app / self-exclude | **Implemented.** Default *entire desktop* uses miniaudio's whole-device loopback in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when mono — so a stereo channel gets genuine stereo end-to-end (no downmix). It inherently captures this app's own incoming voice mix (self-echo). The **per-app modes and the "exclude VoiceCat's own audio" option** instead drive `ProcessLoopbackCapture` (process-specific INCLUDE/EXCLUDE) through the external-feed mixer (`vc_stream_feed_pcm`, `external_feed=1`), which avoids self-echo and supports true "everything except". See below. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). **Supports per-app audio selection** — see below. |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). ReplayKit only ever delivers the *mixed* system stream as `.audioApp`, so **per-app filtering / VoiceOver exclusion is not possible on iOS** (it has no per-app granularity, unlike ScreenCaptureKit). |
### macOS detail — per-app audio selection
ScreenCaptureKit filters audio at the **application** level, so before sharing starts the user
picks a scope in `ScreenSharePickerSheet` (`clients/apple/macOS/VoiceCatMac/Sheets/`):
- **Everything** — whole display, the original behaviour (`SCContentFilter(display:excludingWindows:)`).
- **Only selected apps** — capture just the ticked apps (`init(display:including:exceptingWindows:)`).
- **All except selected apps** — capture everything but the ticked apps
(`init(display:excludingApplications:exceptingWindows:)`).
A dedicated **"Exclude screen reader (VoiceOver) audio"** toggle merges the screen-reader
process(es) into the exclude set (`ScreenAudioCapture.screenReaderBundleIDs` — VoiceOver plus
the speech-synthesis daemon that actually renders the spoken audio). The chosen
`ScreenAudioSelection` is passed into `ScreenAudioCapture`, which builds the matching
`SCContentFilter`. iOS/ReplayKit has no equivalent control (see the table note above).
### Windows detail — per-app audio selection and self-echo
`AppAudioPickerDialog` (`clients/windows/VoiceCat.App/Forms/`) offers the same shape as macOS:
- **Entire desktop** — whole-device miniaudio loopback handled by the core (default path).
- **Only selected apps** — one `ProcessLoopbackCapture` in **INCLUDE** mode per ticked app,
mixed by `ProcessAudioMixer` and fed via `vc_stream_feed_pcm`.
- **All apps except selected** — a **single** `ProcessLoopbackCapture` in **EXCLUDE** mode of
the chosen process tree. WASAPI's `AUDIOCLIENT_PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE`
captures the whole render mix minus that tree *dynamically* (apps launched after sharing
starts are included automatically). The activation params take a **single** target PID, so
exclude is restricted to **one** app — the picker enforces single-selection in this mode.
An **"Exclude VoiceCat's own audio (prevents echo)"** checkbox (default on, enabled for
*entire desktop*) routes the desktop capture through the same EXCLUDE path targeting
VoiceCat's **own** process id (`Environment.ProcessId`) — i.e. "entire desktop except this
app" — which removes the self-echo loop the whole-device path otherwise has. The per-app
INCLUDE modes already never capture this app's tree, so they have no self-echo to remove.
### iOS detail
The extension **captures**, the host app **sends**. Unlike a self-connecting extension, this
keeps a **single session** — the screen-audio share appears as a second stream of the *same*
user (exactly like macOS/Windows), and no credentials are ever persisted to disk.
- The user starts a broadcast from Control Center's screen-record button; we surface it via
`RPSystemBroadcastPickerView` from inside the app (`VoiceControlsView`) for one-tap start.
- The **Broadcast Upload Extension** (`clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift`)
receives `RPSampleBufferType.audioApp` (system/app audio), `.audioMic`, and `.video`. We
consume **`.audioApp`** only and drop video + mic — video is what blows the **~50 MB**
extension memory budget, so an audio-only consumer stays comfortably inside it. The extension
does **not** link `libvoicecat`.
- The extension converts each chunk to the core's canonical format (48 kHz int16 stereo, via
`AVAudioConverter`) and writes it into a lock-free single-producer/single-consumer ring in a
shared **App Group** mmap'd file (`clients/apple/iOS/Shared/BroadcastAudioRing.swift`). It
posts Darwin notifications on start/stop so the host reacts promptly.
- The **host app** owns the stream: its `BroadcastAudioPump` announces the `SCREEN_AUDIO`
stream over the control channel (`StreamAnnounce`), drains the ring, and calls
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the Opus
encode + AEAD + send path. It downmixes to mono when the channel's effective config is mono.
- Mic + voice also run in the host app. When the broadcast stops (`broadcastFinished`), the
extension clears the ring's active flag (and posts a Darwin notification); the host stops
feeding and emits `StreamStop`. The host must be alive to relay — always true while in a
call (the app declares the `audio` background mode).