voice-cat/docs/voice.md

Voice & Media

Real-time audio runs over UDP, secured per security.md. The control channel (TCP/TLS) handles signaling — announcing streams, channel membership, talk state — while UDP carries only the encoded audio frames. This split keeps media latency low and independent of TCP head-of-line blocking.

1. The multi-stream model

A user publishes one or more streams. Each stream is an independent audio source with its own encoder, its own stream_id (unique per user) and ssrc (media-plane id assigned by the server), and is independently mutable/mutable at the receiver.

        User "Alex"                              Receiver "Sam"
   ┌────────────────────┐                   ┌──────────────────────────┐
   │ mic       → enc ───┼──ssrc 1001──▶     │ jitter(1001)→dec→┐        │
   │ desktop   → enc ───┼──ssrc 1002──▶     │ jitter(1002)→dec→┤        │
   │ 2nd mic   → enc ───┼──ssrc 1003──▶     │ jitter(1003)→dec→┴─mix──▶ out
   └────────────────────┘                   └──────────────────────────┘

Stream kinds (v1): MIC, SCREEN_AUDIO (system/desktop audio for listening together), AUX_DEVICE (a second capture device). Receivers can set, per incoming stream: gain, mute, and noise reduction (see §10) — so Sam can turn down Alex's desktop audio while keeping the mic, and independently apply noise suppression to a third user who has a loud fan. The mixer sums all active streams from all users in the channel into the local playback device. All of these receiver-side controls are local to the listener and carry no protocol traffic.

SCREEN_AUDIO capture is platform-specific and covered in §9 — it is supported on Windows, macOS, and iOS (via a ReplayKit broadcast extension).

2. Voice frame format (UDP payload, inside the media AEAD)

A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.

 0      1      2      3      4      5      6      7      8 ...
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
│ type │flags │  codec      │     ssrc (u32)                            │
├──────┴──────┴──────┴──────┼──────┬──────┬──────┬──────┬──────────────┤
│  seq (u16)  │       timestamp (u32, in samples @48k)   │  payload ... │
└─────────────┴──────────────────────────────────────────┴──────────────┘

type    u8   1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
flags   u8   bit0 marker (start of talkspurt) · bit1 FEC-present
             bit2 DTX/comfort-noise · bit3 last-frame-before-stop
codec   u16  0 = OPUS  (room for future codecs)
ssrc    u32  media-plane stream id. Client sends its own ssrc; the server
             validates it against the bound session and relays unchanged.
seq     u16  per-ssrc sequence number, wraps; drives loss detection + reorder
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
payload      one Opus packet (the encoder's output for one frame)

This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's full machinery. The server relays the payload unmodified — it only reads the header to route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once assigned at StreamAnnounce). No server-side decode.

Why client-sends-ssrc is safe

The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are dropped. So identity is anchored by the session binding + transport encryption, not by trusting the header.

3. Per-channel audio configuration

Opus is configured per channel and pushed to clients in JoinChannelResult.audio / StreamAnnounceResult.effective_audio. All members of a channel encode with mutually decodable parameters.

message AudioConfig {
  uint32 codec = 1;               // 0 = OPUS
  ChannelMode mode = 2;           // MONO / STEREO
  uint32 sample_rate = 3;         // 8000/12000/16000/24000/48000 (48000 recommended)
  uint32 bitrate_bps = 4;         // e.g. 24000 (speech) … 128000 (music/stereo)
  uint32 frame_ms = 5;            // 2.5/5/10/20/40/60 (20 default)
  OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY
  bool   fec = 7;                 // in-band forward error correction
  uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness
  bool   dtx = 9;                 // discontinuous transmission (silence suppression)
  uint32 complexity = 10;         // 0..10 encoder complexity
}

Guidance baked into defaults / docs:

• Sample rate: always run Opus at 48 kHz internally. Opus resamples internally anyway; 48 kHz avoids surprises. The sample_rate field mainly constrains capture/narrowband modes for very low bitrate channels. Default 48000.
• Frame size: 20 ms default. Smaller (10 ms) lowers latency at the cost of more per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms.
• Mode/bitrate: speech channels → MONO, VOIP, 24–32 kbps, DTX on, FEC on. Music/screen-audio channels → STEREO, AUDIO, 96–128 kbps, DTX off, FEC optional.
• application: VOIP for talk, AUDIO for music/screen-share, LOWDELAY for monitoring use cases.

4. Packet-loss resilience (Opus 1.6)

Layered, all configurable per channel:

1. In-band FEC — Opus embeds a low-bitrate copy of the previous frame; the decoder recovers a lost packet from the next one (costs one frame of latency on recovery). Tuned by expected_packet_loss.
2. PLC (packet loss concealment) — decoder synthesizes a plausible frame for an unrecovered loss; always on, free.
3. DTX — sender stops transmitting during silence and sends sparse comfort-noise updates; cuts bandwidth and is bandwidth-friendly on busy channels.
4. DRED (Deep REDundancy, optional/feature-gated) — Opus 1.6's ML redundancy carries acoustic features so the decoder can reconstruct longer loss bursts. Heavier CPU; gate behind a features flag and per-channel toggle, off by default.

5. Jitter buffer

Each receiver keeps an adaptive jitter buffer per ssrc.

• Frames are inserted by timestamp; playback reads in order at the device callback rate.
• Target depth adapts to observed network jitter between a configurable min/max latency (channel-level "stability vs latency" knob). A "low-latency" channel runs a shallow buffer; a "stable" channel runs deeper.
• Late frames past the playout point are dropped; gaps are filled by FEC (if the next frame arrived) or PLC.
• The marker flag (start of talkspurt) lets the buffer resynchronize cleanly after silence/DTX without accumulating drift.

 incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer
                                     ▲
                             jitter estimate feeds depth

6. UDP keepalive & NAT

• A KEEPALIVE (type 2) frame flows both directions on the media channel every ~5 s to hold NAT bindings and measure media-path RTT/loss independent of TCP.
• If the media path dies but TCP is alive, the client surfaces a "voice disconnected" state and attempts UDP re-binding (re-derive media keys + fresh UdpBinding) without dropping the control session.
• No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the server is reachable (public IP or port-forward); clients sit behind NAT and initiate, so their bindings are created by their outbound first packet.

7. Talk-state signaling

"Who is talking" can be derived two ways; we use both:

• Implicit: presence of recent voice frames for an ssrc → that stream is "active". The receiver drives talk indicators from the jitter buffer, so they're accurate and need no extra messages.
• Explicit (optional): StreamStateUpdate on TCP for coarse UI state (muted, hold) and for users not currently subscribed to the media. Server-side mute/deafen is authoritative and always signaled on TCP.

8. Capture/playback pipeline (inside the core)

 device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
        → Opus encode → frame header → AEAD → UDP send

 UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
        → per-stream recv-side NS (optional, per user) → per-stream gain/mute
        → mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
          48k, stereo) → device

• Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA). Playback is genuinely stereo end-to-end; capture stays mono (no stereo mic in v1).
• DSP engine: see §11. The original plan was webrtc-audio-processing (AEC + NS + AGC + VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream (confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard abseil-cpp dependency, Linux-tested only — gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1). v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is no AEC, NS, or AGC implementation at all yet — not just a deferred VAD, the whole APM is unbuilt. Real webrtc-audio-processing stays a tracked future swap, behind the same ApmProcessor interface (core/src/audio/apm_processor.h), revisit if/when a Linux build target exists or upstream Windows support matures.
• The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.

10. Noise reduction — two-sided

Noise reduction can be applied at the sender, at the listener, or both — they are independent.

• Sender-side (the talker's choice): the publishing client runs APM noise suppression on its mic before encoding, controlled by that user's own settings. This cleans the signal for everyone and saves bitrate.
• Listener-side, per user (the listener's choice): on the receive path, after decoding each stream and before mixing, the listener can enable an additional NS pass on a specific sender's stream. So even if Alex chose not to denoise his mic, Sam can locally suppress Alex's background noise without affecting how anyone else hears Alex.

Implementation: a per-ssrc APM NS instance on the receive path, instantiated lazily only for streams the listener has flagged. State lives entirely on the listener's machine; toggling it is a local UI action with no protocol message and no effect on other listeners. Because each receive stream is decoded independently before the mixer (voice.md §1), per-user receive NS is a clean drop-in on that per-stream stage.

11. Input activation — VAD and PTT (client-configurable)

Whether the mic transmits is decided locally by the input gate, and the client supports both modes, switchable per client (vc_set_input_mode):

• Voice activation (VAD): v1 implements this as a lightweight, dependency-free energy/RMS-threshold VAD (EnergyVadProcessor, core/src/audio/apm_processor.cpp) — no external DSP dependency, since real webrtc-audio-processing has no working Windows/MSVC build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold (default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms, matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping word tails. DTX naturally complements this — when the gate is closed nothing (or only comfort noise) is sent. This implementation has no AEC — a real limitation versus the originally-planned APM, not just a deferred VAD.
• Push-to-talk (PTT): vc_set_push_to_talk(active) opens/closes the gate directly. The UI exposes a configurable keybind; the core just receives gate open/close.

Gating applies to the MIC stream only — SCREEN_AUDIO/AUX_DEVICE always bypass it (gating a desktop-audio share on the user's own voice activity would silently drop shared music/video audio whenever the user isn't talking, which defeats the feature).

This is purely a send-side, client-local concern — it gates what gets encoded and sent. It needs no protocol support; remote talk indicators are still derived from the presence of received frames (§7), so they work identically under VAD or PTT.

9. System / screen audio capture (SCREEN_AUDIO)

"Listen together" needs to capture the audio another app is playing. The capture mechanism differs per OS, but it always feeds the same Opus-encode → media-AEAD → UDP path as a normal stream; only the source is platform-specific.

Platform	Mechanism	Notes
Windows	WASAPI loopback capture of the default render endpoint (via miniaudio's loopback mode)	Implemented. Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (AUDIOCLIENT_ACTIVATION_PARAMS) would avoid this but miniaudio doesn't expose it — a future enhancement.
macOS	ScreenCaptureKit system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes	OS requires screen-recording permission; capture happens in the main app.
iOS	ReplayKit Broadcast Upload Extension (the Discord mechanism)	See below — separate process, App Group, ~50 MB cap (fine for audio-only).

iOS detail

• The user starts a broadcast from Control Center's screen-record button; we surface it via RPSystemBroadcastPickerView from inside the app for one-tap start.
• The Broadcast Upload Extension receives RPSampleBufferType.audioApp (system/app audio) and .audioMic. We consume .audioApp for SCREEN_AUDIO and drop the video buffers entirely — video is what blows the ~50 MB extension memory budget, so an audio-only consumer stays comfortably inside it.
• The extension is a separate process. It links a minimal slice of the core (Opus encode + media send only — not the full client), reads the active session token and server endpoint from a shared App Group container that the host app wrote at join time, derives its own media keys, and publishes the SCREEN_AUDIO stream directly. The host app announces the stream over its control channel (StreamAnnounce) so the server and peers learn about it.
• Mic + voice continue to run in the host app; only the system-audio share lives in the extension. When the broadcast stops (broadcastFinished), the extension sends a final frame with the last-frame-before-stop flag and the host app emits StreamStop.