docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation: - README: overview, locked decisions, principles, glossary - architecture: shared C++ core + C ABI, native UIs (Swift/C#), threading model, server design (SFU relay) - protocol: TCP/TLS control plane, protobuf Envelope + message catalog, connection lifecycle, extensibility rules - voice: UDP media frame format, per-channel Opus config, multi-stream model, two-sided noise reduction, VAD/PTT, jitter buffer, iOS ReplayKit screen-audio - security: mandatory encryption (TLS 1.3 + exported-key AEAD), TOFU server identity, admin-provisioned accounts, anti-replay - tech-stack: permissive-only deps (mbedTLS, libsodium, opus, miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL - deployment: zero-config self-host (Docker / binary / source) - roadmap: M0-M5 milestones, resolved decisions Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
240
docs/voice.md
Normal file
240
docs/voice.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# Voice & Media
|
||||
|
||||
Real-time audio runs over **UDP**, secured per [security.md](security.md). The control
|
||||
channel (TCP/TLS) handles *signaling* — announcing streams, channel membership, talk state
|
||||
— while UDP carries only the encoded audio frames. This split keeps media latency low and
|
||||
independent of TCP head-of-line blocking.
|
||||
|
||||
## 1. The multi-stream model
|
||||
|
||||
A **user** publishes one or more **streams**. Each stream is an independent audio source
|
||||
with its own encoder, its own `stream_id` (unique per user) and `ssrc` (media-plane id
|
||||
assigned by the server), and is independently mutable/mutable at the receiver.
|
||||
|
||||
```
|
||||
User "Alex" Receiver "Sam"
|
||||
┌────────────────────┐ ┌──────────────────────────┐
|
||||
│ mic → enc ───┼──ssrc 1001──▶ │ jitter(1001)→dec→┐ │
|
||||
│ desktop → enc ───┼──ssrc 1002──▶ │ jitter(1002)→dec→┤ │
|
||||
│ 2nd mic → enc ───┼──ssrc 1003──▶ │ jitter(1003)→dec→┴─mix──▶ out
|
||||
└────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
Stream **kinds** (v1): `MIC`, `SCREEN_AUDIO` (system/desktop audio for listening together),
|
||||
`AUX_DEVICE` (a second capture device). Receivers can set, per incoming stream: **gain**,
|
||||
**mute**, and **noise reduction** (see §10) — so Sam can turn down Alex's desktop audio
|
||||
while keeping the mic, *and* independently apply noise suppression to a third user who has a
|
||||
loud fan. The mixer sums all active streams from all users in the channel into the local
|
||||
playback device. All of these receiver-side controls are **local to the listener** and carry
|
||||
no protocol traffic.
|
||||
|
||||
`SCREEN_AUDIO` capture is platform-specific and covered in §9 — it is supported on Windows,
|
||||
macOS, **and iOS** (via a ReplayKit broadcast extension).
|
||||
|
||||
## 2. Voice frame format (UDP payload, inside the media AEAD)
|
||||
|
||||
A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.
|
||||
|
||||
```
|
||||
0 1 2 3 4 5 6 7 8 ...
|
||||
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
|
||||
│ type │flags │ codec │ ssrc (u32) │
|
||||
├──────┴──────┴──────┴──────┼──────┬──────┬──────┬──────┬──────────────┤
|
||||
│ seq (u16) │ timestamp (u32, in samples @48k) │ payload ... │
|
||||
└─────────────┴──────────────────────────────────────────┴──────────────┘
|
||||
|
||||
type u8 1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
|
||||
flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present
|
||||
bit2 DTX/comfort-noise · bit3 last-frame-before-stop
|
||||
codec u16 0 = OPUS (room for future codecs)
|
||||
ssrc u32 media-plane stream id. Client sends its own ssrc; the server
|
||||
validates it against the bound session and relays unchanged.
|
||||
seq u16 per-ssrc sequence number, wraps; drives loss detection + reorder
|
||||
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
|
||||
payload one Opus packet (the encoder's output for one frame)
|
||||
```
|
||||
|
||||
This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's
|
||||
full machinery. The **server relays the payload unmodified** — it only reads the header to
|
||||
route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once
|
||||
assigned at `StreamAnnounce`). No server-side decode.
|
||||
|
||||
### Why client-sends-ssrc is safe
|
||||
|
||||
The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks
|
||||
that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are
|
||||
dropped. So identity is anchored by the session binding + transport encryption, not by
|
||||
trusting the header.
|
||||
|
||||
## 3. Per-channel audio configuration
|
||||
|
||||
Opus is configured **per channel** and pushed to clients in `JoinChannelResult.audio` /
|
||||
`StreamAnnounceResult.effective_audio`. All members of a channel encode with mutually
|
||||
decodable parameters.
|
||||
|
||||
```proto
|
||||
message AudioConfig {
|
||||
uint32 codec = 1; // 0 = OPUS
|
||||
ChannelMode mode = 2; // MONO / STEREO
|
||||
uint32 sample_rate = 3; // 8000/12000/16000/24000/48000 (48000 recommended)
|
||||
uint32 bitrate_bps = 4; // e.g. 24000 (speech) … 128000 (music/stereo)
|
||||
uint32 frame_ms = 5; // 2.5/5/10/20/40/60 (20 default)
|
||||
OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY
|
||||
bool fec = 7; // in-band forward error correction
|
||||
uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness
|
||||
bool dtx = 9; // discontinuous transmission (silence suppression)
|
||||
uint32 complexity = 10; // 0..10 encoder complexity
|
||||
}
|
||||
```
|
||||
|
||||
Guidance baked into defaults / docs:
|
||||
|
||||
- **Sample rate: always run Opus at 48 kHz internally.** Opus resamples internally anyway;
|
||||
48 kHz avoids surprises. The `sample_rate` field mainly constrains capture/narrowband
|
||||
modes for very low bitrate channels. Default **48000**.
|
||||
- **Frame size: 20 ms default.** Smaller (10 ms) lowers latency at the cost of more
|
||||
per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at
|
||||
the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms.
|
||||
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 24–32 kbps, DTX on, FEC on.
|
||||
Music/screen-audio channels → `STEREO`, `AUDIO`, 96–128 kbps, DTX off, FEC optional.
|
||||
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
|
||||
monitoring use cases.
|
||||
|
||||
## 4. Packet-loss resilience (Opus 1.6)
|
||||
|
||||
Layered, all configurable per channel:
|
||||
|
||||
1. **In-band FEC** — Opus embeds a low-bitrate copy of the previous frame; the decoder
|
||||
recovers a lost packet from the *next* one (costs one frame of latency on recovery).
|
||||
Tuned by `expected_packet_loss`.
|
||||
2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an
|
||||
unrecovered loss; always on, free.
|
||||
3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise
|
||||
updates; cuts bandwidth and is bandwidth-friendly on busy channels.
|
||||
4. **DRED (Deep REDundancy, optional/feature-gated)** — Opus 1.6's ML redundancy carries
|
||||
acoustic features so the decoder can reconstruct longer loss bursts. Heavier CPU; gate
|
||||
behind a `features` flag and per-channel toggle, off by default.
|
||||
|
||||
## 5. Jitter buffer
|
||||
|
||||
Each receiver keeps an **adaptive jitter buffer per ssrc**.
|
||||
|
||||
- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
|
||||
- Target depth adapts to observed network jitter between a configurable **min/max latency**
|
||||
(channel-level "stability vs latency" knob). A "low-latency" channel runs a shallow
|
||||
buffer; a "stable" channel runs deeper.
|
||||
- Late frames past the playout point are dropped; gaps are filled by FEC (if the next frame
|
||||
arrived) or PLC.
|
||||
- The `marker` flag (start of talkspurt) lets the buffer resynchronize cleanly after
|
||||
silence/DTX without accumulating drift.
|
||||
|
||||
```
|
||||
incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer
|
||||
▲
|
||||
jitter estimate feeds depth
|
||||
```
|
||||
|
||||
## 6. UDP keepalive & NAT
|
||||
|
||||
- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to
|
||||
hold NAT bindings and measure media-path RTT/loss independent of TCP.
|
||||
- If the media path dies but TCP is alive, the client surfaces a "voice disconnected"
|
||||
state and attempts UDP re-binding (re-derive media keys + fresh `UdpBinding`) without dropping
|
||||
the control session.
|
||||
- No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the **server** is reachable
|
||||
(public IP or port-forward); **clients** sit behind NAT and initiate, so their bindings
|
||||
are created by their outbound first packet.
|
||||
|
||||
## 7. Talk-state signaling
|
||||
|
||||
"Who is talking" can be derived two ways; we use both:
|
||||
|
||||
- **Implicit:** presence of recent voice frames for an ssrc → that stream is "active". The
|
||||
receiver drives talk indicators from the jitter buffer, so they're accurate and need no
|
||||
extra messages.
|
||||
- **Explicit (optional):** `StreamStateUpdate` on TCP for coarse UI state (muted, hold) and
|
||||
for users not currently subscribed to the media. Server-side mute/deafen is authoritative
|
||||
and always signaled on TCP.
|
||||
|
||||
## 8. Capture/playback pipeline (inside the core)
|
||||
|
||||
```
|
||||
device ─(miniaudio capture, 48k)→ resample? → send-side APM
|
||||
(AEC + NS + AGC + VAD/PTT gate) → Opus encode → frame header → AEAD → UDP send
|
||||
|
||||
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
|
||||
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
|
||||
→ mixer (sum all ssrc) → (miniaudio playback, 48k) → device
|
||||
```
|
||||
|
||||
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
|
||||
- **DSP engine: webrtc-audio-processing (APM)** — the "better one". It provides
|
||||
high-quality **AEC** (acoustic echo cancellation, essential for speaker users), **noise
|
||||
suppression**, **AGC**, and a **VAD** in one tuned module, BSD-licensed. speexdsp is kept
|
||||
only for resampling and as a lightweight jitter-buffer reference. AEC is **in from the
|
||||
start**, not deferred.
|
||||
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
|
||||
|
||||
## 10. Noise reduction — two-sided
|
||||
|
||||
Noise reduction can be applied **at the sender, at the listener, or both** — they are
|
||||
independent.
|
||||
|
||||
- **Sender-side** (the talker's choice): the publishing client runs APM noise suppression on
|
||||
its mic before encoding, controlled by that user's own settings. This cleans the signal for
|
||||
*everyone* and saves bitrate.
|
||||
- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding
|
||||
each stream and *before* mixing, the listener can enable an **additional** NS pass on a
|
||||
**specific** sender's stream. So even if Alex chose not to denoise his mic, Sam can locally
|
||||
suppress Alex's background noise without affecting how anyone else hears Alex.
|
||||
|
||||
Implementation: a per-`ssrc` APM NS instance on the receive path, instantiated lazily only
|
||||
for streams the listener has flagged. State lives entirely on the listener's machine; toggling
|
||||
it is a local UI action with **no protocol message** and no effect on other listeners. Because
|
||||
each receive stream is decoded independently before the mixer (voice.md §1), per-user receive
|
||||
NS is a clean drop-in on that per-stream stage.
|
||||
|
||||
## 11. Input activation — VAD and PTT (client-configurable)
|
||||
|
||||
Whether the mic transmits is decided locally by the **input gate**, and the client supports
|
||||
**both** modes, switchable per client (and ideally per input device):
|
||||
|
||||
- **Voice activation (VAD):** the APM VAD opens the gate when speech is detected, with a
|
||||
configurable threshold and hang-time to avoid clipping word tails. DTX naturally
|
||||
complements this — when the gate is closed nothing (or only comfort noise) is sent.
|
||||
- **Push-to-talk (PTT):** a held key/button opens the gate. The UI exposes a configurable
|
||||
keybind; the core just receives gate open/close.
|
||||
|
||||
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
|
||||
needs **no protocol support**; remote talk indicators are still derived from the presence of
|
||||
received frames (§7), so they work identically under VAD or PTT.
|
||||
|
||||
## 9. System / screen audio capture (`SCREEN_AUDIO`)
|
||||
|
||||
"Listen together" needs to capture the audio another app is playing. The capture mechanism
|
||||
differs per OS, but it always feeds the **same** Opus-encode → media-AEAD → UDP path as a
|
||||
normal stream; only the *source* is platform-specific.
|
||||
|
||||
| Platform | Mechanism | Notes |
|
||||
|----------|-----------|-------|
|
||||
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | Cleanest case; no extra process. Can capture system mix or a specific endpoint. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
||||
|
||||
### iOS detail
|
||||
|
||||
- The user starts a broadcast from Control Center's screen-record button; we surface it via
|
||||
`RPSystemBroadcastPickerView` from inside the app for one-tap start.
|
||||
- The **Broadcast Upload Extension** receives `RPSampleBufferType.audioApp` (system/app
|
||||
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
|
||||
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
|
||||
audio-only consumer stays comfortably inside it.
|
||||
- The extension is a *separate process*. It links a **minimal slice of the core** (Opus
|
||||
encode + media send only — not the full client), reads the active session token and
|
||||
server endpoint from a shared **App Group** container that the host app wrote at join
|
||||
time, derives its own media keys, and publishes the `SCREEN_AUDIO` stream directly. The
|
||||
host app announces the stream over its control channel (`StreamAnnounce`) so the server and
|
||||
peers learn about it.
|
||||
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the
|
||||
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final
|
||||
frame with the `last-frame-before-stop` flag and the host app emits `StreamStop`.
|
||||
Reference in New Issue
Block a user