456 lines
32 KiB
Markdown
456 lines
32 KiB
Markdown
# Voice & Media
|
||
|
||
Real-time audio runs over **UDP**, secured per [security.md](security.md). The control
|
||
channel (TCP/TLS) handles *signaling* — announcing streams, channel membership, talk state
|
||
— while UDP carries only the encoded audio frames. This split keeps media latency low and
|
||
independent of TCP head-of-line blocking.
|
||
|
||
## 1. The multi-stream model
|
||
|
||
A **user** publishes one or more **streams**. Each stream is an independent audio source
|
||
with its own encoder, its own `stream_id` (unique per user) and `ssrc` (media-plane id
|
||
assigned by the server), and is independently mutable/mutable at the receiver.
|
||
|
||
```
|
||
User "Alex" Receiver "Sam"
|
||
┌────────────────────┐ ┌──────────────────────────┐
|
||
│ mic → enc ───┼──ssrc 1001──▶ │ jitter(1001)→dec→┐ │
|
||
│ desktop → enc ───┼──ssrc 1002──▶ │ jitter(1002)→dec→┤ │
|
||
│ 2nd mic → enc ───┼──ssrc 1003──▶ │ jitter(1003)→dec→┴─mix──▶ out
|
||
└────────────────────┘ └──────────────────────────┘
|
||
```
|
||
|
||
Stream **kinds** (v1): `MIC`, `SCREEN_AUDIO` (system/desktop audio for listening together),
|
||
`AUX_DEVICE` (a second capture device). Receivers can set, per incoming stream: **gain**,
|
||
**mute**, and **noise reduction** (see §10) — so Sam can turn down Alex's desktop audio
|
||
while keeping the mic, *and* independently apply noise suppression to a third user who has a
|
||
loud fan. The mixer sums all active streams from all users in the channel into the local
|
||
playback device. All of these receiver-side controls are **local to the listener** and carry
|
||
no protocol traffic.
|
||
|
||
`SCREEN_AUDIO` capture is platform-specific and covered in §9 — it is supported on Windows,
|
||
macOS, **and iOS** (via a ReplayKit broadcast extension).
|
||
|
||
## 2. Voice frame format (UDP payload, inside the media AEAD)
|
||
|
||
A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.
|
||
|
||
The header is **20 bytes** (protocol v2; v1 was 14 bytes with a u16 seq — see note below).
|
||
|
||
```
|
||
0 1 2 3 4 5 6 7 8 ............ 15
|
||
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
|
||
│ type │flags │ codec │ ssrc (u32) │ seq (u64) ──▶ │
|
||
├──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴───────────────┤
|
||
│ ◀── seq (u64) ──┤ timestamp (u32 @48k) │ payload ... │
|
||
└──────────────────┴────────────────────────────────────┴───────────────┘
|
||
bytes [8..15] = seq (u64) [16..19] = timestamp (u32)
|
||
|
||
type u8 1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
|
||
flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present
|
||
bit2 DTX/comfort-noise · bit3 last-frame-before-stop
|
||
codec u16 0 = OPUS (room for future codecs)
|
||
ssrc u32 media-plane stream id. Client sends its own ssrc; the server
|
||
validates it against the bound session and relays unchanged.
|
||
seq u64 full monotonic send counter. This IS the AEAD nonce counter, so the
|
||
receiver derives the nonce directly from it — no rollover guessing.
|
||
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
|
||
payload one Opus packet (the encoder's output for one frame)
|
||
```
|
||
|
||
> **Why u64 (protocol v2).** v1 carried only the low 16 bits of the counter and the
|
||
> receiver zero-extended them to rebuild the AEAD nonce. After 65,536 frames the seq
|
||
> wrapped, the reconstructed nonce diverged from the sealing nonce, and **every frame
|
||
> failed authentication permanently** (no rollover counter). v2 puts the full 64-bit
|
||
> counter on the wire so the nonce is always exact. A v2 server and a v1 client cannot
|
||
> interoperate; the `Hello` handshake rejects on `proto_version` mismatch.
|
||
|
||
This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's
|
||
full machinery. The **server relays the payload unmodified** — it only reads the header to
|
||
route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once
|
||
assigned at `StreamAnnounce`). No server-side decode.
|
||
|
||
### Why client-sends-ssrc is safe
|
||
|
||
The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks
|
||
that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are
|
||
dropped. So identity is anchored by the session binding + transport encryption, not by
|
||
trusting the header.
|
||
|
||
## 3. Per-channel audio configuration
|
||
|
||
Opus is configured **per channel** and pushed to clients in `JoinChannelResult.audio` /
|
||
`StreamAnnounceResult.effective_audio`. All members of a channel encode with mutually
|
||
decodable parameters.
|
||
|
||
```proto
|
||
message AudioConfig {
|
||
uint32 codec = 1; // 0 = OPUS
|
||
ChannelMode mode = 2; // MONO / STEREO
|
||
uint32 sample_rate = 3; // 8000/12000/16000/24000/48000 (48000 recommended)
|
||
uint32 bitrate_bps = 4; // e.g. 24000 (speech) … 128000 (music/stereo)
|
||
uint32 frame_ms = 5; // 2.5/5/10/20/40/60 (20 default)
|
||
OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY
|
||
bool fec = 7; // in-band forward error correction
|
||
uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness
|
||
bool dtx = 9; // discontinuous transmission (silence suppression)
|
||
uint32 complexity = 10; // 0..10 encoder complexity
|
||
}
|
||
```
|
||
|
||
Guidance baked into defaults / docs:
|
||
|
||
- **Sample rate: always run Opus at 48 kHz internally.** Opus resamples internally anyway;
|
||
48 kHz avoids surprises, and the whole audio stack (capture, `vc_stream_feed_pcm`, mixing,
|
||
playback) runs at 48 kHz. The per-channel `sample_rate` field is **channel-authoritative**
|
||
(not a client request) and does *not* change the codec/PCM clock — it caps the encoder's
|
||
audio bandwidth via `OPUS_SET_MAX_BANDWIDTH` (8000 → narrowband ~4 kHz, 16000 → wideband
|
||
~8 kHz, 24000 → super-wideband ~12 kHz, 48000 → full ~20 kHz). This lets a low-bitrate room
|
||
shed out-of-band content while every endpoint keeps a single 48 kHz clock. Default **48000**
|
||
(full band). See `OpusEncoder::init` and `vc_client::opus_params_from_audio_config`.
|
||
- **Frame size: 20 ms default.** Smaller (10 ms) lowers latency at the cost of more
|
||
per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at
|
||
the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms.
|
||
The capture engine runs on a fixed 48 kHz / 20 ms clock (960-sample frames), so the send
|
||
path **reframes** each captured/fed block to the channel's `frame_ms` before encoding
|
||
(accumulating two 960-frames for a 40 ms channel, splitting each into two 480-frames for a
|
||
10 ms channel, etc.). This keeps the hardware/`vc_stream_feed_pcm` contract a single 48 kHz
|
||
clock regardless of the channel's window — see `vc_client::on_capture_frame`.
|
||
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 24–32 kbps, DTX on, FEC on.
|
||
Music/screen-audio channels → `STEREO`, `AUDIO`, 96–128 kbps, DTX off, FEC optional.
|
||
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
|
||
monitoring use cases.
|
||
|
||
## 4. Packet-loss resilience (Opus 1.6)
|
||
|
||
Layered, all configurable per channel:
|
||
|
||
1. **In-band FEC** — the encoder embeds a low-bitrate copy of the current frame in the
|
||
*next* packet (`OPUS_SET_INBAND_FEC`, redundancy scaled by `expected_packet_loss`). On a
|
||
loss, the receiver decodes that copy out of the next already-buffered packet with
|
||
`opus_decode(..., decode_fec=1)` — costing one frame of latency on recovery. Gated on the
|
||
per-stream `fec` flag; if the next packet carries no redundancy libopus yields PLC output,
|
||
so it is at worst a no-op relative to (2).
|
||
2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an
|
||
unrecovered loss; always on, free. The terminal fallback when neither DRED nor FEC applies.
|
||
3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise
|
||
updates; cuts bandwidth and is bandwidth-friendly on busy channels.
|
||
4. **DRED (Deep REDundancy, per-channel toggle)** — Opus 1.6's ML redundancy: the encoder
|
||
embeds 20 ms of acoustic features in every packet (`bool dred` in `AudioConfig`, off by
|
||
default). When a packet is lost, the receiver peeks at the next already-buffered packet,
|
||
parses its DRED extension (`opus_dred_parse`), and reconstructs the lost frame with
|
||
`opus_decoder_dred_decode` — producing significantly better audio than PLC comfort noise
|
||
for single-frame gaps. Heavier CPU on the encoder (~5–10 % at 24 kbps); minimal overhead
|
||
on the decoder (parse is a fast header check on non-DRED packets).
|
||
|
||
When a frame is lost, `AudioEngine::on_playback` tries these recovery paths in quality order,
|
||
falling through on failure: **DRED → in-band FEC → PLC**. DRED and FEC both need the next
|
||
packet already buffered (one frame of look-ahead); when it has not arrived yet, recovery falls
|
||
straight through to PLC.
|
||
|
||
## 5. Jitter buffer
|
||
|
||
Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth playout**
|
||
(`core/src/audio/audio_engine.cpp` — `JitterBuffer` + `AudioEngine::on_playback`).
|
||
|
||
- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
|
||
- **The playout clock is always bounded against the stream's *leading edge* (newest buffered
|
||
frame), never re-synced to the oldest.** The clock free-runs at the playback hardware rate,
|
||
while the sender omits VAD/PTT/DTX silence from its timestamps, so the two diverge across gaps
|
||
and late joins. Two corrections keep latency bounded:
|
||
- **(Re)seed to the leading edge** on first frame, on a talkspurt `marker`, or when the clock
|
||
has run past the newest frame (starved after silence). No artificial prebuffer — latency
|
||
starts as low as possible; buffered frames still play oldest-first.
|
||
- **Frame-skip catch-up:** when the backlog grows past `target + hysteresis` (clock drift,
|
||
bursty arrival, reordering), fast-forward the clock to leave `target` buffered and drop the
|
||
now-stale frames. This is the downward force that prevents latency from ratcheting upward.
|
||
- `target` is the adaptive jitter estimate (EWMA of inter-arrival gap vs. the per-frame gap),
|
||
floored; silence gaps and reordered stragglers are rejected as outliers so they don't inflate
|
||
it. The late-drop window tracks `target` (floored/capped at 500 ms).
|
||
- Late frames past the playout point are dropped; gaps are filled by DRED (if the next frame
|
||
arrived) or PLC.
|
||
- The `marker` flag (start of talkspurt) — set by the sender on the first frame after a
|
||
transmission gap — lets the buffer reseed cleanly after silence/DTX without accumulating drift.
|
||
- Diagnostics per stream: `packets_lost`, `duplicates`, `underruns`, `target_depth_ms`.
|
||
|
||
```
|
||
incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer
|
||
▲
|
||
jitter estimate feeds depth
|
||
```
|
||
|
||
## 6. UDP keepalive & NAT
|
||
|
||
- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to
|
||
hold NAT bindings and measure media-path RTT/loss independent of TCP. The frame is
|
||
plaintext (14-byte header, no payload, no AEAD) — the server identifies the sender by
|
||
its already-verified UDP endpoint (established during the `UdpBinding` handshake). On
|
||
receipt the server bumps the sender's `last_seen` (so media activity defers the TCP
|
||
reaper independently of control-channel traffic) and echoes the frame back so the
|
||
client can measure media-path RTT.
|
||
- If the media path dies but TCP is alive, the client surfaces a "voice disconnected"
|
||
state and attempts UDP re-binding (re-derive media keys + fresh `UdpBinding`) without dropping
|
||
the control session.
|
||
- No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the **server** is reachable
|
||
(public IP or port-forward); **clients** sit behind NAT and initiate, so their bindings
|
||
are created by their outbound first packet.
|
||
|
||
## 7. Talk-state signaling
|
||
|
||
"Who is talking" can be derived two ways; we use both:
|
||
|
||
- **Implicit:** presence of recent voice frames for an ssrc → that stream is "active". The
|
||
receiver drives talk indicators from the jitter buffer, so they're accurate and need no
|
||
extra messages.
|
||
- **Explicit (optional):** `StreamStateUpdate` on TCP for coarse UI state (muted, hold) and
|
||
for users not currently subscribed to the media. Server-side mute/deafen is authoritative
|
||
and always signaled on TCP.
|
||
|
||
## 8. Capture/playback pipeline (inside the core)
|
||
|
||
```
|
||
mic device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
|
||
→ Opus encode → frame header → AEAD → UDP send
|
||
|
||
screen audio ─(WASAPI loopback, 48k, mono or stereo per channel mode)→ Opus encode
|
||
→ frame header → AEAD → UDP send
|
||
|
||
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
|
||
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
|
||
→ mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
|
||
48k, stereo) → device
|
||
```
|
||
|
||
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
|
||
Playback is genuinely stereo end-to-end. **Mic capture** is mono by default; **stereo mic
|
||
capture** is supported via `vc_set_capture_channels(stream_id, 2)` — when enabled, the
|
||
capture device opens in stereo (interleaved L/R). All native clients expose this as a
|
||
per-user toggle (iOS in Settings; the Windows and macOS desktop clients via a "Stereo
|
||
microphone" checkbox in Audio settings — a live toggle there restarts the capture device via
|
||
`vc_audio_restart` so it takes effect immediately). Whether stereo actually reaches the wire
|
||
depends on the **channel's** Opus mode, which decides the encoder's channel count — the mic's
|
||
channel count and the channel's mode are independent knobs:
|
||
- **stereo mic + stereo channel** → real interleaved L/R is encoded directly (no upmix).
|
||
- **mono mic + stereo channel** → the mono frame is upmixed L=R so the Opus bitstream is
|
||
still spec-correct stereo.
|
||
- **stereo mic + mono channel** → the interleaved L/R is folded to mono before the mono
|
||
encoder. (Handing interleaved pairs straight to a mono `opus_encode` would make it read 2×
|
||
the samples it should — wrong pitch / garbage — so the fold keeps the toggle safe on any
|
||
channel.)
|
||
**Screen-audio (`SCREEN_AUDIO`) loopback** captures
|
||
in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
|
||
downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
|
||
genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
|
||
- **iOS audio — one path, always external.** On iOS the core **never opens a miniaudio device**:
|
||
a single `AVAudioEngine` (`IOSAudioEngine`) drives *both* directions, and the core runs fully
|
||
external for the whole connection. This is the single most important property of the iOS audio
|
||
stack — there is no second (miniaudio) path to switch to, so a preset/route change cannot leave
|
||
one direction dropped. The single ordering rule is: `vc_set_external_playback(1)` is set **once
|
||
at connect** (before the session is activated or any remote stream arrives), and every MIC
|
||
stream is started with `vc_stream_desc.external_feed=1`.
|
||
- **core → speaker:** the core's mixer-timer thread decodes+mixes on a ~20 ms cadence and
|
||
delivers the FINAL mixed PCM via `vc_set_mixed_output_sink`; an `AVAudioSourceNode` pulls it
|
||
from a lock-free ring and renders it. This runs the whole time we are connected, so remote
|
||
audio plays even before the user joins voice (kills the "can't hear anyone" race).
|
||
- **mic → core:** when the mic is active a tap on the engine's input node converts to 48 kHz
|
||
int16 (`vc_set_capture_channels` decides mono/stereo) and calls `vc_stream_feed_pcm`.
|
||
- **iOS routing** is still driven from Swift via `AVAudioSession` by the `IOSAudioRouter`
|
||
singleton — miniaudio never touches `AVAudioSession` on iOS. Input port selection
|
||
(`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom),
|
||
polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic
|
||
processing mode (Standard vs `.measurement` Raw), Bluetooth mode (`.allowBluetoothHFP` HFP voice
|
||
vs `.allowBluetoothA2DP` stereo output vs neither), and stereo capture (`.stereo` polar pattern
|
||
+ `setPreferredInput` + `setInputDataSource` → `vc_set_capture_channels`) are all set from
|
||
Swift. Any preset / route / interruption change funnels through one deterministic, Swift-only
|
||
rebuild: `IOSAudioEngine` stops, `IOSAudioRouter.applyConfiguration()` re-applies the
|
||
`AVAudioSession`, the graph is rebuilt against the new route, and the engine restarts. No
|
||
`vc_audio_restart`/`vc_audio_suspend` dance is needed for routing (the core has no hardware
|
||
devices to reopen) — this is the spirit of TeamTalk5's "close then re-init sound devices", but
|
||
entirely inside the Swift engine.
|
||
- **iOS voice processing (AEC/NS/AGC) — native VPIO.** Real iOS echo cancellation, noise
|
||
suppression and AGC come ONLY from Apple's **Voice-Processing I/O audio unit (VPIO)**, which
|
||
`inputNode.setVoiceProcessingEnabled(true)` enables; for it to cancel echo it must own BOTH the
|
||
mic capture and the playback — which the unified engine already does. VPIO forces **mono**, so
|
||
it is engaged only when the active config wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`:
|
||
mono + standard + non-A2DP + the user's master toggle). iOS exposes no per-stage VPIO control,
|
||
so the Advanced UI offers exactly two switches: a master **Voice Processing** (AEC + NS bundled)
|
||
and **AGC** (`isVoiceProcessingAGCEnabled`).
|
||
- **iOS presets** (`IOSAudioRouter.AudioPreset`): **Voice Chat** (VPIO mono, system output incl.
|
||
HFP/wired), **Stereo Mic** (internal stereo built-in mic regardless of output, A2DP-capable, no
|
||
VPIO), **Mono Mic** (internal mono built-in mic regardless of output, A2DP-capable, no VPIO),
|
||
and **Advanced** (every knob manual). A2DP output requires an internal-mic preset (the Bluetooth
|
||
device is output-only); the Stereo/Mono Mic presets fall back to the built-in speaker when no
|
||
external output is connected (`applyA2dpSpeakerFallback`).
|
||
- **iOS implementation invariants:**
|
||
- External playback is enabled before connecting because authentication can start the audio
|
||
engine before the UI receives another turn.
|
||
- AVAudioEngine callbacks may contain multiple codec frames. The mic path writes them to an
|
||
SPSC ring and releases complete 20 ms frames at a steady cadence; it never consumes a partial
|
||
frame, and the pacer is recreated when mono/stereo capture changes.
|
||
- `AVAudioSession` setters can synchronously emit route-change notifications. Configuration is
|
||
re-entrancy guarded, and recovery ignores `.categoryChange`, `.routeConfigurationChange`, and
|
||
`.override` because those reasons are generated by the app's own routing calls. External route
|
||
changes and engine-configuration notifications still rebuild the graph.
|
||
- Stereo capture anchors the built-in mic's stereo data source. It does not call
|
||
`setPreferredInputNumberOfChannels(2)`, which can disrupt A2DP output; the core receives the
|
||
channel count through `vc_set_capture_channels`.
|
||
- The A2DP speaker fallback caches its last output override. Reapplying the same override would
|
||
emit another `.override` notification and recursively trigger recovery.
|
||
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
|
||
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
|
||
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
|
||
`abseil-cpp` dependency, Linux-tested only —
|
||
[gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)).
|
||
v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS,
|
||
or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real
|
||
`webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor`
|
||
interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or
|
||
upstream Windows support matures.
|
||
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
|
||
|
||
## 10. Noise reduction — two-sided
|
||
|
||
Noise reduction can be applied **at the sender, at the listener, or both** — they are
|
||
independent.
|
||
|
||
- **Sender-side** (the talker's choice): the publishing client runs noise suppression on its
|
||
mic before the input gain and the VAD/PTT gate, controlled by that user's own settings
|
||
(`vc_set_input_noise_reduction`). This cleans the signal for *everyone* in one pass and helps
|
||
bitrate/VAD. MIC stream only.
|
||
- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding
|
||
each stream and *before* mixing, the listener can enable an **additional** NS pass on a
|
||
**specific** sender's stream (`vc_set_remote_stream(..., noise_reduction)`). So even if Alex
|
||
chose not to denoise his mic, Sam can locally suppress Alex's background noise without
|
||
affecting how anyone else hears Alex.
|
||
|
||
**Backend: RNNoise** (vendored in [`third_party/rnnoise/`](../third_party/rnnoise), BSD-3 + CC0).
|
||
The original plan was WebRTC's APM, but `webrtc-audio-processing` has no working Windows/MSVC
|
||
build (see §8). RNNoise is a small, dependency-free C library — a hybrid DSP/RNN speech denoiser
|
||
that runs ~60× faster than real time. Both NR paths share one `ApmProcessor` implementation
|
||
(`RnnoiseProcessor`, `core/src/audio/apm_processor.cpp`), selected by `ApmProcessor::create()`
|
||
when the core is built with `VOICECAT_HAS_NS` (a no-op `ApmPassthrough` otherwise). Allocation
|
||
happens at construction; `process_capture()` runs lock-free on the RT thread (architecture.md §3).
|
||
|
||
RNNoise is a **mono, 48 kHz, 480-sample (10 ms)** denoiser. Our engine clock is fixed at 48 kHz
|
||
and every Opus frame size (480/960/1920/2880) is a multiple of 480, so frames are processed as
|
||
whole 480-sample chunks with no resampling. Because it's mono-only:
|
||
- **Send-side:** a stereo mic is downmixed to mono **only when NR is enabled** — with NR off a
|
||
stereo mic keeps full stereo (we never collapse mic quality unless asked).
|
||
- **Receive-side:** NR applies to **voice (MIC) streams only**, gated on the stream *kind* — not
|
||
on its channel count, since a stereo mic with send-side NR off now arrives as stereo voice.
|
||
When enabled on such a stream the decoded stereo frame is folded to mono, denoised, and
|
||
duplicated back across both channels (symmetric with the send-side downmix), so that stream
|
||
plays as mono while NR is on. A **screen-audio share is never voice and is left untouched** —
|
||
denoising music/video with a speech denoiser would mangle it.
|
||
|
||
Implementation: a per-`ssrc` NS instance (`RemoteStream::recv_ns`) on the receive path,
|
||
instantiated lazily only for streams the listener has flagged; the send-side instance
|
||
(`vc_client::mic_ns_`) is built once with the MIC stream and gated by an atomic flag so toggling
|
||
never allocates on the capture callback. State lives entirely on the local machine; toggling
|
||
either is a local UI action with **no protocol message** and no effect on other users. Because
|
||
each receive stream is decoded independently before the mixer (voice.md §1), per-user receive
|
||
NS is a clean drop-in on that per-stream stage.
|
||
|
||
All three receive-side controls (gain, mute, NR) are queryable via `vc_get_remote_stream` —
|
||
the counterpart to `vc_set_remote_stream` — so a UI can reopen its per-stream mix controls at
|
||
the listener's actual current settings (defaults: gain 1.0, unmuted, NR off). Like the setter,
|
||
it carries no protocol traffic.
|
||
|
||
## 11. Input activation — VAD and PTT (client-configurable)
|
||
|
||
Whether the mic transmits is decided locally by the **input gate**, and the client supports
|
||
**both** modes, switchable per client (`vc_set_input_mode`):
|
||
|
||
- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free
|
||
energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no
|
||
external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC
|
||
build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold
|
||
(default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms,
|
||
matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping
|
||
word tails. DTX naturally complements this — when the gate is closed nothing (or only
|
||
comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the
|
||
originally-planned APM, not just a deferred VAD.
|
||
- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI
|
||
exposes a configurable keybind; the core just receives gate open/close.
|
||
|
||
Gating applies to the **MIC stream only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
|
||
(gating a desktop-audio share on the user's own voice activity would silently drop shared
|
||
music/video audio whenever the user isn't talking, which defeats the feature).
|
||
|
||
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
|
||
needs **no protocol support**; remote talk indicators are still derived from the presence of
|
||
received frames (§7), so they work identically under VAD or PTT.
|
||
|
||
## 9. System / screen audio capture (`SCREEN_AUDIO`)
|
||
|
||
"Listen together" needs to capture the audio another app is playing. The capture mechanism
|
||
differs per OS, but it always feeds the **same** Opus-encode → media-AEAD → UDP path as a
|
||
normal stream; only the *source* is platform-specific.
|
||
|
||
| Platform | Mechanism | Notes |
|
||
|----------|-----------|-------|
|
||
| **Windows** | **WASAPI loopback** (whole-device, via miniaudio) **or WASAPI process loopback** (`AUDIOCLIENT_ACTIVATION_PARAMS`, Win10 2004+) for per-app / self-exclude | **Implemented.** Default *entire desktop* uses miniaudio's whole-device loopback in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when mono — so a stereo channel gets genuine stereo end-to-end (no downmix). It inherently captures this app's own incoming voice mix (self-echo). The **per-app modes and the "exclude VoiceCat's own audio" option** instead drive `ProcessLoopbackCapture` (process-specific INCLUDE/EXCLUDE) through the external-feed mixer (`vc_stream_feed_pcm`, `external_feed=1`), which avoids self-echo and supports true "everything except". See below. |
|
||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). **Supports per-app audio selection** — see below. |
|
||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). ReplayKit only ever delivers the *mixed* system stream as `.audioApp`, so **per-app filtering / VoiceOver exclusion is not possible on iOS** (it has no per-app granularity, unlike ScreenCaptureKit). |
|
||
|
||
### macOS detail — per-app audio selection
|
||
|
||
ScreenCaptureKit filters audio at the **application** level, so before sharing starts the user
|
||
picks a scope in `ScreenSharePickerSheet` (`clients/apple/macOS/VoiceCatMac/Sheets/`):
|
||
|
||
- **Everything** — whole display, the original behaviour (`SCContentFilter(display:excludingWindows:)`).
|
||
- **Only selected apps** — capture just the ticked apps (`init(display:including:exceptingWindows:)`).
|
||
- **All except selected apps** — capture everything but the ticked apps
|
||
(`init(display:excludingApplications:exceptingWindows:)`).
|
||
|
||
A dedicated **"Exclude screen reader (VoiceOver) audio"** toggle merges the screen-reader
|
||
process(es) into the exclude set (`ScreenAudioCapture.screenReaderBundleIDs` — VoiceOver plus
|
||
the speech-synthesis daemon that actually renders the spoken audio). The chosen
|
||
`ScreenAudioSelection` is passed into `ScreenAudioCapture`, which builds the matching
|
||
`SCContentFilter`. iOS/ReplayKit has no equivalent control (see the table note above).
|
||
|
||
### Windows detail — per-app audio selection and self-echo
|
||
|
||
`AppAudioPickerDialog` (`clients/windows/VoiceCat.App/Forms/`) offers the same shape as macOS:
|
||
|
||
- **Entire desktop** — whole-device miniaudio loopback handled by the core (default path).
|
||
- **Only selected apps** — one `ProcessLoopbackCapture` in **INCLUDE** mode per ticked app,
|
||
mixed by `ProcessAudioMixer` and fed via `vc_stream_feed_pcm`.
|
||
- **All apps except selected** — a **single** `ProcessLoopbackCapture` in **EXCLUDE** mode of
|
||
the chosen process tree. WASAPI's `AUDIOCLIENT_PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE`
|
||
captures the whole render mix minus that tree *dynamically* (apps launched after sharing
|
||
starts are included automatically). The activation params take a **single** target PID, so
|
||
exclude is restricted to **one** app — the picker enforces single-selection in this mode.
|
||
|
||
An **"Exclude VoiceCat's own audio (prevents echo)"** checkbox (default on, enabled for
|
||
*entire desktop*) routes the desktop capture through the same EXCLUDE path targeting
|
||
VoiceCat's **own** process id (`Environment.ProcessId`) — i.e. "entire desktop except this
|
||
app" — which removes the self-echo loop the whole-device path otherwise has. The per-app
|
||
INCLUDE modes already never capture this app's tree, so they have no self-echo to remove.
|
||
|
||
### iOS detail
|
||
|
||
The extension **captures**, the host app **sends**. Unlike a self-connecting extension, this
|
||
keeps a **single session** — the screen-audio share appears as a second stream of the *same*
|
||
user (exactly like macOS/Windows), and no credentials are ever persisted to disk.
|
||
|
||
- The user starts a broadcast from Control Center's screen-record button; we surface it via
|
||
`RPSystemBroadcastPickerView` from inside the app (`VoiceControlsView`) for one-tap start.
|
||
- The **Broadcast Upload Extension** (`clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift`)
|
||
receives `RPSampleBufferType.audioApp` (system/app audio), `.audioMic`, and `.video`. We
|
||
consume **`.audioApp`** only and drop video + mic — video is what blows the **~50 MB**
|
||
extension memory budget, so an audio-only consumer stays comfortably inside it. The extension
|
||
does **not** link `libvoicecat`.
|
||
- The extension converts each chunk to the core's canonical format (48 kHz int16 stereo, via
|
||
`AVAudioConverter`) and writes it into a lock-free single-producer/single-consumer ring in a
|
||
shared **App Group** mmap'd file (`clients/apple/iOS/Shared/BroadcastAudioRing.swift`). It
|
||
posts Darwin notifications on start/stop so the host reacts promptly.
|
||
- The **host app** owns the stream: its `BroadcastAudioPump` announces the `SCREEN_AUDIO`
|
||
stream over the control channel (`StreamAnnounce`), drains the ring, and calls
|
||
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the Opus
|
||
encode + AEAD + send path. It downmixes to mono when the channel's effective config is mono.
|
||
- Mic + voice also run in the host app. When the broadcast stops (`broadcastFinished`), the
|
||
extension clears the ring's active flag (and posts a Darwin notification); the host stops
|
||
feeding and emits `StreamStop`. The host must be alive to relay — always true while in a
|
||
call (the app declares the `audio` background mode).
|