feat(audio): real noise suppression via vendored RNNoise (send + receive)
The two-sided NR plumbing (RemoteStream::recv_ns + the per-listener vc_set_remote_stream noise_reduction toggle) was wired but inert: ApmProcessor::create() returned a no-op passthrough, because the originally-planned webrtc-audio-processing has no working Windows/macOS build. Drop in RNNoise as the real backend behind the same ApmProcessor interface, lighting up both NR paths. - Vendor RNNoise (BSD-3 + CC0) at third_party/rnnoise/ — the vcpkg port is !windows !arm, so it can't cover our primary targets. Shrunk int8 model (78MB -> 11.7MB via upstream scripts/shrink_model.sh), built as a standalone C static lib with no RTCD (portable scalar path on x86, auto-NEON on arm64) under -DDISABLE_DEBUG_FLOAT. Model is baked in (rnnoise_create(NULL)); no runtime file. - New RnnoiseProcessor (core/src/audio/apm_processor.cpp) selected by ApmProcessor::create() when VOICECAT_HAS_NS. Mono/48kHz/480-sample; our clock is fixed 48kHz and Opus frame sizes are multiples of 480, so no resampling. RT-safe: allocates at construction, lock-free in the capture/playback callbacks. - Receive-side: lit up via the factory; gated to mono streams (a stereo stream is a screen-audio share, not voice). - Send-side (new): vc_set_input_noise_reduction(client, enable) ABI + vc_client::mic_ns_, run before input gain/VAD in on_capture_frame. A stereo mic is downmixed to mono ONLY when NR is on — with NR off a stereo mic keeps full stereo (never collapse mic quality unasked). - Enable C as a project language for the vendored lib. - New noise_suppression test: white noise through ApmProcessor::create() drops ~99.9% RMS. ctest --preset dev green, 28/28. windows-client DLL builds clean with vc_set_input_noise_reduction exported, system-only deps. - Docs synced: voice.md §10, tech-stack.md §1/§5, third_party/README.md, vcpkg.json note, PROGRESS.md, CLAUDE.md. Client on/off UI toggles (Windows/macOS/iOS) are the remaining follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -12,7 +12,8 @@ Concrete library choices with versions and rationale. Everything in the **core**
|
||||
| Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. |
|
||||
| Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. |
|
||||
| Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. |
|
||||
| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) — **planned, not built** | 1.x (standalone APM) | **BSD-3**, but has no working Windows/MSVC build upstream (GCC-only Meson, MinGW support unfinished, hard `abseil-cpp` dep — see roadmap.md §2). v1 ships a lightweight, dependency-free energy/RMS VAD instead (`core/src/audio/apm_processor.cpp`); there is **no AEC, NS, or AGC implementation at all** yet. Real APM stays a tracked future swap behind the same `ApmProcessor` interface. |
|
||||
| Audio DSP — noise suppression (NS) | **RNNoise** (vendored, `third_party/rnnoise/`) | xiph @ `70f1d25` (2026-06) | **BSD-3-Clause + CC0-1.0** (model). Hybrid DSP/RNN speech denoiser, mono/48 kHz, ~60× real time, no deps. The shipped NS backend behind `ApmProcessor` (`RnnoiseProcessor`), used by both send-side mic NR (`vc_set_input_noise_reduction`) and per-listener receive NR (`vc_set_remote_stream`). Vendored (not vcpkg) because the vcpkg port is `!windows !arm`. See [voice.md](voice.md) §10. |
|
||||
| Audio DSP — AEC/AGC/VAD | **webrtc-audio-processing** (APM) — **planned, not built** | 1.x (standalone APM) | **BSD-3**, but has no working Windows/MSVC build upstream (GCC-only Meson, MinGW support unfinished, hard `abseil-cpp` dep — see roadmap.md §2). v1 ships a lightweight, dependency-free energy/RMS VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`); **NS now exists via RNNoise (row above)**, but there is still **no AEC or AGC** (iOS gets AEC/NS/AGC natively from VPIO). Real APM stays a tracked future swap behind the same `ApmProcessor` interface. |
|
||||
| Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) |
|
||||
| Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. |
|
||||
| Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. |
|
||||
@@ -74,7 +75,9 @@ public-domain:
|
||||
|
||||
- **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ ·
|
||||
**miniaudio** — public domain / MIT-0 ✅ · **protobuf** — BSD ✅ ·
|
||||
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅.
|
||||
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅ ·
|
||||
**RNNoise** — BSD-3-Clause (code) + CC0-1.0 (model) ✅, vendored in `third_party/rnnoise/`
|
||||
(not vcpkg — the port is `!windows !arm`; see [`third_party/README.md`](../third_party/README.md)).
|
||||
**webrtc-audio-processing** would be BSD-3 ✅ if/when it's actually built in (see §1) —
|
||||
not a live dependency today, so not part of the resolved vcpkg graph the license scanner
|
||||
below checks.
|
||||
|
||||
@@ -287,17 +287,37 @@ Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth
|
||||
Noise reduction can be applied **at the sender, at the listener, or both** — they are
|
||||
independent.
|
||||
|
||||
- **Sender-side** (the talker's choice): the publishing client runs APM noise suppression on
|
||||
its mic before encoding, controlled by that user's own settings. This cleans the signal for
|
||||
*everyone* and saves bitrate.
|
||||
- **Sender-side** (the talker's choice): the publishing client runs noise suppression on its
|
||||
mic before the input gain and the VAD/PTT gate, controlled by that user's own settings
|
||||
(`vc_set_input_noise_reduction`). This cleans the signal for *everyone* in one pass and helps
|
||||
bitrate/VAD. MIC stream only.
|
||||
- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding
|
||||
each stream and *before* mixing, the listener can enable an **additional** NS pass on a
|
||||
**specific** sender's stream. So even if Alex chose not to denoise his mic, Sam can locally
|
||||
suppress Alex's background noise without affecting how anyone else hears Alex.
|
||||
**specific** sender's stream (`vc_set_remote_stream(..., noise_reduction)`). So even if Alex
|
||||
chose not to denoise his mic, Sam can locally suppress Alex's background noise without
|
||||
affecting how anyone else hears Alex.
|
||||
|
||||
Implementation: a per-`ssrc` APM NS instance on the receive path, instantiated lazily only
|
||||
for streams the listener has flagged. State lives entirely on the listener's machine; toggling
|
||||
it is a local UI action with **no protocol message** and no effect on other listeners. Because
|
||||
**Backend: RNNoise** (vendored in [`third_party/rnnoise/`](../third_party/rnnoise), BSD-3 + CC0).
|
||||
The original plan was WebRTC's APM, but `webrtc-audio-processing` has no working Windows/MSVC
|
||||
build (see §8). RNNoise is a small, dependency-free C library — a hybrid DSP/RNN speech denoiser
|
||||
that runs ~60× faster than real time. Both NR paths share one `ApmProcessor` implementation
|
||||
(`RnnoiseProcessor`, `core/src/audio/apm_processor.cpp`), selected by `ApmProcessor::create()`
|
||||
when the core is built with `VOICECAT_HAS_NS` (a no-op `ApmPassthrough` otherwise). Allocation
|
||||
happens at construction; `process_capture()` runs lock-free on the RT thread (architecture.md §3).
|
||||
|
||||
RNNoise is a **mono, 48 kHz, 480-sample (10 ms)** denoiser. Our engine clock is fixed at 48 kHz
|
||||
and every Opus frame size (480/960/1920/2880) is a multiple of 480, so frames are processed as
|
||||
whole 480-sample chunks with no resampling. Because it's mono-only:
|
||||
- **Send-side:** a stereo mic is downmixed to mono **only when NR is enabled** — with NR off a
|
||||
stereo mic keeps full stereo (we never collapse mic quality unless asked).
|
||||
- **Receive-side:** NR is skipped on stereo streams (a stereo stream is a screen-audio share,
|
||||
not voice).
|
||||
|
||||
Implementation: a per-`ssrc` NS instance (`RemoteStream::recv_ns`) on the receive path,
|
||||
instantiated lazily only for streams the listener has flagged; the send-side instance
|
||||
(`vc_client::mic_ns_`) is built once with the MIC stream and gated by an atomic flag so toggling
|
||||
never allocates on the capture callback. State lives entirely on the local machine; toggling
|
||||
either is a local UI action with **no protocol message** and no effect on other users. Because
|
||||
each receive stream is decoded independently before the mixer (voice.md §1), per-user receive
|
||||
NS is a clean drop-in on that per-stream stage.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user