feat(audio): real noise suppression via vendored RNNoise (send + receive)
Some checks failed
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Build Linux Binaries / linux/arm64 (push) Has been cancelled

The two-sided NR plumbing (RemoteStream::recv_ns + the per-listener
vc_set_remote_stream noise_reduction toggle) was wired but inert:
ApmProcessor::create() returned a no-op passthrough, because the
originally-planned webrtc-audio-processing has no working Windows/macOS
build. Drop in RNNoise as the real backend behind the same ApmProcessor
interface, lighting up both NR paths.

- Vendor RNNoise (BSD-3 + CC0) at third_party/rnnoise/ — the vcpkg port
  is !windows !arm, so it can't cover our primary targets. Shrunk int8
  model (78MB -> 11.7MB via upstream scripts/shrink_model.sh), built as a
  standalone C static lib with no RTCD (portable scalar path on x86,
  auto-NEON on arm64) under -DDISABLE_DEBUG_FLOAT. Model is baked in
  (rnnoise_create(NULL)); no runtime file.
- New RnnoiseProcessor (core/src/audio/apm_processor.cpp) selected by
  ApmProcessor::create() when VOICECAT_HAS_NS. Mono/48kHz/480-sample;
  our clock is fixed 48kHz and Opus frame sizes are multiples of 480, so
  no resampling. RT-safe: allocates at construction, lock-free in the
  capture/playback callbacks.
- Receive-side: lit up via the factory; gated to mono streams (a stereo
  stream is a screen-audio share, not voice).
- Send-side (new): vc_set_input_noise_reduction(client, enable) ABI +
  vc_client::mic_ns_, run before input gain/VAD in on_capture_frame. A
  stereo mic is downmixed to mono ONLY when NR is on — with NR off a
  stereo mic keeps full stereo (never collapse mic quality unasked).
- Enable C as a project language for the vendored lib.
- New noise_suppression test: white noise through ApmProcessor::create()
  drops ~99.9% RMS. ctest --preset dev green, 28/28. windows-client DLL
  builds clean with vc_set_input_noise_reduction exported, system-only deps.
- Docs synced: voice.md §10, tech-stack.md §1/§5, third_party/README.md,
  vcpkg.json note, PROGRESS.md, CLAUDE.md.

Client on/off UI toggles (Windows/macOS/iOS) are the remaining follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-23 13:30:54 +02:00
parent 7249a8fd30
commit bad9c7533a
50 changed files with 381395 additions and 23 deletions

View File

@@ -10,6 +10,26 @@ up instantly. Newest status at the top.
## ▶ Where we left off / next action
- **Done (2026-06-23):** **Remote-stream noise suppression — real backend (RNNoise).** The
two-sided NR plumbing (`RemoteStream::recv_ns` + `vc_set_remote_stream(... noise_reduction)`)
was wired but **inert**`ApmProcessor::create()` returned a no-op passthrough, because the
originally-planned `webrtc-audio-processing` won't build on Windows/macOS. Replaced with
**RNNoise** (BSD-3 + CC0), vendored at `third_party/rnnoise/` (the vcpkg port is `!windows
!arm`), built as a standalone C static lib + `VOICECAT_HAS_NS`. One `RnnoiseProcessor`
(`core/src/audio/apm_processor.cpp`) now backs **both** NR paths:
- **Receive-side** (per-listener, per-`ssrc`): lit up automatically via the factory; gated to
mono streams (`audio_engine.cpp` ~L791).
- **Send-side** (mic, new): `vc_set_input_noise_reduction(client, enable)` ABI +
`vc_client::mic_ns_`, run before input gain/VAD in `on_capture_frame`. A stereo mic is
downmixed to mono **only when NR is on**; with NR off a stereo mic keeps full stereo.
- RNNoise is mono/48 kHz/480-sample; our clock is fixed 48 kHz and Opus frame sizes are all
multiples of 480, so no resampling. RT-safe: alloc at construction, lock-free in the callback.
- **Verify status:** `ctest --preset dev` green — **28/28** (new `noise_suppression` test:
feeds white noise through `ApmProcessor::create()`, measures **99.9%** RMS reduction). Build
clean on the `dev` MinGW preset. **Next (manual):** add the on/off toggles to the client UIs
(Windows Audio Settings dialog, macOS/iOS settings) calling the two ABIs; build `windows-client`
+ `apple-dev` presets to confirm RNNoise compiles under MinGW-DLL and arm64; two-client E2E.
- **Done (2026-06-23):** **Aux outgoing stream (mic + a second input device) — Windows + macOS.**
Users can now transmit a second hardware input device (e.g. line-in / aux) alongside the mic, with
its own device picker and volume, from Audio Settings. **No core/ABI/proto changes** — the aux is