feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback

Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
parent 867557eda1
commit 5f6c223526
19 changed files with 1106 additions and 85 deletions
--- a/docs/README.md
+++ b/docs/README.md
@@ -19,7 +19,7 @@ that implementation can start from a shared, agreed plan.
 | Media transport | **UDP** secured by **TLS-exported keys + ChaCha20-Poly1305 AEAD** — mandatory, no plaintext mode (see [security.md](security.md)) |
 | Crypto libraries | **mbedTLS** (TLS 1.3) + **libsodium** (AEAD, Argon2id, Ed25519) — both permissive, **no GPL/LGPL anywhere** |
 | Voice codec | **Opus** (libopus 1.6), per-channel configurable mono/stereo, bitrate, frame size, FEC/DTX |
-| Audio DSP | **webrtc-audio-processing (APM)** — AEC/NS/AGC/VAD. NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally. Input gate supports **VAD and PTT**, client-configurable. |
+| Audio DSP | **webrtc-audio-processing (APM)** was the plan for AEC/NS/AGC/VAD, but has no working Windows/MSVC build upstream — v1 ships a lightweight energy/RMS VAD instead, no AEC/NS/AGC yet (see [tech-stack.md](tech-stack.md), [roadmap.md](roadmap.md) §2). NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally — this plumbing exists but is currently inert pending a real DSP backend. Input gate supports **VAD and PTT**, client-configurable. |
 | Identity | **Guests + admin-provisioned local accounts** (Argon2id, SQLite). No self-serve registration; guests toggleable per server. |
 | Text | **Ephemeral** — live relay, no server-side history in v1. |
 | Serialization | **Protocol Buffers** for the control plane; **custom binary** for voice frames |
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -126,6 +126,11 @@ Design notes:
  core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
  meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
  exception that needs UI-side cooperation — see below.)
+- **Device enumeration works pre-connect.** `vc_list_devices` needs no live session — device
+  pickers can populate before `vc_connect`. `vc_device.id` is an opaque, internally-encoded
+  handle (currently a hex-encoded `ma_device_id`) — always round-trip an id that came from
+  `vc_list_devices`/`vc_get_stream_audio_config`; never construct one by hand. Tolerate an
+  empty list (a machine can legitimately have zero input or output devices).
 - **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
  caller-allocated or returned with a paired `vc_free`.

--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -77,6 +77,13 @@ Settled and reflected throughout the docs:
  directory). (deployment.md §3)
 - **Bitrate ceiling:** server-config `opus.limits.max_bitrate_bps`. (deployment.md §2)
 - **Name:** "VoiceCat" stays as the internal placeholder.
+- **DSP engine, superseded (2026-06-16):** the "webrtc-audio-processing (APM)" decision above
+  (AEC + NS/AGC/VAD in one module) could not be carried out — it has no working Windows/MSVC
+  build upstream (GCC-only Meson build, MinGW support unfinished, hard `abseil-cpp` dependency,
+  Linux-tested only). v1 ships a lightweight, dependency-free energy/RMS VAD instead, behind
+  the same `ApmProcessor` interface; there is **no AEC/NS/AGC implementation at all** yet. Real
+  `webrtc-audio-processing` stays a tracked future swap (e.g. if/when a Linux build target
+  exists). (voice.md §8, §11)

 ## 3. Open questions

--- a/docs/tech-stack.md
+++ b/docs/tech-stack.md
@@ -12,7 +12,7 @@ Concrete library choices with versions and rationale. Everything in the **core**
 | Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. |
 | Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. |
 | Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. |
-| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) | 1.x (standalone APM) | **BSD-3.** The primary DSP engine: high-quality acoustic echo cancellation, noise suppression, AGC, and VAD in one tuned module. Used **send-side** (clean the mic) and **receive-side per user** (listener-chosen NS on a specific stream — voice.md §10). AEC is in from the start, not deferred. |
+| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) — **planned, not built** | 1.x (standalone APM) | **BSD-3**, but has no working Windows/MSVC build upstream (GCC-only Meson, MinGW support unfinished, hard `abseil-cpp` dep — see roadmap.md §2). v1 ships a lightweight, dependency-free energy/RMS VAD instead (`core/src/audio/apm_processor.cpp`); there is **no AEC, NS, or AGC implementation at all** yet. Real APM stays a tracked future swap behind the same `ApmProcessor` interface. |
 | Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) |
 | Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. |
 | Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. |
@@ -56,7 +56,7 @@ explicit resampling (speexdsp/libsamplerate) is only needed when a device can't
 | Tool | Use |
 |------|-----|
 | **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. |
-| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, webrtc-audio-processing, speexdsp, asio, miniaudio). Reproducible across OSes. |
+| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, asio, miniaudio — see `vcpkg.json`). `webrtc-audio-processing`/`speexdsp` are **not** in the manifest: no working vcpkg port / no working Windows/MSVC build exists upstream for the former; the latter was never actually wired up (the lightweight VAD needs no resampler). Reproducible across OSes. |
 | **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). |
 | **clang-format / clang-tidy** | Style + static analysis on the core. |
 | **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). |
@@ -69,9 +69,11 @@ dependencies are permitted.** Every dependency below is BSD / MIT / ISC / Apache
 public-domain:

 - **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ ·
-  **miniaudio** — public domain / MIT-0 ✅ · **speexdsp** — BSD ✅ · **protobuf** — BSD ✅ ·
-  **SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅ ·
-  **webrtc-audio-processing** — BSD-3 ✅ (heavier build, but core to the DSP path).
+  **miniaudio** — public domain / MIT-0 ✅ · **protobuf** — BSD ✅ ·
+  **SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅.
+  **webrtc-audio-processing** would be BSD-3 ✅ if/when it's actually built in (see §1) —
+  not a live dependency today, so not part of the resolved vcpkg graph the license scanner
+  below checks.
 - **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag
  in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for
  one entirely.
--- a/docs/voice.md
+++ b/docs/voice.md
@@ -159,20 +159,27 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**.
 ## 8. Capture/playback pipeline (inside the core)

 ```
- device ─(miniaudio capture, 48k)→ resample? → send-side APM
-        (AEC + NS + AGC + VAD/PTT gate) → Opus encode → frame header → AEAD → UDP send
+ device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
+        → Opus encode → frame header → AEAD → UDP send

 UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
        → per-stream recv-side NS (optional, per user) → per-stream gain/mute
-        → mixer (sum all ssrc) → (miniaudio playback, 48k) → device
+        → mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
+          48k, stereo) → device
 ```

 - Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
- **DSP engine: webrtc-audio-processing (APM)** — the "better one". It provides
-  high-quality **AEC** (acoustic echo cancellation, essential for speaker users), **noise
-  suppression**, **AGC**, and a **VAD** in one tuned module, BSD-licensed. speexdsp is kept
-  only for resampling and as a lightweight jitter-buffer reference. AEC is **in from the
-  start**, not deferred.
+  Playback is genuinely stereo end-to-end; capture stays mono (no stereo mic in v1).
+- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
+  VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
+  (confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
+  `abseil-cpp` dependency, Linux-tested only —
+  [gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)).
+  v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS,
+  or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real
+  `webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor`
+  interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or
+  upstream Windows support matures.
 - The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.

 ## 10. Noise reduction — two-sided
@@ -197,13 +204,23 @@ NS is a clean drop-in on that per-stream stage.
 ## 11. Input activation — VAD and PTT (client-configurable)

 Whether the mic transmits is decided locally by the **input gate**, and the client supports
-**both** modes, switchable per client (and ideally per input device):
+**both** modes, switchable per client (`vc_set_input_mode`):

- **Voice activation (VAD):** the APM VAD opens the gate when speech is detected, with a
-  configurable threshold and hang-time to avoid clipping word tails. DTX naturally
-  complements this — when the gate is closed nothing (or only comfort noise) is sent.
- **Push-to-talk (PTT):** a held key/button opens the gate. The UI exposes a configurable
-  keybind; the core just receives gate open/close.
+- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free
+  energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no
+  external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC
+  build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold
+  (default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms,
+  matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping
+  word tails. DTX naturally complements this — when the gate is closed nothing (or only
+  comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the
+  originally-planned APM, not just a deferred VAD.
+- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI
+  exposes a configurable keybind; the core just receives gate open/close.
+
+Gating applies to the **MIC stream only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
+(gating a desktop-audio share on the user's own voice activity would silently drop shared
+music/video audio whenever the user isn't talking, which defeats the feature).

 This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
 needs **no protocol support**; remote talk indicators are still derived from the presence of
@@ -217,7 +234,7 @@ normal stream; only the *source* is platform-specific.

 | Platform | Mechanism | Notes |
 |----------|-----------|-------|
-| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | Cleanest case; no extra process. Can capture system mix or a specific endpoint. |
+| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
 | **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
 | **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |