feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback

Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:

- Device enumeration (vc_list_devices) + input device selection
  (vc_set_input_device), backed by AudioEngine::enumerate_devices() via
  miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
  ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
  webrtc-audio-processing (the originally-planned APM) has no working
  Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
  abseil-cpp dependency), so VAD is a new lightweight, dependency-free
  energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
  interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
  stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
  stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
  miniaudio's loopback device type), replacing test-only injection as the
  production capture path.

Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).

Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.

Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.

Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-16 16:11:52 +02:00
parent 867557eda1
commit 5f6c223526
19 changed files with 1106 additions and 85 deletions

View File

@@ -10,19 +10,29 @@ up instantly. Newest status at the top.
## ▶ Where we left off / next action
- **Done:** **M3 — multi-stream & per-channel tuning** ✓ complete (2026-06-16). See the M3
section below for the full file-by-file change list. `ctest --test-dir build/m1-dev`
**11/11 tests** green (3 consecutive full-suite runs), including the new
`test_m3_multistream` (real `vc_client`s, not raw sockets — same lesson as M2: ABI-level
coverage is what proves the client library, not just the wire protocol).
**Two items intentionally still open** (carried forward, not silently dropped):
- `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices`
(device enumeration + VAD/PTT input gate) remain `VC_ERR_NOT_IMPLEMENTED` — explicitly
scoped out of this M3 pass; revisit in a future milestone.
- Stereo Opus is now wire-correct end-to-end (a channel configured `MODE_STEREO` really
encodes/decodes 2-channel Opus packets), but `AudioEngine`'s playback mixer/output device
stays mono internally — stereo streams are downmixed (avg L/R) immediately after decode,
before mixing. True stereo *playback output* is a follow-up, not part of M3.
- **Done:** **Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI
loopback** ✓ complete (2026-06-16). Closes all three items M3 explicitly carried forward as
out of scope (see the dated section below for the full file-by-file change list).
`ctest --test-dir build/m1-dev`**12/12 tests** green (3 consecutive full-suite runs),
including the new `test_vad_ptt_devices` (real `vc_client`s against a real server, plus a
white-box `AudioEngine` stereo-mix check — same ABI-level-coverage lesson as M2/M3).
Manually verified live: `vccli --list-devices` against real hardware, and
`vccli --voice --input-mode vad` connecting/streaming without incident.
**Still explicitly out of scope** (carried forward, not silently dropped):
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream; v1 ships a
lightweight energy/RMS VAD instead (see docs/roadmap.md §2, docs/voice.md §8/§11). There is
**no AEC, NS, or AGC implementation at all**, not just a deferred VAD.
`vc_set_remote_stream(..., noise_reduction)`'s per-stream NS toggle is unaffected by this
pass and stays exactly as inert as it was after M3 (`ApmPassthrough`, no PCM modification).
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — this pass is
Windows-only for real loopback capture; other platforms keep `vc_test_inject_capture` as
the only way to feed `SCREEN_AUDIO`.
- Process-specific WASAPI loopback — miniaudio's loopback mode captures the whole render
endpoint (including this app's own incoming voice mix), not a single process.
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
(mutex lock, heap allocation, blocking `sendto` on the miniaudio real-time callback
thread) — predates this work, documented but not fixed; fixing it needs the lock-free
ring-buffer hand-off `docs/architecture.md §3` specifies, a separate, larger refactor.
- **Next:** **M4 — native clients** (Windows C#, macOS/iOS Swift). See `docs/roadmap.md §M4`.
---
@@ -233,6 +243,92 @@ not treated as pre-existing-and-out-of-scope:
---
## Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ (completed 2026-06-16)
Closes all three items the M3 section above explicitly carried forward as out of scope.
**Exit verification:** `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive
full-suite runs), including the new `test_vad_ptt_devices` (device enumeration + VAD/PTT gate
through real `vc_client`s against a real server, plus a white-box `AudioEngine` stereo-mix
check — no audio hardware needed for that last part). Also verified 5 consecutive standalone
runs of the new test alone, no flakes. Manually verified live on Windows: `vccli
--list-devices` against real hardware (3 input / 4 output devices, correct `is_default`
flags), and `vccli --voice --input-mode vad` connecting + streaming without incident.
- [x] **Device enumeration** (`vc_list_devices`) — `AudioEngine::enumerate_devices(bool
capture)` (static, works without a running engine — inits a throwaway `ma_context` via
`ma_context_get_devices`). `device_id`/`vc_device.id` is an opaque hex-encoded raw
`ma_device_id` (not the device name — names aren't guaranteed unique); documented as an
internal contract callers must round-trip, never construct by hand. `vc_client::list_devices`
works in any connection state (no `VC_STATE_CONNECTED` gate) since device pickers need to
populate pre-connect. `vc_free_device_list` is now a real free (was a no-op stub).
- [x] **Input device selection** (`vc_set_input_device`) — stores the device id on the
targeted `LocalStream` (new field); for the MIC stream, if the engine is already running,
restarts it (`stop()` + `ensure_audio_running()`) to pick up the new device. Simplified: it
restarts unconditionally rather than trying to detect whether the device id actually
changed (`AudioEngine` has no getter for "current device").
- [x] **VAD/PTT input gate** (`vc_set_input_mode`, `vc_set_push_to_talk`) — new
`EnergyVadProcessor` (`core/src/audio/apm_processor.cpp`) implementing the existing
`ApmProcessor` interface: energy/RMS threshold (default ~0.025 normalized) + hang-time
(default 300 ms, matching `kTalkHangoverMs`). New factory `ApmProcessor::create_vad()`,
kept separate from `create()` (which recv-side per-stream NS still uses, unaffected by this
pass). `vc_client` gained `current_input_mode_`/`ptt_active_`/`mic_vad_`; the gate is
inserted in `on_capture_frame`, **MIC-only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
(gating a desktop-audio share on the user's own voice activity would silently drop shared
music/video audio). `last_capture_ms` (drives the talk indicator) is now updated *after* the
gate check, not before, so a VAD/PTT-closed frame never shows as "talking". `mic_vad_` is
constructed once the MIC stream's `StreamAnnounceResult` lands (on `io_thread_`), not lazily
inside the capture path.
- [x] **True stereo playback** — `AudioParams::channels` split into `capture_channels` (stays
1) and `playback_channels` (now 2, unconditionally). `AudioEngine::on_playback` no longer
downmixes decoded stereo streams to mono before mixing — stereo decode output is mixed
directly (L→L, R→R); mono decode output is upmixed (duplicated into both channels). Falls
back to a 1-channel playback device once if the 2-channel `ma_device_init` fails (unusual
hardware). New test-only `AudioEngine::mix_for_test()` exposes the mixer for white-box
testing without a real `ma_device`.
- [x] **WASAPI loopback capture for `SCREEN_AUDIO`** — new `VOICECAT_HAS_LOOPBACK` macro
(`core/CMakeLists.txt`, Windows-only). `AudioEngine` gained a separate `loopback_device_`
(own lifecycle, decoupled from the mic capture/playback devices) with
`start_loopback_capture()`/`stop_loopback_capture()`, using miniaudio's
`ma_device_type_loopback` against the default render endpoint. Its callback feeds
`capture_cb_` directly (same pattern as the real mic capture device), **not** through
`inject_capture()`'s test-only ring. Wired into `vc_client::handle_stream_announce_result`
(start, alongside `ensure_audio_running()`) and `stream_stop` (stop) for
`VC_STREAM_SCREEN_AUDIO`. Non-Windows builds keep `vc_test_inject_capture` as the only way to
feed `SCREEN_AUDIO`.
- [x] `tools/vccli/src/main.cpp` — new flags `--list-devices`, `--input-device`,
`--input-mode vad|ptt`, `--share-screen-audio`; while `--voice` is running, a background
stdin-reader thread accepts `ptt on`/`ptt off`/`mode vad`/`mode ptt` (the most portable way
to drive PTT interactively from a headless CLI — no SIGUSR1 equivalent on Windows). Also
prints `VC_EVENT_TALK_STATE`. Known minor caveat: on Windows the stdin-reader thread is
detached (not joined) on exit, since `std::getline` can't be interrupted from another thread
— a `vc_client*` use-after-free is theoretically possible if a command line arrives in the
brief window between teardown and process exit; acceptable for a headless test/dev tool.
- [x] `tests/test_vad_ptt_devices.cpp` — new test covering all four items above; registered in
`tests/CMakeLists.txt`. `tests/test_smoke.cpp`'s device-list assertion is now conditional on
`VOICECAT_HAS_AUDIO` (was a hard `VC_ERR_NOT_IMPLEMENTED` assertion) — `VC_OK` only, never
`count > 0` (a headless CI build agent may legitimately report zero audio devices).
**Still explicitly out of scope** (carried forward, not silently dropped):
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream (see
docs/roadmap.md §2's superseding decision-log entry). There is **no AEC, NS, or AGC
implementation at all**, not just a deferred VAD. The per-stream NS toggle
(`vc_set_remote_stream(..., noise_reduction)`) is unaffected by this pass and stays exactly
as inert as it was after M3 (`ApmPassthrough`, no PCM modification) — don't mistake this
pass for having fixed it.
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — Windows-only loopback in
this pass.
- Process-specific WASAPI loopback — whole-device capture only; inherently captures this
app's own incoming voice mix along with everything else playing.
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
(mutex lock, heap allocation for the stereo-upmix path, blocking `sendto`, all on the
miniaudio real-time callback thread) — predates this work (was already present in M2/M3);
documented here explicitly rather than silently carried forward again. Fixing it properly
needs the lock-free ring-buffer hand-off `docs/architecture.md §3` specifies — a separate,
larger refactor, out of scope for this pass.
---
## Decisions log
All architecture/scope decisions are settled and recorded in