feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
122
PROGRESS.md
122
PROGRESS.md
@@ -10,19 +10,29 @@ up instantly. Newest status at the top.
|
||||
|
||||
## ▶ Where we left off / next action
|
||||
|
||||
- **Done:** **M3 — multi-stream & per-channel tuning** ✓ complete (2026-06-16). See the M3
|
||||
section below for the full file-by-file change list. `ctest --test-dir build/m1-dev` —
|
||||
**11/11 tests** green (3 consecutive full-suite runs), including the new
|
||||
`test_m3_multistream` (real `vc_client`s, not raw sockets — same lesson as M2: ABI-level
|
||||
coverage is what proves the client library, not just the wire protocol).
|
||||
**Two items intentionally still open** (carried forward, not silently dropped):
|
||||
- `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices`
|
||||
(device enumeration + VAD/PTT input gate) remain `VC_ERR_NOT_IMPLEMENTED` — explicitly
|
||||
scoped out of this M3 pass; revisit in a future milestone.
|
||||
- Stereo Opus is now wire-correct end-to-end (a channel configured `MODE_STEREO` really
|
||||
encodes/decodes 2-channel Opus packets), but `AudioEngine`'s playback mixer/output device
|
||||
stays mono internally — stereo streams are downmixed (avg L/R) immediately after decode,
|
||||
before mixing. True stereo *playback output* is a follow-up, not part of M3.
|
||||
- **Done:** **Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI
|
||||
loopback** ✓ complete (2026-06-16). Closes all three items M3 explicitly carried forward as
|
||||
out of scope (see the dated section below for the full file-by-file change list).
|
||||
`ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive full-suite runs),
|
||||
including the new `test_vad_ptt_devices` (real `vc_client`s against a real server, plus a
|
||||
white-box `AudioEngine` stereo-mix check — same ABI-level-coverage lesson as M2/M3).
|
||||
Manually verified live: `vccli --list-devices` against real hardware, and
|
||||
`vccli --voice --input-mode vad` connecting/streaming without incident.
|
||||
**Still explicitly out of scope** (carried forward, not silently dropped):
|
||||
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream; v1 ships a
|
||||
lightweight energy/RMS VAD instead (see docs/roadmap.md §2, docs/voice.md §8/§11). There is
|
||||
**no AEC, NS, or AGC implementation at all**, not just a deferred VAD.
|
||||
`vc_set_remote_stream(..., noise_reduction)`'s per-stream NS toggle is unaffected by this
|
||||
pass and stays exactly as inert as it was after M3 (`ApmPassthrough`, no PCM modification).
|
||||
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — this pass is
|
||||
Windows-only for real loopback capture; other platforms keep `vc_test_inject_capture` as
|
||||
the only way to feed `SCREEN_AUDIO`.
|
||||
- Process-specific WASAPI loopback — miniaudio's loopback mode captures the whole render
|
||||
endpoint (including this app's own incoming voice mix), not a single process.
|
||||
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
|
||||
(mutex lock, heap allocation, blocking `sendto` on the miniaudio real-time callback
|
||||
thread) — predates this work, documented but not fixed; fixing it needs the lock-free
|
||||
ring-buffer hand-off `docs/architecture.md §3` specifies, a separate, larger refactor.
|
||||
- **Next:** **M4 — native clients** (Windows C#, macOS/iOS Swift). See `docs/roadmap.md §M4`.
|
||||
|
||||
---
|
||||
@@ -233,6 +243,92 @@ not treated as pre-existing-and-out-of-scope:
|
||||
|
||||
---
|
||||
|
||||
## Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ (completed 2026-06-16)
|
||||
|
||||
Closes all three items the M3 section above explicitly carried forward as out of scope.
|
||||
|
||||
**Exit verification:** `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive
|
||||
full-suite runs), including the new `test_vad_ptt_devices` (device enumeration + VAD/PTT gate
|
||||
through real `vc_client`s against a real server, plus a white-box `AudioEngine` stereo-mix
|
||||
check — no audio hardware needed for that last part). Also verified 5 consecutive standalone
|
||||
runs of the new test alone, no flakes. Manually verified live on Windows: `vccli
|
||||
--list-devices` against real hardware (3 input / 4 output devices, correct `is_default`
|
||||
flags), and `vccli --voice --input-mode vad` connecting + streaming without incident.
|
||||
|
||||
- [x] **Device enumeration** (`vc_list_devices`) — `AudioEngine::enumerate_devices(bool
|
||||
capture)` (static, works without a running engine — inits a throwaway `ma_context` via
|
||||
`ma_context_get_devices`). `device_id`/`vc_device.id` is an opaque hex-encoded raw
|
||||
`ma_device_id` (not the device name — names aren't guaranteed unique); documented as an
|
||||
internal contract callers must round-trip, never construct by hand. `vc_client::list_devices`
|
||||
works in any connection state (no `VC_STATE_CONNECTED` gate) since device pickers need to
|
||||
populate pre-connect. `vc_free_device_list` is now a real free (was a no-op stub).
|
||||
- [x] **Input device selection** (`vc_set_input_device`) — stores the device id on the
|
||||
targeted `LocalStream` (new field); for the MIC stream, if the engine is already running,
|
||||
restarts it (`stop()` + `ensure_audio_running()`) to pick up the new device. Simplified: it
|
||||
restarts unconditionally rather than trying to detect whether the device id actually
|
||||
changed (`AudioEngine` has no getter for "current device").
|
||||
- [x] **VAD/PTT input gate** (`vc_set_input_mode`, `vc_set_push_to_talk`) — new
|
||||
`EnergyVadProcessor` (`core/src/audio/apm_processor.cpp`) implementing the existing
|
||||
`ApmProcessor` interface: energy/RMS threshold (default ~0.025 normalized) + hang-time
|
||||
(default 300 ms, matching `kTalkHangoverMs`). New factory `ApmProcessor::create_vad()`,
|
||||
kept separate from `create()` (which recv-side per-stream NS still uses, unaffected by this
|
||||
pass). `vc_client` gained `current_input_mode_`/`ptt_active_`/`mic_vad_`; the gate is
|
||||
inserted in `on_capture_frame`, **MIC-only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
|
||||
(gating a desktop-audio share on the user's own voice activity would silently drop shared
|
||||
music/video audio). `last_capture_ms` (drives the talk indicator) is now updated *after* the
|
||||
gate check, not before, so a VAD/PTT-closed frame never shows as "talking". `mic_vad_` is
|
||||
constructed once the MIC stream's `StreamAnnounceResult` lands (on `io_thread_`), not lazily
|
||||
inside the capture path.
|
||||
- [x] **True stereo playback** — `AudioParams::channels` split into `capture_channels` (stays
|
||||
1) and `playback_channels` (now 2, unconditionally). `AudioEngine::on_playback` no longer
|
||||
downmixes decoded stereo streams to mono before mixing — stereo decode output is mixed
|
||||
directly (L→L, R→R); mono decode output is upmixed (duplicated into both channels). Falls
|
||||
back to a 1-channel playback device once if the 2-channel `ma_device_init` fails (unusual
|
||||
hardware). New test-only `AudioEngine::mix_for_test()` exposes the mixer for white-box
|
||||
testing without a real `ma_device`.
|
||||
- [x] **WASAPI loopback capture for `SCREEN_AUDIO`** — new `VOICECAT_HAS_LOOPBACK` macro
|
||||
(`core/CMakeLists.txt`, Windows-only). `AudioEngine` gained a separate `loopback_device_`
|
||||
(own lifecycle, decoupled from the mic capture/playback devices) with
|
||||
`start_loopback_capture()`/`stop_loopback_capture()`, using miniaudio's
|
||||
`ma_device_type_loopback` against the default render endpoint. Its callback feeds
|
||||
`capture_cb_` directly (same pattern as the real mic capture device), **not** through
|
||||
`inject_capture()`'s test-only ring. Wired into `vc_client::handle_stream_announce_result`
|
||||
(start, alongside `ensure_audio_running()`) and `stream_stop` (stop) for
|
||||
`VC_STREAM_SCREEN_AUDIO`. Non-Windows builds keep `vc_test_inject_capture` as the only way to
|
||||
feed `SCREEN_AUDIO`.
|
||||
- [x] `tools/vccli/src/main.cpp` — new flags `--list-devices`, `--input-device`,
|
||||
`--input-mode vad|ptt`, `--share-screen-audio`; while `--voice` is running, a background
|
||||
stdin-reader thread accepts `ptt on`/`ptt off`/`mode vad`/`mode ptt` (the most portable way
|
||||
to drive PTT interactively from a headless CLI — no SIGUSR1 equivalent on Windows). Also
|
||||
prints `VC_EVENT_TALK_STATE`. Known minor caveat: on Windows the stdin-reader thread is
|
||||
detached (not joined) on exit, since `std::getline` can't be interrupted from another thread
|
||||
— a `vc_client*` use-after-free is theoretically possible if a command line arrives in the
|
||||
brief window between teardown and process exit; acceptable for a headless test/dev tool.
|
||||
- [x] `tests/test_vad_ptt_devices.cpp` — new test covering all four items above; registered in
|
||||
`tests/CMakeLists.txt`. `tests/test_smoke.cpp`'s device-list assertion is now conditional on
|
||||
`VOICECAT_HAS_AUDIO` (was a hard `VC_ERR_NOT_IMPLEMENTED` assertion) — `VC_OK` only, never
|
||||
`count > 0` (a headless CI build agent may legitimately report zero audio devices).
|
||||
|
||||
**Still explicitly out of scope** (carried forward, not silently dropped):
|
||||
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream (see
|
||||
docs/roadmap.md §2's superseding decision-log entry). There is **no AEC, NS, or AGC
|
||||
implementation at all**, not just a deferred VAD. The per-stream NS toggle
|
||||
(`vc_set_remote_stream(..., noise_reduction)`) is unaffected by this pass and stays exactly
|
||||
as inert as it was after M3 (`ApmPassthrough`, no PCM modification) — don't mistake this
|
||||
pass for having fixed it.
|
||||
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — Windows-only loopback in
|
||||
this pass.
|
||||
- Process-specific WASAPI loopback — whole-device capture only; inherently captures this
|
||||
app's own incoming voice mix along with everything else playing.
|
||||
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
|
||||
(mutex lock, heap allocation for the stereo-upmix path, blocking `sendto`, all on the
|
||||
miniaudio real-time callback thread) — predates this work (was already present in M2/M3);
|
||||
documented here explicitly rather than silently carried forward again. Fixing it properly
|
||||
needs the lock-free ring-buffer hand-off `docs/architecture.md §3` specifies — a separate,
|
||||
larger refactor, out of scope for this pass.
|
||||
|
||||
---
|
||||
|
||||
## Decisions log
|
||||
|
||||
All architecture/scope decisions are settled and recorded in
|
||||
|
||||
Reference in New Issue
Block a user