diff --git a/CLAUDE.md b/CLAUDE.md index 7996cb4..dbaf005 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,11 +4,13 @@ Auto-loaded each session. This is the **map**: build commands, architecture at a where everything is. For the *working method* read [`AGENTS.md`](AGENTS.md); for *what's done and what's next* read [`PROGRESS.md`](PROGRESS.md); for *design* read [`docs/`](docs/). -> **One-line status:** M2 voice/media plane is complete and verified through the real client -> library, not just a raw-socket harness (`ctest --test-dir build/m1-dev` green — 10/10 tests, -> including `test_voice_client_abi` driving two real `vc_client`s end-to-end, and `vccli -> --voice` manually verified live). Next up is **M3** (multi-stream, per-channel tuning, -> listener-side NR). See [`PROGRESS.md`](PROGRESS.md). +> **One-line status:** M3 (multi-stream, per-channel tuning) is complete, plus a follow-up +> pass closing the device enumeration / VAD-PTT input gate / stereo playback / WASAPI loopback +> gaps it left open (`ctest --test-dir build/m1-dev` green — 12/12 tests, including +> `test_vad_ptt_devices`, and `vccli --voice`/`--list-devices` manually verified live). Real +> `webrtc-audio-processing` (AEC/NS/AGC) is still unbuilt — no working Windows/MSVC port +> upstream — so v1 ships a lightweight energy/RMS VAD instead. Next up is **M4** (native +> clients). See [`PROGRESS.md`](PROGRESS.md). VoiceCat = self-hosted native voice & text chat (TeamSpeak/Mumble-style). Plain TCP (control) + UDP (media), no WebRTC, encrypted by default. A shared C++ core (`libvoicecat`) drives diff --git a/PROGRESS.md b/PROGRESS.md index 9f39959..17be6cf 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -10,19 +10,29 @@ up instantly. Newest status at the top. ## ▶ Where we left off / next action -- **Done:** **M3 — multi-stream & per-channel tuning** ✓ complete (2026-06-16). See the M3 - section below for the full file-by-file change list. `ctest --test-dir build/m1-dev` — - **11/11 tests** green (3 consecutive full-suite runs), including the new - `test_m3_multistream` (real `vc_client`s, not raw sockets — same lesson as M2: ABI-level - coverage is what proves the client library, not just the wire protocol). - **Two items intentionally still open** (carried forward, not silently dropped): - - `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices` - (device enumeration + VAD/PTT input gate) remain `VC_ERR_NOT_IMPLEMENTED` — explicitly - scoped out of this M3 pass; revisit in a future milestone. - - Stereo Opus is now wire-correct end-to-end (a channel configured `MODE_STEREO` really - encodes/decodes 2-channel Opus packets), but `AudioEngine`'s playback mixer/output device - stays mono internally — stereo streams are downmixed (avg L/R) immediately after decode, - before mixing. True stereo *playback output* is a follow-up, not part of M3. +- **Done:** **Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI + loopback** ✓ complete (2026-06-16). Closes all three items M3 explicitly carried forward as + out of scope (see the dated section below for the full file-by-file change list). + `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive full-suite runs), + including the new `test_vad_ptt_devices` (real `vc_client`s against a real server, plus a + white-box `AudioEngine` stereo-mix check — same ABI-level-coverage lesson as M2/M3). + Manually verified live: `vccli --list-devices` against real hardware, and + `vccli --voice --input-mode vad` connecting/streaming without incident. + **Still explicitly out of scope** (carried forward, not silently dropped): + - Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream; v1 ships a + lightweight energy/RMS VAD instead (see docs/roadmap.md §2, docs/voice.md §8/§11). There is + **no AEC, NS, or AGC implementation at all**, not just a deferred VAD. + `vc_set_remote_stream(..., noise_reduction)`'s per-stream NS toggle is unaffected by this + pass and stays exactly as inert as it was after M3 (`ApmPassthrough`, no PCM modification). + - macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — this pass is + Windows-only for real loopback capture; other platforms keep `vc_test_inject_capture` as + the only way to feed `SCREEN_AUDIO`. + - Process-specific WASAPI loopback — miniaudio's loopback mode captures the whole render + endpoint (including this app's own incoming voice mix), not a single process. + - The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture` + (mutex lock, heap allocation, blocking `sendto` on the miniaudio real-time callback + thread) — predates this work, documented but not fixed; fixing it needs the lock-free + ring-buffer hand-off `docs/architecture.md §3` specifies, a separate, larger refactor. - **Next:** **M4 — native clients** (Windows C#, macOS/iOS Swift). See `docs/roadmap.md §M4`. --- @@ -233,6 +243,92 @@ not treated as pre-existing-and-out-of-scope: --- +## Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ (completed 2026-06-16) + +Closes all three items the M3 section above explicitly carried forward as out of scope. + +**Exit verification:** `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive +full-suite runs), including the new `test_vad_ptt_devices` (device enumeration + VAD/PTT gate +through real `vc_client`s against a real server, plus a white-box `AudioEngine` stereo-mix +check — no audio hardware needed for that last part). Also verified 5 consecutive standalone +runs of the new test alone, no flakes. Manually verified live on Windows: `vccli +--list-devices` against real hardware (3 input / 4 output devices, correct `is_default` +flags), and `vccli --voice --input-mode vad` connecting + streaming without incident. + +- [x] **Device enumeration** (`vc_list_devices`) — `AudioEngine::enumerate_devices(bool + capture)` (static, works without a running engine — inits a throwaway `ma_context` via + `ma_context_get_devices`). `device_id`/`vc_device.id` is an opaque hex-encoded raw + `ma_device_id` (not the device name — names aren't guaranteed unique); documented as an + internal contract callers must round-trip, never construct by hand. `vc_client::list_devices` + works in any connection state (no `VC_STATE_CONNECTED` gate) since device pickers need to + populate pre-connect. `vc_free_device_list` is now a real free (was a no-op stub). +- [x] **Input device selection** (`vc_set_input_device`) — stores the device id on the + targeted `LocalStream` (new field); for the MIC stream, if the engine is already running, + restarts it (`stop()` + `ensure_audio_running()`) to pick up the new device. Simplified: it + restarts unconditionally rather than trying to detect whether the device id actually + changed (`AudioEngine` has no getter for "current device"). +- [x] **VAD/PTT input gate** (`vc_set_input_mode`, `vc_set_push_to_talk`) — new + `EnergyVadProcessor` (`core/src/audio/apm_processor.cpp`) implementing the existing + `ApmProcessor` interface: energy/RMS threshold (default ~0.025 normalized) + hang-time + (default 300 ms, matching `kTalkHangoverMs`). New factory `ApmProcessor::create_vad()`, + kept separate from `create()` (which recv-side per-stream NS still uses, unaffected by this + pass). `vc_client` gained `current_input_mode_`/`ptt_active_`/`mic_vad_`; the gate is + inserted in `on_capture_frame`, **MIC-only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it + (gating a desktop-audio share on the user's own voice activity would silently drop shared + music/video audio). `last_capture_ms` (drives the talk indicator) is now updated *after* the + gate check, not before, so a VAD/PTT-closed frame never shows as "talking". `mic_vad_` is + constructed once the MIC stream's `StreamAnnounceResult` lands (on `io_thread_`), not lazily + inside the capture path. +- [x] **True stereo playback** — `AudioParams::channels` split into `capture_channels` (stays + 1) and `playback_channels` (now 2, unconditionally). `AudioEngine::on_playback` no longer + downmixes decoded stereo streams to mono before mixing — stereo decode output is mixed + directly (L→L, R→R); mono decode output is upmixed (duplicated into both channels). Falls + back to a 1-channel playback device once if the 2-channel `ma_device_init` fails (unusual + hardware). New test-only `AudioEngine::mix_for_test()` exposes the mixer for white-box + testing without a real `ma_device`. +- [x] **WASAPI loopback capture for `SCREEN_AUDIO`** — new `VOICECAT_HAS_LOOPBACK` macro + (`core/CMakeLists.txt`, Windows-only). `AudioEngine` gained a separate `loopback_device_` + (own lifecycle, decoupled from the mic capture/playback devices) with + `start_loopback_capture()`/`stop_loopback_capture()`, using miniaudio's + `ma_device_type_loopback` against the default render endpoint. Its callback feeds + `capture_cb_` directly (same pattern as the real mic capture device), **not** through + `inject_capture()`'s test-only ring. Wired into `vc_client::handle_stream_announce_result` + (start, alongside `ensure_audio_running()`) and `stream_stop` (stop) for + `VC_STREAM_SCREEN_AUDIO`. Non-Windows builds keep `vc_test_inject_capture` as the only way to + feed `SCREEN_AUDIO`. +- [x] `tools/vccli/src/main.cpp` — new flags `--list-devices`, `--input-device`, + `--input-mode vad|ptt`, `--share-screen-audio`; while `--voice` is running, a background + stdin-reader thread accepts `ptt on`/`ptt off`/`mode vad`/`mode ptt` (the most portable way + to drive PTT interactively from a headless CLI — no SIGUSR1 equivalent on Windows). Also + prints `VC_EVENT_TALK_STATE`. Known minor caveat: on Windows the stdin-reader thread is + detached (not joined) on exit, since `std::getline` can't be interrupted from another thread + — a `vc_client*` use-after-free is theoretically possible if a command line arrives in the + brief window between teardown and process exit; acceptable for a headless test/dev tool. +- [x] `tests/test_vad_ptt_devices.cpp` — new test covering all four items above; registered in + `tests/CMakeLists.txt`. `tests/test_smoke.cpp`'s device-list assertion is now conditional on + `VOICECAT_HAS_AUDIO` (was a hard `VC_ERR_NOT_IMPLEMENTED` assertion) — `VC_OK` only, never + `count > 0` (a headless CI build agent may legitimately report zero audio devices). + +**Still explicitly out of scope** (carried forward, not silently dropped): +- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream (see + docs/roadmap.md §2's superseding decision-log entry). There is **no AEC, NS, or AGC + implementation at all**, not just a deferred VAD. The per-stream NS toggle + (`vc_set_remote_stream(..., noise_reduction)`) is unaffected by this pass and stays exactly + as inert as it was after M3 (`ApmPassthrough`, no PCM modification) — don't mistake this + pass for having fixed it. +- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — Windows-only loopback in + this pass. +- Process-specific WASAPI loopback — whole-device capture only; inherently captures this + app's own incoming voice mix along with everything else playing. +- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture` + (mutex lock, heap allocation for the stereo-upmix path, blocking `sendto`, all on the + miniaudio real-time callback thread) — predates this work (was already present in M2/M3); + documented here explicitly rather than silently carried forward again. Fixing it properly + needs the lock-free ring-buffer hand-off `docs/architecture.md §3` specifies — a separate, + larger refactor, out of scope for this pass. + +--- + ## Decisions log All architecture/scope decisions are settled and recorded in diff --git a/core/CMakeLists.txt b/core/CMakeLists.txt index c800368..b2871a5 100644 --- a/core/CMakeLists.txt +++ b/core/CMakeLists.txt @@ -60,6 +60,11 @@ if(VOICECAT_USE_VCPKG_DEPS) if(WIN32) # AcceptEx / GetAcceptExSockaddrs live in mswsock; ws2_32 covers the base Winsock API. target_link_libraries(voicecat PRIVATE ws2_32 mswsock) + # Real desktop-audio loopback capture for SCREEN_AUDIO (miniaudio's ma_device_type_loopback + # is WASAPI-only). Other platforms keep vc_test_inject_capture as the only way to feed + # SCREEN_AUDIO until a per-platform loopback path is built (macOS: ScreenCaptureKit, per + # docs/voice.md §9 — not in scope yet). + target_compile_definitions(voicecat PUBLIC VOICECAT_HAS_LOOPBACK) endif() # Signal to C++ code that the real networking/crypto stack is available. diff --git a/core/src/audio/apm_processor.cpp b/core/src/audio/apm_processor.cpp index 7156a2c..7911e9d 100644 --- a/core/src/audio/apm_processor.cpp +++ b/core/src/audio/apm_processor.cpp @@ -1,7 +1,18 @@ #include "audio/apm_processor.h" +#include +#include + namespace voicecat::audio { +namespace { +int64_t steady_now_ms() { + return std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()) + .count(); +} +} // namespace + // ── ApmPassthrough ──────────────────────────────────────────────────────────── // No-op: returns true (VAD always open), does not modify PCM. // Replaced by WebrtcApmProcessor when VOICECAT_HAS_APM is defined. @@ -11,11 +22,47 @@ class ApmPassthrough final : public ApmProcessor { bool process_capture(int16_t*, int, int) override { return true; } }; +// ── EnergyVadProcessor ────────────────────────────────────────────────────── +// Lightweight, dependency-free energy/RMS VAD — see apm_processor.h's create_vad() doc comment +// for why this exists instead of a real APM. No AEC (process_render is a no-op); doesn't modify +// the PCM it's given, only inspects it. +class EnergyVadProcessor final : public ApmProcessor { + public: + EnergyVadProcessor(float rms_threshold, int64_t hang_time_ms) + : threshold_(rms_threshold), hang_time_ms_(hang_time_ms) {} + + void process_render(const int16_t*, int, int) override {} + + bool process_capture(int16_t* pcm, int samples, int /*sample_rate*/) override { + if (samples > 0) { + double sum_sq = 0.0; + for (int i = 0; i < samples; ++i) { + double s = static_cast(pcm[i]) / 32768.0; + sum_sq += s * s; + } + double rms = std::sqrt(sum_sq / samples); + if (rms >= threshold_) last_voice_ms_ = steady_now_ms(); + } + return (steady_now_ms() - last_voice_ms_) < hang_time_ms_; + } + + private: + float threshold_; + int64_t hang_time_ms_; + int64_t last_voice_ms_ = 0; // epoch start -> gate begins closed until first loud frame +}; + std::unique_ptr ApmProcessor::create() { #ifdef VOICECAT_HAS_APM - // TODO(M3): return std::make_unique(); + // TODO: return std::make_unique(); — see create_vad()'s doc comment for + // why this isn't wired up yet (no working Windows/MSVC build upstream). #endif return std::make_unique(); } +std::unique_ptr ApmProcessor::create_vad(float rms_threshold, + int64_t hang_time_ms) { + return std::make_unique(rms_threshold, hang_time_ms); +} + } // namespace voicecat::audio diff --git a/core/src/audio/apm_processor.h b/core/src/audio/apm_processor.h index 2c19f95..880ab20 100644 --- a/core/src/audio/apm_processor.h +++ b/core/src/audio/apm_processor.h @@ -24,8 +24,24 @@ class ApmProcessor { // Returns false → caller should skip encode/send (silence gate). virtual bool process_capture(int16_t* pcm, int samples, int sample_rate) = 0; - // Factory: returns a real APM if VOICECAT_HAS_APM is defined, else a passthrough. + // Factory: returns a real APM if VOICECAT_HAS_APM is defined, else a passthrough. Used for + // recv-side per-stream noise reduction (docs/voice.md §10) — gating doesn't apply there, so + // this stays a passthrough until a real APM/NS backend exists (still inert; see + // PROGRESS.md). Do not use this for the send-side VAD gate — see create_vad() below. static std::unique_ptr create(); + + // Factory for the send-side input gate (docs/voice.md §11): a lightweight, dependency-free + // energy/RMS VAD with configurable threshold + hang-time. webrtc-audio-processing (the + // originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson build, + // unfinished MinGW support, hard abseil-cpp dependency — see PROGRESS.md), so this is the + // real v1 implementation behind the same ApmProcessor interface, not a passthrough. No AEC + // — process_render() is a no-op here; that's a real limitation versus the originally-planned + // APM, not just a deferred VAD. + // rms_threshold: normalized 0.0-1.0 RMS-of-int16-range; default ~0.025. + // hang_time_ms: how long the gate stays open after the last loud frame; default 300 ms + // (matches AudioEngine's kTalkHangoverMs so "talking" and "gate open" agree). + static std::unique_ptr create_vad(float rms_threshold = 0.025f, + int64_t hang_time_ms = 300); }; } // namespace voicecat::audio diff --git a/core/src/audio/audio_engine.cpp b/core/src/audio/audio_engine.cpp index a4701dc..2b231b2 100644 --- a/core/src/audio/audio_engine.cpp +++ b/core/src/audio/audio_engine.cpp @@ -17,6 +17,43 @@ int64_t now_ms() { std::chrono::steady_clock::now().time_since_epoch()) .count(); } + +#ifdef VOICECAT_HAS_AUDIO +// device_id encoding (DeviceInfo::id / AudioParams::*_device_id): a hex string of the raw +// ma_device_id bytes. Opaque on purpose — names aren't guaranteed unique, and this is the only +// stable handle miniaudio accepts back for device selection. Internal contract only; never +// exposed as anything other than an opaque round-tripped string at the C ABI boundary. +std::string hex_encode_device_id(const ma_device_id& id) { + static constexpr char kHex[] = "0123456789abcdef"; + const auto* bytes = reinterpret_cast(&id); + std::string out; + out.reserve(sizeof(ma_device_id) * 2); + for (size_t i = 0; i < sizeof(ma_device_id); ++i) { + out.push_back(kHex[bytes[i] >> 4]); + out.push_back(kHex[bytes[i] & 0xF]); + } + return out; +} + +bool hex_decode_device_id(const std::string& hex, ma_device_id* out) { + if (hex.size() != sizeof(ma_device_id) * 2) return false; + std::memset(out, 0, sizeof(ma_device_id)); + auto* bytes = reinterpret_cast(out); + auto nibble = [](char c) -> int { + if (c >= '0' && c <= '9') return c - '0'; + if (c >= 'a' && c <= 'f') return c - 'a' + 10; + if (c >= 'A' && c <= 'F') return c - 'A' + 10; + return -1; + }; + for (size_t i = 0; i < sizeof(ma_device_id); ++i) { + int hi = nibble(hex[i * 2]); + int lo = nibble(hex[i * 2 + 1]); + if (hi < 0 || lo < 0) return false; + bytes[i] = static_cast((hi << 4) | lo); + } + return true; +} +#endif // VOICECAT_HAS_AUDIO } // namespace // ── JitterBuffer ───────────────────────────────────────────────────────────── @@ -88,13 +125,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) { #ifdef VOICECAT_HAS_AUDIO // ── Capture device ────────────────────────────────────────────────────── + ma_device_id cap_id{}; + bool have_cap_id = !p.capture_device_id.empty() && + hex_decode_device_id(p.capture_device_id, &cap_id); + ma_device_config cap_cfg = ma_device_config_init(ma_device_type_capture); cap_cfg.capture.format = ma_format_s16; - cap_cfg.capture.channels = p.channels; + cap_cfg.capture.channels = p.capture_channels; cap_cfg.sampleRate = p.sample_rate; cap_cfg.dataCallback = capture_data_cb; cap_cfg.pUserData = this; - cap_cfg.capture.pDeviceID = nullptr; // always default for now + cap_cfg.capture.pDeviceID = have_cap_id ? &cap_id : nullptr; // null = default device if (ma_device_init(nullptr, &cap_cfg, &capture_device_) == MA_SUCCESS) { if (ma_device_start(&capture_device_) == MA_SUCCESS) { @@ -105,13 +146,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) { } // ── Playback device ───────────────────────────────────────────────────── + ma_device_id pb_id{}; + bool have_pb_id = !p.playback_device_id.empty() && + hex_decode_device_id(p.playback_device_id, &pb_id); + ma_device_config pb_cfg = ma_device_config_init(ma_device_type_playback); pb_cfg.playback.format = ma_format_s16; - pb_cfg.playback.channels = p.channels; + pb_cfg.playback.channels = p.playback_channels; pb_cfg.sampleRate = p.sample_rate; pb_cfg.dataCallback = playback_data_cb; pb_cfg.pUserData = this; - pb_cfg.playback.pDeviceID = nullptr; + pb_cfg.playback.pDeviceID = have_pb_id ? &pb_id : nullptr; if (ma_device_init(nullptr, &pb_cfg, &playback_device_) == MA_SUCCESS) { if (ma_device_start(&playback_device_) == MA_SUCCESS) { @@ -119,12 +164,57 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) { } else { ma_device_uninit(&playback_device_); } + } else if (p.playback_channels != 1) { + // Fallback: some unusual hardware may not accept the requested channel count even + // though WASAPI shared mode normally remixes transparently. Retry once at mono rather + // than leaving playback dead. + pb_cfg.playback.channels = 1; + if (ma_device_init(nullptr, &pb_cfg, &playback_device_) == MA_SUCCESS) { + if (ma_device_start(&playback_device_) == MA_SUCCESS) { + playback_started_ = true; + params_.playback_channels = 1; + } else { + ma_device_uninit(&playback_device_); + } + } } #endif // VOICECAT_HAS_AUDIO return true; } +std::vector AudioEngine::enumerate_devices(bool capture) { + std::vector result; +#ifdef VOICECAT_HAS_AUDIO + ma_context ctx; + if (ma_context_init(nullptr, 0, nullptr, &ctx) != MA_SUCCESS) return result; + + ma_device_info* playback_infos = nullptr; + ma_uint32 playback_count = 0; + ma_device_info* capture_infos = nullptr; + ma_uint32 capture_count = 0; + + if (ma_context_get_devices(&ctx, &playback_infos, &playback_count, &capture_infos, + &capture_count) == MA_SUCCESS) { + ma_device_info* infos = capture ? capture_infos : playback_infos; + ma_uint32 count = capture ? capture_count : playback_count; + result.reserve(count); + for (ma_uint32 i = 0; i < count; ++i) { + DeviceInfo d; + d.id = hex_encode_device_id(infos[i].id); + d.name = infos[i].name; + d.is_default = infos[i].isDefault != 0; + result.push_back(std::move(d)); + } + } + + ma_context_uninit(&ctx); +#else + (void)capture; +#endif // VOICECAT_HAS_AUDIO + return result; +} + void AudioEngine::stop() { if (!running_.exchange(false)) return; @@ -263,22 +353,25 @@ void AudioEngine::playback_data_cb(ma_device* dev, void* out, } void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) { - std::memset(out, 0, frames * params_.channels * sizeof(int16_t)); + const uint32_t pb_channels = params_.playback_channels; + std::memset(out, 0, frames * pb_channels * sizeof(int16_t)); std::unique_lock lk(streams_mu_, std::try_to_lock); if (!lk) return; // contended: emit silence this period #ifdef VOICECAT_HAS_OPUS - std::vector mix(frames * params_.channels, 0); + std::vector mix(frames * pb_channels, 0); for (auto& [ssrc, stream] : streams_) { if (stream.mute || !stream.decoder.valid()) continue; - // M3: a stream's Opus channel count (mono/stereo, per-channel AudioConfig) may differ - // from the engine-wide playback channel count (always mono in M3 — see PROGRESS.md). + // A stream's Opus channel count (mono/stereo, per-channel AudioConfig) may differ from + // the engine-wide playback channel count (now genuinely stereo — see docs/voice.md §8). // Decode into a buffer sized for the *decoder's* channel count (opus_decode's // frame_size parameter is samples-per-channel, not total samples — pass `frames`, - // not `frames * channels`), then convert at this boundary. + // not `frames * channels`), then convert at this mix boundary: true stereo decode + // output is mixed in directly (no downmix); mono decode output is upmixed L=R (mirrors + // the capture-side upmix in vc_client::on_capture_frame). int dec_channels = std::max(1, stream.decoder.channels()); auto maybe_frame = stream.jitter.pop(stream.playout_ts); std::vector pcm(frames * static_cast(dec_channels)); @@ -302,20 +395,26 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) { float g = stream.gain; for (int i = 0; i < n; ++i) { - // Downmix decoder output to the engine's mono accumulator if needed - // (average L/R); upmix is unnecessary since the mix buffer is per-channel. - int32_t sample = (dec_channels == 2) - ? (static_cast(pcm[i * 2]) + - static_cast(pcm[i * 2 + 1])) / 2 - : static_cast(pcm[i]); - for (uint32_t c = 0; c < params_.channels; ++c) - mix[i * params_.channels + c] += static_cast(sample * g); + if (dec_channels == 2) { + int32_t l = static_cast(static_cast(pcm[i * 2]) * g); + int32_t r = static_cast(static_cast(pcm[i * 2 + 1]) * g); + if (pb_channels == 2) { + mix[i * 2] += l; + mix[i * 2 + 1] += r; + } else { + mix[i] += (l + r) / 2; // playback device fell back to mono + } + } else { + int32_t sample = static_cast(static_cast(pcm[i]) * g); + for (uint32_t c = 0; c < pb_channels; ++c) + mix[i * pb_channels + c] += sample; // upmix mono -> all playback channels + } } } stream.playout_ts += frames; } - for (ma_uint32 i = 0; i < frames * params_.channels; ++i) + for (ma_uint32 i = 0; i < frames * pb_channels; ++i) out[i] = static_cast(std::clamp(mix[i], -32768, 32767)); #else (void)out; @@ -323,6 +422,62 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) { #endif } +#ifdef VOICECAT_HAS_LOOPBACK +void AudioEngine::loopback_data_cb(ma_device* dev, void* /*out*/, const void* in, + ma_uint32 frame_count) { + auto* self = static_cast(dev->pUserData); + self->on_loopback(static_cast(in), frame_count); +} + +void AudioEngine::on_loopback(const int16_t* pcm, ma_uint32 frames) { + // Same pattern as the real mic capture callback (on_capture) — call capture_cb_ directly, + // NOT through inject_capture()'s test-only ring (see audio_engine.h). + if (capture_cb_) capture_cb_(loopback_kind_, pcm, static_cast(frames)); +} + +bool AudioEngine::start_loopback_capture(int kind) { + if (loopback_started_) return false; // already running; stop_loopback_capture() first + + ma_device_config cfg = ma_device_config_init(ma_device_type_loopback); + cfg.capture.format = ma_format_s16; + cfg.capture.channels = 1; // let miniaudio's converter remix from the system's mix format + cfg.sampleRate = params_.sample_rate; + cfg.dataCallback = loopback_data_cb; + cfg.pUserData = this; + // pDeviceID left null: captures the default render endpoint (docs/voice.md §9). Windows + // 10 2004+ supports process-specific loopback via AUDIOCLIENT_ACTIVATION_PARAMS, but + // miniaudio's loopback mode only exposes whole-device capture — a future enhancement, not + // this pass. Whole-device loopback inherently captures this app's own incoming voice mix + // along with everything else playing — an accepted self-echo-loop characteristic of + // desktop-audio capture, not a bug. + + if (ma_device_init(nullptr, &cfg, &loopback_device_) != MA_SUCCESS) return false; + if (ma_device_start(&loopback_device_) != MA_SUCCESS) { + ma_device_uninit(&loopback_device_); + return false; + } + + loopback_kind_ = kind; + loopback_started_ = true; + return true; +} + +void AudioEngine::stop_loopback_capture() { + if (!loopback_started_) return; + ma_device_stop(&loopback_device_); + ma_device_uninit(&loopback_device_); + loopback_started_ = false; +} +#else // !VOICECAT_HAS_LOOPBACK +bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; } +void AudioEngine::stop_loopback_capture() {} +#endif // VOICECAT_HAS_LOOPBACK + #endif // VOICECAT_HAS_AUDIO +#ifndef VOICECAT_HAS_AUDIO +bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; } +void AudioEngine::stop_loopback_capture() {} +#endif + } // namespace voicecat::audio diff --git a/core/src/audio/audio_engine.h b/core/src/audio/audio_engine.h index 56192b1..fc7567f 100644 --- a/core/src/audio/audio_engine.h +++ b/core/src/audio/audio_engine.h @@ -75,10 +75,21 @@ class JitterBuffer { // ── AudioParams ────────────────────────────────────────────────────────────── struct AudioParams { uint32_t sample_rate = 48000; - uint32_t channels = 1; + uint32_t capture_channels = 1; // no stereo capture device (mic) in this pass + uint32_t playback_channels = 2; // true stereo output (see audio_engine.cpp on_playback) uint32_t frame_ms = 20; - std::string capture_device_id; // "" = default - std::string playback_device_id; // "" = default + std::string capture_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices + std::string playback_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices +}; + +// One enumerated device, returned by AudioEngine::enumerate_devices(). `id` is an internal, +// opaque hex-encoded ma_device_id — callers must always round-trip an id that came from +// enumerate_devices(); never construct one by hand (names aren't guaranteed unique, so the id +// is the only stable handle miniaudio accepts back for device selection). +struct DeviceInfo { + std::string id; + std::string name; + bool is_default = false; }; // ── AudioEngine ────────────────────────────────────────────────────────────── @@ -105,6 +116,18 @@ class AudioEngine { bool running() const { return running_.load(std::memory_order_acquire); } + // Enumerate input or output devices (for UI pickers / vc_list_devices). Static: works + // before any AudioEngine instance is running (device pickers need to populate pre-connect). + // Inits a throwaway ma_context if VOICECAT_HAS_AUDIO; returns {} otherwise. See DeviceInfo + // above for the `id` encoding contract. + static std::vector enumerate_devices(bool capture); + + // Real desktop-audio loopback capture (Windows/WASAPI only, VOICECAT_HAS_LOOPBACK). Feeds + // `kind`'s capture_cb_ directly, same pattern as the real mic capture device — NOT routed + // through inject_capture()'s test-only ring. No-op (returns false) when unsupported. + bool start_loopback_capture(int kind); + void stop_loopback_capture(); + // Inject synthetic PCM directly into the capture pipeline (bypasses real device). // Thread-safe; can be called from any thread including tests. `kind` selects which local // stream's injection tap to feed (each gets its own ring buffer); the 2-arg overload @@ -139,6 +162,13 @@ class AudioEngine { void init_recv_stream(uint32_t ssrc, const codec::OpusParams& p); #endif +#ifdef VOICECAT_HAS_AUDIO + // TEST-ONLY — exposes the playback mixer without a real ma_device, so tests can verify + // stereo mixing end-to-end (no audio hardware needed). Same logic the real playback + // callback uses; safe to call any time after start() (no ma_device touched). + void mix_for_test(int16_t* out, uint32_t frames) { on_playback(out, frames); } +#endif + private: #ifdef VOICECAT_HAS_AUDIO static void capture_data_cb(ma_device*, void*, const void*, ma_uint32); @@ -150,6 +180,18 @@ class AudioEngine { ma_device playback_device_{}; bool capture_started_ = false; bool playback_started_ = false; + +#ifdef VOICECAT_HAS_LOOPBACK + // Desktop-audio loopback capture (SCREEN_AUDIO) — own lifecycle, decoupled from + // capture_device_/playback_device_ start/stop (a screen-share can start/stop independently + // of the mic and of whether anything is currently playing back). + static void loopback_data_cb(ma_device*, void*, const void*, ma_uint32); + void on_loopback(const int16_t* pcm, ma_uint32 frames); + + ma_device loopback_device_{}; + bool loopback_started_ = false; + int loopback_kind_ = 0; +#endif #endif AudioParams params_{}; diff --git a/core/src/core/client.cpp b/core/src/core/client.cpp index cd7bb6e..dd3e152 100644 --- a/core/src/core/client.cpp +++ b/core/src/core/client.cpp @@ -657,11 +657,26 @@ void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples) { // playing while the user's mic is muted (docs §M3 scope decision). if (kind == static_cast(VC_STREAM_MIC) && self_mic_muted_.load(std::memory_order_acquire)) return; + + // Send-side input gate (docs/voice.md §11) — MIC only. SCREEN_AUDIO/AUX_DEVICE always + // bypass this: gating a screen-share on the user's own voice activity would silently drop + // shared music/video audio whenever the user isn't talking, which defeats the feature. + if (kind == static_cast(VC_STREAM_MIC)) { + if (current_input_mode_.load(std::memory_order_acquire) == VC_INPUT_PUSH_TO_TALK) { + if (!ptt_active_.load(std::memory_order_acquire)) return; // gate closed + } else if (mic_vad_) { + // EnergyVadProcessor never writes through the pointer (see apm_processor.cpp); the + // const_cast is safe and avoids splitting ApmProcessor's interface just for this. + if (!mic_vad_->process_capture(const_cast(pcm), samples, 48000)) return; + } + } + if (!media_send_crypto_) return; int fd = udp_fd_.load(std::memory_order_acquire); if (fd == -1) return; auto& ls = it->second; + // Updated only after the gate above, so a VAD/PTT-closed frame never shows as "talking". ls.last_capture_ms.store(client_now_ms(), std::memory_order_relaxed); uint8_t opus_buf[1500]; @@ -711,9 +726,15 @@ void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples) { void vc_client::ensure_audio_running() { if (audio_engine_.running()) return; voicecat::audio::AudioParams p; - p.sample_rate = 48000; - p.channels = 1; - p.frame_ms = 20; + p.sample_rate = 48000; + p.capture_channels = 1; // no stereo capture device in this pass + p.playback_channels = 2; // true stereo output (audio_engine.cpp on_playback) + p.frame_ms = 20; + { + std::lock_guard lk(local_streams_mu_); + auto it = local_streams_.find(static_cast(VC_STREAM_MIC)); + if (it != local_streams_.end()) p.capture_device_id = it->second.capture_device_id; + } audio_engine_.start(p, [this](int kind, const int16_t* pcm, int samples) { on_capture_frame(kind, pcm, samples); }); @@ -818,11 +839,12 @@ void vc_client::handle_stream_announce_result(uint64_t req_id, uint32_t self_uid; uint32_t emit_stream_id; bool ok_to_emit = false; + int kind; { std::lock_guard lk(local_streams_mu_); auto pit = pending_announce_kind_.find(req_id); if (pit == pending_announce_kind_.end()) return; // stray/duplicate — ignore - int kind = pit->second; + kind = pit->second; pending_announce_kind_.erase(pit); auto sit = local_streams_.find(kind); @@ -849,9 +871,18 @@ void vc_client::handle_stream_announce_result(uint64_t req_id, self_uid = self_user_id_; emit_stream_id = ls.stream_id; ok_to_emit = true; + + // Construct the MIC VAD once, here on io_thread_ (not the RT capture callback) — see + // client.h's comment on mic_vad_. + if (kind == static_cast(VC_STREAM_MIC) && !mic_vad_) { + mic_vad_ = voicecat::audio::ApmProcessor::create_vad(); + } } ensure_audio_running(); + if (kind == static_cast(VC_STREAM_SCREEN_AUDIO)) { + audio_engine_.start_loopback_capture(kind); + } if (ok_to_emit) { vc_event ev{}; @@ -865,14 +896,22 @@ void vc_client::handle_stream_announce_result(uint64_t req_id, vc_result vc_client::stream_stop(uint32_t stream_id) { if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED; + int stopped_kind = -1; { std::lock_guard lk(local_streams_mu_); + for (auto& [k, ls] : local_streams_) { + if (ls.stream_id == stream_id) { stopped_kind = k; break; } + } LocalStream* ls = find_local_stream_by_id(stream_id); if (!ls || !ls->active.load(std::memory_order_acquire)) return VC_ERR_INVALID_ARG; ls->active.store(false, std::memory_order_release); ls->encoder.destroy(); } + if (stopped_kind == static_cast(VC_STREAM_SCREEN_AUDIO)) { + audio_engine_.stop_loopback_capture(); + } + voicecat::v1::Envelope req; req.set_request_id(next_req_id_++); req.mutable_stream_stop()->set_stream_id(stream_id); @@ -894,9 +933,38 @@ vc_client::LocalStream* vc_client::find_local_stream_by_id(uint32_t stream_id) { return nullptr; } -vc_result vc_client::set_input_device(uint32_t, const char*) { return VC_ERR_NOT_IMPLEMENTED; } -vc_result vc_client::set_input_mode(vc_input_mode) { return VC_ERR_NOT_IMPLEMENTED; } -vc_result vc_client::set_push_to_talk(bool) { return VC_ERR_NOT_IMPLEMENTED; } +vc_result vc_client::set_input_device(uint32_t stream_id, const char* device_id) { + int kind = -1; + { + std::lock_guard lk(local_streams_mu_); + LocalStream* ls = find_local_stream_by_id(stream_id); + if (!ls) return VC_ERR_INVALID_ARG; + ls->capture_device_id = device_id ? device_id : ""; + for (auto& [k, entry] : local_streams_) { + if (&entry == ls) { kind = k; break; } + } + } + // Only the real capture device (MIC) is affected by device selection — SCREEN_AUDIO uses + // loopback capture (no input device to pick) and AUX_DEVICE isn't backed by a real device + // path yet. Restart the engine unconditionally when it's already running so the new device + // id takes effect; AudioEngine doesn't expose a getter for "is this the same device" so we + // don't try to skip the restart when it happens to be a no-op change. + if (kind == static_cast(VC_STREAM_MIC) && audio_engine_.running()) { + audio_engine_.stop(); + ensure_audio_running(); + } + return VC_OK; +} + +vc_result vc_client::set_input_mode(vc_input_mode mode) { + current_input_mode_.store(mode, std::memory_order_release); + return VC_OK; +} + +vc_result vc_client::set_push_to_talk(bool active) { + ptt_active_.store(active, std::memory_order_release); + return VC_OK; +} vc_result vc_client::set_self_mute(bool mic_muted, bool deafened) { if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED; @@ -984,10 +1052,31 @@ vc_result vc_client::test_inject_capture(uint32_t stream_id, const int16_t* pcm, return VC_OK; } -vc_result vc_client::list_devices(vc_device_kind, vc_device_list* out) { +vc_result vc_client::list_devices(vc_device_kind kind, vc_device_list* out) { +#ifdef VOICECAT_HAS_AUDIO + // Works in any connection state — device pickers need to populate pre-connect. + auto devices = voicecat::audio::AudioEngine::enumerate_devices(kind == VC_DEVICE_INPUT); + + auto* items = new vc_device[devices.size()]; + for (size_t i = 0; i < devices.size(); ++i) { + const auto& d = devices[i]; + auto* id = new char[d.id.size() + 1]; + auto* name = new char[d.name.size() + 1]; + std::memcpy(id, d.id.c_str(), d.id.size() + 1); + std::memcpy(name, d.name.c_str(), d.name.size() + 1); + items[i].id = id; + items[i].name = name; + items[i].is_default = d.is_default ? 1 : 0; + } + out->items = items; + out->count = devices.size(); + return VC_OK; +#else + (void)kind; out->items = nullptr; out->count = 0; return VC_ERR_NOT_IMPLEMENTED; +#endif } void vc_client::run_talk_timer() { diff --git a/core/src/core/client.h b/core/src/core/client.h index f80f1d1..c1bebde 100644 --- a/core/src/core/client.h +++ b/core/src/core/client.h @@ -147,9 +147,16 @@ struct vc_client { // vc_get_stream_audio_config() has something to read back for our own streams. voicecat::codec::OpusParams effective_params; - // Talk-indicator edge detection (docs/voice.md §7) — updated in on_capture_frame. + // Talk-indicator edge detection (docs/voice.md §7) — updated in on_capture_frame, after + // the VAD/PTT gate so a gated-closed frame doesn't show as "talking". std::atomic last_capture_ms{0}; bool talking = false; + + // Device-enumeration follow-up: the device this stream's capture should use ("" = + // default). Only meaningful for VC_STREAM_MIC today (the real capture device); set via + // vc_set_input_device. Opaque id from AudioEngine::enumerate_devices — see + // audio_engine.h's DeviceInfo doc comment. + std::string capture_device_id; }; mutable std::mutex local_streams_mu_; std::unordered_map local_streams_; // keyed by vc_stream_kind @@ -174,6 +181,14 @@ struct vc_client { std::atomic self_mic_muted_{false}; std::atomic self_deafened_{false}; + // Follow-up to M3: send-side input gate (docs/voice.md §11). MIC-only — SCREEN_AUDIO/ + // AUX_DEVICE are never gated (see PROGRESS.md for the rationale). Pure local state, no + // protocol traffic. mic_vad_ is constructed once the MIC stream's StreamAnnounceResult + // lands (handle_stream_announce_result, on io_thread_ — not the RT capture callback). + std::atomic current_input_mode_{VC_INPUT_VOICE_ACTIVATION}; + std::atomic ptt_active_{false}; + std::unique_ptr mic_vad_; + // teardown_voice() is called both from run_io()'s own cleanup (on the io_thread_, when // the read loop exits) and from disconnect() (on the caller's thread) -- without // serializing those two call sites, both can see udp_thread_/talk_timer_thread_ as diff --git a/core/src/voicecat.cpp b/core/src/voicecat.cpp index 098c5b0..fa4ecbb 100644 --- a/core/src/voicecat.cpp +++ b/core/src/voicecat.cpp @@ -140,8 +140,12 @@ vc_result vc_list_devices(vc_client* c, vc_device_kind kind, vc_device_list* out } void vc_free_device_list(vc_device_list* list) { - if (list == nullptr) return; - /* Stub: no allocation yet. Real impl frees list->items here. */ + if (list == nullptr || list->items == nullptr) return; + for (size_t i = 0; i < list->count; ++i) { + delete[] list->items[i].id; + delete[] list->items[i].name; + } + delete[] list->items; list->items = nullptr; list->count = 0; } diff --git a/docs/README.md b/docs/README.md index db9ecf2..e8d1ef9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -19,7 +19,7 @@ that implementation can start from a shared, agreed plan. | Media transport | **UDP** secured by **TLS-exported keys + ChaCha20-Poly1305 AEAD** — mandatory, no plaintext mode (see [security.md](security.md)) | | Crypto libraries | **mbedTLS** (TLS 1.3) + **libsodium** (AEAD, Argon2id, Ed25519) — both permissive, **no GPL/LGPL anywhere** | | Voice codec | **Opus** (libopus 1.6), per-channel configurable mono/stereo, bitrate, frame size, FEC/DTX | -| Audio DSP | **webrtc-audio-processing (APM)** — AEC/NS/AGC/VAD. NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally. Input gate supports **VAD and PTT**, client-configurable. | +| Audio DSP | **webrtc-audio-processing (APM)** was the plan for AEC/NS/AGC/VAD, but has no working Windows/MSVC build upstream — v1 ships a lightweight energy/RMS VAD instead, no AEC/NS/AGC yet (see [tech-stack.md](tech-stack.md), [roadmap.md](roadmap.md) §2). NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally — this plumbing exists but is currently inert pending a real DSP backend. Input gate supports **VAD and PTT**, client-configurable. | | Identity | **Guests + admin-provisioned local accounts** (Argon2id, SQLite). No self-serve registration; guests toggleable per server. | | Text | **Ephemeral** — live relay, no server-side history in v1. | | Serialization | **Protocol Buffers** for the control plane; **custom binary** for voice frames | diff --git a/docs/architecture.md b/docs/architecture.md index 2ad12ee..56fb6a5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -126,6 +126,11 @@ Design notes: core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders meters/state*. This keeps the real-time path identical on every OS. (iOS is the one exception that needs UI-side cooperation — see below.) +- **Device enumeration works pre-connect.** `vc_list_devices` needs no live session — device + pickers can populate before `vc_connect`. `vc_device.id` is an opaque, internally-encoded + handle (currently a hex-encoded `ma_device_id`) — always round-trip an id that came from + `vc_list_devices`/`vc_get_stream_audio_config`; never construct one by hand. Tolerate an + empty list (a machine can legitimately have zero input or output devices). - **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are caller-allocated or returned with a paired `vc_free`. diff --git a/docs/roadmap.md b/docs/roadmap.md index 9959a81..5f98dc0 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -77,6 +77,13 @@ Settled and reflected throughout the docs: directory). (deployment.md §3) - **Bitrate ceiling:** server-config `opus.limits.max_bitrate_bps`. (deployment.md §2) - **Name:** "VoiceCat" stays as the internal placeholder. +- **DSP engine, superseded (2026-06-16):** the "webrtc-audio-processing (APM)" decision above + (AEC + NS/AGC/VAD in one module) could not be carried out — it has no working Windows/MSVC + build upstream (GCC-only Meson build, MinGW support unfinished, hard `abseil-cpp` dependency, + Linux-tested only). v1 ships a lightweight, dependency-free energy/RMS VAD instead, behind + the same `ApmProcessor` interface; there is **no AEC/NS/AGC implementation at all** yet. Real + `webrtc-audio-processing` stays a tracked future swap (e.g. if/when a Linux build target + exists). (voice.md §8, §11) ## 3. Open questions diff --git a/docs/tech-stack.md b/docs/tech-stack.md index 4716358..ea2527c 100644 --- a/docs/tech-stack.md +++ b/docs/tech-stack.md @@ -12,7 +12,7 @@ Concrete library choices with versions and rationale. Everything in the **core** | Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. | | Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. | | Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. | -| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) | 1.x (standalone APM) | **BSD-3.** The primary DSP engine: high-quality acoustic echo cancellation, noise suppression, AGC, and VAD in one tuned module. Used **send-side** (clean the mic) and **receive-side per user** (listener-chosen NS on a specific stream — voice.md §10). AEC is in from the start, not deferred. | +| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) — **planned, not built** | 1.x (standalone APM) | **BSD-3**, but has no working Windows/MSVC build upstream (GCC-only Meson, MinGW support unfinished, hard `abseil-cpp` dep — see roadmap.md §2). v1 ships a lightweight, dependency-free energy/RMS VAD instead (`core/src/audio/apm_processor.cpp`); there is **no AEC, NS, or AGC implementation at all** yet. Real APM stays a tracked future swap behind the same `ApmProcessor` interface. | | Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) | | Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. | | Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. | @@ -56,7 +56,7 @@ explicit resampling (speexdsp/libsamplerate) is only needed when a device can't | Tool | Use | |------|-----| | **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. | -| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, webrtc-audio-processing, speexdsp, asio, miniaudio). Reproducible across OSes. | +| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, asio, miniaudio — see `vcpkg.json`). `webrtc-audio-processing`/`speexdsp` are **not** in the manifest: no working vcpkg port / no working Windows/MSVC build exists upstream for the former; the latter was never actually wired up (the lightweight VAD needs no resampler). Reproducible across OSes. | | **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). | | **clang-format / clang-tidy** | Style + static analysis on the core. | | **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). | @@ -69,9 +69,11 @@ dependencies are permitted.** Every dependency below is BSD / MIT / ISC / Apache public-domain: - **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ · - **miniaudio** — public domain / MIT-0 ✅ · **speexdsp** — BSD ✅ · **protobuf** — BSD ✅ · - **SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅ · - **webrtc-audio-processing** — BSD-3 ✅ (heavier build, but core to the DSP path). + **miniaudio** — public domain / MIT-0 ✅ · **protobuf** — BSD ✅ · + **SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅. + **webrtc-audio-processing** would be BSD-3 ✅ if/when it's actually built in (see §1) — + not a live dependency today, so not part of the resolved vcpkg graph the license scanner + below checks. - **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for one entirely. diff --git a/docs/voice.md b/docs/voice.md index a2ffa93..69a2db0 100644 --- a/docs/voice.md +++ b/docs/voice.md @@ -159,20 +159,27 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**. ## 8. Capture/playback pipeline (inside the core) ``` - device ─(miniaudio capture, 48k)→ resample? → send-side APM - (AEC + NS + AGC + VAD/PTT gate) → Opus encode → frame header → AEAD → UDP send + device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate + → Opus encode → frame header → AEAD → UDP send UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode → per-stream recv-side NS (optional, per user) → per-stream gain/mute - → mixer (sum all ssrc) → (miniaudio playback, 48k) → device + → mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback, + 48k, stereo) → device ``` - Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA). -- **DSP engine: webrtc-audio-processing (APM)** — the "better one". It provides - high-quality **AEC** (acoustic echo cancellation, essential for speaker users), **noise - suppression**, **AGC**, and a **VAD** in one tuned module, BSD-licensed. speexdsp is kept - only for resampling and as a lightweight jitter-buffer reference. AEC is **in from the - start**, not deferred. + Playback is genuinely stereo end-to-end; capture stays mono (no stereo mic in v1). +- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC + + VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream + (confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard + `abseil-cpp` dependency, Linux-tested only — + [gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)). + v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS, + or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real + `webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor` + interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or + upstream Windows support matures. - The mixer sums decoded streams; clipping is handled by soft limiting on the master bus. ## 10. Noise reduction — two-sided @@ -197,13 +204,23 @@ NS is a clean drop-in on that per-stream stage. ## 11. Input activation — VAD and PTT (client-configurable) Whether the mic transmits is decided locally by the **input gate**, and the client supports -**both** modes, switchable per client (and ideally per input device): +**both** modes, switchable per client (`vc_set_input_mode`): -- **Voice activation (VAD):** the APM VAD opens the gate when speech is detected, with a - configurable threshold and hang-time to avoid clipping word tails. DTX naturally - complements this — when the gate is closed nothing (or only comfort noise) is sent. -- **Push-to-talk (PTT):** a held key/button opens the gate. The UI exposes a configurable - keybind; the core just receives gate open/close. +- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free + energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no + external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC + build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold + (default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms, + matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping + word tails. DTX naturally complements this — when the gate is closed nothing (or only + comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the + originally-planned APM, not just a deferred VAD. +- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI + exposes a configurable keybind; the core just receives gate open/close. + +Gating applies to the **MIC stream only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it +(gating a desktop-audio share on the user's own voice activity would silently drop shared +music/video audio whenever the user isn't talking, which defeats the feature). This is purely a send-side, client-local concern — it gates what gets encoded and sent. It needs **no protocol support**; remote talk indicators are still derived from the presence of @@ -217,7 +234,7 @@ normal stream; only the *source* is platform-specific. | Platform | Mechanism | Notes | |----------|-----------|-------| -| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | Cleanest case; no extra process. Can capture system mix or a specific endpoint. | +| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. | | **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. | | **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). | diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index f06c3e5..fdd4fde 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -86,4 +86,13 @@ if(VOICECAT_USE_VCPKG_DEPS) target_include_directories(test_m3_multistream PRIVATE ${VC_TEST_INTERNAL_INCLUDES}) add_test(NAME m3_multistream COMMAND test_m3_multistream) set_tests_properties(m3_multistream PROPERTIES TIMEOUT 60) + + # Post-M3 follow-up: device enumeration, VAD/PTT input gate, true stereo playback mixing — + # the items PROGRESS.md's M3 section explicitly carried forward as out of scope. + add_executable(test_vad_ptt_devices test_vad_ptt_devices.cpp) + target_link_libraries(test_vad_ptt_devices PRIVATE voicecat::server) + target_compile_features(test_vad_ptt_devices PRIVATE cxx_std_20) + target_include_directories(test_vad_ptt_devices PRIVATE ${VC_TEST_INTERNAL_INCLUDES}) + add_test(NAME vad_ptt_devices COMMAND test_vad_ptt_devices) + set_tests_properties(vad_ptt_devices PROPERTIES TIMEOUT 60) endif() diff --git a/tests/test_smoke.cpp b/tests/test_smoke.cpp index 7a76ffb..fa960ae 100644 --- a/tests/test_smoke.cpp +++ b/tests/test_smoke.cpp @@ -60,9 +60,17 @@ int main() { vc_result rc_join = vc_join_channel(c, 1, nullptr); CHECK(rc_join == VC_ERR_NOT_IMPLEMENTED || rc_join == VC_ERR_NOT_CONNECTED); - vc_device_list dl; - CHECK(vc_list_devices(c, VC_DEVICE_INPUT, &dl) == VC_ERR_NOT_IMPLEMENTED); + vc_device_list dl{}; + vc_result rc_devices = vc_list_devices(c, VC_DEVICE_INPUT, &dl); +#ifdef VOICECAT_HAS_AUDIO + // Real device enumeration is wired up once miniaudio is linked in (post-M3 follow-up). + // Never assert count > 0 here — a headless CI build agent may legitimately report zero + // audio devices; only that the call itself succeeded. + CHECK(rc_devices == VC_OK); +#else + CHECK(rc_devices == VC_ERR_NOT_IMPLEMENTED); CHECK(dl.count == 0); +#endif vc_free_device_list(&dl); vc_client_destroy(c); diff --git a/tests/test_vad_ptt_devices.cpp b/tests/test_vad_ptt_devices.cpp new file mode 100644 index 0000000..79a2fe7 --- /dev/null +++ b/tests/test_vad_ptt_devices.cpp @@ -0,0 +1,396 @@ +/* + * test_vad_ptt_devices — closes M3's "explicitly out of scope" gaps (PROGRESS.md): device + * enumeration, the VAD/PTT send-side input gate, and true stereo playback mixing. + * + * Mirrors test_m3_multistream.cpp's approach (real vc_client instances against a real + * in-process server, not raw sockets) for the ABI-level pieces, plus a white-box AudioEngine + * test for the stereo mixer (no audio hardware needed — see AudioEngine::mix_for_test). + * + * 1. Device enumeration (vc_list_devices) works pre-connect, for both kinds, and tolerates + * an empty list (headless CI build agents may have zero audio devices) — VC_OK is the + * only thing asserted, never count > 0. + * 2. VAD gate: under VC_INPUT_VOICE_ACTIVATION (the default), silent PCM never reaches the + * peer (no talking edge); loud PCM does. + * 3. PTT gate: under VC_INPUT_PUSH_TO_TALK, loud PCM is gated closed until + * vc_set_push_to_talk(1); then it reaches the peer. + * 4. Stereo playback mixer: white-box (AudioEngine directly) — a genuinely stereo decoded + * stream survives into the mix without being downmixed to mono. + */ +#include + +#ifdef VOICECAT_HAS_NET + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "voicecat.h" +#include "server.h" +#include "db.h" + +#ifdef VOICECAT_HAS_AUDIO +#include "audio/audio_engine.h" +#endif +#ifdef VOICECAT_HAS_OPUS +#include "codec/opus_codec.h" +#endif + +// ── Event tracking (same shape as test_m3_multistream.cpp) ────────────────────── + +struct TalkEvent { + uint32_t user_id; + uint32_t stream_id; + bool talking; +}; + +struct EventStore { + std::mutex mu; + std::condition_variable cv; + + bool auth_ok{false}; + uint32_t self_user_id{0}; + bool channel_list_received{false}; + bool saw_stream_started{false}; + std::vector talk_events; + bool disconnected{false}; + + const char* label{nullptr}; +}; + +static void on_event(void* user, const vc_event* ev) { + auto* s = static_cast(user); + std::lock_guard lk(s->mu); + switch (ev->type) { + case VC_EVENT_AUTH_RESULT: + s->auth_ok = (ev->result == VC_OK); + s->self_user_id = ev->user_id; + break; + case VC_EVENT_CHANNEL_LIST: + s->channel_list_received = true; + break; + case VC_EVENT_STREAM_STARTED: + s->saw_stream_started = true; + break; + case VC_EVENT_TALK_STATE: + s->talk_events.push_back({ev->user_id, ev->stream_id, ev->u32a != 0}); + break; + case VC_EVENT_DISCONNECTED: + s->disconnected = true; + break; + default: + break; + } + s->cv.notify_all(); +} + +template +static bool wait_for(EventStore& s, Pred pred, int timeout_ms) { + auto deadline = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout_ms); + std::unique_lock lk(s.mu); + return s.cv.wait_until(lk, deadline, [&] { return pred(s); }); +} + +static std::vector make_sine_frame(int frame_idx, float freq_hz, + int frame_samples = 960) { + std::vector pcm(frame_samples); + for (int i = 0; i < frame_samples; ++i) { + float t = static_cast(frame_idx * frame_samples + i) / 48000.0f; + pcm[i] = static_cast(std::sin(2.0f * 3.14159265f * freq_hz * t) * 16000.0f); + } + return pcm; +} + +static std::vector make_silence_frame(int frame_samples = 960) { + return std::vector(frame_samples, 0); +} + +// Did `talking==true` ever fire for (user_id, stream_id) at index >= `from`? +static bool saw_talking_true(EventStore& s, uint32_t user_id, uint32_t stream_id, size_t from) { + std::lock_guard lk(s.mu); + for (size_t i = from; i < s.talk_events.size(); ++i) { + auto& e = s.talk_events[i]; + if (e.user_id == user_id && e.stream_id == stream_id && e.talking) return true; + } + return false; +} + +static size_t talk_event_count(EventStore& s) { + std::lock_guard lk(s.mu); + return s.talk_events.size(); +} + +// ── Test harness ────────────────────────────────────────────────────────────── + +static int g_failures = 0; +#define CHECK(cond) \ + do { \ + if (!(cond)) { \ + std::printf("FAIL: %s (%s:%d)\n", #cond, __FILE__, __LINE__); \ + ++g_failures; \ + } \ + } while (0) + +// ── 1. Device enumeration (no server needed) ──────────────────────────────────── +static void test_device_enumeration() { + vc_config cfg{"test-devices", "0.1", VC_LOG_OFF}; + vc_callbacks cb{}; + vc_client* c = vc_client_create(&cfg, cb); + CHECK(c != nullptr); + + for (vc_device_kind kind : {VC_DEVICE_INPUT, VC_DEVICE_OUTPUT}) { + vc_device_list dl{}; + vc_result r = vc_list_devices(c, kind, &dl); +#ifdef VOICECAT_HAS_AUDIO + CHECK(r == VC_OK); + // Headless CI build agents may legitimately report zero devices — never assert + // count > 0, only that the call itself succeeded and the list is well-formed. + for (size_t i = 0; i < dl.count; ++i) { + CHECK(dl.items[i].id != nullptr); + CHECK(dl.items[i].name != nullptr); + } +#else + CHECK(r == VC_ERR_NOT_IMPLEMENTED); +#endif + vc_free_device_list(&dl); + vc_free_device_list(&dl); // idempotent — must not crash on a second call + } + + vc_client_destroy(c); + std::printf("test_device_enumeration: ok\n"); +} + +// ── 4. Stereo playback mixer (white-box, no audio hardware needed) ────────────── +#if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS) +static void test_stereo_mix() { + voicecat::audio::AudioEngine engine; + voicecat::audio::AudioParams p; + p.sample_rate = 48000; + p.capture_channels = 1; + p.playback_channels = 2; + p.frame_ms = 20; + CHECK(engine.start(p)); // capture_cb intentionally omitted — not exercised here + + voicecat::codec::OpusParams stereo_params; + stereo_params.stereo = true; + int frame_samples = voicecat::codec::opus_frame_samples(stereo_params); + + voicecat::codec::OpusEncoder enc; + CHECK(enc.init(stereo_params)); + + // Loud left channel, silent right channel — a real downmix would average them into a + // single audible-but-quieter centered sample; true stereo should keep them distinct. + std::vector interleaved(static_cast(frame_samples) * 2); + for (int i = 0; i < frame_samples; ++i) { + float t = static_cast(i) / 48000.0f; + interleaved[i * 2] = static_cast(std::sin(2.0f * 3.14159265f * 440.0f * t) * 20000.0f); + interleaved[i * 2 + 1] = 0; + } + + uint8_t opus_buf[1500]; + int opus_len = enc.encode(interleaved.data(), frame_samples, opus_buf, sizeof(opus_buf)); + CHECK(opus_len > 0); + + engine.init_recv_stream(/*ssrc=*/1, stereo_params); + + voicecat::audio::JitterBuffer::Frame f; + f.seq = 0; + f.timestamp = 0; + f.fec_present = false; + f.payload.assign(opus_buf, opus_buf + opus_len); + engine.push_recv_frame(1, std::move(f)); + + std::vector out(static_cast(frame_samples) * 2, 0); + engine.mix_for_test(out.data(), static_cast(frame_samples)); + + // If the engine downmixed (old M3 behavior), every L/R pair would be identical (the + // average of a loud sample and 0). True stereo should show a clear, consistent L != R + // difference across the frame. + int64_t total_diff = 0; + for (int i = 0; i < frame_samples; ++i) + total_diff += std::abs(static_cast(out[i * 2]) - static_cast(out[i * 2 + 1])); + CHECK(total_diff > static_cast(frame_samples) * 1000); // well above decode noise + + engine.remove_stream(1); + engine.stop(); + std::printf("test_stereo_mix: ok (total_diff=%lld)\n", static_cast(total_diff)); +} +#endif // VOICECAT_HAS_AUDIO && VOICECAT_HAS_OPUS + +// ── 2/3. VAD + PTT gate, through the real ABI against a real server ───────────── +static void test_vad_and_ptt_gate() { + auto tmp = std::filesystem::temp_directory_path() / + ("vctest_vadptt_" + std::to_string( + std::chrono::steady_clock::now().time_since_epoch().count())); + std::filesystem::create_directories(tmp); + std::string data_dir = tmp.string(); + + std::atomic bound_port{0}; + std::mutex ready_mu; + std::condition_variable ready_cv; + bool ready{false}; + + voicecat::server::Config cfg; + cfg.data_dir = data_dir; + cfg.bind_port = 0; + cfg.media_port = 0; + cfg.server_name = "VoiceCat-VadPttTest"; + cfg.allow_guests = true; + cfg.on_ready = [&](uint16_t p) { + bound_port.store(p); + { std::lock_guard lk(ready_mu); ready = true; } + ready_cv.notify_all(); + }; + + voicecat::server::Server server(cfg); + std::thread server_thread([&] { server.run(); }); + + { + std::unique_lock lk(ready_mu); + bool ok = ready_cv.wait_for(lk, std::chrono::seconds(10), [&] { return ready; }); + if (!ok) { + std::printf("FAIL: server did not become ready within 10s\n"); + ++g_failures; + server.stop(); + server_thread.join(); + std::filesystem::remove_all(tmp); + return; + } + } + + uint16_t port = bound_port.load(); + std::printf("test_vad_and_ptt_gate: server ready on :%u\n", port); + + EventStore evA; + evA.label = "A"; + vc_callbacks cbA{on_event, nullptr, &evA}; + vc_config cfgA{"test-A", "0.1", VC_LOG_OFF}; + vc_client* clientA = vc_client_create(&cfgA, cbA); + CHECK(clientA != nullptr); + CHECK(vc_connect(clientA, "127.0.0.1", port) == VC_OK); + CHECK(vc_authenticate_guest(clientA, "VP-A") == VC_OK); + CHECK(wait_for(evA, [](EventStore& s) { return s.auth_ok; }, 8000)); + CHECK(wait_for(evA, [](EventStore& s) { return s.channel_list_received; }, 3000)); + + EventStore evB; + evB.label = "B"; + vc_callbacks cbB{on_event, nullptr, &evB}; + vc_config cfgB{"test-B", "0.1", VC_LOG_OFF}; + vc_client* clientB = vc_client_create(&cfgB, cbB); + CHECK(clientB != nullptr); + CHECK(vc_connect(clientB, "127.0.0.1", port) == VC_OK); + CHECK(vc_authenticate_guest(clientB, "VP-B") == VC_OK); + CHECK(wait_for(evB, [](EventStore& s) { return s.auth_ok; }, 8000)); + CHECK(wait_for(evB, [](EventStore& s) { return s.channel_list_received; }, 3000)); + + uint32_t a_uid = 0; + { std::lock_guard lk(evA.mu); a_uid = evA.self_user_id; } + + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + + vc_stream_desc mic_desc{}; + mic_desc.kind = VC_STREAM_MIC; + mic_desc.label = "mic"; + uint32_t mic_sid = 0; + CHECK(vc_stream_start(clientA, &mic_desc, &mic_sid) == VC_OK); + CHECK(wait_for(evB, [](EventStore& s) { return s.saw_stream_started; }, 5000)); + CHECK(wait_for(evA, [](EventStore& s) { return s.saw_stream_started; }, 5000)); + + // ── 2a. VAD mode (default), silent PCM: must NOT reach B as a talking edge ── + CHECK(vc_set_input_mode(clientA, VC_INPUT_VOICE_ACTIVATION) == VC_OK); + for (int i = 0; i < 15; ++i) { + auto silence = make_silence_frame(); + CHECK(vc_test_inject_capture(clientA, mic_sid, silence.data(), silence.size()) == VC_OK); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + CHECK(!saw_talking_true(evB, a_uid, mic_sid, 0)); + + // ── 2b. VAD mode, loud PCM: must reach B as a talking edge ────────────────── + size_t mark = talk_event_count(evB); + for (int i = 0; i < 20; ++i) { + auto loud = make_sine_frame(i, 440.0f); + CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + CHECK(wait_for(evB, [&](EventStore& s) { + for (size_t i = mark; i < s.talk_events.size(); ++i) { + auto& e = s.talk_events[i]; + if (e.user_id == a_uid && e.stream_id == mic_sid && e.talking) return true; + } + return false; + }, 3000)); + + // ── 3a. PTT mode, key up: loud PCM must NOT reach B as a new talking edge ─── + CHECK(vc_set_input_mode(clientA, VC_INPUT_PUSH_TO_TALK) == VC_OK); + CHECK(vc_set_push_to_talk(clientA, 0) == VC_OK); + // Let any in-flight VAD-driven talking state lapse (hang-time ~300ms) before measuring. + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + mark = talk_event_count(evB); + for (int i = 0; i < 20; ++i) { + auto loud = make_sine_frame(i, 440.0f); + CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + CHECK(!saw_talking_true(evB, a_uid, mic_sid, mark)); + + // ── 3b. PTT mode, key down: loud PCM must reach B as a talking edge ───────── + CHECK(vc_set_push_to_talk(clientA, 1) == VC_OK); + mark = talk_event_count(evB); + for (int i = 0; i < 20; ++i) { + auto loud = make_sine_frame(i, 440.0f); + CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + CHECK(wait_for(evB, [&](EventStore& s) { + for (size_t i = mark; i < s.talk_events.size(); ++i) { + auto& e = s.talk_events[i]; + if (e.user_id == a_uid && e.stream_id == mic_sid && e.talking) return true; + } + return false; + }, 3000)); + + { std::lock_guard lk(evA.mu); CHECK(!evA.disconnected); } + { std::lock_guard lk(evB.mu); CHECK(!evB.disconnected); } + + vc_disconnect(clientA); + vc_disconnect(clientB); + vc_client_destroy(clientA); + vc_client_destroy(clientB); + + server.stop(); + server_thread.join(); + std::filesystem::remove_all(tmp); + + std::printf("test_vad_and_ptt_gate: done\n"); +} + +int main() { + test_device_enumeration(); +#if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS) + test_stereo_mix(); +#endif + test_vad_and_ptt_gate(); + + if (g_failures == 0) { + std::printf("vad_ptt_devices: all checks passed\n"); + return 0; + } + std::printf("vad_ptt_devices: %d failure(s)\n", g_failures); + return 1; +} + +#else // !VOICECAT_HAS_NET + +int main() { + std::printf("vad_ptt_devices: SKIP (VOICECAT_HAS_NET not defined)\n"); + return 0; +} + +#endif // VOICECAT_HAS_NET diff --git a/tools/vccli/src/main.cpp b/tools/vccli/src/main.cpp index a7e22c5..5718ade 100644 --- a/tools/vccli/src/main.cpp +++ b/tools/vccli/src/main.cpp @@ -10,6 +10,8 @@ #include #include #include +#include +#include #include #include @@ -58,6 +60,10 @@ void on_event(void* user, const vc_event* ev) { case VC_EVENT_TEXT_MESSAGE: std::printf("[text] from=%u: %s\n", ev->user_id, ev->text ? ev->text : ""); break; + case VC_EVENT_TALK_STATE: + std::printf("[voice] talk state: user_id=%u stream_id=%u talking=%u\n", ev->user_id, + ev->stream_id, ev->u32a); + break; case VC_EVENT_ERROR: std::fprintf(stderr, "[error] rc=%d: %s\n", ev->result, ev->text ? ev->text : ""); break; @@ -82,14 +88,58 @@ bool wait_until(std::atomic& flag, int timeout_ms) { void print_usage() { std::printf( "usage: vccli [--host H] [--port P] [--nick NAME] [--channel ID]\n" - " [--voice] [--mute] [--text MSG]\n" - " --host H server host (default 127.0.0.1)\n" - " --port P server TCP port (default 8384)\n" - " --nick NAME guest nickname (default vccli-test)\n" - " --channel ID channel to join after auth (default 1, Lobby)\n" - " --voice start a MIC stream and stay connected until Ctrl+C\n" - " --mute start with the mic muted (only meaningful with --voice)\n" - " --text MSG send MSG to the channel, then exit\n"); + " [--voice] [--mute] [--text MSG] [--list-devices]\n" + " [--input-device ID] [--input-mode vad|ptt] [--share-screen-audio]\n" + " --host H server host (default 127.0.0.1)\n" + " --port P server TCP port (default 8384)\n" + " --nick NAME guest nickname (default vccli-test)\n" + " --channel ID channel to join after auth (default 1, Lobby)\n" + " --voice start a MIC stream and stay connected until Ctrl+C\n" + " --mute start with the mic muted (only meaningful with --voice)\n" + " --text MSG send MSG to the channel, then exit\n" + " --list-devices print input/output devices (vc_list_devices) and exit\n" + " --input-device ID use device ID (from --list-devices) for the MIC stream\n" + " --input-mode vad|ptt send-side input gate mode (default vad)\n" + " --share-screen-audio also start a SCREEN_AUDIO stream (WASAPI loopback on Windows)\n" + "\n" + "While --voice is running, stdin accepts: \"ptt on\", \"ptt off\", \"mode vad\",\n" + "\"mode ptt\" (PTT key state can't be held interactively in a headless CLI, so it's\n" + "toggled via these commands instead).\n"); +} + +void print_device_list(vc_client* c, vc_device_kind kind, const char* label) { + vc_device_list dl{}; + vc_result r = vc_list_devices(c, kind, &dl); + std::printf("%s devices: vc_list_devices -> %d (%s), count=%zu\n", label, r, + vc_result_string(r), dl.count); + for (size_t i = 0; i < dl.count; ++i) { + std::printf(" [%s] %s%s\n", dl.items[i].id, dl.items[i].name, + dl.items[i].is_default ? " (default)" : ""); + } + vc_free_device_list(&dl); +} + +// Background stdin reader for --voice mode: the most portable way to drive PTT/mode toggles +// interactively from a headless CLI (no SIGUSR1 equivalent on Windows). +void run_stdin_commands(vc_client* c, std::atomic& stop) { + std::string line; + while (!stop.load() && std::getline(std::cin, line)) { + if (line == "ptt on") { + vc_set_push_to_talk(c, 1); + std::printf("[ptt] on\n"); + } else if (line == "ptt off") { + vc_set_push_to_talk(c, 0); + std::printf("[ptt] off\n"); + } else if (line == "mode vad") { + vc_set_input_mode(c, VC_INPUT_VOICE_ACTIVATION); + std::printf("[mode] vad\n"); + } else if (line == "mode ptt") { + vc_set_input_mode(c, VC_INPUT_PUSH_TO_TALK); + std::printf("[mode] ptt\n"); + } else if (!line.empty()) { + std::fprintf(stderr, "unknown command: %s\n", line.c_str()); + } + } } } // namespace @@ -107,6 +157,11 @@ int main(int argc, char** argv) { bool start_muted = false; std::string text_msg; bool have_text = false; + bool list_devices = false; + std::string input_device; + bool have_input_device = false; + vc_input_mode input_mode = VC_INPUT_VOICE_ACTIVATION; + bool share_screen_audio = false; for (int i = 1; i < argc; ++i) { std::string a = argv[i]; @@ -118,6 +173,14 @@ int main(int argc, char** argv) { else if (a == "--voice") voice_mode = true; else if (a == "--mute") start_muted = true; else if (a == "--text") { text_msg = next(); have_text = true; } + else if (a == "--list-devices") list_devices = true; + else if (a == "--input-device") { input_device = next(); have_input_device = true; } + else if (a == "--input-mode") { + std::string m = next(); + if (m == "ptt") input_mode = VC_INPUT_PUSH_TO_TALK; + else if (m != "vad") { std::fprintf(stderr, "--input-mode must be vad|ptt\n"); return 1; } + } + else if (a == "--share-screen-audio") share_screen_audio = true; else if (a == "--help" || a == "-h") { print_usage(); return 0; } else { std::fprintf(stderr, "unknown flag: %s\n", a.c_str()); print_usage(); return 1; } } @@ -143,6 +206,14 @@ int main(int argc, char** argv) { return 1; } + if (list_devices) { + // Device enumeration works pre-connect (no server needed) — see docs/voice.md. + print_device_list(c, VC_DEVICE_INPUT, "input"); + print_device_list(c, VC_DEVICE_OUTPUT, "output"); + vc_client_destroy(c); + return 0; + } + vc_result r = vc_connect(c, host.c_str(), port); std::printf("vc_connect(%s:%u) -> %d (%s)\n", host.c_str(), port, r, vc_result_string(r)); if (r != VC_OK) { @@ -178,9 +249,13 @@ int main(int argc, char** argv) { if (start_muted) vc_set_self_mute(c, 1, 0); + r = vc_set_input_mode(c, input_mode); + std::printf("vc_set_input_mode(%s) -> %d (%s)\n", + input_mode == VC_INPUT_PUSH_TO_TALK ? "ptt" : "vad", r, vc_result_string(r)); + vc_stream_desc desc{}; desc.kind = VC_STREAM_MIC; - desc.device_id = nullptr; + desc.device_id = have_input_device ? input_device.c_str() : nullptr; desc.label = "Microphone"; uint32_t stream_id = 0; @@ -188,12 +263,43 @@ int main(int argc, char** argv) { std::printf("vc_stream_start -> %d (%s), stream_id=%u\n", r, vc_result_string(r), stream_id); - std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n"); + if (have_input_device && r == VC_OK) { + r = vc_set_input_device(c, stream_id, input_device.c_str()); + std::printf("vc_set_input_device -> %d (%s)\n", r, vc_result_string(r)); + } + + uint32_t screen_stream_id = 0; + if (share_screen_audio) { + vc_stream_desc sdesc{}; + sdesc.kind = VC_STREAM_SCREEN_AUDIO; + sdesc.label = "Desktop audio"; + r = vc_stream_start(c, &sdesc, &screen_stream_id); + std::printf("vc_stream_start(SCREEN_AUDIO) -> %d (%s), stream_id=%u\n", r, + vc_result_string(r), screen_stream_id); + } + + std::thread stdin_thread(run_stdin_commands, c, std::ref(g_stop)); + + std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n" + "(type \"ptt on\"/\"ptt off\"/\"mode vad\"/\"mode ptt\" to drive the input gate)\n"); while (!g_stop.load()) { std::this_thread::sleep_for(std::chrono::milliseconds(200)); } + if (share_screen_audio) vc_stream_stop(c, screen_stream_id); vc_stream_stop(c, stream_id); + + // Unblock the stdin reader and let it exit before the client is torn down. + if (stdin_thread.joinable()) { +#if defined(_WIN32) + // std::getline on a console stdin blocks indefinitely; on Windows there's no + // portable way to interrupt it from another thread, so detach rather than join — + // process exit reclaims the thread. + stdin_thread.detach(); +#else + stdin_thread.join(); +#endif + } } vc_disconnect(c);