1 Commits

Author SHA1 Message Date
5f6c223526 feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:

- Device enumeration (vc_list_devices) + input device selection
  (vc_set_input_device), backed by AudioEngine::enumerate_devices() via
  miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
  ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
  webrtc-audio-processing (the originally-planned APM) has no working
  Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
  abseil-cpp dependency), so VAD is a new lightweight, dependency-free
  energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
  interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
  stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
  stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
  miniaudio's loopback device type), replacing test-only injection as the
  production capture path.

Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).

Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.

Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.

Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
19 changed files with 1106 additions and 85 deletions

View File

@@ -4,11 +4,13 @@ Auto-loaded each session. This is the **map**: build commands, architecture at a
where everything is. For the *working method* read [`AGENTS.md`](AGENTS.md); for *what's done
and what's next* read [`PROGRESS.md`](PROGRESS.md); for *design* read [`docs/`](docs/).
> **One-line status:** M2 voice/media plane is complete and verified through the real client
> library, not just a raw-socket harness (`ctest --test-dir build/m1-dev` green — 10/10 tests,
> including `test_voice_client_abi` driving two real `vc_client`s end-to-end, and `vccli
> --voice` manually verified live). Next up is **M3** (multi-stream, per-channel tuning,
> listener-side NR). See [`PROGRESS.md`](PROGRESS.md).
> **One-line status:** M3 (multi-stream, per-channel tuning) is complete, plus a follow-up
> pass closing the device enumeration / VAD-PTT input gate / stereo playback / WASAPI loopback
> gaps it left open (`ctest --test-dir build/m1-dev` green — 12/12 tests, including
> `test_vad_ptt_devices`, and `vccli --voice`/`--list-devices` manually verified live). Real
> `webrtc-audio-processing` (AEC/NS/AGC) is still unbuilt — no working Windows/MSVC port
> upstream — so v1 ships a lightweight energy/RMS VAD instead. Next up is **M4** (native
> clients). See [`PROGRESS.md`](PROGRESS.md).
VoiceCat = self-hosted native voice & text chat (TeamSpeak/Mumble-style). Plain TCP (control)
+ UDP (media), no WebRTC, encrypted by default. A shared C++ core (`libvoicecat`) drives

View File

@@ -10,19 +10,29 @@ up instantly. Newest status at the top.
## ▶ Where we left off / next action
- **Done:** **M3 — multi-stream & per-channel tuning** ✓ complete (2026-06-16). See the M3
section below for the full file-by-file change list. `ctest --test-dir build/m1-dev`
**11/11 tests** green (3 consecutive full-suite runs), including the new
`test_m3_multistream` (real `vc_client`s, not raw sockets — same lesson as M2: ABI-level
coverage is what proves the client library, not just the wire protocol).
**Two items intentionally still open** (carried forward, not silently dropped):
- `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices`
(device enumeration + VAD/PTT input gate) remain `VC_ERR_NOT_IMPLEMENTED` — explicitly
scoped out of this M3 pass; revisit in a future milestone.
- Stereo Opus is now wire-correct end-to-end (a channel configured `MODE_STEREO` really
encodes/decodes 2-channel Opus packets), but `AudioEngine`'s playback mixer/output device
stays mono internally — stereo streams are downmixed (avg L/R) immediately after decode,
before mixing. True stereo *playback output* is a follow-up, not part of M3.
- **Done:** **Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI
loopback** ✓ complete (2026-06-16). Closes all three items M3 explicitly carried forward as
out of scope (see the dated section below for the full file-by-file change list).
`ctest --test-dir build/m1-dev`**12/12 tests** green (3 consecutive full-suite runs),
including the new `test_vad_ptt_devices` (real `vc_client`s against a real server, plus a
white-box `AudioEngine` stereo-mix check — same ABI-level-coverage lesson as M2/M3).
Manually verified live: `vccli --list-devices` against real hardware, and
`vccli --voice --input-mode vad` connecting/streaming without incident.
**Still explicitly out of scope** (carried forward, not silently dropped):
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream; v1 ships a
lightweight energy/RMS VAD instead (see docs/roadmap.md §2, docs/voice.md §8/§11). There is
**no AEC, NS, or AGC implementation at all**, not just a deferred VAD.
`vc_set_remote_stream(..., noise_reduction)`'s per-stream NS toggle is unaffected by this
pass and stays exactly as inert as it was after M3 (`ApmPassthrough`, no PCM modification).
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — this pass is
Windows-only for real loopback capture; other platforms keep `vc_test_inject_capture` as
the only way to feed `SCREEN_AUDIO`.
- Process-specific WASAPI loopback — miniaudio's loopback mode captures the whole render
endpoint (including this app's own incoming voice mix), not a single process.
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
(mutex lock, heap allocation, blocking `sendto` on the miniaudio real-time callback
thread) — predates this work, documented but not fixed; fixing it needs the lock-free
ring-buffer hand-off `docs/architecture.md §3` specifies, a separate, larger refactor.
- **Next:** **M4 — native clients** (Windows C#, macOS/iOS Swift). See `docs/roadmap.md §M4`.
---
@@ -233,6 +243,92 @@ not treated as pre-existing-and-out-of-scope:
---
## Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ (completed 2026-06-16)
Closes all three items the M3 section above explicitly carried forward as out of scope.
**Exit verification:** `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive
full-suite runs), including the new `test_vad_ptt_devices` (device enumeration + VAD/PTT gate
through real `vc_client`s against a real server, plus a white-box `AudioEngine` stereo-mix
check — no audio hardware needed for that last part). Also verified 5 consecutive standalone
runs of the new test alone, no flakes. Manually verified live on Windows: `vccli
--list-devices` against real hardware (3 input / 4 output devices, correct `is_default`
flags), and `vccli --voice --input-mode vad` connecting + streaming without incident.
- [x] **Device enumeration** (`vc_list_devices`) — `AudioEngine::enumerate_devices(bool
capture)` (static, works without a running engine — inits a throwaway `ma_context` via
`ma_context_get_devices`). `device_id`/`vc_device.id` is an opaque hex-encoded raw
`ma_device_id` (not the device name — names aren't guaranteed unique); documented as an
internal contract callers must round-trip, never construct by hand. `vc_client::list_devices`
works in any connection state (no `VC_STATE_CONNECTED` gate) since device pickers need to
populate pre-connect. `vc_free_device_list` is now a real free (was a no-op stub).
- [x] **Input device selection** (`vc_set_input_device`) — stores the device id on the
targeted `LocalStream` (new field); for the MIC stream, if the engine is already running,
restarts it (`stop()` + `ensure_audio_running()`) to pick up the new device. Simplified: it
restarts unconditionally rather than trying to detect whether the device id actually
changed (`AudioEngine` has no getter for "current device").
- [x] **VAD/PTT input gate** (`vc_set_input_mode`, `vc_set_push_to_talk`) — new
`EnergyVadProcessor` (`core/src/audio/apm_processor.cpp`) implementing the existing
`ApmProcessor` interface: energy/RMS threshold (default ~0.025 normalized) + hang-time
(default 300 ms, matching `kTalkHangoverMs`). New factory `ApmProcessor::create_vad()`,
kept separate from `create()` (which recv-side per-stream NS still uses, unaffected by this
pass). `vc_client` gained `current_input_mode_`/`ptt_active_`/`mic_vad_`; the gate is
inserted in `on_capture_frame`, **MIC-only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
(gating a desktop-audio share on the user's own voice activity would silently drop shared
music/video audio). `last_capture_ms` (drives the talk indicator) is now updated *after* the
gate check, not before, so a VAD/PTT-closed frame never shows as "talking". `mic_vad_` is
constructed once the MIC stream's `StreamAnnounceResult` lands (on `io_thread_`), not lazily
inside the capture path.
- [x] **True stereo playback** — `AudioParams::channels` split into `capture_channels` (stays
1) and `playback_channels` (now 2, unconditionally). `AudioEngine::on_playback` no longer
downmixes decoded stereo streams to mono before mixing — stereo decode output is mixed
directly (L→L, R→R); mono decode output is upmixed (duplicated into both channels). Falls
back to a 1-channel playback device once if the 2-channel `ma_device_init` fails (unusual
hardware). New test-only `AudioEngine::mix_for_test()` exposes the mixer for white-box
testing without a real `ma_device`.
- [x] **WASAPI loopback capture for `SCREEN_AUDIO`** — new `VOICECAT_HAS_LOOPBACK` macro
(`core/CMakeLists.txt`, Windows-only). `AudioEngine` gained a separate `loopback_device_`
(own lifecycle, decoupled from the mic capture/playback devices) with
`start_loopback_capture()`/`stop_loopback_capture()`, using miniaudio's
`ma_device_type_loopback` against the default render endpoint. Its callback feeds
`capture_cb_` directly (same pattern as the real mic capture device), **not** through
`inject_capture()`'s test-only ring. Wired into `vc_client::handle_stream_announce_result`
(start, alongside `ensure_audio_running()`) and `stream_stop` (stop) for
`VC_STREAM_SCREEN_AUDIO`. Non-Windows builds keep `vc_test_inject_capture` as the only way to
feed `SCREEN_AUDIO`.
- [x] `tools/vccli/src/main.cpp` — new flags `--list-devices`, `--input-device`,
`--input-mode vad|ptt`, `--share-screen-audio`; while `--voice` is running, a background
stdin-reader thread accepts `ptt on`/`ptt off`/`mode vad`/`mode ptt` (the most portable way
to drive PTT interactively from a headless CLI — no SIGUSR1 equivalent on Windows). Also
prints `VC_EVENT_TALK_STATE`. Known minor caveat: on Windows the stdin-reader thread is
detached (not joined) on exit, since `std::getline` can't be interrupted from another thread
— a `vc_client*` use-after-free is theoretically possible if a command line arrives in the
brief window between teardown and process exit; acceptable for a headless test/dev tool.
- [x] `tests/test_vad_ptt_devices.cpp` — new test covering all four items above; registered in
`tests/CMakeLists.txt`. `tests/test_smoke.cpp`'s device-list assertion is now conditional on
`VOICECAT_HAS_AUDIO` (was a hard `VC_ERR_NOT_IMPLEMENTED` assertion) — `VC_OK` only, never
`count > 0` (a headless CI build agent may legitimately report zero audio devices).
**Still explicitly out of scope** (carried forward, not silently dropped):
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream (see
docs/roadmap.md §2's superseding decision-log entry). There is **no AEC, NS, or AGC
implementation at all**, not just a deferred VAD. The per-stream NS toggle
(`vc_set_remote_stream(..., noise_reduction)`) is unaffected by this pass and stays exactly
as inert as it was after M3 (`ApmPassthrough`, no PCM modification) — don't mistake this
pass for having fixed it.
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — Windows-only loopback in
this pass.
- Process-specific WASAPI loopback — whole-device capture only; inherently captures this
app's own incoming voice mix along with everything else playing.
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
(mutex lock, heap allocation for the stereo-upmix path, blocking `sendto`, all on the
miniaudio real-time callback thread) — predates this work (was already present in M2/M3);
documented here explicitly rather than silently carried forward again. Fixing it properly
needs the lock-free ring-buffer hand-off `docs/architecture.md §3` specifies — a separate,
larger refactor, out of scope for this pass.
---
## Decisions log
All architecture/scope decisions are settled and recorded in

View File

@@ -60,6 +60,11 @@ if(VOICECAT_USE_VCPKG_DEPS)
if(WIN32)
# AcceptEx / GetAcceptExSockaddrs live in mswsock; ws2_32 covers the base Winsock API.
target_link_libraries(voicecat PRIVATE ws2_32 mswsock)
# Real desktop-audio loopback capture for SCREEN_AUDIO (miniaudio's ma_device_type_loopback
# is WASAPI-only). Other platforms keep vc_test_inject_capture as the only way to feed
# SCREEN_AUDIO until a per-platform loopback path is built (macOS: ScreenCaptureKit, per
# docs/voice.md §9 — not in scope yet).
target_compile_definitions(voicecat PUBLIC VOICECAT_HAS_LOOPBACK)
endif()
# Signal to C++ code that the real networking/crypto stack is available.

View File

@@ -1,7 +1,18 @@
#include "audio/apm_processor.h"
#include <chrono>
#include <cmath>
namespace voicecat::audio {
namespace {
int64_t steady_now_ms() {
return std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now().time_since_epoch())
.count();
}
} // namespace
// ── ApmPassthrough ────────────────────────────────────────────────────────────
// No-op: returns true (VAD always open), does not modify PCM.
// Replaced by WebrtcApmProcessor when VOICECAT_HAS_APM is defined.
@@ -11,11 +22,47 @@ class ApmPassthrough final : public ApmProcessor {
bool process_capture(int16_t*, int, int) override { return true; }
};
// ── EnergyVadProcessor ──────────────────────────────────────────────────────
// Lightweight, dependency-free energy/RMS VAD — see apm_processor.h's create_vad() doc comment
// for why this exists instead of a real APM. No AEC (process_render is a no-op); doesn't modify
// the PCM it's given, only inspects it.
class EnergyVadProcessor final : public ApmProcessor {
public:
EnergyVadProcessor(float rms_threshold, int64_t hang_time_ms)
: threshold_(rms_threshold), hang_time_ms_(hang_time_ms) {}
void process_render(const int16_t*, int, int) override {}
bool process_capture(int16_t* pcm, int samples, int /*sample_rate*/) override {
if (samples > 0) {
double sum_sq = 0.0;
for (int i = 0; i < samples; ++i) {
double s = static_cast<double>(pcm[i]) / 32768.0;
sum_sq += s * s;
}
double rms = std::sqrt(sum_sq / samples);
if (rms >= threshold_) last_voice_ms_ = steady_now_ms();
}
return (steady_now_ms() - last_voice_ms_) < hang_time_ms_;
}
private:
float threshold_;
int64_t hang_time_ms_;
int64_t last_voice_ms_ = 0; // epoch start -> gate begins closed until first loud frame
};
std::unique_ptr<ApmProcessor> ApmProcessor::create() {
#ifdef VOICECAT_HAS_APM
// TODO(M3): return std::make_unique<WebrtcApmProcessor>();
// TODO: return std::make_unique<WebrtcApmProcessor>(); — see create_vad()'s doc comment for
// why this isn't wired up yet (no working Windows/MSVC build upstream).
#endif
return std::make_unique<ApmPassthrough>();
}
std::unique_ptr<ApmProcessor> ApmProcessor::create_vad(float rms_threshold,
int64_t hang_time_ms) {
return std::make_unique<EnergyVadProcessor>(rms_threshold, hang_time_ms);
}
} // namespace voicecat::audio

View File

@@ -24,8 +24,24 @@ class ApmProcessor {
// Returns false → caller should skip encode/send (silence gate).
virtual bool process_capture(int16_t* pcm, int samples, int sample_rate) = 0;
// Factory: returns a real APM if VOICECAT_HAS_APM is defined, else a passthrough.
// Factory: returns a real APM if VOICECAT_HAS_APM is defined, else a passthrough. Used for
// recv-side per-stream noise reduction (docs/voice.md §10) — gating doesn't apply there, so
// this stays a passthrough until a real APM/NS backend exists (still inert; see
// PROGRESS.md). Do not use this for the send-side VAD gate — see create_vad() below.
static std::unique_ptr<ApmProcessor> create();
// Factory for the send-side input gate (docs/voice.md §11): a lightweight, dependency-free
// energy/RMS VAD with configurable threshold + hang-time. webrtc-audio-processing (the
// originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson build,
// unfinished MinGW support, hard abseil-cpp dependency — see PROGRESS.md), so this is the
// real v1 implementation behind the same ApmProcessor interface, not a passthrough. No AEC
// — process_render() is a no-op here; that's a real limitation versus the originally-planned
// APM, not just a deferred VAD.
// rms_threshold: normalized 0.0-1.0 RMS-of-int16-range; default ~0.025.
// hang_time_ms: how long the gate stays open after the last loud frame; default 300 ms
// (matches AudioEngine's kTalkHangoverMs so "talking" and "gate open" agree).
static std::unique_ptr<ApmProcessor> create_vad(float rms_threshold = 0.025f,
int64_t hang_time_ms = 300);
};
} // namespace voicecat::audio

View File

@@ -17,6 +17,43 @@ int64_t now_ms() {
std::chrono::steady_clock::now().time_since_epoch())
.count();
}
#ifdef VOICECAT_HAS_AUDIO
// device_id encoding (DeviceInfo::id / AudioParams::*_device_id): a hex string of the raw
// ma_device_id bytes. Opaque on purpose — names aren't guaranteed unique, and this is the only
// stable handle miniaudio accepts back for device selection. Internal contract only; never
// exposed as anything other than an opaque round-tripped string at the C ABI boundary.
std::string hex_encode_device_id(const ma_device_id& id) {
static constexpr char kHex[] = "0123456789abcdef";
const auto* bytes = reinterpret_cast<const uint8_t*>(&id);
std::string out;
out.reserve(sizeof(ma_device_id) * 2);
for (size_t i = 0; i < sizeof(ma_device_id); ++i) {
out.push_back(kHex[bytes[i] >> 4]);
out.push_back(kHex[bytes[i] & 0xF]);
}
return out;
}
bool hex_decode_device_id(const std::string& hex, ma_device_id* out) {
if (hex.size() != sizeof(ma_device_id) * 2) return false;
std::memset(out, 0, sizeof(ma_device_id));
auto* bytes = reinterpret_cast<uint8_t*>(out);
auto nibble = [](char c) -> int {
if (c >= '0' && c <= '9') return c - '0';
if (c >= 'a' && c <= 'f') return c - 'a' + 10;
if (c >= 'A' && c <= 'F') return c - 'A' + 10;
return -1;
};
for (size_t i = 0; i < sizeof(ma_device_id); ++i) {
int hi = nibble(hex[i * 2]);
int lo = nibble(hex[i * 2 + 1]);
if (hi < 0 || lo < 0) return false;
bytes[i] = static_cast<uint8_t>((hi << 4) | lo);
}
return true;
}
#endif // VOICECAT_HAS_AUDIO
} // namespace
// ── JitterBuffer ─────────────────────────────────────────────────────────────
@@ -88,13 +125,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
#ifdef VOICECAT_HAS_AUDIO
// ── Capture device ──────────────────────────────────────────────────────
ma_device_id cap_id{};
bool have_cap_id = !p.capture_device_id.empty() &&
hex_decode_device_id(p.capture_device_id, &cap_id);
ma_device_config cap_cfg = ma_device_config_init(ma_device_type_capture);
cap_cfg.capture.format = ma_format_s16;
cap_cfg.capture.channels = p.channels;
cap_cfg.capture.channels = p.capture_channels;
cap_cfg.sampleRate = p.sample_rate;
cap_cfg.dataCallback = capture_data_cb;
cap_cfg.pUserData = this;
cap_cfg.capture.pDeviceID = nullptr; // always default for now
cap_cfg.capture.pDeviceID = have_cap_id ? &cap_id : nullptr; // null = default device
if (ma_device_init(nullptr, &cap_cfg, &capture_device_) == MA_SUCCESS) {
if (ma_device_start(&capture_device_) == MA_SUCCESS) {
@@ -105,13 +146,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
}
// ── Playback device ─────────────────────────────────────────────────────
ma_device_id pb_id{};
bool have_pb_id = !p.playback_device_id.empty() &&
hex_decode_device_id(p.playback_device_id, &pb_id);
ma_device_config pb_cfg = ma_device_config_init(ma_device_type_playback);
pb_cfg.playback.format = ma_format_s16;
pb_cfg.playback.channels = p.channels;
pb_cfg.playback.channels = p.playback_channels;
pb_cfg.sampleRate = p.sample_rate;
pb_cfg.dataCallback = playback_data_cb;
pb_cfg.pUserData = this;
pb_cfg.playback.pDeviceID = nullptr;
pb_cfg.playback.pDeviceID = have_pb_id ? &pb_id : nullptr;
if (ma_device_init(nullptr, &pb_cfg, &playback_device_) == MA_SUCCESS) {
if (ma_device_start(&playback_device_) == MA_SUCCESS) {
@@ -119,12 +164,57 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
} else {
ma_device_uninit(&playback_device_);
}
} else if (p.playback_channels != 1) {
// Fallback: some unusual hardware may not accept the requested channel count even
// though WASAPI shared mode normally remixes transparently. Retry once at mono rather
// than leaving playback dead.
pb_cfg.playback.channels = 1;
if (ma_device_init(nullptr, &pb_cfg, &playback_device_) == MA_SUCCESS) {
if (ma_device_start(&playback_device_) == MA_SUCCESS) {
playback_started_ = true;
params_.playback_channels = 1;
} else {
ma_device_uninit(&playback_device_);
}
}
}
#endif // VOICECAT_HAS_AUDIO
return true;
}
std::vector<DeviceInfo> AudioEngine::enumerate_devices(bool capture) {
std::vector<DeviceInfo> result;
#ifdef VOICECAT_HAS_AUDIO
ma_context ctx;
if (ma_context_init(nullptr, 0, nullptr, &ctx) != MA_SUCCESS) return result;
ma_device_info* playback_infos = nullptr;
ma_uint32 playback_count = 0;
ma_device_info* capture_infos = nullptr;
ma_uint32 capture_count = 0;
if (ma_context_get_devices(&ctx, &playback_infos, &playback_count, &capture_infos,
&capture_count) == MA_SUCCESS) {
ma_device_info* infos = capture ? capture_infos : playback_infos;
ma_uint32 count = capture ? capture_count : playback_count;
result.reserve(count);
for (ma_uint32 i = 0; i < count; ++i) {
DeviceInfo d;
d.id = hex_encode_device_id(infos[i].id);
d.name = infos[i].name;
d.is_default = infos[i].isDefault != 0;
result.push_back(std::move(d));
}
}
ma_context_uninit(&ctx);
#else
(void)capture;
#endif // VOICECAT_HAS_AUDIO
return result;
}
void AudioEngine::stop() {
if (!running_.exchange(false)) return;
@@ -263,22 +353,25 @@ void AudioEngine::playback_data_cb(ma_device* dev, void* out,
}
void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
std::memset(out, 0, frames * params_.channels * sizeof(int16_t));
const uint32_t pb_channels = params_.playback_channels;
std::memset(out, 0, frames * pb_channels * sizeof(int16_t));
std::unique_lock lk(streams_mu_, std::try_to_lock);
if (!lk) return; // contended: emit silence this period
#ifdef VOICECAT_HAS_OPUS
std::vector<int32_t> mix(frames * params_.channels, 0);
std::vector<int32_t> mix(frames * pb_channels, 0);
for (auto& [ssrc, stream] : streams_) {
if (stream.mute || !stream.decoder.valid()) continue;
// M3: a stream's Opus channel count (mono/stereo, per-channel AudioConfig) may differ
// from the engine-wide playback channel count (always mono in M3 — see PROGRESS.md).
// A stream's Opus channel count (mono/stereo, per-channel AudioConfig) may differ from
// the engine-wide playback channel count (now genuinely stereo — see docs/voice.md §8).
// Decode into a buffer sized for the *decoder's* channel count (opus_decode's
// frame_size parameter is samples-per-channel, not total samples — pass `frames`,
// not `frames * channels`), then convert at this boundary.
// not `frames * channels`), then convert at this mix boundary: true stereo decode
// output is mixed in directly (no downmix); mono decode output is upmixed L=R (mirrors
// the capture-side upmix in vc_client::on_capture_frame).
int dec_channels = std::max(1, stream.decoder.channels());
auto maybe_frame = stream.jitter.pop(stream.playout_ts);
std::vector<int16_t> pcm(frames * static_cast<ma_uint32>(dec_channels));
@@ -302,20 +395,26 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
float g = stream.gain;
for (int i = 0; i < n; ++i) {
// Downmix decoder output to the engine's mono accumulator if needed
// (average L/R); upmix is unnecessary since the mix buffer is per-channel.
int32_t sample = (dec_channels == 2)
? (static_cast<int32_t>(pcm[i * 2]) +
static_cast<int32_t>(pcm[i * 2 + 1])) / 2
: static_cast<int32_t>(pcm[i]);
for (uint32_t c = 0; c < params_.channels; ++c)
mix[i * params_.channels + c] += static_cast<int32_t>(sample * g);
if (dec_channels == 2) {
int32_t l = static_cast<int32_t>(static_cast<float>(pcm[i * 2]) * g);
int32_t r = static_cast<int32_t>(static_cast<float>(pcm[i * 2 + 1]) * g);
if (pb_channels == 2) {
mix[i * 2] += l;
mix[i * 2 + 1] += r;
} else {
mix[i] += (l + r) / 2; // playback device fell back to mono
}
} else {
int32_t sample = static_cast<int32_t>(static_cast<float>(pcm[i]) * g);
for (uint32_t c = 0; c < pb_channels; ++c)
mix[i * pb_channels + c] += sample; // upmix mono -> all playback channels
}
}
}
stream.playout_ts += frames;
}
for (ma_uint32 i = 0; i < frames * params_.channels; ++i)
for (ma_uint32 i = 0; i < frames * pb_channels; ++i)
out[i] = static_cast<int16_t>(std::clamp(mix[i], -32768, 32767));
#else
(void)out;
@@ -323,6 +422,62 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
#endif
}
#ifdef VOICECAT_HAS_LOOPBACK
void AudioEngine::loopback_data_cb(ma_device* dev, void* /*out*/, const void* in,
ma_uint32 frame_count) {
auto* self = static_cast<AudioEngine*>(dev->pUserData);
self->on_loopback(static_cast<const int16_t*>(in), frame_count);
}
void AudioEngine::on_loopback(const int16_t* pcm, ma_uint32 frames) {
// Same pattern as the real mic capture callback (on_capture) — call capture_cb_ directly,
// NOT through inject_capture()'s test-only ring (see audio_engine.h).
if (capture_cb_) capture_cb_(loopback_kind_, pcm, static_cast<int>(frames));
}
bool AudioEngine::start_loopback_capture(int kind) {
if (loopback_started_) return false; // already running; stop_loopback_capture() first
ma_device_config cfg = ma_device_config_init(ma_device_type_loopback);
cfg.capture.format = ma_format_s16;
cfg.capture.channels = 1; // let miniaudio's converter remix from the system's mix format
cfg.sampleRate = params_.sample_rate;
cfg.dataCallback = loopback_data_cb;
cfg.pUserData = this;
// pDeviceID left null: captures the default render endpoint (docs/voice.md §9). Windows
// 10 2004+ supports process-specific loopback via AUDIOCLIENT_ACTIVATION_PARAMS, but
// miniaudio's loopback mode only exposes whole-device capture — a future enhancement, not
// this pass. Whole-device loopback inherently captures this app's own incoming voice mix
// along with everything else playing — an accepted self-echo-loop characteristic of
// desktop-audio capture, not a bug.
if (ma_device_init(nullptr, &cfg, &loopback_device_) != MA_SUCCESS) return false;
if (ma_device_start(&loopback_device_) != MA_SUCCESS) {
ma_device_uninit(&loopback_device_);
return false;
}
loopback_kind_ = kind;
loopback_started_ = true;
return true;
}
void AudioEngine::stop_loopback_capture() {
if (!loopback_started_) return;
ma_device_stop(&loopback_device_);
ma_device_uninit(&loopback_device_);
loopback_started_ = false;
}
#else // !VOICECAT_HAS_LOOPBACK
bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; }
void AudioEngine::stop_loopback_capture() {}
#endif // VOICECAT_HAS_LOOPBACK
#endif // VOICECAT_HAS_AUDIO
#ifndef VOICECAT_HAS_AUDIO
bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; }
void AudioEngine::stop_loopback_capture() {}
#endif
} // namespace voicecat::audio

View File

@@ -75,10 +75,21 @@ class JitterBuffer {
// ── AudioParams ──────────────────────────────────────────────────────────────
struct AudioParams {
uint32_t sample_rate = 48000;
uint32_t channels = 1;
uint32_t capture_channels = 1; // no stereo capture device (mic) in this pass
uint32_t playback_channels = 2; // true stereo output (see audio_engine.cpp on_playback)
uint32_t frame_ms = 20;
std::string capture_device_id; // "" = default
std::string playback_device_id; // "" = default
std::string capture_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
std::string playback_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
};
// One enumerated device, returned by AudioEngine::enumerate_devices(). `id` is an internal,
// opaque hex-encoded ma_device_id — callers must always round-trip an id that came from
// enumerate_devices(); never construct one by hand (names aren't guaranteed unique, so the id
// is the only stable handle miniaudio accepts back for device selection).
struct DeviceInfo {
std::string id;
std::string name;
bool is_default = false;
};
// ── AudioEngine ──────────────────────────────────────────────────────────────
@@ -105,6 +116,18 @@ class AudioEngine {
bool running() const { return running_.load(std::memory_order_acquire); }
// Enumerate input or output devices (for UI pickers / vc_list_devices). Static: works
// before any AudioEngine instance is running (device pickers need to populate pre-connect).
// Inits a throwaway ma_context if VOICECAT_HAS_AUDIO; returns {} otherwise. See DeviceInfo
// above for the `id` encoding contract.
static std::vector<DeviceInfo> enumerate_devices(bool capture);
// Real desktop-audio loopback capture (Windows/WASAPI only, VOICECAT_HAS_LOOPBACK). Feeds
// `kind`'s capture_cb_ directly, same pattern as the real mic capture device — NOT routed
// through inject_capture()'s test-only ring. No-op (returns false) when unsupported.
bool start_loopback_capture(int kind);
void stop_loopback_capture();
// Inject synthetic PCM directly into the capture pipeline (bypasses real device).
// Thread-safe; can be called from any thread including tests. `kind` selects which local
// stream's injection tap to feed (each gets its own ring buffer); the 2-arg overload
@@ -139,6 +162,13 @@ class AudioEngine {
void init_recv_stream(uint32_t ssrc, const codec::OpusParams& p);
#endif
#ifdef VOICECAT_HAS_AUDIO
// TEST-ONLY — exposes the playback mixer without a real ma_device, so tests can verify
// stereo mixing end-to-end (no audio hardware needed). Same logic the real playback
// callback uses; safe to call any time after start() (no ma_device touched).
void mix_for_test(int16_t* out, uint32_t frames) { on_playback(out, frames); }
#endif
private:
#ifdef VOICECAT_HAS_AUDIO
static void capture_data_cb(ma_device*, void*, const void*, ma_uint32);
@@ -150,6 +180,18 @@ class AudioEngine {
ma_device playback_device_{};
bool capture_started_ = false;
bool playback_started_ = false;
#ifdef VOICECAT_HAS_LOOPBACK
// Desktop-audio loopback capture (SCREEN_AUDIO) — own lifecycle, decoupled from
// capture_device_/playback_device_ start/stop (a screen-share can start/stop independently
// of the mic and of whether anything is currently playing back).
static void loopback_data_cb(ma_device*, void*, const void*, ma_uint32);
void on_loopback(const int16_t* pcm, ma_uint32 frames);
ma_device loopback_device_{};
bool loopback_started_ = false;
int loopback_kind_ = 0;
#endif
#endif
AudioParams params_{};

View File

@@ -657,11 +657,26 @@ void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples) {
// playing while the user's mic is muted (docs §M3 scope decision).
if (kind == static_cast<int>(VC_STREAM_MIC) &&
self_mic_muted_.load(std::memory_order_acquire)) return;
// Send-side input gate (docs/voice.md §11) — MIC only. SCREEN_AUDIO/AUX_DEVICE always
// bypass this: gating a screen-share on the user's own voice activity would silently drop
// shared music/video audio whenever the user isn't talking, which defeats the feature.
if (kind == static_cast<int>(VC_STREAM_MIC)) {
if (current_input_mode_.load(std::memory_order_acquire) == VC_INPUT_PUSH_TO_TALK) {
if (!ptt_active_.load(std::memory_order_acquire)) return; // gate closed
} else if (mic_vad_) {
// EnergyVadProcessor never writes through the pointer (see apm_processor.cpp); the
// const_cast is safe and avoids splitting ApmProcessor's interface just for this.
if (!mic_vad_->process_capture(const_cast<int16_t*>(pcm), samples, 48000)) return;
}
}
if (!media_send_crypto_) return;
int fd = udp_fd_.load(std::memory_order_acquire);
if (fd == -1) return;
auto& ls = it->second;
// Updated only after the gate above, so a VAD/PTT-closed frame never shows as "talking".
ls.last_capture_ms.store(client_now_ms(), std::memory_order_relaxed);
uint8_t opus_buf[1500];
@@ -711,9 +726,15 @@ void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples) {
void vc_client::ensure_audio_running() {
if (audio_engine_.running()) return;
voicecat::audio::AudioParams p;
p.sample_rate = 48000;
p.channels = 1;
p.frame_ms = 20;
p.sample_rate = 48000;
p.capture_channels = 1; // no stereo capture device in this pass
p.playback_channels = 2; // true stereo output (audio_engine.cpp on_playback)
p.frame_ms = 20;
{
std::lock_guard lk(local_streams_mu_);
auto it = local_streams_.find(static_cast<int>(VC_STREAM_MIC));
if (it != local_streams_.end()) p.capture_device_id = it->second.capture_device_id;
}
audio_engine_.start(p, [this](int kind, const int16_t* pcm, int samples) {
on_capture_frame(kind, pcm, samples);
});
@@ -818,11 +839,12 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
uint32_t self_uid;
uint32_t emit_stream_id;
bool ok_to_emit = false;
int kind;
{
std::lock_guard lk(local_streams_mu_);
auto pit = pending_announce_kind_.find(req_id);
if (pit == pending_announce_kind_.end()) return; // stray/duplicate — ignore
int kind = pit->second;
kind = pit->second;
pending_announce_kind_.erase(pit);
auto sit = local_streams_.find(kind);
@@ -849,9 +871,18 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
self_uid = self_user_id_;
emit_stream_id = ls.stream_id;
ok_to_emit = true;
// Construct the MIC VAD once, here on io_thread_ (not the RT capture callback) — see
// client.h's comment on mic_vad_.
if (kind == static_cast<int>(VC_STREAM_MIC) && !mic_vad_) {
mic_vad_ = voicecat::audio::ApmProcessor::create_vad();
}
}
ensure_audio_running();
if (kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO)) {
audio_engine_.start_loopback_capture(kind);
}
if (ok_to_emit) {
vc_event ev{};
@@ -865,14 +896,22 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
vc_result vc_client::stream_stop(uint32_t stream_id) {
if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED;
int stopped_kind = -1;
{
std::lock_guard lk(local_streams_mu_);
for (auto& [k, ls] : local_streams_) {
if (ls.stream_id == stream_id) { stopped_kind = k; break; }
}
LocalStream* ls = find_local_stream_by_id(stream_id);
if (!ls || !ls->active.load(std::memory_order_acquire)) return VC_ERR_INVALID_ARG;
ls->active.store(false, std::memory_order_release);
ls->encoder.destroy();
}
if (stopped_kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO)) {
audio_engine_.stop_loopback_capture();
}
voicecat::v1::Envelope req;
req.set_request_id(next_req_id_++);
req.mutable_stream_stop()->set_stream_id(stream_id);
@@ -894,9 +933,38 @@ vc_client::LocalStream* vc_client::find_local_stream_by_id(uint32_t stream_id) {
return nullptr;
}
vc_result vc_client::set_input_device(uint32_t, const char*) { return VC_ERR_NOT_IMPLEMENTED; }
vc_result vc_client::set_input_mode(vc_input_mode) { return VC_ERR_NOT_IMPLEMENTED; }
vc_result vc_client::set_push_to_talk(bool) { return VC_ERR_NOT_IMPLEMENTED; }
vc_result vc_client::set_input_device(uint32_t stream_id, const char* device_id) {
int kind = -1;
{
std::lock_guard lk(local_streams_mu_);
LocalStream* ls = find_local_stream_by_id(stream_id);
if (!ls) return VC_ERR_INVALID_ARG;
ls->capture_device_id = device_id ? device_id : "";
for (auto& [k, entry] : local_streams_) {
if (&entry == ls) { kind = k; break; }
}
}
// Only the real capture device (MIC) is affected by device selection — SCREEN_AUDIO uses
// loopback capture (no input device to pick) and AUX_DEVICE isn't backed by a real device
// path yet. Restart the engine unconditionally when it's already running so the new device
// id takes effect; AudioEngine doesn't expose a getter for "is this the same device" so we
// don't try to skip the restart when it happens to be a no-op change.
if (kind == static_cast<int>(VC_STREAM_MIC) && audio_engine_.running()) {
audio_engine_.stop();
ensure_audio_running();
}
return VC_OK;
}
vc_result vc_client::set_input_mode(vc_input_mode mode) {
current_input_mode_.store(mode, std::memory_order_release);
return VC_OK;
}
vc_result vc_client::set_push_to_talk(bool active) {
ptt_active_.store(active, std::memory_order_release);
return VC_OK;
}
vc_result vc_client::set_self_mute(bool mic_muted, bool deafened) {
if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED;
@@ -984,10 +1052,31 @@ vc_result vc_client::test_inject_capture(uint32_t stream_id, const int16_t* pcm,
return VC_OK;
}
vc_result vc_client::list_devices(vc_device_kind, vc_device_list* out) {
vc_result vc_client::list_devices(vc_device_kind kind, vc_device_list* out) {
#ifdef VOICECAT_HAS_AUDIO
// Works in any connection state — device pickers need to populate pre-connect.
auto devices = voicecat::audio::AudioEngine::enumerate_devices(kind == VC_DEVICE_INPUT);
auto* items = new vc_device[devices.size()];
for (size_t i = 0; i < devices.size(); ++i) {
const auto& d = devices[i];
auto* id = new char[d.id.size() + 1];
auto* name = new char[d.name.size() + 1];
std::memcpy(id, d.id.c_str(), d.id.size() + 1);
std::memcpy(name, d.name.c_str(), d.name.size() + 1);
items[i].id = id;
items[i].name = name;
items[i].is_default = d.is_default ? 1 : 0;
}
out->items = items;
out->count = devices.size();
return VC_OK;
#else
(void)kind;
out->items = nullptr;
out->count = 0;
return VC_ERR_NOT_IMPLEMENTED;
#endif
}
void vc_client::run_talk_timer() {

View File

@@ -147,9 +147,16 @@ struct vc_client {
// vc_get_stream_audio_config() has something to read back for our own streams.
voicecat::codec::OpusParams effective_params;
// Talk-indicator edge detection (docs/voice.md §7) — updated in on_capture_frame.
// Talk-indicator edge detection (docs/voice.md §7) — updated in on_capture_frame, after
// the VAD/PTT gate so a gated-closed frame doesn't show as "talking".
std::atomic<int64_t> last_capture_ms{0};
bool talking = false;
// Device-enumeration follow-up: the device this stream's capture should use ("" =
// default). Only meaningful for VC_STREAM_MIC today (the real capture device); set via
// vc_set_input_device. Opaque id from AudioEngine::enumerate_devices — see
// audio_engine.h's DeviceInfo doc comment.
std::string capture_device_id;
};
mutable std::mutex local_streams_mu_;
std::unordered_map<int, LocalStream> local_streams_; // keyed by vc_stream_kind
@@ -174,6 +181,14 @@ struct vc_client {
std::atomic<bool> self_mic_muted_{false};
std::atomic<bool> self_deafened_{false};
// Follow-up to M3: send-side input gate (docs/voice.md §11). MIC-only — SCREEN_AUDIO/
// AUX_DEVICE are never gated (see PROGRESS.md for the rationale). Pure local state, no
// protocol traffic. mic_vad_ is constructed once the MIC stream's StreamAnnounceResult
// lands (handle_stream_announce_result, on io_thread_ — not the RT capture callback).
std::atomic<vc_input_mode> current_input_mode_{VC_INPUT_VOICE_ACTIVATION};
std::atomic<bool> ptt_active_{false};
std::unique_ptr<voicecat::audio::ApmProcessor> mic_vad_;
// teardown_voice() is called both from run_io()'s own cleanup (on the io_thread_, when
// the read loop exits) and from disconnect() (on the caller's thread) -- without
// serializing those two call sites, both can see udp_thread_/talk_timer_thread_ as

View File

@@ -140,8 +140,12 @@ vc_result vc_list_devices(vc_client* c, vc_device_kind kind, vc_device_list* out
}
void vc_free_device_list(vc_device_list* list) {
if (list == nullptr) return;
/* Stub: no allocation yet. Real impl frees list->items here. */
if (list == nullptr || list->items == nullptr) return;
for (size_t i = 0; i < list->count; ++i) {
delete[] list->items[i].id;
delete[] list->items[i].name;
}
delete[] list->items;
list->items = nullptr;
list->count = 0;
}

View File

@@ -19,7 +19,7 @@ that implementation can start from a shared, agreed plan.
| Media transport | **UDP** secured by **TLS-exported keys + ChaCha20-Poly1305 AEAD** — mandatory, no plaintext mode (see [security.md](security.md)) |
| Crypto libraries | **mbedTLS** (TLS 1.3) + **libsodium** (AEAD, Argon2id, Ed25519) — both permissive, **no GPL/LGPL anywhere** |
| Voice codec | **Opus** (libopus 1.6), per-channel configurable mono/stereo, bitrate, frame size, FEC/DTX |
| Audio DSP | **webrtc-audio-processing (APM)** — AEC/NS/AGC/VAD. NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally. Input gate supports **VAD and PTT**, client-configurable. |
| Audio DSP | **webrtc-audio-processing (APM)** was the plan for AEC/NS/AGC/VAD, but has no working Windows/MSVC build upstream — v1 ships a lightweight energy/RMS VAD instead, no AEC/NS/AGC yet (see [tech-stack.md](tech-stack.md), [roadmap.md](roadmap.md) §2). NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally — this plumbing exists but is currently inert pending a real DSP backend. Input gate supports **VAD and PTT**, client-configurable. |
| Identity | **Guests + admin-provisioned local accounts** (Argon2id, SQLite). No self-serve registration; guests toggleable per server. |
| Text | **Ephemeral** — live relay, no server-side history in v1. |
| Serialization | **Protocol Buffers** for the control plane; **custom binary** for voice frames |

View File

@@ -126,6 +126,11 @@ Design notes:
core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
exception that needs UI-side cooperation — see below.)
- **Device enumeration works pre-connect.** `vc_list_devices` needs no live session — device
pickers can populate before `vc_connect`. `vc_device.id` is an opaque, internally-encoded
handle (currently a hex-encoded `ma_device_id`) — always round-trip an id that came from
`vc_list_devices`/`vc_get_stream_audio_config`; never construct one by hand. Tolerate an
empty list (a machine can legitimately have zero input or output devices).
- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
caller-allocated or returned with a paired `vc_free`.

View File

@@ -77,6 +77,13 @@ Settled and reflected throughout the docs:
directory). (deployment.md §3)
- **Bitrate ceiling:** server-config `opus.limits.max_bitrate_bps`. (deployment.md §2)
- **Name:** "VoiceCat" stays as the internal placeholder.
- **DSP engine, superseded (2026-06-16):** the "webrtc-audio-processing (APM)" decision above
(AEC + NS/AGC/VAD in one module) could not be carried out — it has no working Windows/MSVC
build upstream (GCC-only Meson build, MinGW support unfinished, hard `abseil-cpp` dependency,
Linux-tested only). v1 ships a lightweight, dependency-free energy/RMS VAD instead, behind
the same `ApmProcessor` interface; there is **no AEC/NS/AGC implementation at all** yet. Real
`webrtc-audio-processing` stays a tracked future swap (e.g. if/when a Linux build target
exists). (voice.md §8, §11)
## 3. Open questions

View File

@@ -12,7 +12,7 @@ Concrete library choices with versions and rationale. Everything in the **core**
| Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. |
| Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. |
| Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. |
| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) | 1.x (standalone APM) | **BSD-3.** The primary DSP engine: high-quality acoustic echo cancellation, noise suppression, AGC, and VAD in one tuned module. Used **send-side** (clean the mic) and **receive-side per user** (listener-chosen NS on a specific stream — voice.md §10). AEC is in from the start, not deferred. |
| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM)**planned, not built** | 1.x (standalone APM) | **BSD-3**, but has no working Windows/MSVC build upstream (GCC-only Meson, MinGW support unfinished, hard `abseil-cpp` dep — see roadmap.md §2). v1 ships a lightweight, dependency-free energy/RMS VAD instead (`core/src/audio/apm_processor.cpp`); there is **no AEC, NS, or AGC implementation at all** yet. Real APM stays a tracked future swap behind the same `ApmProcessor` interface. |
| Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) |
| Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. |
| Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. |
@@ -56,7 +56,7 @@ explicit resampling (speexdsp/libsamplerate) is only needed when a device can't
| Tool | Use |
|------|-----|
| **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. |
| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, webrtc-audio-processing, speexdsp, asio, miniaudio). Reproducible across OSes. |
| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, asio, miniaudio — see `vcpkg.json`). `webrtc-audio-processing`/`speexdsp` are **not** in the manifest: no working vcpkg port / no working Windows/MSVC build exists upstream for the former; the latter was never actually wired up (the lightweight VAD needs no resampler). Reproducible across OSes. |
| **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). |
| **clang-format / clang-tidy** | Style + static analysis on the core. |
| **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). |
@@ -69,9 +69,11 @@ dependencies are permitted.** Every dependency below is BSD / MIT / ISC / Apache
public-domain:
- **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ ·
**miniaudio** — public domain / MIT-0 ✅ · **speexdsp** — BSD ✅ · **protobuf** — BSD ✅ ·
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅ ·
**webrtc-audio-processing** BSD-3 ✅ (heavier build, but core to the DSP path).
**miniaudio** — public domain / MIT-0 ✅ · **protobuf** — BSD ✅ ·
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅.
**webrtc-audio-processing** would be BSD-3 ✅ if/when it's actually built in (see §1) —
not a live dependency today, so not part of the resolved vcpkg graph the license scanner
below checks.
- **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag
in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for
one entirely.

View File

@@ -159,20 +159,27 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**.
## 8. Capture/playback pipeline (inside the core)
```
device ─(miniaudio capture, 48k)→ resample? → send-side APM
(AEC + NS + AGC + VAD/PTT gate) → Opus encode → frame header → AEAD → UDP send
device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
→ Opus encode → frame header → AEAD → UDP send
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
→ mixer (sum all ssrc) → (miniaudio playback, 48k) → device
→ mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
48k, stereo) → device
```
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
- **DSP engine: webrtc-audio-processing (APM)** — the "better one". It provides
high-quality **AEC** (acoustic echo cancellation, essential for speaker users), **noise
suppression**, **AGC**, and a **VAD** in one tuned module, BSD-licensed. speexdsp is kept
only for resampling and as a lightweight jitter-buffer reference. AEC is **in from the
start**, not deferred.
Playback is genuinely stereo end-to-end; capture stays mono (no stereo mic in v1).
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
`abseil-cpp` dependency, Linux-tested only —
[gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)).
v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS,
or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real
`webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor`
interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or
upstream Windows support matures.
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
## 10. Noise reduction — two-sided
@@ -197,13 +204,23 @@ NS is a clean drop-in on that per-stream stage.
## 11. Input activation — VAD and PTT (client-configurable)
Whether the mic transmits is decided locally by the **input gate**, and the client supports
**both** modes, switchable per client (and ideally per input device):
**both** modes, switchable per client (`vc_set_input_mode`):
- **Voice activation (VAD):** the APM VAD opens the gate when speech is detected, with a
configurable threshold and hang-time to avoid clipping word tails. DTX naturally
complements this — when the gate is closed nothing (or only comfort noise) is sent.
- **Push-to-talk (PTT):** a held key/button opens the gate. The UI exposes a configurable
keybind; the core just receives gate open/close.
- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free
energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no
external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC
build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold
(default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms,
matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping
word tails. DTX naturally complements this — when the gate is closed nothing (or only
comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the
originally-planned APM, not just a deferred VAD.
- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI
exposes a configurable keybind; the core just receives gate open/close.
Gating applies to the **MIC stream only**`SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
(gating a desktop-audio share on the user's own voice activity would silently drop shared
music/video audio whenever the user isn't talking, which defeats the feature).
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
needs **no protocol support**; remote talk indicators are still derived from the presence of
@@ -217,7 +234,7 @@ normal stream; only the *source* is platform-specific.
| Platform | Mechanism | Notes |
|----------|-----------|-------|
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | Cleanest case; no extra process. Can capture system mix or a specific endpoint. |
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |

View File

@@ -86,4 +86,13 @@ if(VOICECAT_USE_VCPKG_DEPS)
target_include_directories(test_m3_multistream PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME m3_multistream COMMAND test_m3_multistream)
set_tests_properties(m3_multistream PROPERTIES TIMEOUT 60)
# Post-M3 follow-up: device enumeration, VAD/PTT input gate, true stereo playback mixing —
# the items PROGRESS.md's M3 section explicitly carried forward as out of scope.
add_executable(test_vad_ptt_devices test_vad_ptt_devices.cpp)
target_link_libraries(test_vad_ptt_devices PRIVATE voicecat::server)
target_compile_features(test_vad_ptt_devices PRIVATE cxx_std_20)
target_include_directories(test_vad_ptt_devices PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME vad_ptt_devices COMMAND test_vad_ptt_devices)
set_tests_properties(vad_ptt_devices PROPERTIES TIMEOUT 60)
endif()

View File

@@ -60,9 +60,17 @@ int main() {
vc_result rc_join = vc_join_channel(c, 1, nullptr);
CHECK(rc_join == VC_ERR_NOT_IMPLEMENTED || rc_join == VC_ERR_NOT_CONNECTED);
vc_device_list dl;
CHECK(vc_list_devices(c, VC_DEVICE_INPUT, &dl) == VC_ERR_NOT_IMPLEMENTED);
vc_device_list dl{};
vc_result rc_devices = vc_list_devices(c, VC_DEVICE_INPUT, &dl);
#ifdef VOICECAT_HAS_AUDIO
// Real device enumeration is wired up once miniaudio is linked in (post-M3 follow-up).
// Never assert count > 0 here — a headless CI build agent may legitimately report zero
// audio devices; only that the call itself succeeded.
CHECK(rc_devices == VC_OK);
#else
CHECK(rc_devices == VC_ERR_NOT_IMPLEMENTED);
CHECK(dl.count == 0);
#endif
vc_free_device_list(&dl);
vc_client_destroy(c);

View File

@@ -0,0 +1,396 @@
/*
* test_vad_ptt_devices — closes M3's "explicitly out of scope" gaps (PROGRESS.md): device
* enumeration, the VAD/PTT send-side input gate, and true stereo playback mixing.
*
* Mirrors test_m3_multistream.cpp's approach (real vc_client instances against a real
* in-process server, not raw sockets) for the ABI-level pieces, plus a white-box AudioEngine
* test for the stereo mixer (no audio hardware needed — see AudioEngine::mix_for_test).
*
* 1. Device enumeration (vc_list_devices) works pre-connect, for both kinds, and tolerates
* an empty list (headless CI build agents may have zero audio devices) — VC_OK is the
* only thing asserted, never count > 0.
* 2. VAD gate: under VC_INPUT_VOICE_ACTIVATION (the default), silent PCM never reaches the
* peer (no talking edge); loud PCM does.
* 3. PTT gate: under VC_INPUT_PUSH_TO_TALK, loud PCM is gated closed until
* vc_set_push_to_talk(1); then it reaches the peer.
* 4. Stereo playback mixer: white-box (AudioEngine directly) — a genuinely stereo decoded
* stream survives into the mix without being downmixed to mono.
*/
#include <cstdio>
#ifdef VOICECAT_HAS_NET
#include <atomic>
#include <chrono>
#include <cmath>
#include <condition_variable>
#include <cstdlib>
#include <filesystem>
#include <mutex>
#include <string>
#include <thread>
#include <vector>
#include "voicecat.h"
#include "server.h"
#include "db.h"
#ifdef VOICECAT_HAS_AUDIO
#include "audio/audio_engine.h"
#endif
#ifdef VOICECAT_HAS_OPUS
#include "codec/opus_codec.h"
#endif
// ── Event tracking (same shape as test_m3_multistream.cpp) ──────────────────────
struct TalkEvent {
uint32_t user_id;
uint32_t stream_id;
bool talking;
};
struct EventStore {
std::mutex mu;
std::condition_variable cv;
bool auth_ok{false};
uint32_t self_user_id{0};
bool channel_list_received{false};
bool saw_stream_started{false};
std::vector<TalkEvent> talk_events;
bool disconnected{false};
const char* label{nullptr};
};
static void on_event(void* user, const vc_event* ev) {
auto* s = static_cast<EventStore*>(user);
std::lock_guard lk(s->mu);
switch (ev->type) {
case VC_EVENT_AUTH_RESULT:
s->auth_ok = (ev->result == VC_OK);
s->self_user_id = ev->user_id;
break;
case VC_EVENT_CHANNEL_LIST:
s->channel_list_received = true;
break;
case VC_EVENT_STREAM_STARTED:
s->saw_stream_started = true;
break;
case VC_EVENT_TALK_STATE:
s->talk_events.push_back({ev->user_id, ev->stream_id, ev->u32a != 0});
break;
case VC_EVENT_DISCONNECTED:
s->disconnected = true;
break;
default:
break;
}
s->cv.notify_all();
}
template <typename Pred>
static bool wait_for(EventStore& s, Pred pred, int timeout_ms) {
auto deadline = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout_ms);
std::unique_lock lk(s.mu);
return s.cv.wait_until(lk, deadline, [&] { return pred(s); });
}
static std::vector<int16_t> make_sine_frame(int frame_idx, float freq_hz,
int frame_samples = 960) {
std::vector<int16_t> pcm(frame_samples);
for (int i = 0; i < frame_samples; ++i) {
float t = static_cast<float>(frame_idx * frame_samples + i) / 48000.0f;
pcm[i] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * freq_hz * t) * 16000.0f);
}
return pcm;
}
static std::vector<int16_t> make_silence_frame(int frame_samples = 960) {
return std::vector<int16_t>(frame_samples, 0);
}
// Did `talking==true` ever fire for (user_id, stream_id) at index >= `from`?
static bool saw_talking_true(EventStore& s, uint32_t user_id, uint32_t stream_id, size_t from) {
std::lock_guard lk(s.mu);
for (size_t i = from; i < s.talk_events.size(); ++i) {
auto& e = s.talk_events[i];
if (e.user_id == user_id && e.stream_id == stream_id && e.talking) return true;
}
return false;
}
static size_t talk_event_count(EventStore& s) {
std::lock_guard lk(s.mu);
return s.talk_events.size();
}
// ── Test harness ──────────────────────────────────────────────────────────────
static int g_failures = 0;
#define CHECK(cond) \
do { \
if (!(cond)) { \
std::printf("FAIL: %s (%s:%d)\n", #cond, __FILE__, __LINE__); \
++g_failures; \
} \
} while (0)
// ── 1. Device enumeration (no server needed) ────────────────────────────────────
static void test_device_enumeration() {
vc_config cfg{"test-devices", "0.1", VC_LOG_OFF};
vc_callbacks cb{};
vc_client* c = vc_client_create(&cfg, cb);
CHECK(c != nullptr);
for (vc_device_kind kind : {VC_DEVICE_INPUT, VC_DEVICE_OUTPUT}) {
vc_device_list dl{};
vc_result r = vc_list_devices(c, kind, &dl);
#ifdef VOICECAT_HAS_AUDIO
CHECK(r == VC_OK);
// Headless CI build agents may legitimately report zero devices — never assert
// count > 0, only that the call itself succeeded and the list is well-formed.
for (size_t i = 0; i < dl.count; ++i) {
CHECK(dl.items[i].id != nullptr);
CHECK(dl.items[i].name != nullptr);
}
#else
CHECK(r == VC_ERR_NOT_IMPLEMENTED);
#endif
vc_free_device_list(&dl);
vc_free_device_list(&dl); // idempotent — must not crash on a second call
}
vc_client_destroy(c);
std::printf("test_device_enumeration: ok\n");
}
// ── 4. Stereo playback mixer (white-box, no audio hardware needed) ──────────────
#if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS)
static void test_stereo_mix() {
voicecat::audio::AudioEngine engine;
voicecat::audio::AudioParams p;
p.sample_rate = 48000;
p.capture_channels = 1;
p.playback_channels = 2;
p.frame_ms = 20;
CHECK(engine.start(p)); // capture_cb intentionally omitted — not exercised here
voicecat::codec::OpusParams stereo_params;
stereo_params.stereo = true;
int frame_samples = voicecat::codec::opus_frame_samples(stereo_params);
voicecat::codec::OpusEncoder enc;
CHECK(enc.init(stereo_params));
// Loud left channel, silent right channel — a real downmix would average them into a
// single audible-but-quieter centered sample; true stereo should keep them distinct.
std::vector<int16_t> interleaved(static_cast<size_t>(frame_samples) * 2);
for (int i = 0; i < frame_samples; ++i) {
float t = static_cast<float>(i) / 48000.0f;
interleaved[i * 2] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * 440.0f * t) * 20000.0f);
interleaved[i * 2 + 1] = 0;
}
uint8_t opus_buf[1500];
int opus_len = enc.encode(interleaved.data(), frame_samples, opus_buf, sizeof(opus_buf));
CHECK(opus_len > 0);
engine.init_recv_stream(/*ssrc=*/1, stereo_params);
voicecat::audio::JitterBuffer::Frame f;
f.seq = 0;
f.timestamp = 0;
f.fec_present = false;
f.payload.assign(opus_buf, opus_buf + opus_len);
engine.push_recv_frame(1, std::move(f));
std::vector<int16_t> out(static_cast<size_t>(frame_samples) * 2, 0);
engine.mix_for_test(out.data(), static_cast<uint32_t>(frame_samples));
// If the engine downmixed (old M3 behavior), every L/R pair would be identical (the
// average of a loud sample and 0). True stereo should show a clear, consistent L != R
// difference across the frame.
int64_t total_diff = 0;
for (int i = 0; i < frame_samples; ++i)
total_diff += std::abs(static_cast<int>(out[i * 2]) - static_cast<int>(out[i * 2 + 1]));
CHECK(total_diff > static_cast<int64_t>(frame_samples) * 1000); // well above decode noise
engine.remove_stream(1);
engine.stop();
std::printf("test_stereo_mix: ok (total_diff=%lld)\n", static_cast<long long>(total_diff));
}
#endif // VOICECAT_HAS_AUDIO && VOICECAT_HAS_OPUS
// ── 2/3. VAD + PTT gate, through the real ABI against a real server ─────────────
static void test_vad_and_ptt_gate() {
auto tmp = std::filesystem::temp_directory_path() /
("vctest_vadptt_" + std::to_string(
std::chrono::steady_clock::now().time_since_epoch().count()));
std::filesystem::create_directories(tmp);
std::string data_dir = tmp.string();
std::atomic<uint16_t> bound_port{0};
std::mutex ready_mu;
std::condition_variable ready_cv;
bool ready{false};
voicecat::server::Config cfg;
cfg.data_dir = data_dir;
cfg.bind_port = 0;
cfg.media_port = 0;
cfg.server_name = "VoiceCat-VadPttTest";
cfg.allow_guests = true;
cfg.on_ready = [&](uint16_t p) {
bound_port.store(p);
{ std::lock_guard lk(ready_mu); ready = true; }
ready_cv.notify_all();
};
voicecat::server::Server server(cfg);
std::thread server_thread([&] { server.run(); });
{
std::unique_lock lk(ready_mu);
bool ok = ready_cv.wait_for(lk, std::chrono::seconds(10), [&] { return ready; });
if (!ok) {
std::printf("FAIL: server did not become ready within 10s\n");
++g_failures;
server.stop();
server_thread.join();
std::filesystem::remove_all(tmp);
return;
}
}
uint16_t port = bound_port.load();
std::printf("test_vad_and_ptt_gate: server ready on :%u\n", port);
EventStore evA;
evA.label = "A";
vc_callbacks cbA{on_event, nullptr, &evA};
vc_config cfgA{"test-A", "0.1", VC_LOG_OFF};
vc_client* clientA = vc_client_create(&cfgA, cbA);
CHECK(clientA != nullptr);
CHECK(vc_connect(clientA, "127.0.0.1", port) == VC_OK);
CHECK(vc_authenticate_guest(clientA, "VP-A") == VC_OK);
CHECK(wait_for(evA, [](EventStore& s) { return s.auth_ok; }, 8000));
CHECK(wait_for(evA, [](EventStore& s) { return s.channel_list_received; }, 3000));
EventStore evB;
evB.label = "B";
vc_callbacks cbB{on_event, nullptr, &evB};
vc_config cfgB{"test-B", "0.1", VC_LOG_OFF};
vc_client* clientB = vc_client_create(&cfgB, cbB);
CHECK(clientB != nullptr);
CHECK(vc_connect(clientB, "127.0.0.1", port) == VC_OK);
CHECK(vc_authenticate_guest(clientB, "VP-B") == VC_OK);
CHECK(wait_for(evB, [](EventStore& s) { return s.auth_ok; }, 8000));
CHECK(wait_for(evB, [](EventStore& s) { return s.channel_list_received; }, 3000));
uint32_t a_uid = 0;
{ std::lock_guard lk(evA.mu); a_uid = evA.self_user_id; }
std::this_thread::sleep_for(std::chrono::milliseconds(500));
vc_stream_desc mic_desc{};
mic_desc.kind = VC_STREAM_MIC;
mic_desc.label = "mic";
uint32_t mic_sid = 0;
CHECK(vc_stream_start(clientA, &mic_desc, &mic_sid) == VC_OK);
CHECK(wait_for(evB, [](EventStore& s) { return s.saw_stream_started; }, 5000));
CHECK(wait_for(evA, [](EventStore& s) { return s.saw_stream_started; }, 5000));
// ── 2a. VAD mode (default), silent PCM: must NOT reach B as a talking edge ──
CHECK(vc_set_input_mode(clientA, VC_INPUT_VOICE_ACTIVATION) == VC_OK);
for (int i = 0; i < 15; ++i) {
auto silence = make_silence_frame();
CHECK(vc_test_inject_capture(clientA, mic_sid, silence.data(), silence.size()) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(20));
}
CHECK(!saw_talking_true(evB, a_uid, mic_sid, 0));
// ── 2b. VAD mode, loud PCM: must reach B as a talking edge ──────────────────
size_t mark = talk_event_count(evB);
for (int i = 0; i < 20; ++i) {
auto loud = make_sine_frame(i, 440.0f);
CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(20));
}
CHECK(wait_for(evB, [&](EventStore& s) {
for (size_t i = mark; i < s.talk_events.size(); ++i) {
auto& e = s.talk_events[i];
if (e.user_id == a_uid && e.stream_id == mic_sid && e.talking) return true;
}
return false;
}, 3000));
// ── 3a. PTT mode, key up: loud PCM must NOT reach B as a new talking edge ───
CHECK(vc_set_input_mode(clientA, VC_INPUT_PUSH_TO_TALK) == VC_OK);
CHECK(vc_set_push_to_talk(clientA, 0) == VC_OK);
// Let any in-flight VAD-driven talking state lapse (hang-time ~300ms) before measuring.
std::this_thread::sleep_for(std::chrono::milliseconds(500));
mark = talk_event_count(evB);
for (int i = 0; i < 20; ++i) {
auto loud = make_sine_frame(i, 440.0f);
CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(20));
}
std::this_thread::sleep_for(std::chrono::milliseconds(200));
CHECK(!saw_talking_true(evB, a_uid, mic_sid, mark));
// ── 3b. PTT mode, key down: loud PCM must reach B as a talking edge ─────────
CHECK(vc_set_push_to_talk(clientA, 1) == VC_OK);
mark = talk_event_count(evB);
for (int i = 0; i < 20; ++i) {
auto loud = make_sine_frame(i, 440.0f);
CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(20));
}
CHECK(wait_for(evB, [&](EventStore& s) {
for (size_t i = mark; i < s.talk_events.size(); ++i) {
auto& e = s.talk_events[i];
if (e.user_id == a_uid && e.stream_id == mic_sid && e.talking) return true;
}
return false;
}, 3000));
{ std::lock_guard lk(evA.mu); CHECK(!evA.disconnected); }
{ std::lock_guard lk(evB.mu); CHECK(!evB.disconnected); }
vc_disconnect(clientA);
vc_disconnect(clientB);
vc_client_destroy(clientA);
vc_client_destroy(clientB);
server.stop();
server_thread.join();
std::filesystem::remove_all(tmp);
std::printf("test_vad_and_ptt_gate: done\n");
}
int main() {
test_device_enumeration();
#if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS)
test_stereo_mix();
#endif
test_vad_and_ptt_gate();
if (g_failures == 0) {
std::printf("vad_ptt_devices: all checks passed\n");
return 0;
}
std::printf("vad_ptt_devices: %d failure(s)\n", g_failures);
return 1;
}
#else // !VOICECAT_HAS_NET
int main() {
std::printf("vad_ptt_devices: SKIP (VOICECAT_HAS_NET not defined)\n");
return 0;
}
#endif // VOICECAT_HAS_NET

View File

@@ -10,6 +10,8 @@
#include <csignal>
#include <cstdio>
#include <cstring>
#include <functional>
#include <iostream>
#include <string>
#include <thread>
@@ -58,6 +60,10 @@ void on_event(void* user, const vc_event* ev) {
case VC_EVENT_TEXT_MESSAGE:
std::printf("[text] from=%u: %s\n", ev->user_id, ev->text ? ev->text : "");
break;
case VC_EVENT_TALK_STATE:
std::printf("[voice] talk state: user_id=%u stream_id=%u talking=%u\n", ev->user_id,
ev->stream_id, ev->u32a);
break;
case VC_EVENT_ERROR:
std::fprintf(stderr, "[error] rc=%d: %s\n", ev->result, ev->text ? ev->text : "");
break;
@@ -82,14 +88,58 @@ bool wait_until(std::atomic<bool>& flag, int timeout_ms) {
void print_usage() {
std::printf(
"usage: vccli [--host H] [--port P] [--nick NAME] [--channel ID]\n"
" [--voice] [--mute] [--text MSG]\n"
" --host H server host (default 127.0.0.1)\n"
" --port P server TCP port (default 8384)\n"
" --nick NAME guest nickname (default vccli-test)\n"
" --channel ID channel to join after auth (default 1, Lobby)\n"
" --voice start a MIC stream and stay connected until Ctrl+C\n"
" --mute start with the mic muted (only meaningful with --voice)\n"
" --text MSG send MSG to the channel, then exit\n");
" [--voice] [--mute] [--text MSG] [--list-devices]\n"
" [--input-device ID] [--input-mode vad|ptt] [--share-screen-audio]\n"
" --host H server host (default 127.0.0.1)\n"
" --port P server TCP port (default 8384)\n"
" --nick NAME guest nickname (default vccli-test)\n"
" --channel ID channel to join after auth (default 1, Lobby)\n"
" --voice start a MIC stream and stay connected until Ctrl+C\n"
" --mute start with the mic muted (only meaningful with --voice)\n"
" --text MSG send MSG to the channel, then exit\n"
" --list-devices print input/output devices (vc_list_devices) and exit\n"
" --input-device ID use device ID (from --list-devices) for the MIC stream\n"
" --input-mode vad|ptt send-side input gate mode (default vad)\n"
" --share-screen-audio also start a SCREEN_AUDIO stream (WASAPI loopback on Windows)\n"
"\n"
"While --voice is running, stdin accepts: \"ptt on\", \"ptt off\", \"mode vad\",\n"
"\"mode ptt\" (PTT key state can't be held interactively in a headless CLI, so it's\n"
"toggled via these commands instead).\n");
}
void print_device_list(vc_client* c, vc_device_kind kind, const char* label) {
vc_device_list dl{};
vc_result r = vc_list_devices(c, kind, &dl);
std::printf("%s devices: vc_list_devices -> %d (%s), count=%zu\n", label, r,
vc_result_string(r), dl.count);
for (size_t i = 0; i < dl.count; ++i) {
std::printf(" [%s] %s%s\n", dl.items[i].id, dl.items[i].name,
dl.items[i].is_default ? " (default)" : "");
}
vc_free_device_list(&dl);
}
// Background stdin reader for --voice mode: the most portable way to drive PTT/mode toggles
// interactively from a headless CLI (no SIGUSR1 equivalent on Windows).
void run_stdin_commands(vc_client* c, std::atomic<bool>& stop) {
std::string line;
while (!stop.load() && std::getline(std::cin, line)) {
if (line == "ptt on") {
vc_set_push_to_talk(c, 1);
std::printf("[ptt] on\n");
} else if (line == "ptt off") {
vc_set_push_to_talk(c, 0);
std::printf("[ptt] off\n");
} else if (line == "mode vad") {
vc_set_input_mode(c, VC_INPUT_VOICE_ACTIVATION);
std::printf("[mode] vad\n");
} else if (line == "mode ptt") {
vc_set_input_mode(c, VC_INPUT_PUSH_TO_TALK);
std::printf("[mode] ptt\n");
} else if (!line.empty()) {
std::fprintf(stderr, "unknown command: %s\n", line.c_str());
}
}
}
} // namespace
@@ -107,6 +157,11 @@ int main(int argc, char** argv) {
bool start_muted = false;
std::string text_msg;
bool have_text = false;
bool list_devices = false;
std::string input_device;
bool have_input_device = false;
vc_input_mode input_mode = VC_INPUT_VOICE_ACTIVATION;
bool share_screen_audio = false;
for (int i = 1; i < argc; ++i) {
std::string a = argv[i];
@@ -118,6 +173,14 @@ int main(int argc, char** argv) {
else if (a == "--voice") voice_mode = true;
else if (a == "--mute") start_muted = true;
else if (a == "--text") { text_msg = next(); have_text = true; }
else if (a == "--list-devices") list_devices = true;
else if (a == "--input-device") { input_device = next(); have_input_device = true; }
else if (a == "--input-mode") {
std::string m = next();
if (m == "ptt") input_mode = VC_INPUT_PUSH_TO_TALK;
else if (m != "vad") { std::fprintf(stderr, "--input-mode must be vad|ptt\n"); return 1; }
}
else if (a == "--share-screen-audio") share_screen_audio = true;
else if (a == "--help" || a == "-h") { print_usage(); return 0; }
else { std::fprintf(stderr, "unknown flag: %s\n", a.c_str()); print_usage(); return 1; }
}
@@ -143,6 +206,14 @@ int main(int argc, char** argv) {
return 1;
}
if (list_devices) {
// Device enumeration works pre-connect (no server needed) — see docs/voice.md.
print_device_list(c, VC_DEVICE_INPUT, "input");
print_device_list(c, VC_DEVICE_OUTPUT, "output");
vc_client_destroy(c);
return 0;
}
vc_result r = vc_connect(c, host.c_str(), port);
std::printf("vc_connect(%s:%u) -> %d (%s)\n", host.c_str(), port, r, vc_result_string(r));
if (r != VC_OK) {
@@ -178,9 +249,13 @@ int main(int argc, char** argv) {
if (start_muted) vc_set_self_mute(c, 1, 0);
r = vc_set_input_mode(c, input_mode);
std::printf("vc_set_input_mode(%s) -> %d (%s)\n",
input_mode == VC_INPUT_PUSH_TO_TALK ? "ptt" : "vad", r, vc_result_string(r));
vc_stream_desc desc{};
desc.kind = VC_STREAM_MIC;
desc.device_id = nullptr;
desc.device_id = have_input_device ? input_device.c_str() : nullptr;
desc.label = "Microphone";
uint32_t stream_id = 0;
@@ -188,12 +263,43 @@ int main(int argc, char** argv) {
std::printf("vc_stream_start -> %d (%s), stream_id=%u\n", r, vc_result_string(r),
stream_id);
std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n");
if (have_input_device && r == VC_OK) {
r = vc_set_input_device(c, stream_id, input_device.c_str());
std::printf("vc_set_input_device -> %d (%s)\n", r, vc_result_string(r));
}
uint32_t screen_stream_id = 0;
if (share_screen_audio) {
vc_stream_desc sdesc{};
sdesc.kind = VC_STREAM_SCREEN_AUDIO;
sdesc.label = "Desktop audio";
r = vc_stream_start(c, &sdesc, &screen_stream_id);
std::printf("vc_stream_start(SCREEN_AUDIO) -> %d (%s), stream_id=%u\n", r,
vc_result_string(r), screen_stream_id);
}
std::thread stdin_thread(run_stdin_commands, c, std::ref(g_stop));
std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n"
"(type \"ptt on\"/\"ptt off\"/\"mode vad\"/\"mode ptt\" to drive the input gate)\n");
while (!g_stop.load()) {
std::this_thread::sleep_for(std::chrono::milliseconds(200));
}
if (share_screen_audio) vc_stream_stop(c, screen_stream_id);
vc_stream_stop(c, stream_id);
// Unblock the stdin reader and let it exit before the client is torn down.
if (stdin_thread.joinable()) {
#if defined(_WIN32)
// std::getline on a console stdin blocks indefinitely; on Windows there's no
// portable way to interrupt it from another thread, so detach rather than join —
// process exit reclaims the thread.
stdin_thread.detach();
#else
stdin_thread.join();
#endif
}
}
vc_disconnect(c);