Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback

Browse Source 
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:

- Device enumeration (vc_list_devices) + input device selection
  (vc_set_input_device), backed by AudioEngine::enumerate_devices() via
  miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
  ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
  webrtc-audio-processing (the originally-planned APM) has no working
  Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
  abseil-cpp dependency), so VAD is a new lightweight, dependency-free
  energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
  interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
  stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
  stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
  miniaudio's loopback device type), replacing test-only injection as the
  production capture path.

Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).

Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.

Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.

Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Talon
2026-06-16 16:11:52 +02:00
parent 867557eda1
commit 5f6c223526
19 changed files with 1106 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files

126
tools/vccli/src/main.cpp
View File

@@ -10,6 +10,8 @@
#include <csignal>
#include <cstdio>
#include <cstring>
#include <functional>
#include <iostream>
#include <string>
#include <thread>
@@ -58,6 +60,10 @@ void on_event(void* user, const vc_event* ev) {
case VC_EVENT_TEXT_MESSAGE:
std::printf("[text] from=%u: %s\n", ev->user_id, ev->text ? ev->text : "");
break;
case VC_EVENT_TALK_STATE:
std::printf("[voice] talk state: user_id=%u stream_id=%u talking=%u\n", ev->user_id,
ev->stream_id, ev->u32a);
break;
case VC_EVENT_ERROR:
std::fprintf(stderr, "[error] rc=%d: %s\n", ev->result, ev->text ? ev->text : "");
break;
@@ -82,14 +88,58 @@ bool wait_until(std::atomic<bool>& flag, int timeout_ms) {
void print_usage() {
std::printf(
"usage: vccli [--host H] [--port P] [--nick NAME] [--channel ID]\n"
" [--voice] [--mute] [--text MSG]\n"
" --host H server host (default 127.0.0.1)\n"
" --port P server TCP port (default 8384)\n"
" --nick NAME guest nickname (default vccli-test)\n"
" --channel ID channel to join after auth (default 1, Lobby)\n"
" --voice start a MIC stream and stay connected until Ctrl+C\n"
" --mute start with the mic muted (only meaningful with --voice)\n"
" --text MSG send MSG to the channel, then exit\n");
" [--voice] [--mute] [--text MSG] [--list-devices]\n"
" [--input-device ID] [--input-mode vad|ptt] [--share-screen-audio]\n"
" --host H server host (default 127.0.0.1)\n"
" --port P server TCP port (default 8384)\n"
" --nick NAME guest nickname (default vccli-test)\n"
" --channel ID channel to join after auth (default 1, Lobby)\n"
" --voice start a MIC stream and stay connected until Ctrl+C\n"
" --mute start with the mic muted (only meaningful with --voice)\n"
" --text MSG send MSG to the channel, then exit\n"
" --list-devices print input/output devices (vc_list_devices) and exit\n"
" --input-device ID use device ID (from --list-devices) for the MIC stream\n"
" --input-mode vad|ptt send-side input gate mode (default vad)\n"
" --share-screen-audio also start a SCREEN_AUDIO stream (WASAPI loopback on Windows)\n"
"\n"
"While --voice is running, stdin accepts: \"ptt on\", \"ptt off\", \"mode vad\",\n"
"\"mode ptt\" (PTT key state can't be held interactively in a headless CLI, so it's\n"
"toggled via these commands instead).\n");
}
void print_device_list(vc_client* c, vc_device_kind kind, const char* label) {
vc_device_list dl{};
vc_result r = vc_list_devices(c, kind, &dl);
std::printf("%s devices: vc_list_devices -> %d (%s), count=%zu\n", label, r,
vc_result_string(r), dl.count);
for (size_t i = 0; i < dl.count; ++i) {
std::printf(" [%s] %s%s\n", dl.items[i].id, dl.items[i].name,
dl.items[i].is_default ? " (default)" : "");
}
vc_free_device_list(&dl);
}
// Background stdin reader for --voice mode: the most portable way to drive PTT/mode toggles
// interactively from a headless CLI (no SIGUSR1 equivalent on Windows).
void run_stdin_commands(vc_client* c, std::atomic<bool>& stop) {
std::string line;
while (!stop.load() && std::getline(std::cin, line)) {
if (line == "ptt on") {
vc_set_push_to_talk(c, 1);
std::printf("[ptt] on\n");
} else if (line == "ptt off") {
vc_set_push_to_talk(c, 0);
std::printf("[ptt] off\n");
} else if (line == "mode vad") {
vc_set_input_mode(c, VC_INPUT_VOICE_ACTIVATION);
std::printf("[mode] vad\n");
} else if (line == "mode ptt") {
vc_set_input_mode(c, VC_INPUT_PUSH_TO_TALK);
std::printf("[mode] ptt\n");
} else if (!line.empty()) {
std::fprintf(stderr, "unknown command: %s\n", line.c_str());
}
}
}
} // namespace
@@ -107,6 +157,11 @@ int main(int argc, char** argv) {
bool start_muted = false;
std::string text_msg;
bool have_text = false;
bool list_devices = false;
std::string input_device;
bool have_input_device = false;
vc_input_mode input_mode = VC_INPUT_VOICE_ACTIVATION;
bool share_screen_audio = false;
for (int i = 1; i < argc; ++i) {
std::string a = argv[i];
@@ -118,6 +173,14 @@ int main(int argc, char** argv) {
else if (a == "--voice") voice_mode = true;
else if (a == "--mute") start_muted = true;
else if (a == "--text") { text_msg = next(); have_text = true; }
else if (a == "--list-devices") list_devices = true;
else if (a == "--input-device") { input_device = next(); have_input_device = true; }
else if (a == "--input-mode") {
std::string m = next();
if (m == "ptt") input_mode = VC_INPUT_PUSH_TO_TALK;
else if (m != "vad") { std::fprintf(stderr, "--input-mode must be vad|ptt\n"); return 1; }
}
else if (a == "--share-screen-audio") share_screen_audio = true;
else if (a == "--help" || a == "-h") { print_usage(); return 0; }
else { std::fprintf(stderr, "unknown flag: %s\n", a.c_str()); print_usage(); return 1; }
}
@@ -143,6 +206,14 @@ int main(int argc, char** argv) {
return 1;
}
if (list_devices) {
// Device enumeration works pre-connect (no server needed) — see docs/voice.md.
print_device_list(c, VC_DEVICE_INPUT, "input");
print_device_list(c, VC_DEVICE_OUTPUT, "output");
vc_client_destroy(c);
return 0;
}
vc_result r = vc_connect(c, host.c_str(), port);
std::printf("vc_connect(%s:%u) -> %d (%s)\n", host.c_str(), port, r, vc_result_string(r));
if (r != VC_OK) {
@@ -178,9 +249,13 @@ int main(int argc, char** argv) {
if (start_muted) vc_set_self_mute(c, 1, 0);
r = vc_set_input_mode(c, input_mode);
std::printf("vc_set_input_mode(%s) -> %d (%s)\n",
input_mode == VC_INPUT_PUSH_TO_TALK ? "ptt" : "vad", r, vc_result_string(r));
vc_stream_desc desc{};
desc.kind = VC_STREAM_MIC;
desc.device_id = nullptr;
desc.device_id = have_input_device ? input_device.c_str() : nullptr;
desc.label = "Microphone";
uint32_t stream_id = 0;
@@ -188,12 +263,43 @@ int main(int argc, char** argv) {
std::printf("vc_stream_start -> %d (%s), stream_id=%u\n", r, vc_result_string(r),
stream_id);
std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n");
if (have_input_device && r == VC_OK) {
r = vc_set_input_device(c, stream_id, input_device.c_str());
std::printf("vc_set_input_device -> %d (%s)\n", r, vc_result_string(r));
}
uint32_t screen_stream_id = 0;
if (share_screen_audio) {
vc_stream_desc sdesc{};
sdesc.kind = VC_STREAM_SCREEN_AUDIO;
sdesc.label = "Desktop audio";
r = vc_stream_start(c, &sdesc, &screen_stream_id);
std::printf("vc_stream_start(SCREEN_AUDIO) -> %d (%s), stream_id=%u\n", r,
vc_result_string(r), screen_stream_id);
}
std::thread stdin_thread(run_stdin_commands, c, std::ref(g_stop));
std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n"
"(type \"ptt on\"/\"ptt off\"/\"mode vad\"/\"mode ptt\" to drive the input gate)\n");
while (!g_stop.load()) {
std::this_thread::sleep_for(std::chrono::milliseconds(200));
}
if (share_screen_audio) vc_stream_stop(c, screen_stream_id);
vc_stream_stop(c, stream_id);
// Unblock the stdin reader and let it exit before the client is torn down.
if (stdin_thread.joinable()) {
#if defined(_WIN32)
// std::getline on a console stdin blocks indefinitely; on Windows there's no
// portable way to interrupt it from another thread, so detach rather than join —
// process exit reclaims the thread.
stdin_thread.detach();
#else
stdin_thread.join();
#endif
}
}
vc_disconnect(c);