Compare commits
1 Commits
feat/m3-mu
...
post-m3-va
| Author | SHA1 | Date | |
|---|---|---|---|
| 5f6c223526 |
12
CLAUDE.md
12
CLAUDE.md
@@ -4,11 +4,13 @@ Auto-loaded each session. This is the **map**: build commands, architecture at a
|
|||||||
where everything is. For the *working method* read [`AGENTS.md`](AGENTS.md); for *what's done
|
where everything is. For the *working method* read [`AGENTS.md`](AGENTS.md); for *what's done
|
||||||
and what's next* read [`PROGRESS.md`](PROGRESS.md); for *design* read [`docs/`](docs/).
|
and what's next* read [`PROGRESS.md`](PROGRESS.md); for *design* read [`docs/`](docs/).
|
||||||
|
|
||||||
> **One-line status:** M2 voice/media plane is complete and verified through the real client
|
> **One-line status:** M3 (multi-stream, per-channel tuning) is complete, plus a follow-up
|
||||||
> library, not just a raw-socket harness (`ctest --test-dir build/m1-dev` green — 10/10 tests,
|
> pass closing the device enumeration / VAD-PTT input gate / stereo playback / WASAPI loopback
|
||||||
> including `test_voice_client_abi` driving two real `vc_client`s end-to-end, and `vccli
|
> gaps it left open (`ctest --test-dir build/m1-dev` green — 12/12 tests, including
|
||||||
> --voice` manually verified live). Next up is **M3** (multi-stream, per-channel tuning,
|
> `test_vad_ptt_devices`, and `vccli --voice`/`--list-devices` manually verified live). Real
|
||||||
> listener-side NR). See [`PROGRESS.md`](PROGRESS.md).
|
> `webrtc-audio-processing` (AEC/NS/AGC) is still unbuilt — no working Windows/MSVC port
|
||||||
|
> upstream — so v1 ships a lightweight energy/RMS VAD instead. Next up is **M4** (native
|
||||||
|
> clients). See [`PROGRESS.md`](PROGRESS.md).
|
||||||
|
|
||||||
VoiceCat = self-hosted native voice & text chat (TeamSpeak/Mumble-style). Plain TCP (control)
|
VoiceCat = self-hosted native voice & text chat (TeamSpeak/Mumble-style). Plain TCP (control)
|
||||||
+ UDP (media), no WebRTC, encrypted by default. A shared C++ core (`libvoicecat`) drives
|
+ UDP (media), no WebRTC, encrypted by default. A shared C++ core (`libvoicecat`) drives
|
||||||
|
|||||||
122
PROGRESS.md
122
PROGRESS.md
@@ -10,19 +10,29 @@ up instantly. Newest status at the top.
|
|||||||
|
|
||||||
## ▶ Where we left off / next action
|
## ▶ Where we left off / next action
|
||||||
|
|
||||||
- **Done:** **M3 — multi-stream & per-channel tuning** ✓ complete (2026-06-16). See the M3
|
- **Done:** **Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI
|
||||||
section below for the full file-by-file change list. `ctest --test-dir build/m1-dev` —
|
loopback** ✓ complete (2026-06-16). Closes all three items M3 explicitly carried forward as
|
||||||
**11/11 tests** green (3 consecutive full-suite runs), including the new
|
out of scope (see the dated section below for the full file-by-file change list).
|
||||||
`test_m3_multistream` (real `vc_client`s, not raw sockets — same lesson as M2: ABI-level
|
`ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive full-suite runs),
|
||||||
coverage is what proves the client library, not just the wire protocol).
|
including the new `test_vad_ptt_devices` (real `vc_client`s against a real server, plus a
|
||||||
**Two items intentionally still open** (carried forward, not silently dropped):
|
white-box `AudioEngine` stereo-mix check — same ABI-level-coverage lesson as M2/M3).
|
||||||
- `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices`
|
Manually verified live: `vccli --list-devices` against real hardware, and
|
||||||
(device enumeration + VAD/PTT input gate) remain `VC_ERR_NOT_IMPLEMENTED` — explicitly
|
`vccli --voice --input-mode vad` connecting/streaming without incident.
|
||||||
scoped out of this M3 pass; revisit in a future milestone.
|
**Still explicitly out of scope** (carried forward, not silently dropped):
|
||||||
- Stereo Opus is now wire-correct end-to-end (a channel configured `MODE_STEREO` really
|
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream; v1 ships a
|
||||||
encodes/decodes 2-channel Opus packets), but `AudioEngine`'s playback mixer/output device
|
lightweight energy/RMS VAD instead (see docs/roadmap.md §2, docs/voice.md §8/§11). There is
|
||||||
stays mono internally — stereo streams are downmixed (avg L/R) immediately after decode,
|
**no AEC, NS, or AGC implementation at all**, not just a deferred VAD.
|
||||||
before mixing. True stereo *playback output* is a follow-up, not part of M3.
|
`vc_set_remote_stream(..., noise_reduction)`'s per-stream NS toggle is unaffected by this
|
||||||
|
pass and stays exactly as inert as it was after M3 (`ApmPassthrough`, no PCM modification).
|
||||||
|
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — this pass is
|
||||||
|
Windows-only for real loopback capture; other platforms keep `vc_test_inject_capture` as
|
||||||
|
the only way to feed `SCREEN_AUDIO`.
|
||||||
|
- Process-specific WASAPI loopback — miniaudio's loopback mode captures the whole render
|
||||||
|
endpoint (including this app's own incoming voice mix), not a single process.
|
||||||
|
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
|
||||||
|
(mutex lock, heap allocation, blocking `sendto` on the miniaudio real-time callback
|
||||||
|
thread) — predates this work, documented but not fixed; fixing it needs the lock-free
|
||||||
|
ring-buffer hand-off `docs/architecture.md §3` specifies, a separate, larger refactor.
|
||||||
- **Next:** **M4 — native clients** (Windows C#, macOS/iOS Swift). See `docs/roadmap.md §M4`.
|
- **Next:** **M4 — native clients** (Windows C#, macOS/iOS Swift). See `docs/roadmap.md §M4`.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -233,6 +243,92 @@ not treated as pre-existing-and-out-of-scope:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ (completed 2026-06-16)
|
||||||
|
|
||||||
|
Closes all three items the M3 section above explicitly carried forward as out of scope.
|
||||||
|
|
||||||
|
**Exit verification:** `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive
|
||||||
|
full-suite runs), including the new `test_vad_ptt_devices` (device enumeration + VAD/PTT gate
|
||||||
|
through real `vc_client`s against a real server, plus a white-box `AudioEngine` stereo-mix
|
||||||
|
check — no audio hardware needed for that last part). Also verified 5 consecutive standalone
|
||||||
|
runs of the new test alone, no flakes. Manually verified live on Windows: `vccli
|
||||||
|
--list-devices` against real hardware (3 input / 4 output devices, correct `is_default`
|
||||||
|
flags), and `vccli --voice --input-mode vad` connecting + streaming without incident.
|
||||||
|
|
||||||
|
- [x] **Device enumeration** (`vc_list_devices`) — `AudioEngine::enumerate_devices(bool
|
||||||
|
capture)` (static, works without a running engine — inits a throwaway `ma_context` via
|
||||||
|
`ma_context_get_devices`). `device_id`/`vc_device.id` is an opaque hex-encoded raw
|
||||||
|
`ma_device_id` (not the device name — names aren't guaranteed unique); documented as an
|
||||||
|
internal contract callers must round-trip, never construct by hand. `vc_client::list_devices`
|
||||||
|
works in any connection state (no `VC_STATE_CONNECTED` gate) since device pickers need to
|
||||||
|
populate pre-connect. `vc_free_device_list` is now a real free (was a no-op stub).
|
||||||
|
- [x] **Input device selection** (`vc_set_input_device`) — stores the device id on the
|
||||||
|
targeted `LocalStream` (new field); for the MIC stream, if the engine is already running,
|
||||||
|
restarts it (`stop()` + `ensure_audio_running()`) to pick up the new device. Simplified: it
|
||||||
|
restarts unconditionally rather than trying to detect whether the device id actually
|
||||||
|
changed (`AudioEngine` has no getter for "current device").
|
||||||
|
- [x] **VAD/PTT input gate** (`vc_set_input_mode`, `vc_set_push_to_talk`) — new
|
||||||
|
`EnergyVadProcessor` (`core/src/audio/apm_processor.cpp`) implementing the existing
|
||||||
|
`ApmProcessor` interface: energy/RMS threshold (default ~0.025 normalized) + hang-time
|
||||||
|
(default 300 ms, matching `kTalkHangoverMs`). New factory `ApmProcessor::create_vad()`,
|
||||||
|
kept separate from `create()` (which recv-side per-stream NS still uses, unaffected by this
|
||||||
|
pass). `vc_client` gained `current_input_mode_`/`ptt_active_`/`mic_vad_`; the gate is
|
||||||
|
inserted in `on_capture_frame`, **MIC-only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
|
||||||
|
(gating a desktop-audio share on the user's own voice activity would silently drop shared
|
||||||
|
music/video audio). `last_capture_ms` (drives the talk indicator) is now updated *after* the
|
||||||
|
gate check, not before, so a VAD/PTT-closed frame never shows as "talking". `mic_vad_` is
|
||||||
|
constructed once the MIC stream's `StreamAnnounceResult` lands (on `io_thread_`), not lazily
|
||||||
|
inside the capture path.
|
||||||
|
- [x] **True stereo playback** — `AudioParams::channels` split into `capture_channels` (stays
|
||||||
|
1) and `playback_channels` (now 2, unconditionally). `AudioEngine::on_playback` no longer
|
||||||
|
downmixes decoded stereo streams to mono before mixing — stereo decode output is mixed
|
||||||
|
directly (L→L, R→R); mono decode output is upmixed (duplicated into both channels). Falls
|
||||||
|
back to a 1-channel playback device once if the 2-channel `ma_device_init` fails (unusual
|
||||||
|
hardware). New test-only `AudioEngine::mix_for_test()` exposes the mixer for white-box
|
||||||
|
testing without a real `ma_device`.
|
||||||
|
- [x] **WASAPI loopback capture for `SCREEN_AUDIO`** — new `VOICECAT_HAS_LOOPBACK` macro
|
||||||
|
(`core/CMakeLists.txt`, Windows-only). `AudioEngine` gained a separate `loopback_device_`
|
||||||
|
(own lifecycle, decoupled from the mic capture/playback devices) with
|
||||||
|
`start_loopback_capture()`/`stop_loopback_capture()`, using miniaudio's
|
||||||
|
`ma_device_type_loopback` against the default render endpoint. Its callback feeds
|
||||||
|
`capture_cb_` directly (same pattern as the real mic capture device), **not** through
|
||||||
|
`inject_capture()`'s test-only ring. Wired into `vc_client::handle_stream_announce_result`
|
||||||
|
(start, alongside `ensure_audio_running()`) and `stream_stop` (stop) for
|
||||||
|
`VC_STREAM_SCREEN_AUDIO`. Non-Windows builds keep `vc_test_inject_capture` as the only way to
|
||||||
|
feed `SCREEN_AUDIO`.
|
||||||
|
- [x] `tools/vccli/src/main.cpp` — new flags `--list-devices`, `--input-device`,
|
||||||
|
`--input-mode vad|ptt`, `--share-screen-audio`; while `--voice` is running, a background
|
||||||
|
stdin-reader thread accepts `ptt on`/`ptt off`/`mode vad`/`mode ptt` (the most portable way
|
||||||
|
to drive PTT interactively from a headless CLI — no SIGUSR1 equivalent on Windows). Also
|
||||||
|
prints `VC_EVENT_TALK_STATE`. Known minor caveat: on Windows the stdin-reader thread is
|
||||||
|
detached (not joined) on exit, since `std::getline` can't be interrupted from another thread
|
||||||
|
— a `vc_client*` use-after-free is theoretically possible if a command line arrives in the
|
||||||
|
brief window between teardown and process exit; acceptable for a headless test/dev tool.
|
||||||
|
- [x] `tests/test_vad_ptt_devices.cpp` — new test covering all four items above; registered in
|
||||||
|
`tests/CMakeLists.txt`. `tests/test_smoke.cpp`'s device-list assertion is now conditional on
|
||||||
|
`VOICECAT_HAS_AUDIO` (was a hard `VC_ERR_NOT_IMPLEMENTED` assertion) — `VC_OK` only, never
|
||||||
|
`count > 0` (a headless CI build agent may legitimately report zero audio devices).
|
||||||
|
|
||||||
|
**Still explicitly out of scope** (carried forward, not silently dropped):
|
||||||
|
- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream (see
|
||||||
|
docs/roadmap.md §2's superseding decision-log entry). There is **no AEC, NS, or AGC
|
||||||
|
implementation at all**, not just a deferred VAD. The per-stream NS toggle
|
||||||
|
(`vc_set_remote_stream(..., noise_reduction)`) is unaffected by this pass and stays exactly
|
||||||
|
as inert as it was after M3 (`ApmPassthrough`, no PCM modification) — don't mistake this
|
||||||
|
pass for having fixed it.
|
||||||
|
- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — Windows-only loopback in
|
||||||
|
this pass.
|
||||||
|
- Process-specific WASAPI loopback — whole-device capture only; inherently captures this
|
||||||
|
app's own incoming voice mix along with everything else playing.
|
||||||
|
- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture`
|
||||||
|
(mutex lock, heap allocation for the stereo-upmix path, blocking `sendto`, all on the
|
||||||
|
miniaudio real-time callback thread) — predates this work (was already present in M2/M3);
|
||||||
|
documented here explicitly rather than silently carried forward again. Fixing it properly
|
||||||
|
needs the lock-free ring-buffer hand-off `docs/architecture.md §3` specifies — a separate,
|
||||||
|
larger refactor, out of scope for this pass.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Decisions log
|
## Decisions log
|
||||||
|
|
||||||
All architecture/scope decisions are settled and recorded in
|
All architecture/scope decisions are settled and recorded in
|
||||||
|
|||||||
@@ -60,6 +60,11 @@ if(VOICECAT_USE_VCPKG_DEPS)
|
|||||||
if(WIN32)
|
if(WIN32)
|
||||||
# AcceptEx / GetAcceptExSockaddrs live in mswsock; ws2_32 covers the base Winsock API.
|
# AcceptEx / GetAcceptExSockaddrs live in mswsock; ws2_32 covers the base Winsock API.
|
||||||
target_link_libraries(voicecat PRIVATE ws2_32 mswsock)
|
target_link_libraries(voicecat PRIVATE ws2_32 mswsock)
|
||||||
|
# Real desktop-audio loopback capture for SCREEN_AUDIO (miniaudio's ma_device_type_loopback
|
||||||
|
# is WASAPI-only). Other platforms keep vc_test_inject_capture as the only way to feed
|
||||||
|
# SCREEN_AUDIO until a per-platform loopback path is built (macOS: ScreenCaptureKit, per
|
||||||
|
# docs/voice.md §9 — not in scope yet).
|
||||||
|
target_compile_definitions(voicecat PUBLIC VOICECAT_HAS_LOOPBACK)
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
# Signal to C++ code that the real networking/crypto stack is available.
|
# Signal to C++ code that the real networking/crypto stack is available.
|
||||||
|
|||||||
@@ -1,7 +1,18 @@
|
|||||||
#include "audio/apm_processor.h"
|
#include "audio/apm_processor.h"
|
||||||
|
|
||||||
|
#include <chrono>
|
||||||
|
#include <cmath>
|
||||||
|
|
||||||
namespace voicecat::audio {
|
namespace voicecat::audio {
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
int64_t steady_now_ms() {
|
||||||
|
return std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||||
|
std::chrono::steady_clock::now().time_since_epoch())
|
||||||
|
.count();
|
||||||
|
}
|
||||||
|
} // namespace
|
||||||
|
|
||||||
// ── ApmPassthrough ────────────────────────────────────────────────────────────
|
// ── ApmPassthrough ────────────────────────────────────────────────────────────
|
||||||
// No-op: returns true (VAD always open), does not modify PCM.
|
// No-op: returns true (VAD always open), does not modify PCM.
|
||||||
// Replaced by WebrtcApmProcessor when VOICECAT_HAS_APM is defined.
|
// Replaced by WebrtcApmProcessor when VOICECAT_HAS_APM is defined.
|
||||||
@@ -11,11 +22,47 @@ class ApmPassthrough final : public ApmProcessor {
|
|||||||
bool process_capture(int16_t*, int, int) override { return true; }
|
bool process_capture(int16_t*, int, int) override { return true; }
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// ── EnergyVadProcessor ──────────────────────────────────────────────────────
|
||||||
|
// Lightweight, dependency-free energy/RMS VAD — see apm_processor.h's create_vad() doc comment
|
||||||
|
// for why this exists instead of a real APM. No AEC (process_render is a no-op); doesn't modify
|
||||||
|
// the PCM it's given, only inspects it.
|
||||||
|
class EnergyVadProcessor final : public ApmProcessor {
|
||||||
|
public:
|
||||||
|
EnergyVadProcessor(float rms_threshold, int64_t hang_time_ms)
|
||||||
|
: threshold_(rms_threshold), hang_time_ms_(hang_time_ms) {}
|
||||||
|
|
||||||
|
void process_render(const int16_t*, int, int) override {}
|
||||||
|
|
||||||
|
bool process_capture(int16_t* pcm, int samples, int /*sample_rate*/) override {
|
||||||
|
if (samples > 0) {
|
||||||
|
double sum_sq = 0.0;
|
||||||
|
for (int i = 0; i < samples; ++i) {
|
||||||
|
double s = static_cast<double>(pcm[i]) / 32768.0;
|
||||||
|
sum_sq += s * s;
|
||||||
|
}
|
||||||
|
double rms = std::sqrt(sum_sq / samples);
|
||||||
|
if (rms >= threshold_) last_voice_ms_ = steady_now_ms();
|
||||||
|
}
|
||||||
|
return (steady_now_ms() - last_voice_ms_) < hang_time_ms_;
|
||||||
|
}
|
||||||
|
|
||||||
|
private:
|
||||||
|
float threshold_;
|
||||||
|
int64_t hang_time_ms_;
|
||||||
|
int64_t last_voice_ms_ = 0; // epoch start -> gate begins closed until first loud frame
|
||||||
|
};
|
||||||
|
|
||||||
std::unique_ptr<ApmProcessor> ApmProcessor::create() {
|
std::unique_ptr<ApmProcessor> ApmProcessor::create() {
|
||||||
#ifdef VOICECAT_HAS_APM
|
#ifdef VOICECAT_HAS_APM
|
||||||
// TODO(M3): return std::make_unique<WebrtcApmProcessor>();
|
// TODO: return std::make_unique<WebrtcApmProcessor>(); — see create_vad()'s doc comment for
|
||||||
|
// why this isn't wired up yet (no working Windows/MSVC build upstream).
|
||||||
#endif
|
#endif
|
||||||
return std::make_unique<ApmPassthrough>();
|
return std::make_unique<ApmPassthrough>();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
std::unique_ptr<ApmProcessor> ApmProcessor::create_vad(float rms_threshold,
|
||||||
|
int64_t hang_time_ms) {
|
||||||
|
return std::make_unique<EnergyVadProcessor>(rms_threshold, hang_time_ms);
|
||||||
|
}
|
||||||
|
|
||||||
} // namespace voicecat::audio
|
} // namespace voicecat::audio
|
||||||
|
|||||||
@@ -24,8 +24,24 @@ class ApmProcessor {
|
|||||||
// Returns false → caller should skip encode/send (silence gate).
|
// Returns false → caller should skip encode/send (silence gate).
|
||||||
virtual bool process_capture(int16_t* pcm, int samples, int sample_rate) = 0;
|
virtual bool process_capture(int16_t* pcm, int samples, int sample_rate) = 0;
|
||||||
|
|
||||||
// Factory: returns a real APM if VOICECAT_HAS_APM is defined, else a passthrough.
|
// Factory: returns a real APM if VOICECAT_HAS_APM is defined, else a passthrough. Used for
|
||||||
|
// recv-side per-stream noise reduction (docs/voice.md §10) — gating doesn't apply there, so
|
||||||
|
// this stays a passthrough until a real APM/NS backend exists (still inert; see
|
||||||
|
// PROGRESS.md). Do not use this for the send-side VAD gate — see create_vad() below.
|
||||||
static std::unique_ptr<ApmProcessor> create();
|
static std::unique_ptr<ApmProcessor> create();
|
||||||
|
|
||||||
|
// Factory for the send-side input gate (docs/voice.md §11): a lightweight, dependency-free
|
||||||
|
// energy/RMS VAD with configurable threshold + hang-time. webrtc-audio-processing (the
|
||||||
|
// originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson build,
|
||||||
|
// unfinished MinGW support, hard abseil-cpp dependency — see PROGRESS.md), so this is the
|
||||||
|
// real v1 implementation behind the same ApmProcessor interface, not a passthrough. No AEC
|
||||||
|
// — process_render() is a no-op here; that's a real limitation versus the originally-planned
|
||||||
|
// APM, not just a deferred VAD.
|
||||||
|
// rms_threshold: normalized 0.0-1.0 RMS-of-int16-range; default ~0.025.
|
||||||
|
// hang_time_ms: how long the gate stays open after the last loud frame; default 300 ms
|
||||||
|
// (matches AudioEngine's kTalkHangoverMs so "talking" and "gate open" agree).
|
||||||
|
static std::unique_ptr<ApmProcessor> create_vad(float rms_threshold = 0.025f,
|
||||||
|
int64_t hang_time_ms = 300);
|
||||||
};
|
};
|
||||||
|
|
||||||
} // namespace voicecat::audio
|
} // namespace voicecat::audio
|
||||||
|
|||||||
@@ -17,6 +17,43 @@ int64_t now_ms() {
|
|||||||
std::chrono::steady_clock::now().time_since_epoch())
|
std::chrono::steady_clock::now().time_since_epoch())
|
||||||
.count();
|
.count();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
|
// device_id encoding (DeviceInfo::id / AudioParams::*_device_id): a hex string of the raw
|
||||||
|
// ma_device_id bytes. Opaque on purpose — names aren't guaranteed unique, and this is the only
|
||||||
|
// stable handle miniaudio accepts back for device selection. Internal contract only; never
|
||||||
|
// exposed as anything other than an opaque round-tripped string at the C ABI boundary.
|
||||||
|
std::string hex_encode_device_id(const ma_device_id& id) {
|
||||||
|
static constexpr char kHex[] = "0123456789abcdef";
|
||||||
|
const auto* bytes = reinterpret_cast<const uint8_t*>(&id);
|
||||||
|
std::string out;
|
||||||
|
out.reserve(sizeof(ma_device_id) * 2);
|
||||||
|
for (size_t i = 0; i < sizeof(ma_device_id); ++i) {
|
||||||
|
out.push_back(kHex[bytes[i] >> 4]);
|
||||||
|
out.push_back(kHex[bytes[i] & 0xF]);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool hex_decode_device_id(const std::string& hex, ma_device_id* out) {
|
||||||
|
if (hex.size() != sizeof(ma_device_id) * 2) return false;
|
||||||
|
std::memset(out, 0, sizeof(ma_device_id));
|
||||||
|
auto* bytes = reinterpret_cast<uint8_t*>(out);
|
||||||
|
auto nibble = [](char c) -> int {
|
||||||
|
if (c >= '0' && c <= '9') return c - '0';
|
||||||
|
if (c >= 'a' && c <= 'f') return c - 'a' + 10;
|
||||||
|
if (c >= 'A' && c <= 'F') return c - 'A' + 10;
|
||||||
|
return -1;
|
||||||
|
};
|
||||||
|
for (size_t i = 0; i < sizeof(ma_device_id); ++i) {
|
||||||
|
int hi = nibble(hex[i * 2]);
|
||||||
|
int lo = nibble(hex[i * 2 + 1]);
|
||||||
|
if (hi < 0 || lo < 0) return false;
|
||||||
|
bytes[i] = static_cast<uint8_t>((hi << 4) | lo);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
#endif // VOICECAT_HAS_AUDIO
|
||||||
} // namespace
|
} // namespace
|
||||||
|
|
||||||
// ── JitterBuffer ─────────────────────────────────────────────────────────────
|
// ── JitterBuffer ─────────────────────────────────────────────────────────────
|
||||||
@@ -88,13 +125,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
|
|||||||
|
|
||||||
#ifdef VOICECAT_HAS_AUDIO
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
// ── Capture device ──────────────────────────────────────────────────────
|
// ── Capture device ──────────────────────────────────────────────────────
|
||||||
|
ma_device_id cap_id{};
|
||||||
|
bool have_cap_id = !p.capture_device_id.empty() &&
|
||||||
|
hex_decode_device_id(p.capture_device_id, &cap_id);
|
||||||
|
|
||||||
ma_device_config cap_cfg = ma_device_config_init(ma_device_type_capture);
|
ma_device_config cap_cfg = ma_device_config_init(ma_device_type_capture);
|
||||||
cap_cfg.capture.format = ma_format_s16;
|
cap_cfg.capture.format = ma_format_s16;
|
||||||
cap_cfg.capture.channels = p.channels;
|
cap_cfg.capture.channels = p.capture_channels;
|
||||||
cap_cfg.sampleRate = p.sample_rate;
|
cap_cfg.sampleRate = p.sample_rate;
|
||||||
cap_cfg.dataCallback = capture_data_cb;
|
cap_cfg.dataCallback = capture_data_cb;
|
||||||
cap_cfg.pUserData = this;
|
cap_cfg.pUserData = this;
|
||||||
cap_cfg.capture.pDeviceID = nullptr; // always default for now
|
cap_cfg.capture.pDeviceID = have_cap_id ? &cap_id : nullptr; // null = default device
|
||||||
|
|
||||||
if (ma_device_init(nullptr, &cap_cfg, &capture_device_) == MA_SUCCESS) {
|
if (ma_device_init(nullptr, &cap_cfg, &capture_device_) == MA_SUCCESS) {
|
||||||
if (ma_device_start(&capture_device_) == MA_SUCCESS) {
|
if (ma_device_start(&capture_device_) == MA_SUCCESS) {
|
||||||
@@ -105,13 +146,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ── Playback device ─────────────────────────────────────────────────────
|
// ── Playback device ─────────────────────────────────────────────────────
|
||||||
|
ma_device_id pb_id{};
|
||||||
|
bool have_pb_id = !p.playback_device_id.empty() &&
|
||||||
|
hex_decode_device_id(p.playback_device_id, &pb_id);
|
||||||
|
|
||||||
ma_device_config pb_cfg = ma_device_config_init(ma_device_type_playback);
|
ma_device_config pb_cfg = ma_device_config_init(ma_device_type_playback);
|
||||||
pb_cfg.playback.format = ma_format_s16;
|
pb_cfg.playback.format = ma_format_s16;
|
||||||
pb_cfg.playback.channels = p.channels;
|
pb_cfg.playback.channels = p.playback_channels;
|
||||||
pb_cfg.sampleRate = p.sample_rate;
|
pb_cfg.sampleRate = p.sample_rate;
|
||||||
pb_cfg.dataCallback = playback_data_cb;
|
pb_cfg.dataCallback = playback_data_cb;
|
||||||
pb_cfg.pUserData = this;
|
pb_cfg.pUserData = this;
|
||||||
pb_cfg.playback.pDeviceID = nullptr;
|
pb_cfg.playback.pDeviceID = have_pb_id ? &pb_id : nullptr;
|
||||||
|
|
||||||
if (ma_device_init(nullptr, &pb_cfg, &playback_device_) == MA_SUCCESS) {
|
if (ma_device_init(nullptr, &pb_cfg, &playback_device_) == MA_SUCCESS) {
|
||||||
if (ma_device_start(&playback_device_) == MA_SUCCESS) {
|
if (ma_device_start(&playback_device_) == MA_SUCCESS) {
|
||||||
@@ -119,12 +164,57 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
|
|||||||
} else {
|
} else {
|
||||||
ma_device_uninit(&playback_device_);
|
ma_device_uninit(&playback_device_);
|
||||||
}
|
}
|
||||||
|
} else if (p.playback_channels != 1) {
|
||||||
|
// Fallback: some unusual hardware may not accept the requested channel count even
|
||||||
|
// though WASAPI shared mode normally remixes transparently. Retry once at mono rather
|
||||||
|
// than leaving playback dead.
|
||||||
|
pb_cfg.playback.channels = 1;
|
||||||
|
if (ma_device_init(nullptr, &pb_cfg, &playback_device_) == MA_SUCCESS) {
|
||||||
|
if (ma_device_start(&playback_device_) == MA_SUCCESS) {
|
||||||
|
playback_started_ = true;
|
||||||
|
params_.playback_channels = 1;
|
||||||
|
} else {
|
||||||
|
ma_device_uninit(&playback_device_);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
#endif // VOICECAT_HAS_AUDIO
|
#endif // VOICECAT_HAS_AUDIO
|
||||||
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
std::vector<DeviceInfo> AudioEngine::enumerate_devices(bool capture) {
|
||||||
|
std::vector<DeviceInfo> result;
|
||||||
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
|
ma_context ctx;
|
||||||
|
if (ma_context_init(nullptr, 0, nullptr, &ctx) != MA_SUCCESS) return result;
|
||||||
|
|
||||||
|
ma_device_info* playback_infos = nullptr;
|
||||||
|
ma_uint32 playback_count = 0;
|
||||||
|
ma_device_info* capture_infos = nullptr;
|
||||||
|
ma_uint32 capture_count = 0;
|
||||||
|
|
||||||
|
if (ma_context_get_devices(&ctx, &playback_infos, &playback_count, &capture_infos,
|
||||||
|
&capture_count) == MA_SUCCESS) {
|
||||||
|
ma_device_info* infos = capture ? capture_infos : playback_infos;
|
||||||
|
ma_uint32 count = capture ? capture_count : playback_count;
|
||||||
|
result.reserve(count);
|
||||||
|
for (ma_uint32 i = 0; i < count; ++i) {
|
||||||
|
DeviceInfo d;
|
||||||
|
d.id = hex_encode_device_id(infos[i].id);
|
||||||
|
d.name = infos[i].name;
|
||||||
|
d.is_default = infos[i].isDefault != 0;
|
||||||
|
result.push_back(std::move(d));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
ma_context_uninit(&ctx);
|
||||||
|
#else
|
||||||
|
(void)capture;
|
||||||
|
#endif // VOICECAT_HAS_AUDIO
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
void AudioEngine::stop() {
|
void AudioEngine::stop() {
|
||||||
if (!running_.exchange(false)) return;
|
if (!running_.exchange(false)) return;
|
||||||
|
|
||||||
@@ -263,22 +353,25 @@ void AudioEngine::playback_data_cb(ma_device* dev, void* out,
|
|||||||
}
|
}
|
||||||
|
|
||||||
void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
|
void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
|
||||||
std::memset(out, 0, frames * params_.channels * sizeof(int16_t));
|
const uint32_t pb_channels = params_.playback_channels;
|
||||||
|
std::memset(out, 0, frames * pb_channels * sizeof(int16_t));
|
||||||
|
|
||||||
std::unique_lock lk(streams_mu_, std::try_to_lock);
|
std::unique_lock lk(streams_mu_, std::try_to_lock);
|
||||||
if (!lk) return; // contended: emit silence this period
|
if (!lk) return; // contended: emit silence this period
|
||||||
|
|
||||||
#ifdef VOICECAT_HAS_OPUS
|
#ifdef VOICECAT_HAS_OPUS
|
||||||
std::vector<int32_t> mix(frames * params_.channels, 0);
|
std::vector<int32_t> mix(frames * pb_channels, 0);
|
||||||
|
|
||||||
for (auto& [ssrc, stream] : streams_) {
|
for (auto& [ssrc, stream] : streams_) {
|
||||||
if (stream.mute || !stream.decoder.valid()) continue;
|
if (stream.mute || !stream.decoder.valid()) continue;
|
||||||
|
|
||||||
// M3: a stream's Opus channel count (mono/stereo, per-channel AudioConfig) may differ
|
// A stream's Opus channel count (mono/stereo, per-channel AudioConfig) may differ from
|
||||||
// from the engine-wide playback channel count (always mono in M3 — see PROGRESS.md).
|
// the engine-wide playback channel count (now genuinely stereo — see docs/voice.md §8).
|
||||||
// Decode into a buffer sized for the *decoder's* channel count (opus_decode's
|
// Decode into a buffer sized for the *decoder's* channel count (opus_decode's
|
||||||
// frame_size parameter is samples-per-channel, not total samples — pass `frames`,
|
// frame_size parameter is samples-per-channel, not total samples — pass `frames`,
|
||||||
// not `frames * channels`), then convert at this boundary.
|
// not `frames * channels`), then convert at this mix boundary: true stereo decode
|
||||||
|
// output is mixed in directly (no downmix); mono decode output is upmixed L=R (mirrors
|
||||||
|
// the capture-side upmix in vc_client::on_capture_frame).
|
||||||
int dec_channels = std::max(1, stream.decoder.channels());
|
int dec_channels = std::max(1, stream.decoder.channels());
|
||||||
auto maybe_frame = stream.jitter.pop(stream.playout_ts);
|
auto maybe_frame = stream.jitter.pop(stream.playout_ts);
|
||||||
std::vector<int16_t> pcm(frames * static_cast<ma_uint32>(dec_channels));
|
std::vector<int16_t> pcm(frames * static_cast<ma_uint32>(dec_channels));
|
||||||
@@ -302,20 +395,26 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
|
|||||||
|
|
||||||
float g = stream.gain;
|
float g = stream.gain;
|
||||||
for (int i = 0; i < n; ++i) {
|
for (int i = 0; i < n; ++i) {
|
||||||
// Downmix decoder output to the engine's mono accumulator if needed
|
if (dec_channels == 2) {
|
||||||
// (average L/R); upmix is unnecessary since the mix buffer is per-channel.
|
int32_t l = static_cast<int32_t>(static_cast<float>(pcm[i * 2]) * g);
|
||||||
int32_t sample = (dec_channels == 2)
|
int32_t r = static_cast<int32_t>(static_cast<float>(pcm[i * 2 + 1]) * g);
|
||||||
? (static_cast<int32_t>(pcm[i * 2]) +
|
if (pb_channels == 2) {
|
||||||
static_cast<int32_t>(pcm[i * 2 + 1])) / 2
|
mix[i * 2] += l;
|
||||||
: static_cast<int32_t>(pcm[i]);
|
mix[i * 2 + 1] += r;
|
||||||
for (uint32_t c = 0; c < params_.channels; ++c)
|
} else {
|
||||||
mix[i * params_.channels + c] += static_cast<int32_t>(sample * g);
|
mix[i] += (l + r) / 2; // playback device fell back to mono
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
int32_t sample = static_cast<int32_t>(static_cast<float>(pcm[i]) * g);
|
||||||
|
for (uint32_t c = 0; c < pb_channels; ++c)
|
||||||
|
mix[i * pb_channels + c] += sample; // upmix mono -> all playback channels
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
stream.playout_ts += frames;
|
stream.playout_ts += frames;
|
||||||
}
|
}
|
||||||
|
|
||||||
for (ma_uint32 i = 0; i < frames * params_.channels; ++i)
|
for (ma_uint32 i = 0; i < frames * pb_channels; ++i)
|
||||||
out[i] = static_cast<int16_t>(std::clamp(mix[i], -32768, 32767));
|
out[i] = static_cast<int16_t>(std::clamp(mix[i], -32768, 32767));
|
||||||
#else
|
#else
|
||||||
(void)out;
|
(void)out;
|
||||||
@@ -323,6 +422,62 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
|
|||||||
#endif
|
#endif
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#ifdef VOICECAT_HAS_LOOPBACK
|
||||||
|
void AudioEngine::loopback_data_cb(ma_device* dev, void* /*out*/, const void* in,
|
||||||
|
ma_uint32 frame_count) {
|
||||||
|
auto* self = static_cast<AudioEngine*>(dev->pUserData);
|
||||||
|
self->on_loopback(static_cast<const int16_t*>(in), frame_count);
|
||||||
|
}
|
||||||
|
|
||||||
|
void AudioEngine::on_loopback(const int16_t* pcm, ma_uint32 frames) {
|
||||||
|
// Same pattern as the real mic capture callback (on_capture) — call capture_cb_ directly,
|
||||||
|
// NOT through inject_capture()'s test-only ring (see audio_engine.h).
|
||||||
|
if (capture_cb_) capture_cb_(loopback_kind_, pcm, static_cast<int>(frames));
|
||||||
|
}
|
||||||
|
|
||||||
|
bool AudioEngine::start_loopback_capture(int kind) {
|
||||||
|
if (loopback_started_) return false; // already running; stop_loopback_capture() first
|
||||||
|
|
||||||
|
ma_device_config cfg = ma_device_config_init(ma_device_type_loopback);
|
||||||
|
cfg.capture.format = ma_format_s16;
|
||||||
|
cfg.capture.channels = 1; // let miniaudio's converter remix from the system's mix format
|
||||||
|
cfg.sampleRate = params_.sample_rate;
|
||||||
|
cfg.dataCallback = loopback_data_cb;
|
||||||
|
cfg.pUserData = this;
|
||||||
|
// pDeviceID left null: captures the default render endpoint (docs/voice.md §9). Windows
|
||||||
|
// 10 2004+ supports process-specific loopback via AUDIOCLIENT_ACTIVATION_PARAMS, but
|
||||||
|
// miniaudio's loopback mode only exposes whole-device capture — a future enhancement, not
|
||||||
|
// this pass. Whole-device loopback inherently captures this app's own incoming voice mix
|
||||||
|
// along with everything else playing — an accepted self-echo-loop characteristic of
|
||||||
|
// desktop-audio capture, not a bug.
|
||||||
|
|
||||||
|
if (ma_device_init(nullptr, &cfg, &loopback_device_) != MA_SUCCESS) return false;
|
||||||
|
if (ma_device_start(&loopback_device_) != MA_SUCCESS) {
|
||||||
|
ma_device_uninit(&loopback_device_);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
loopback_kind_ = kind;
|
||||||
|
loopback_started_ = true;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
void AudioEngine::stop_loopback_capture() {
|
||||||
|
if (!loopback_started_) return;
|
||||||
|
ma_device_stop(&loopback_device_);
|
||||||
|
ma_device_uninit(&loopback_device_);
|
||||||
|
loopback_started_ = false;
|
||||||
|
}
|
||||||
|
#else // !VOICECAT_HAS_LOOPBACK
|
||||||
|
bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; }
|
||||||
|
void AudioEngine::stop_loopback_capture() {}
|
||||||
|
#endif // VOICECAT_HAS_LOOPBACK
|
||||||
|
|
||||||
#endif // VOICECAT_HAS_AUDIO
|
#endif // VOICECAT_HAS_AUDIO
|
||||||
|
|
||||||
|
#ifndef VOICECAT_HAS_AUDIO
|
||||||
|
bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; }
|
||||||
|
void AudioEngine::stop_loopback_capture() {}
|
||||||
|
#endif
|
||||||
|
|
||||||
} // namespace voicecat::audio
|
} // namespace voicecat::audio
|
||||||
|
|||||||
@@ -75,10 +75,21 @@ class JitterBuffer {
|
|||||||
// ── AudioParams ──────────────────────────────────────────────────────────────
|
// ── AudioParams ──────────────────────────────────────────────────────────────
|
||||||
struct AudioParams {
|
struct AudioParams {
|
||||||
uint32_t sample_rate = 48000;
|
uint32_t sample_rate = 48000;
|
||||||
uint32_t channels = 1;
|
uint32_t capture_channels = 1; // no stereo capture device (mic) in this pass
|
||||||
|
uint32_t playback_channels = 2; // true stereo output (see audio_engine.cpp on_playback)
|
||||||
uint32_t frame_ms = 20;
|
uint32_t frame_ms = 20;
|
||||||
std::string capture_device_id; // "" = default
|
std::string capture_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
|
||||||
std::string playback_device_id; // "" = default
|
std::string playback_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
|
||||||
|
};
|
||||||
|
|
||||||
|
// One enumerated device, returned by AudioEngine::enumerate_devices(). `id` is an internal,
|
||||||
|
// opaque hex-encoded ma_device_id — callers must always round-trip an id that came from
|
||||||
|
// enumerate_devices(); never construct one by hand (names aren't guaranteed unique, so the id
|
||||||
|
// is the only stable handle miniaudio accepts back for device selection).
|
||||||
|
struct DeviceInfo {
|
||||||
|
std::string id;
|
||||||
|
std::string name;
|
||||||
|
bool is_default = false;
|
||||||
};
|
};
|
||||||
|
|
||||||
// ── AudioEngine ──────────────────────────────────────────────────────────────
|
// ── AudioEngine ──────────────────────────────────────────────────────────────
|
||||||
@@ -105,6 +116,18 @@ class AudioEngine {
|
|||||||
|
|
||||||
bool running() const { return running_.load(std::memory_order_acquire); }
|
bool running() const { return running_.load(std::memory_order_acquire); }
|
||||||
|
|
||||||
|
// Enumerate input or output devices (for UI pickers / vc_list_devices). Static: works
|
||||||
|
// before any AudioEngine instance is running (device pickers need to populate pre-connect).
|
||||||
|
// Inits a throwaway ma_context if VOICECAT_HAS_AUDIO; returns {} otherwise. See DeviceInfo
|
||||||
|
// above for the `id` encoding contract.
|
||||||
|
static std::vector<DeviceInfo> enumerate_devices(bool capture);
|
||||||
|
|
||||||
|
// Real desktop-audio loopback capture (Windows/WASAPI only, VOICECAT_HAS_LOOPBACK). Feeds
|
||||||
|
// `kind`'s capture_cb_ directly, same pattern as the real mic capture device — NOT routed
|
||||||
|
// through inject_capture()'s test-only ring. No-op (returns false) when unsupported.
|
||||||
|
bool start_loopback_capture(int kind);
|
||||||
|
void stop_loopback_capture();
|
||||||
|
|
||||||
// Inject synthetic PCM directly into the capture pipeline (bypasses real device).
|
// Inject synthetic PCM directly into the capture pipeline (bypasses real device).
|
||||||
// Thread-safe; can be called from any thread including tests. `kind` selects which local
|
// Thread-safe; can be called from any thread including tests. `kind` selects which local
|
||||||
// stream's injection tap to feed (each gets its own ring buffer); the 2-arg overload
|
// stream's injection tap to feed (each gets its own ring buffer); the 2-arg overload
|
||||||
@@ -139,6 +162,13 @@ class AudioEngine {
|
|||||||
void init_recv_stream(uint32_t ssrc, const codec::OpusParams& p);
|
void init_recv_stream(uint32_t ssrc, const codec::OpusParams& p);
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
|
// TEST-ONLY — exposes the playback mixer without a real ma_device, so tests can verify
|
||||||
|
// stereo mixing end-to-end (no audio hardware needed). Same logic the real playback
|
||||||
|
// callback uses; safe to call any time after start() (no ma_device touched).
|
||||||
|
void mix_for_test(int16_t* out, uint32_t frames) { on_playback(out, frames); }
|
||||||
|
#endif
|
||||||
|
|
||||||
private:
|
private:
|
||||||
#ifdef VOICECAT_HAS_AUDIO
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
static void capture_data_cb(ma_device*, void*, const void*, ma_uint32);
|
static void capture_data_cb(ma_device*, void*, const void*, ma_uint32);
|
||||||
@@ -150,6 +180,18 @@ class AudioEngine {
|
|||||||
ma_device playback_device_{};
|
ma_device playback_device_{};
|
||||||
bool capture_started_ = false;
|
bool capture_started_ = false;
|
||||||
bool playback_started_ = false;
|
bool playback_started_ = false;
|
||||||
|
|
||||||
|
#ifdef VOICECAT_HAS_LOOPBACK
|
||||||
|
// Desktop-audio loopback capture (SCREEN_AUDIO) — own lifecycle, decoupled from
|
||||||
|
// capture_device_/playback_device_ start/stop (a screen-share can start/stop independently
|
||||||
|
// of the mic and of whether anything is currently playing back).
|
||||||
|
static void loopback_data_cb(ma_device*, void*, const void*, ma_uint32);
|
||||||
|
void on_loopback(const int16_t* pcm, ma_uint32 frames);
|
||||||
|
|
||||||
|
ma_device loopback_device_{};
|
||||||
|
bool loopback_started_ = false;
|
||||||
|
int loopback_kind_ = 0;
|
||||||
|
#endif
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
AudioParams params_{};
|
AudioParams params_{};
|
||||||
|
|||||||
@@ -657,11 +657,26 @@ void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples) {
|
|||||||
// playing while the user's mic is muted (docs §M3 scope decision).
|
// playing while the user's mic is muted (docs §M3 scope decision).
|
||||||
if (kind == static_cast<int>(VC_STREAM_MIC) &&
|
if (kind == static_cast<int>(VC_STREAM_MIC) &&
|
||||||
self_mic_muted_.load(std::memory_order_acquire)) return;
|
self_mic_muted_.load(std::memory_order_acquire)) return;
|
||||||
|
|
||||||
|
// Send-side input gate (docs/voice.md §11) — MIC only. SCREEN_AUDIO/AUX_DEVICE always
|
||||||
|
// bypass this: gating a screen-share on the user's own voice activity would silently drop
|
||||||
|
// shared music/video audio whenever the user isn't talking, which defeats the feature.
|
||||||
|
if (kind == static_cast<int>(VC_STREAM_MIC)) {
|
||||||
|
if (current_input_mode_.load(std::memory_order_acquire) == VC_INPUT_PUSH_TO_TALK) {
|
||||||
|
if (!ptt_active_.load(std::memory_order_acquire)) return; // gate closed
|
||||||
|
} else if (mic_vad_) {
|
||||||
|
// EnergyVadProcessor never writes through the pointer (see apm_processor.cpp); the
|
||||||
|
// const_cast is safe and avoids splitting ApmProcessor's interface just for this.
|
||||||
|
if (!mic_vad_->process_capture(const_cast<int16_t*>(pcm), samples, 48000)) return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (!media_send_crypto_) return;
|
if (!media_send_crypto_) return;
|
||||||
int fd = udp_fd_.load(std::memory_order_acquire);
|
int fd = udp_fd_.load(std::memory_order_acquire);
|
||||||
if (fd == -1) return;
|
if (fd == -1) return;
|
||||||
|
|
||||||
auto& ls = it->second;
|
auto& ls = it->second;
|
||||||
|
// Updated only after the gate above, so a VAD/PTT-closed frame never shows as "talking".
|
||||||
ls.last_capture_ms.store(client_now_ms(), std::memory_order_relaxed);
|
ls.last_capture_ms.store(client_now_ms(), std::memory_order_relaxed);
|
||||||
|
|
||||||
uint8_t opus_buf[1500];
|
uint8_t opus_buf[1500];
|
||||||
@@ -712,8 +727,14 @@ void vc_client::ensure_audio_running() {
|
|||||||
if (audio_engine_.running()) return;
|
if (audio_engine_.running()) return;
|
||||||
voicecat::audio::AudioParams p;
|
voicecat::audio::AudioParams p;
|
||||||
p.sample_rate = 48000;
|
p.sample_rate = 48000;
|
||||||
p.channels = 1;
|
p.capture_channels = 1; // no stereo capture device in this pass
|
||||||
|
p.playback_channels = 2; // true stereo output (audio_engine.cpp on_playback)
|
||||||
p.frame_ms = 20;
|
p.frame_ms = 20;
|
||||||
|
{
|
||||||
|
std::lock_guard lk(local_streams_mu_);
|
||||||
|
auto it = local_streams_.find(static_cast<int>(VC_STREAM_MIC));
|
||||||
|
if (it != local_streams_.end()) p.capture_device_id = it->second.capture_device_id;
|
||||||
|
}
|
||||||
audio_engine_.start(p, [this](int kind, const int16_t* pcm, int samples) {
|
audio_engine_.start(p, [this](int kind, const int16_t* pcm, int samples) {
|
||||||
on_capture_frame(kind, pcm, samples);
|
on_capture_frame(kind, pcm, samples);
|
||||||
});
|
});
|
||||||
@@ -818,11 +839,12 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
|
|||||||
uint32_t self_uid;
|
uint32_t self_uid;
|
||||||
uint32_t emit_stream_id;
|
uint32_t emit_stream_id;
|
||||||
bool ok_to_emit = false;
|
bool ok_to_emit = false;
|
||||||
|
int kind;
|
||||||
{
|
{
|
||||||
std::lock_guard lk(local_streams_mu_);
|
std::lock_guard lk(local_streams_mu_);
|
||||||
auto pit = pending_announce_kind_.find(req_id);
|
auto pit = pending_announce_kind_.find(req_id);
|
||||||
if (pit == pending_announce_kind_.end()) return; // stray/duplicate — ignore
|
if (pit == pending_announce_kind_.end()) return; // stray/duplicate — ignore
|
||||||
int kind = pit->second;
|
kind = pit->second;
|
||||||
pending_announce_kind_.erase(pit);
|
pending_announce_kind_.erase(pit);
|
||||||
|
|
||||||
auto sit = local_streams_.find(kind);
|
auto sit = local_streams_.find(kind);
|
||||||
@@ -849,9 +871,18 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
|
|||||||
self_uid = self_user_id_;
|
self_uid = self_user_id_;
|
||||||
emit_stream_id = ls.stream_id;
|
emit_stream_id = ls.stream_id;
|
||||||
ok_to_emit = true;
|
ok_to_emit = true;
|
||||||
|
|
||||||
|
// Construct the MIC VAD once, here on io_thread_ (not the RT capture callback) — see
|
||||||
|
// client.h's comment on mic_vad_.
|
||||||
|
if (kind == static_cast<int>(VC_STREAM_MIC) && !mic_vad_) {
|
||||||
|
mic_vad_ = voicecat::audio::ApmProcessor::create_vad();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
ensure_audio_running();
|
ensure_audio_running();
|
||||||
|
if (kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO)) {
|
||||||
|
audio_engine_.start_loopback_capture(kind);
|
||||||
|
}
|
||||||
|
|
||||||
if (ok_to_emit) {
|
if (ok_to_emit) {
|
||||||
vc_event ev{};
|
vc_event ev{};
|
||||||
@@ -865,14 +896,22 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
|
|||||||
vc_result vc_client::stream_stop(uint32_t stream_id) {
|
vc_result vc_client::stream_stop(uint32_t stream_id) {
|
||||||
if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED;
|
if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED;
|
||||||
|
|
||||||
|
int stopped_kind = -1;
|
||||||
{
|
{
|
||||||
std::lock_guard lk(local_streams_mu_);
|
std::lock_guard lk(local_streams_mu_);
|
||||||
|
for (auto& [k, ls] : local_streams_) {
|
||||||
|
if (ls.stream_id == stream_id) { stopped_kind = k; break; }
|
||||||
|
}
|
||||||
LocalStream* ls = find_local_stream_by_id(stream_id);
|
LocalStream* ls = find_local_stream_by_id(stream_id);
|
||||||
if (!ls || !ls->active.load(std::memory_order_acquire)) return VC_ERR_INVALID_ARG;
|
if (!ls || !ls->active.load(std::memory_order_acquire)) return VC_ERR_INVALID_ARG;
|
||||||
ls->active.store(false, std::memory_order_release);
|
ls->active.store(false, std::memory_order_release);
|
||||||
ls->encoder.destroy();
|
ls->encoder.destroy();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (stopped_kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO)) {
|
||||||
|
audio_engine_.stop_loopback_capture();
|
||||||
|
}
|
||||||
|
|
||||||
voicecat::v1::Envelope req;
|
voicecat::v1::Envelope req;
|
||||||
req.set_request_id(next_req_id_++);
|
req.set_request_id(next_req_id_++);
|
||||||
req.mutable_stream_stop()->set_stream_id(stream_id);
|
req.mutable_stream_stop()->set_stream_id(stream_id);
|
||||||
@@ -894,9 +933,38 @@ vc_client::LocalStream* vc_client::find_local_stream_by_id(uint32_t stream_id) {
|
|||||||
return nullptr;
|
return nullptr;
|
||||||
}
|
}
|
||||||
|
|
||||||
vc_result vc_client::set_input_device(uint32_t, const char*) { return VC_ERR_NOT_IMPLEMENTED; }
|
vc_result vc_client::set_input_device(uint32_t stream_id, const char* device_id) {
|
||||||
vc_result vc_client::set_input_mode(vc_input_mode) { return VC_ERR_NOT_IMPLEMENTED; }
|
int kind = -1;
|
||||||
vc_result vc_client::set_push_to_talk(bool) { return VC_ERR_NOT_IMPLEMENTED; }
|
{
|
||||||
|
std::lock_guard lk(local_streams_mu_);
|
||||||
|
LocalStream* ls = find_local_stream_by_id(stream_id);
|
||||||
|
if (!ls) return VC_ERR_INVALID_ARG;
|
||||||
|
ls->capture_device_id = device_id ? device_id : "";
|
||||||
|
for (auto& [k, entry] : local_streams_) {
|
||||||
|
if (&entry == ls) { kind = k; break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Only the real capture device (MIC) is affected by device selection — SCREEN_AUDIO uses
|
||||||
|
// loopback capture (no input device to pick) and AUX_DEVICE isn't backed by a real device
|
||||||
|
// path yet. Restart the engine unconditionally when it's already running so the new device
|
||||||
|
// id takes effect; AudioEngine doesn't expose a getter for "is this the same device" so we
|
||||||
|
// don't try to skip the restart when it happens to be a no-op change.
|
||||||
|
if (kind == static_cast<int>(VC_STREAM_MIC) && audio_engine_.running()) {
|
||||||
|
audio_engine_.stop();
|
||||||
|
ensure_audio_running();
|
||||||
|
}
|
||||||
|
return VC_OK;
|
||||||
|
}
|
||||||
|
|
||||||
|
vc_result vc_client::set_input_mode(vc_input_mode mode) {
|
||||||
|
current_input_mode_.store(mode, std::memory_order_release);
|
||||||
|
return VC_OK;
|
||||||
|
}
|
||||||
|
|
||||||
|
vc_result vc_client::set_push_to_talk(bool active) {
|
||||||
|
ptt_active_.store(active, std::memory_order_release);
|
||||||
|
return VC_OK;
|
||||||
|
}
|
||||||
|
|
||||||
vc_result vc_client::set_self_mute(bool mic_muted, bool deafened) {
|
vc_result vc_client::set_self_mute(bool mic_muted, bool deafened) {
|
||||||
if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED;
|
if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED;
|
||||||
@@ -984,10 +1052,31 @@ vc_result vc_client::test_inject_capture(uint32_t stream_id, const int16_t* pcm,
|
|||||||
return VC_OK;
|
return VC_OK;
|
||||||
}
|
}
|
||||||
|
|
||||||
vc_result vc_client::list_devices(vc_device_kind, vc_device_list* out) {
|
vc_result vc_client::list_devices(vc_device_kind kind, vc_device_list* out) {
|
||||||
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
|
// Works in any connection state — device pickers need to populate pre-connect.
|
||||||
|
auto devices = voicecat::audio::AudioEngine::enumerate_devices(kind == VC_DEVICE_INPUT);
|
||||||
|
|
||||||
|
auto* items = new vc_device[devices.size()];
|
||||||
|
for (size_t i = 0; i < devices.size(); ++i) {
|
||||||
|
const auto& d = devices[i];
|
||||||
|
auto* id = new char[d.id.size() + 1];
|
||||||
|
auto* name = new char[d.name.size() + 1];
|
||||||
|
std::memcpy(id, d.id.c_str(), d.id.size() + 1);
|
||||||
|
std::memcpy(name, d.name.c_str(), d.name.size() + 1);
|
||||||
|
items[i].id = id;
|
||||||
|
items[i].name = name;
|
||||||
|
items[i].is_default = d.is_default ? 1 : 0;
|
||||||
|
}
|
||||||
|
out->items = items;
|
||||||
|
out->count = devices.size();
|
||||||
|
return VC_OK;
|
||||||
|
#else
|
||||||
|
(void)kind;
|
||||||
out->items = nullptr;
|
out->items = nullptr;
|
||||||
out->count = 0;
|
out->count = 0;
|
||||||
return VC_ERR_NOT_IMPLEMENTED;
|
return VC_ERR_NOT_IMPLEMENTED;
|
||||||
|
#endif
|
||||||
}
|
}
|
||||||
|
|
||||||
void vc_client::run_talk_timer() {
|
void vc_client::run_talk_timer() {
|
||||||
|
|||||||
@@ -147,9 +147,16 @@ struct vc_client {
|
|||||||
// vc_get_stream_audio_config() has something to read back for our own streams.
|
// vc_get_stream_audio_config() has something to read back for our own streams.
|
||||||
voicecat::codec::OpusParams effective_params;
|
voicecat::codec::OpusParams effective_params;
|
||||||
|
|
||||||
// Talk-indicator edge detection (docs/voice.md §7) — updated in on_capture_frame.
|
// Talk-indicator edge detection (docs/voice.md §7) — updated in on_capture_frame, after
|
||||||
|
// the VAD/PTT gate so a gated-closed frame doesn't show as "talking".
|
||||||
std::atomic<int64_t> last_capture_ms{0};
|
std::atomic<int64_t> last_capture_ms{0};
|
||||||
bool talking = false;
|
bool talking = false;
|
||||||
|
|
||||||
|
// Device-enumeration follow-up: the device this stream's capture should use ("" =
|
||||||
|
// default). Only meaningful for VC_STREAM_MIC today (the real capture device); set via
|
||||||
|
// vc_set_input_device. Opaque id from AudioEngine::enumerate_devices — see
|
||||||
|
// audio_engine.h's DeviceInfo doc comment.
|
||||||
|
std::string capture_device_id;
|
||||||
};
|
};
|
||||||
mutable std::mutex local_streams_mu_;
|
mutable std::mutex local_streams_mu_;
|
||||||
std::unordered_map<int, LocalStream> local_streams_; // keyed by vc_stream_kind
|
std::unordered_map<int, LocalStream> local_streams_; // keyed by vc_stream_kind
|
||||||
@@ -174,6 +181,14 @@ struct vc_client {
|
|||||||
std::atomic<bool> self_mic_muted_{false};
|
std::atomic<bool> self_mic_muted_{false};
|
||||||
std::atomic<bool> self_deafened_{false};
|
std::atomic<bool> self_deafened_{false};
|
||||||
|
|
||||||
|
// Follow-up to M3: send-side input gate (docs/voice.md §11). MIC-only — SCREEN_AUDIO/
|
||||||
|
// AUX_DEVICE are never gated (see PROGRESS.md for the rationale). Pure local state, no
|
||||||
|
// protocol traffic. mic_vad_ is constructed once the MIC stream's StreamAnnounceResult
|
||||||
|
// lands (handle_stream_announce_result, on io_thread_ — not the RT capture callback).
|
||||||
|
std::atomic<vc_input_mode> current_input_mode_{VC_INPUT_VOICE_ACTIVATION};
|
||||||
|
std::atomic<bool> ptt_active_{false};
|
||||||
|
std::unique_ptr<voicecat::audio::ApmProcessor> mic_vad_;
|
||||||
|
|
||||||
// teardown_voice() is called both from run_io()'s own cleanup (on the io_thread_, when
|
// teardown_voice() is called both from run_io()'s own cleanup (on the io_thread_, when
|
||||||
// the read loop exits) and from disconnect() (on the caller's thread) -- without
|
// the read loop exits) and from disconnect() (on the caller's thread) -- without
|
||||||
// serializing those two call sites, both can see udp_thread_/talk_timer_thread_ as
|
// serializing those two call sites, both can see udp_thread_/talk_timer_thread_ as
|
||||||
|
|||||||
@@ -140,8 +140,12 @@ vc_result vc_list_devices(vc_client* c, vc_device_kind kind, vc_device_list* out
|
|||||||
}
|
}
|
||||||
|
|
||||||
void vc_free_device_list(vc_device_list* list) {
|
void vc_free_device_list(vc_device_list* list) {
|
||||||
if (list == nullptr) return;
|
if (list == nullptr || list->items == nullptr) return;
|
||||||
/* Stub: no allocation yet. Real impl frees list->items here. */
|
for (size_t i = 0; i < list->count; ++i) {
|
||||||
|
delete[] list->items[i].id;
|
||||||
|
delete[] list->items[i].name;
|
||||||
|
}
|
||||||
|
delete[] list->items;
|
||||||
list->items = nullptr;
|
list->items = nullptr;
|
||||||
list->count = 0;
|
list->count = 0;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ that implementation can start from a shared, agreed plan.
|
|||||||
| Media transport | **UDP** secured by **TLS-exported keys + ChaCha20-Poly1305 AEAD** — mandatory, no plaintext mode (see [security.md](security.md)) |
|
| Media transport | **UDP** secured by **TLS-exported keys + ChaCha20-Poly1305 AEAD** — mandatory, no plaintext mode (see [security.md](security.md)) |
|
||||||
| Crypto libraries | **mbedTLS** (TLS 1.3) + **libsodium** (AEAD, Argon2id, Ed25519) — both permissive, **no GPL/LGPL anywhere** |
|
| Crypto libraries | **mbedTLS** (TLS 1.3) + **libsodium** (AEAD, Argon2id, Ed25519) — both permissive, **no GPL/LGPL anywhere** |
|
||||||
| Voice codec | **Opus** (libopus 1.6), per-channel configurable mono/stereo, bitrate, frame size, FEC/DTX |
|
| Voice codec | **Opus** (libopus 1.6), per-channel configurable mono/stereo, bitrate, frame size, FEC/DTX |
|
||||||
| Audio DSP | **webrtc-audio-processing (APM)** — AEC/NS/AGC/VAD. NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally. Input gate supports **VAD and PTT**, client-configurable. |
|
| Audio DSP | **webrtc-audio-processing (APM)** was the plan for AEC/NS/AGC/VAD, but has no working Windows/MSVC build upstream — v1 ships a lightweight energy/RMS VAD instead, no AEC/NS/AGC yet (see [tech-stack.md](tech-stack.md), [roadmap.md](roadmap.md) §2). NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally — this plumbing exists but is currently inert pending a real DSP backend. Input gate supports **VAD and PTT**, client-configurable. |
|
||||||
| Identity | **Guests + admin-provisioned local accounts** (Argon2id, SQLite). No self-serve registration; guests toggleable per server. |
|
| Identity | **Guests + admin-provisioned local accounts** (Argon2id, SQLite). No self-serve registration; guests toggleable per server. |
|
||||||
| Text | **Ephemeral** — live relay, no server-side history in v1. |
|
| Text | **Ephemeral** — live relay, no server-side history in v1. |
|
||||||
| Serialization | **Protocol Buffers** for the control plane; **custom binary** for voice frames |
|
| Serialization | **Protocol Buffers** for the control plane; **custom binary** for voice frames |
|
||||||
|
|||||||
@@ -126,6 +126,11 @@ Design notes:
|
|||||||
core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
|
core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
|
||||||
meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
|
meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
|
||||||
exception that needs UI-side cooperation — see below.)
|
exception that needs UI-side cooperation — see below.)
|
||||||
|
- **Device enumeration works pre-connect.** `vc_list_devices` needs no live session — device
|
||||||
|
pickers can populate before `vc_connect`. `vc_device.id` is an opaque, internally-encoded
|
||||||
|
handle (currently a hex-encoded `ma_device_id`) — always round-trip an id that came from
|
||||||
|
`vc_list_devices`/`vc_get_stream_audio_config`; never construct one by hand. Tolerate an
|
||||||
|
empty list (a machine can legitimately have zero input or output devices).
|
||||||
- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
|
- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
|
||||||
caller-allocated or returned with a paired `vc_free`.
|
caller-allocated or returned with a paired `vc_free`.
|
||||||
|
|
||||||
|
|||||||
@@ -77,6 +77,13 @@ Settled and reflected throughout the docs:
|
|||||||
directory). (deployment.md §3)
|
directory). (deployment.md §3)
|
||||||
- **Bitrate ceiling:** server-config `opus.limits.max_bitrate_bps`. (deployment.md §2)
|
- **Bitrate ceiling:** server-config `opus.limits.max_bitrate_bps`. (deployment.md §2)
|
||||||
- **Name:** "VoiceCat" stays as the internal placeholder.
|
- **Name:** "VoiceCat" stays as the internal placeholder.
|
||||||
|
- **DSP engine, superseded (2026-06-16):** the "webrtc-audio-processing (APM)" decision above
|
||||||
|
(AEC + NS/AGC/VAD in one module) could not be carried out — it has no working Windows/MSVC
|
||||||
|
build upstream (GCC-only Meson build, MinGW support unfinished, hard `abseil-cpp` dependency,
|
||||||
|
Linux-tested only). v1 ships a lightweight, dependency-free energy/RMS VAD instead, behind
|
||||||
|
the same `ApmProcessor` interface; there is **no AEC/NS/AGC implementation at all** yet. Real
|
||||||
|
`webrtc-audio-processing` stays a tracked future swap (e.g. if/when a Linux build target
|
||||||
|
exists). (voice.md §8, §11)
|
||||||
|
|
||||||
## 3. Open questions
|
## 3. Open questions
|
||||||
|
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ Concrete library choices with versions and rationale. Everything in the **core**
|
|||||||
| Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. |
|
| Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. |
|
||||||
| Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. |
|
| Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. |
|
||||||
| Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. |
|
| Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. |
|
||||||
| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) | 1.x (standalone APM) | **BSD-3.** The primary DSP engine: high-quality acoustic echo cancellation, noise suppression, AGC, and VAD in one tuned module. Used **send-side** (clean the mic) and **receive-side per user** (listener-chosen NS on a specific stream — voice.md §10). AEC is in from the start, not deferred. |
|
| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) — **planned, not built** | 1.x (standalone APM) | **BSD-3**, but has no working Windows/MSVC build upstream (GCC-only Meson, MinGW support unfinished, hard `abseil-cpp` dep — see roadmap.md §2). v1 ships a lightweight, dependency-free energy/RMS VAD instead (`core/src/audio/apm_processor.cpp`); there is **no AEC, NS, or AGC implementation at all** yet. Real APM stays a tracked future swap behind the same `ApmProcessor` interface. |
|
||||||
| Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) |
|
| Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) |
|
||||||
| Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. |
|
| Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. |
|
||||||
| Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. |
|
| Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. |
|
||||||
@@ -56,7 +56,7 @@ explicit resampling (speexdsp/libsamplerate) is only needed when a device can't
|
|||||||
| Tool | Use |
|
| Tool | Use |
|
||||||
|------|-----|
|
|------|-----|
|
||||||
| **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. |
|
| **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. |
|
||||||
| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, webrtc-audio-processing, speexdsp, asio, miniaudio). Reproducible across OSes. |
|
| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, asio, miniaudio — see `vcpkg.json`). `webrtc-audio-processing`/`speexdsp` are **not** in the manifest: no working vcpkg port / no working Windows/MSVC build exists upstream for the former; the latter was never actually wired up (the lightweight VAD needs no resampler). Reproducible across OSes. |
|
||||||
| **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). |
|
| **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). |
|
||||||
| **clang-format / clang-tidy** | Style + static analysis on the core. |
|
| **clang-format / clang-tidy** | Style + static analysis on the core. |
|
||||||
| **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). |
|
| **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). |
|
||||||
@@ -69,9 +69,11 @@ dependencies are permitted.** Every dependency below is BSD / MIT / ISC / Apache
|
|||||||
public-domain:
|
public-domain:
|
||||||
|
|
||||||
- **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ ·
|
- **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ ·
|
||||||
**miniaudio** — public domain / MIT-0 ✅ · **speexdsp** — BSD ✅ · **protobuf** — BSD ✅ ·
|
**miniaudio** — public domain / MIT-0 ✅ · **protobuf** — BSD ✅ ·
|
||||||
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅ ·
|
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅.
|
||||||
**webrtc-audio-processing** — BSD-3 ✅ (heavier build, but core to the DSP path).
|
**webrtc-audio-processing** would be BSD-3 ✅ if/when it's actually built in (see §1) —
|
||||||
|
not a live dependency today, so not part of the resolved vcpkg graph the license scanner
|
||||||
|
below checks.
|
||||||
- **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag
|
- **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag
|
||||||
in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for
|
in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for
|
||||||
one entirely.
|
one entirely.
|
||||||
|
|||||||
@@ -159,20 +159,27 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**.
|
|||||||
## 8. Capture/playback pipeline (inside the core)
|
## 8. Capture/playback pipeline (inside the core)
|
||||||
|
|
||||||
```
|
```
|
||||||
device ─(miniaudio capture, 48k)→ resample? → send-side APM
|
device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
|
||||||
(AEC + NS + AGC + VAD/PTT gate) → Opus encode → frame header → AEAD → UDP send
|
→ Opus encode → frame header → AEAD → UDP send
|
||||||
|
|
||||||
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
|
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
|
||||||
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
|
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
|
||||||
→ mixer (sum all ssrc) → (miniaudio playback, 48k) → device
|
→ mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
|
||||||
|
48k, stereo) → device
|
||||||
```
|
```
|
||||||
|
|
||||||
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
|
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
|
||||||
- **DSP engine: webrtc-audio-processing (APM)** — the "better one". It provides
|
Playback is genuinely stereo end-to-end; capture stays mono (no stereo mic in v1).
|
||||||
high-quality **AEC** (acoustic echo cancellation, essential for speaker users), **noise
|
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
|
||||||
suppression**, **AGC**, and a **VAD** in one tuned module, BSD-licensed. speexdsp is kept
|
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
|
||||||
only for resampling and as a lightweight jitter-buffer reference. AEC is **in from the
|
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
|
||||||
start**, not deferred.
|
`abseil-cpp` dependency, Linux-tested only —
|
||||||
|
[gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)).
|
||||||
|
v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS,
|
||||||
|
or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real
|
||||||
|
`webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor`
|
||||||
|
interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or
|
||||||
|
upstream Windows support matures.
|
||||||
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
|
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
|
||||||
|
|
||||||
## 10. Noise reduction — two-sided
|
## 10. Noise reduction — two-sided
|
||||||
@@ -197,13 +204,23 @@ NS is a clean drop-in on that per-stream stage.
|
|||||||
## 11. Input activation — VAD and PTT (client-configurable)
|
## 11. Input activation — VAD and PTT (client-configurable)
|
||||||
|
|
||||||
Whether the mic transmits is decided locally by the **input gate**, and the client supports
|
Whether the mic transmits is decided locally by the **input gate**, and the client supports
|
||||||
**both** modes, switchable per client (and ideally per input device):
|
**both** modes, switchable per client (`vc_set_input_mode`):
|
||||||
|
|
||||||
- **Voice activation (VAD):** the APM VAD opens the gate when speech is detected, with a
|
- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free
|
||||||
configurable threshold and hang-time to avoid clipping word tails. DTX naturally
|
energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no
|
||||||
complements this — when the gate is closed nothing (or only comfort noise) is sent.
|
external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC
|
||||||
- **Push-to-talk (PTT):** a held key/button opens the gate. The UI exposes a configurable
|
build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold
|
||||||
keybind; the core just receives gate open/close.
|
(default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms,
|
||||||
|
matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping
|
||||||
|
word tails. DTX naturally complements this — when the gate is closed nothing (or only
|
||||||
|
comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the
|
||||||
|
originally-planned APM, not just a deferred VAD.
|
||||||
|
- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI
|
||||||
|
exposes a configurable keybind; the core just receives gate open/close.
|
||||||
|
|
||||||
|
Gating applies to the **MIC stream only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
|
||||||
|
(gating a desktop-audio share on the user's own voice activity would silently drop shared
|
||||||
|
music/video audio whenever the user isn't talking, which defeats the feature).
|
||||||
|
|
||||||
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
|
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
|
||||||
needs **no protocol support**; remote talk indicators are still derived from the presence of
|
needs **no protocol support**; remote talk indicators are still derived from the presence of
|
||||||
@@ -217,7 +234,7 @@ normal stream; only the *source* is platform-specific.
|
|||||||
|
|
||||||
| Platform | Mechanism | Notes |
|
| Platform | Mechanism | Notes |
|
||||||
|----------|-----------|-------|
|
|----------|-----------|-------|
|
||||||
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | Cleanest case; no extra process. Can capture system mix or a specific endpoint. |
|
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
|
||||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
|
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
|
||||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
||||||
|
|
||||||
|
|||||||
@@ -86,4 +86,13 @@ if(VOICECAT_USE_VCPKG_DEPS)
|
|||||||
target_include_directories(test_m3_multistream PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
|
target_include_directories(test_m3_multistream PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
|
||||||
add_test(NAME m3_multistream COMMAND test_m3_multistream)
|
add_test(NAME m3_multistream COMMAND test_m3_multistream)
|
||||||
set_tests_properties(m3_multistream PROPERTIES TIMEOUT 60)
|
set_tests_properties(m3_multistream PROPERTIES TIMEOUT 60)
|
||||||
|
|
||||||
|
# Post-M3 follow-up: device enumeration, VAD/PTT input gate, true stereo playback mixing —
|
||||||
|
# the items PROGRESS.md's M3 section explicitly carried forward as out of scope.
|
||||||
|
add_executable(test_vad_ptt_devices test_vad_ptt_devices.cpp)
|
||||||
|
target_link_libraries(test_vad_ptt_devices PRIVATE voicecat::server)
|
||||||
|
target_compile_features(test_vad_ptt_devices PRIVATE cxx_std_20)
|
||||||
|
target_include_directories(test_vad_ptt_devices PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
|
||||||
|
add_test(NAME vad_ptt_devices COMMAND test_vad_ptt_devices)
|
||||||
|
set_tests_properties(vad_ptt_devices PROPERTIES TIMEOUT 60)
|
||||||
endif()
|
endif()
|
||||||
|
|||||||
@@ -60,9 +60,17 @@ int main() {
|
|||||||
vc_result rc_join = vc_join_channel(c, 1, nullptr);
|
vc_result rc_join = vc_join_channel(c, 1, nullptr);
|
||||||
CHECK(rc_join == VC_ERR_NOT_IMPLEMENTED || rc_join == VC_ERR_NOT_CONNECTED);
|
CHECK(rc_join == VC_ERR_NOT_IMPLEMENTED || rc_join == VC_ERR_NOT_CONNECTED);
|
||||||
|
|
||||||
vc_device_list dl;
|
vc_device_list dl{};
|
||||||
CHECK(vc_list_devices(c, VC_DEVICE_INPUT, &dl) == VC_ERR_NOT_IMPLEMENTED);
|
vc_result rc_devices = vc_list_devices(c, VC_DEVICE_INPUT, &dl);
|
||||||
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
|
// Real device enumeration is wired up once miniaudio is linked in (post-M3 follow-up).
|
||||||
|
// Never assert count > 0 here — a headless CI build agent may legitimately report zero
|
||||||
|
// audio devices; only that the call itself succeeded.
|
||||||
|
CHECK(rc_devices == VC_OK);
|
||||||
|
#else
|
||||||
|
CHECK(rc_devices == VC_ERR_NOT_IMPLEMENTED);
|
||||||
CHECK(dl.count == 0);
|
CHECK(dl.count == 0);
|
||||||
|
#endif
|
||||||
vc_free_device_list(&dl);
|
vc_free_device_list(&dl);
|
||||||
|
|
||||||
vc_client_destroy(c);
|
vc_client_destroy(c);
|
||||||
|
|||||||
396
tests/test_vad_ptt_devices.cpp
Normal file
396
tests/test_vad_ptt_devices.cpp
Normal file
@@ -0,0 +1,396 @@
|
|||||||
|
/*
|
||||||
|
* test_vad_ptt_devices — closes M3's "explicitly out of scope" gaps (PROGRESS.md): device
|
||||||
|
* enumeration, the VAD/PTT send-side input gate, and true stereo playback mixing.
|
||||||
|
*
|
||||||
|
* Mirrors test_m3_multistream.cpp's approach (real vc_client instances against a real
|
||||||
|
* in-process server, not raw sockets) for the ABI-level pieces, plus a white-box AudioEngine
|
||||||
|
* test for the stereo mixer (no audio hardware needed — see AudioEngine::mix_for_test).
|
||||||
|
*
|
||||||
|
* 1. Device enumeration (vc_list_devices) works pre-connect, for both kinds, and tolerates
|
||||||
|
* an empty list (headless CI build agents may have zero audio devices) — VC_OK is the
|
||||||
|
* only thing asserted, never count > 0.
|
||||||
|
* 2. VAD gate: under VC_INPUT_VOICE_ACTIVATION (the default), silent PCM never reaches the
|
||||||
|
* peer (no talking edge); loud PCM does.
|
||||||
|
* 3. PTT gate: under VC_INPUT_PUSH_TO_TALK, loud PCM is gated closed until
|
||||||
|
* vc_set_push_to_talk(1); then it reaches the peer.
|
||||||
|
* 4. Stereo playback mixer: white-box (AudioEngine directly) — a genuinely stereo decoded
|
||||||
|
* stream survives into the mix without being downmixed to mono.
|
||||||
|
*/
|
||||||
|
#include <cstdio>
|
||||||
|
|
||||||
|
#ifdef VOICECAT_HAS_NET
|
||||||
|
|
||||||
|
#include <atomic>
|
||||||
|
#include <chrono>
|
||||||
|
#include <cmath>
|
||||||
|
#include <condition_variable>
|
||||||
|
#include <cstdlib>
|
||||||
|
#include <filesystem>
|
||||||
|
#include <mutex>
|
||||||
|
#include <string>
|
||||||
|
#include <thread>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
#include "voicecat.h"
|
||||||
|
#include "server.h"
|
||||||
|
#include "db.h"
|
||||||
|
|
||||||
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
|
#include "audio/audio_engine.h"
|
||||||
|
#endif
|
||||||
|
#ifdef VOICECAT_HAS_OPUS
|
||||||
|
#include "codec/opus_codec.h"
|
||||||
|
#endif
|
||||||
|
|
||||||
|
// ── Event tracking (same shape as test_m3_multistream.cpp) ──────────────────────
|
||||||
|
|
||||||
|
struct TalkEvent {
|
||||||
|
uint32_t user_id;
|
||||||
|
uint32_t stream_id;
|
||||||
|
bool talking;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct EventStore {
|
||||||
|
std::mutex mu;
|
||||||
|
std::condition_variable cv;
|
||||||
|
|
||||||
|
bool auth_ok{false};
|
||||||
|
uint32_t self_user_id{0};
|
||||||
|
bool channel_list_received{false};
|
||||||
|
bool saw_stream_started{false};
|
||||||
|
std::vector<TalkEvent> talk_events;
|
||||||
|
bool disconnected{false};
|
||||||
|
|
||||||
|
const char* label{nullptr};
|
||||||
|
};
|
||||||
|
|
||||||
|
static void on_event(void* user, const vc_event* ev) {
|
||||||
|
auto* s = static_cast<EventStore*>(user);
|
||||||
|
std::lock_guard lk(s->mu);
|
||||||
|
switch (ev->type) {
|
||||||
|
case VC_EVENT_AUTH_RESULT:
|
||||||
|
s->auth_ok = (ev->result == VC_OK);
|
||||||
|
s->self_user_id = ev->user_id;
|
||||||
|
break;
|
||||||
|
case VC_EVENT_CHANNEL_LIST:
|
||||||
|
s->channel_list_received = true;
|
||||||
|
break;
|
||||||
|
case VC_EVENT_STREAM_STARTED:
|
||||||
|
s->saw_stream_started = true;
|
||||||
|
break;
|
||||||
|
case VC_EVENT_TALK_STATE:
|
||||||
|
s->talk_events.push_back({ev->user_id, ev->stream_id, ev->u32a != 0});
|
||||||
|
break;
|
||||||
|
case VC_EVENT_DISCONNECTED:
|
||||||
|
s->disconnected = true;
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
s->cv.notify_all();
|
||||||
|
}
|
||||||
|
|
||||||
|
template <typename Pred>
|
||||||
|
static bool wait_for(EventStore& s, Pred pred, int timeout_ms) {
|
||||||
|
auto deadline = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout_ms);
|
||||||
|
std::unique_lock lk(s.mu);
|
||||||
|
return s.cv.wait_until(lk, deadline, [&] { return pred(s); });
|
||||||
|
}
|
||||||
|
|
||||||
|
static std::vector<int16_t> make_sine_frame(int frame_idx, float freq_hz,
|
||||||
|
int frame_samples = 960) {
|
||||||
|
std::vector<int16_t> pcm(frame_samples);
|
||||||
|
for (int i = 0; i < frame_samples; ++i) {
|
||||||
|
float t = static_cast<float>(frame_idx * frame_samples + i) / 48000.0f;
|
||||||
|
pcm[i] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * freq_hz * t) * 16000.0f);
|
||||||
|
}
|
||||||
|
return pcm;
|
||||||
|
}
|
||||||
|
|
||||||
|
static std::vector<int16_t> make_silence_frame(int frame_samples = 960) {
|
||||||
|
return std::vector<int16_t>(frame_samples, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Did `talking==true` ever fire for (user_id, stream_id) at index >= `from`?
|
||||||
|
static bool saw_talking_true(EventStore& s, uint32_t user_id, uint32_t stream_id, size_t from) {
|
||||||
|
std::lock_guard lk(s.mu);
|
||||||
|
for (size_t i = from; i < s.talk_events.size(); ++i) {
|
||||||
|
auto& e = s.talk_events[i];
|
||||||
|
if (e.user_id == user_id && e.stream_id == stream_id && e.talking) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
static size_t talk_event_count(EventStore& s) {
|
||||||
|
std::lock_guard lk(s.mu);
|
||||||
|
return s.talk_events.size();
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Test harness ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
static int g_failures = 0;
|
||||||
|
#define CHECK(cond) \
|
||||||
|
do { \
|
||||||
|
if (!(cond)) { \
|
||||||
|
std::printf("FAIL: %s (%s:%d)\n", #cond, __FILE__, __LINE__); \
|
||||||
|
++g_failures; \
|
||||||
|
} \
|
||||||
|
} while (0)
|
||||||
|
|
||||||
|
// ── 1. Device enumeration (no server needed) ────────────────────────────────────
|
||||||
|
static void test_device_enumeration() {
|
||||||
|
vc_config cfg{"test-devices", "0.1", VC_LOG_OFF};
|
||||||
|
vc_callbacks cb{};
|
||||||
|
vc_client* c = vc_client_create(&cfg, cb);
|
||||||
|
CHECK(c != nullptr);
|
||||||
|
|
||||||
|
for (vc_device_kind kind : {VC_DEVICE_INPUT, VC_DEVICE_OUTPUT}) {
|
||||||
|
vc_device_list dl{};
|
||||||
|
vc_result r = vc_list_devices(c, kind, &dl);
|
||||||
|
#ifdef VOICECAT_HAS_AUDIO
|
||||||
|
CHECK(r == VC_OK);
|
||||||
|
// Headless CI build agents may legitimately report zero devices — never assert
|
||||||
|
// count > 0, only that the call itself succeeded and the list is well-formed.
|
||||||
|
for (size_t i = 0; i < dl.count; ++i) {
|
||||||
|
CHECK(dl.items[i].id != nullptr);
|
||||||
|
CHECK(dl.items[i].name != nullptr);
|
||||||
|
}
|
||||||
|
#else
|
||||||
|
CHECK(r == VC_ERR_NOT_IMPLEMENTED);
|
||||||
|
#endif
|
||||||
|
vc_free_device_list(&dl);
|
||||||
|
vc_free_device_list(&dl); // idempotent — must not crash on a second call
|
||||||
|
}
|
||||||
|
|
||||||
|
vc_client_destroy(c);
|
||||||
|
std::printf("test_device_enumeration: ok\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── 4. Stereo playback mixer (white-box, no audio hardware needed) ──────────────
|
||||||
|
#if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS)
|
||||||
|
static void test_stereo_mix() {
|
||||||
|
voicecat::audio::AudioEngine engine;
|
||||||
|
voicecat::audio::AudioParams p;
|
||||||
|
p.sample_rate = 48000;
|
||||||
|
p.capture_channels = 1;
|
||||||
|
p.playback_channels = 2;
|
||||||
|
p.frame_ms = 20;
|
||||||
|
CHECK(engine.start(p)); // capture_cb intentionally omitted — not exercised here
|
||||||
|
|
||||||
|
voicecat::codec::OpusParams stereo_params;
|
||||||
|
stereo_params.stereo = true;
|
||||||
|
int frame_samples = voicecat::codec::opus_frame_samples(stereo_params);
|
||||||
|
|
||||||
|
voicecat::codec::OpusEncoder enc;
|
||||||
|
CHECK(enc.init(stereo_params));
|
||||||
|
|
||||||
|
// Loud left channel, silent right channel — a real downmix would average them into a
|
||||||
|
// single audible-but-quieter centered sample; true stereo should keep them distinct.
|
||||||
|
std::vector<int16_t> interleaved(static_cast<size_t>(frame_samples) * 2);
|
||||||
|
for (int i = 0; i < frame_samples; ++i) {
|
||||||
|
float t = static_cast<float>(i) / 48000.0f;
|
||||||
|
interleaved[i * 2] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * 440.0f * t) * 20000.0f);
|
||||||
|
interleaved[i * 2 + 1] = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
uint8_t opus_buf[1500];
|
||||||
|
int opus_len = enc.encode(interleaved.data(), frame_samples, opus_buf, sizeof(opus_buf));
|
||||||
|
CHECK(opus_len > 0);
|
||||||
|
|
||||||
|
engine.init_recv_stream(/*ssrc=*/1, stereo_params);
|
||||||
|
|
||||||
|
voicecat::audio::JitterBuffer::Frame f;
|
||||||
|
f.seq = 0;
|
||||||
|
f.timestamp = 0;
|
||||||
|
f.fec_present = false;
|
||||||
|
f.payload.assign(opus_buf, opus_buf + opus_len);
|
||||||
|
engine.push_recv_frame(1, std::move(f));
|
||||||
|
|
||||||
|
std::vector<int16_t> out(static_cast<size_t>(frame_samples) * 2, 0);
|
||||||
|
engine.mix_for_test(out.data(), static_cast<uint32_t>(frame_samples));
|
||||||
|
|
||||||
|
// If the engine downmixed (old M3 behavior), every L/R pair would be identical (the
|
||||||
|
// average of a loud sample and 0). True stereo should show a clear, consistent L != R
|
||||||
|
// difference across the frame.
|
||||||
|
int64_t total_diff = 0;
|
||||||
|
for (int i = 0; i < frame_samples; ++i)
|
||||||
|
total_diff += std::abs(static_cast<int>(out[i * 2]) - static_cast<int>(out[i * 2 + 1]));
|
||||||
|
CHECK(total_diff > static_cast<int64_t>(frame_samples) * 1000); // well above decode noise
|
||||||
|
|
||||||
|
engine.remove_stream(1);
|
||||||
|
engine.stop();
|
||||||
|
std::printf("test_stereo_mix: ok (total_diff=%lld)\n", static_cast<long long>(total_diff));
|
||||||
|
}
|
||||||
|
#endif // VOICECAT_HAS_AUDIO && VOICECAT_HAS_OPUS
|
||||||
|
|
||||||
|
// ── 2/3. VAD + PTT gate, through the real ABI against a real server ─────────────
|
||||||
|
static void test_vad_and_ptt_gate() {
|
||||||
|
auto tmp = std::filesystem::temp_directory_path() /
|
||||||
|
("vctest_vadptt_" + std::to_string(
|
||||||
|
std::chrono::steady_clock::now().time_since_epoch().count()));
|
||||||
|
std::filesystem::create_directories(tmp);
|
||||||
|
std::string data_dir = tmp.string();
|
||||||
|
|
||||||
|
std::atomic<uint16_t> bound_port{0};
|
||||||
|
std::mutex ready_mu;
|
||||||
|
std::condition_variable ready_cv;
|
||||||
|
bool ready{false};
|
||||||
|
|
||||||
|
voicecat::server::Config cfg;
|
||||||
|
cfg.data_dir = data_dir;
|
||||||
|
cfg.bind_port = 0;
|
||||||
|
cfg.media_port = 0;
|
||||||
|
cfg.server_name = "VoiceCat-VadPttTest";
|
||||||
|
cfg.allow_guests = true;
|
||||||
|
cfg.on_ready = [&](uint16_t p) {
|
||||||
|
bound_port.store(p);
|
||||||
|
{ std::lock_guard lk(ready_mu); ready = true; }
|
||||||
|
ready_cv.notify_all();
|
||||||
|
};
|
||||||
|
|
||||||
|
voicecat::server::Server server(cfg);
|
||||||
|
std::thread server_thread([&] { server.run(); });
|
||||||
|
|
||||||
|
{
|
||||||
|
std::unique_lock lk(ready_mu);
|
||||||
|
bool ok = ready_cv.wait_for(lk, std::chrono::seconds(10), [&] { return ready; });
|
||||||
|
if (!ok) {
|
||||||
|
std::printf("FAIL: server did not become ready within 10s\n");
|
||||||
|
++g_failures;
|
||||||
|
server.stop();
|
||||||
|
server_thread.join();
|
||||||
|
std::filesystem::remove_all(tmp);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
uint16_t port = bound_port.load();
|
||||||
|
std::printf("test_vad_and_ptt_gate: server ready on :%u\n", port);
|
||||||
|
|
||||||
|
EventStore evA;
|
||||||
|
evA.label = "A";
|
||||||
|
vc_callbacks cbA{on_event, nullptr, &evA};
|
||||||
|
vc_config cfgA{"test-A", "0.1", VC_LOG_OFF};
|
||||||
|
vc_client* clientA = vc_client_create(&cfgA, cbA);
|
||||||
|
CHECK(clientA != nullptr);
|
||||||
|
CHECK(vc_connect(clientA, "127.0.0.1", port) == VC_OK);
|
||||||
|
CHECK(vc_authenticate_guest(clientA, "VP-A") == VC_OK);
|
||||||
|
CHECK(wait_for(evA, [](EventStore& s) { return s.auth_ok; }, 8000));
|
||||||
|
CHECK(wait_for(evA, [](EventStore& s) { return s.channel_list_received; }, 3000));
|
||||||
|
|
||||||
|
EventStore evB;
|
||||||
|
evB.label = "B";
|
||||||
|
vc_callbacks cbB{on_event, nullptr, &evB};
|
||||||
|
vc_config cfgB{"test-B", "0.1", VC_LOG_OFF};
|
||||||
|
vc_client* clientB = vc_client_create(&cfgB, cbB);
|
||||||
|
CHECK(clientB != nullptr);
|
||||||
|
CHECK(vc_connect(clientB, "127.0.0.1", port) == VC_OK);
|
||||||
|
CHECK(vc_authenticate_guest(clientB, "VP-B") == VC_OK);
|
||||||
|
CHECK(wait_for(evB, [](EventStore& s) { return s.auth_ok; }, 8000));
|
||||||
|
CHECK(wait_for(evB, [](EventStore& s) { return s.channel_list_received; }, 3000));
|
||||||
|
|
||||||
|
uint32_t a_uid = 0;
|
||||||
|
{ std::lock_guard lk(evA.mu); a_uid = evA.self_user_id; }
|
||||||
|
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(500));
|
||||||
|
|
||||||
|
vc_stream_desc mic_desc{};
|
||||||
|
mic_desc.kind = VC_STREAM_MIC;
|
||||||
|
mic_desc.label = "mic";
|
||||||
|
uint32_t mic_sid = 0;
|
||||||
|
CHECK(vc_stream_start(clientA, &mic_desc, &mic_sid) == VC_OK);
|
||||||
|
CHECK(wait_for(evB, [](EventStore& s) { return s.saw_stream_started; }, 5000));
|
||||||
|
CHECK(wait_for(evA, [](EventStore& s) { return s.saw_stream_started; }, 5000));
|
||||||
|
|
||||||
|
// ── 2a. VAD mode (default), silent PCM: must NOT reach B as a talking edge ──
|
||||||
|
CHECK(vc_set_input_mode(clientA, VC_INPUT_VOICE_ACTIVATION) == VC_OK);
|
||||||
|
for (int i = 0; i < 15; ++i) {
|
||||||
|
auto silence = make_silence_frame();
|
||||||
|
CHECK(vc_test_inject_capture(clientA, mic_sid, silence.data(), silence.size()) == VC_OK);
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(20));
|
||||||
|
}
|
||||||
|
CHECK(!saw_talking_true(evB, a_uid, mic_sid, 0));
|
||||||
|
|
||||||
|
// ── 2b. VAD mode, loud PCM: must reach B as a talking edge ──────────────────
|
||||||
|
size_t mark = talk_event_count(evB);
|
||||||
|
for (int i = 0; i < 20; ++i) {
|
||||||
|
auto loud = make_sine_frame(i, 440.0f);
|
||||||
|
CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK);
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(20));
|
||||||
|
}
|
||||||
|
CHECK(wait_for(evB, [&](EventStore& s) {
|
||||||
|
for (size_t i = mark; i < s.talk_events.size(); ++i) {
|
||||||
|
auto& e = s.talk_events[i];
|
||||||
|
if (e.user_id == a_uid && e.stream_id == mic_sid && e.talking) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}, 3000));
|
||||||
|
|
||||||
|
// ── 3a. PTT mode, key up: loud PCM must NOT reach B as a new talking edge ───
|
||||||
|
CHECK(vc_set_input_mode(clientA, VC_INPUT_PUSH_TO_TALK) == VC_OK);
|
||||||
|
CHECK(vc_set_push_to_talk(clientA, 0) == VC_OK);
|
||||||
|
// Let any in-flight VAD-driven talking state lapse (hang-time ~300ms) before measuring.
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(500));
|
||||||
|
mark = talk_event_count(evB);
|
||||||
|
for (int i = 0; i < 20; ++i) {
|
||||||
|
auto loud = make_sine_frame(i, 440.0f);
|
||||||
|
CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK);
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(20));
|
||||||
|
}
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(200));
|
||||||
|
CHECK(!saw_talking_true(evB, a_uid, mic_sid, mark));
|
||||||
|
|
||||||
|
// ── 3b. PTT mode, key down: loud PCM must reach B as a talking edge ─────────
|
||||||
|
CHECK(vc_set_push_to_talk(clientA, 1) == VC_OK);
|
||||||
|
mark = talk_event_count(evB);
|
||||||
|
for (int i = 0; i < 20; ++i) {
|
||||||
|
auto loud = make_sine_frame(i, 440.0f);
|
||||||
|
CHECK(vc_test_inject_capture(clientA, mic_sid, loud.data(), loud.size()) == VC_OK);
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(20));
|
||||||
|
}
|
||||||
|
CHECK(wait_for(evB, [&](EventStore& s) {
|
||||||
|
for (size_t i = mark; i < s.talk_events.size(); ++i) {
|
||||||
|
auto& e = s.talk_events[i];
|
||||||
|
if (e.user_id == a_uid && e.stream_id == mic_sid && e.talking) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}, 3000));
|
||||||
|
|
||||||
|
{ std::lock_guard lk(evA.mu); CHECK(!evA.disconnected); }
|
||||||
|
{ std::lock_guard lk(evB.mu); CHECK(!evB.disconnected); }
|
||||||
|
|
||||||
|
vc_disconnect(clientA);
|
||||||
|
vc_disconnect(clientB);
|
||||||
|
vc_client_destroy(clientA);
|
||||||
|
vc_client_destroy(clientB);
|
||||||
|
|
||||||
|
server.stop();
|
||||||
|
server_thread.join();
|
||||||
|
std::filesystem::remove_all(tmp);
|
||||||
|
|
||||||
|
std::printf("test_vad_and_ptt_gate: done\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
int main() {
|
||||||
|
test_device_enumeration();
|
||||||
|
#if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS)
|
||||||
|
test_stereo_mix();
|
||||||
|
#endif
|
||||||
|
test_vad_and_ptt_gate();
|
||||||
|
|
||||||
|
if (g_failures == 0) {
|
||||||
|
std::printf("vad_ptt_devices: all checks passed\n");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
std::printf("vad_ptt_devices: %d failure(s)\n", g_failures);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
#else // !VOICECAT_HAS_NET
|
||||||
|
|
||||||
|
int main() {
|
||||||
|
std::printf("vad_ptt_devices: SKIP (VOICECAT_HAS_NET not defined)\n");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
#endif // VOICECAT_HAS_NET
|
||||||
@@ -10,6 +10,8 @@
|
|||||||
#include <csignal>
|
#include <csignal>
|
||||||
#include <cstdio>
|
#include <cstdio>
|
||||||
#include <cstring>
|
#include <cstring>
|
||||||
|
#include <functional>
|
||||||
|
#include <iostream>
|
||||||
#include <string>
|
#include <string>
|
||||||
#include <thread>
|
#include <thread>
|
||||||
|
|
||||||
@@ -58,6 +60,10 @@ void on_event(void* user, const vc_event* ev) {
|
|||||||
case VC_EVENT_TEXT_MESSAGE:
|
case VC_EVENT_TEXT_MESSAGE:
|
||||||
std::printf("[text] from=%u: %s\n", ev->user_id, ev->text ? ev->text : "");
|
std::printf("[text] from=%u: %s\n", ev->user_id, ev->text ? ev->text : "");
|
||||||
break;
|
break;
|
||||||
|
case VC_EVENT_TALK_STATE:
|
||||||
|
std::printf("[voice] talk state: user_id=%u stream_id=%u talking=%u\n", ev->user_id,
|
||||||
|
ev->stream_id, ev->u32a);
|
||||||
|
break;
|
||||||
case VC_EVENT_ERROR:
|
case VC_EVENT_ERROR:
|
||||||
std::fprintf(stderr, "[error] rc=%d: %s\n", ev->result, ev->text ? ev->text : "");
|
std::fprintf(stderr, "[error] rc=%d: %s\n", ev->result, ev->text ? ev->text : "");
|
||||||
break;
|
break;
|
||||||
@@ -82,14 +88,58 @@ bool wait_until(std::atomic<bool>& flag, int timeout_ms) {
|
|||||||
void print_usage() {
|
void print_usage() {
|
||||||
std::printf(
|
std::printf(
|
||||||
"usage: vccli [--host H] [--port P] [--nick NAME] [--channel ID]\n"
|
"usage: vccli [--host H] [--port P] [--nick NAME] [--channel ID]\n"
|
||||||
" [--voice] [--mute] [--text MSG]\n"
|
" [--voice] [--mute] [--text MSG] [--list-devices]\n"
|
||||||
|
" [--input-device ID] [--input-mode vad|ptt] [--share-screen-audio]\n"
|
||||||
" --host H server host (default 127.0.0.1)\n"
|
" --host H server host (default 127.0.0.1)\n"
|
||||||
" --port P server TCP port (default 8384)\n"
|
" --port P server TCP port (default 8384)\n"
|
||||||
" --nick NAME guest nickname (default vccli-test)\n"
|
" --nick NAME guest nickname (default vccli-test)\n"
|
||||||
" --channel ID channel to join after auth (default 1, Lobby)\n"
|
" --channel ID channel to join after auth (default 1, Lobby)\n"
|
||||||
" --voice start a MIC stream and stay connected until Ctrl+C\n"
|
" --voice start a MIC stream and stay connected until Ctrl+C\n"
|
||||||
" --mute start with the mic muted (only meaningful with --voice)\n"
|
" --mute start with the mic muted (only meaningful with --voice)\n"
|
||||||
" --text MSG send MSG to the channel, then exit\n");
|
" --text MSG send MSG to the channel, then exit\n"
|
||||||
|
" --list-devices print input/output devices (vc_list_devices) and exit\n"
|
||||||
|
" --input-device ID use device ID (from --list-devices) for the MIC stream\n"
|
||||||
|
" --input-mode vad|ptt send-side input gate mode (default vad)\n"
|
||||||
|
" --share-screen-audio also start a SCREEN_AUDIO stream (WASAPI loopback on Windows)\n"
|
||||||
|
"\n"
|
||||||
|
"While --voice is running, stdin accepts: \"ptt on\", \"ptt off\", \"mode vad\",\n"
|
||||||
|
"\"mode ptt\" (PTT key state can't be held interactively in a headless CLI, so it's\n"
|
||||||
|
"toggled via these commands instead).\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
void print_device_list(vc_client* c, vc_device_kind kind, const char* label) {
|
||||||
|
vc_device_list dl{};
|
||||||
|
vc_result r = vc_list_devices(c, kind, &dl);
|
||||||
|
std::printf("%s devices: vc_list_devices -> %d (%s), count=%zu\n", label, r,
|
||||||
|
vc_result_string(r), dl.count);
|
||||||
|
for (size_t i = 0; i < dl.count; ++i) {
|
||||||
|
std::printf(" [%s] %s%s\n", dl.items[i].id, dl.items[i].name,
|
||||||
|
dl.items[i].is_default ? " (default)" : "");
|
||||||
|
}
|
||||||
|
vc_free_device_list(&dl);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Background stdin reader for --voice mode: the most portable way to drive PTT/mode toggles
|
||||||
|
// interactively from a headless CLI (no SIGUSR1 equivalent on Windows).
|
||||||
|
void run_stdin_commands(vc_client* c, std::atomic<bool>& stop) {
|
||||||
|
std::string line;
|
||||||
|
while (!stop.load() && std::getline(std::cin, line)) {
|
||||||
|
if (line == "ptt on") {
|
||||||
|
vc_set_push_to_talk(c, 1);
|
||||||
|
std::printf("[ptt] on\n");
|
||||||
|
} else if (line == "ptt off") {
|
||||||
|
vc_set_push_to_talk(c, 0);
|
||||||
|
std::printf("[ptt] off\n");
|
||||||
|
} else if (line == "mode vad") {
|
||||||
|
vc_set_input_mode(c, VC_INPUT_VOICE_ACTIVATION);
|
||||||
|
std::printf("[mode] vad\n");
|
||||||
|
} else if (line == "mode ptt") {
|
||||||
|
vc_set_input_mode(c, VC_INPUT_PUSH_TO_TALK);
|
||||||
|
std::printf("[mode] ptt\n");
|
||||||
|
} else if (!line.empty()) {
|
||||||
|
std::fprintf(stderr, "unknown command: %s\n", line.c_str());
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
} // namespace
|
} // namespace
|
||||||
@@ -107,6 +157,11 @@ int main(int argc, char** argv) {
|
|||||||
bool start_muted = false;
|
bool start_muted = false;
|
||||||
std::string text_msg;
|
std::string text_msg;
|
||||||
bool have_text = false;
|
bool have_text = false;
|
||||||
|
bool list_devices = false;
|
||||||
|
std::string input_device;
|
||||||
|
bool have_input_device = false;
|
||||||
|
vc_input_mode input_mode = VC_INPUT_VOICE_ACTIVATION;
|
||||||
|
bool share_screen_audio = false;
|
||||||
|
|
||||||
for (int i = 1; i < argc; ++i) {
|
for (int i = 1; i < argc; ++i) {
|
||||||
std::string a = argv[i];
|
std::string a = argv[i];
|
||||||
@@ -118,6 +173,14 @@ int main(int argc, char** argv) {
|
|||||||
else if (a == "--voice") voice_mode = true;
|
else if (a == "--voice") voice_mode = true;
|
||||||
else if (a == "--mute") start_muted = true;
|
else if (a == "--mute") start_muted = true;
|
||||||
else if (a == "--text") { text_msg = next(); have_text = true; }
|
else if (a == "--text") { text_msg = next(); have_text = true; }
|
||||||
|
else if (a == "--list-devices") list_devices = true;
|
||||||
|
else if (a == "--input-device") { input_device = next(); have_input_device = true; }
|
||||||
|
else if (a == "--input-mode") {
|
||||||
|
std::string m = next();
|
||||||
|
if (m == "ptt") input_mode = VC_INPUT_PUSH_TO_TALK;
|
||||||
|
else if (m != "vad") { std::fprintf(stderr, "--input-mode must be vad|ptt\n"); return 1; }
|
||||||
|
}
|
||||||
|
else if (a == "--share-screen-audio") share_screen_audio = true;
|
||||||
else if (a == "--help" || a == "-h") { print_usage(); return 0; }
|
else if (a == "--help" || a == "-h") { print_usage(); return 0; }
|
||||||
else { std::fprintf(stderr, "unknown flag: %s\n", a.c_str()); print_usage(); return 1; }
|
else { std::fprintf(stderr, "unknown flag: %s\n", a.c_str()); print_usage(); return 1; }
|
||||||
}
|
}
|
||||||
@@ -143,6 +206,14 @@ int main(int argc, char** argv) {
|
|||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (list_devices) {
|
||||||
|
// Device enumeration works pre-connect (no server needed) — see docs/voice.md.
|
||||||
|
print_device_list(c, VC_DEVICE_INPUT, "input");
|
||||||
|
print_device_list(c, VC_DEVICE_OUTPUT, "output");
|
||||||
|
vc_client_destroy(c);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
vc_result r = vc_connect(c, host.c_str(), port);
|
vc_result r = vc_connect(c, host.c_str(), port);
|
||||||
std::printf("vc_connect(%s:%u) -> %d (%s)\n", host.c_str(), port, r, vc_result_string(r));
|
std::printf("vc_connect(%s:%u) -> %d (%s)\n", host.c_str(), port, r, vc_result_string(r));
|
||||||
if (r != VC_OK) {
|
if (r != VC_OK) {
|
||||||
@@ -178,9 +249,13 @@ int main(int argc, char** argv) {
|
|||||||
|
|
||||||
if (start_muted) vc_set_self_mute(c, 1, 0);
|
if (start_muted) vc_set_self_mute(c, 1, 0);
|
||||||
|
|
||||||
|
r = vc_set_input_mode(c, input_mode);
|
||||||
|
std::printf("vc_set_input_mode(%s) -> %d (%s)\n",
|
||||||
|
input_mode == VC_INPUT_PUSH_TO_TALK ? "ptt" : "vad", r, vc_result_string(r));
|
||||||
|
|
||||||
vc_stream_desc desc{};
|
vc_stream_desc desc{};
|
||||||
desc.kind = VC_STREAM_MIC;
|
desc.kind = VC_STREAM_MIC;
|
||||||
desc.device_id = nullptr;
|
desc.device_id = have_input_device ? input_device.c_str() : nullptr;
|
||||||
desc.label = "Microphone";
|
desc.label = "Microphone";
|
||||||
|
|
||||||
uint32_t stream_id = 0;
|
uint32_t stream_id = 0;
|
||||||
@@ -188,12 +263,43 @@ int main(int argc, char** argv) {
|
|||||||
std::printf("vc_stream_start -> %d (%s), stream_id=%u\n", r, vc_result_string(r),
|
std::printf("vc_stream_start -> %d (%s), stream_id=%u\n", r, vc_result_string(r),
|
||||||
stream_id);
|
stream_id);
|
||||||
|
|
||||||
std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n");
|
if (have_input_device && r == VC_OK) {
|
||||||
|
r = vc_set_input_device(c, stream_id, input_device.c_str());
|
||||||
|
std::printf("vc_set_input_device -> %d (%s)\n", r, vc_result_string(r));
|
||||||
|
}
|
||||||
|
|
||||||
|
uint32_t screen_stream_id = 0;
|
||||||
|
if (share_screen_audio) {
|
||||||
|
vc_stream_desc sdesc{};
|
||||||
|
sdesc.kind = VC_STREAM_SCREEN_AUDIO;
|
||||||
|
sdesc.label = "Desktop audio";
|
||||||
|
r = vc_stream_start(c, &sdesc, &screen_stream_id);
|
||||||
|
std::printf("vc_stream_start(SCREEN_AUDIO) -> %d (%s), stream_id=%u\n", r,
|
||||||
|
vc_result_string(r), screen_stream_id);
|
||||||
|
}
|
||||||
|
|
||||||
|
std::thread stdin_thread(run_stdin_commands, c, std::ref(g_stop));
|
||||||
|
|
||||||
|
std::printf("voice mode: streaming mic, listening for remote streams. Ctrl+C to stop.\n"
|
||||||
|
"(type \"ptt on\"/\"ptt off\"/\"mode vad\"/\"mode ptt\" to drive the input gate)\n");
|
||||||
while (!g_stop.load()) {
|
while (!g_stop.load()) {
|
||||||
std::this_thread::sleep_for(std::chrono::milliseconds(200));
|
std::this_thread::sleep_for(std::chrono::milliseconds(200));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (share_screen_audio) vc_stream_stop(c, screen_stream_id);
|
||||||
vc_stream_stop(c, stream_id);
|
vc_stream_stop(c, stream_id);
|
||||||
|
|
||||||
|
// Unblock the stdin reader and let it exit before the client is torn down.
|
||||||
|
if (stdin_thread.joinable()) {
|
||||||
|
#if defined(_WIN32)
|
||||||
|
// std::getline on a console stdin blocks indefinitely; on Windows there's no
|
||||||
|
// portable way to interrupt it from another thread, so detach rather than join —
|
||||||
|
// process exit reclaims the thread.
|
||||||
|
stdin_thread.detach();
|
||||||
|
#else
|
||||||
|
stdin_thread.join();
|
||||||
|
#endif
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
vc_disconnect(c);
|
vc_disconnect(c);
|
||||||
|
|||||||
Reference in New Issue
Block a user