feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
/*
|
|
|
|
|
* audio/apm_processor.h — Send-side audio processing module (AEC/NS/AGC/VAD).
|
|
|
|
|
*
|
|
|
|
|
* Design: docs/voice.md §11. Uses webrtc-audio-processing when VOICECAT_HAS_APM is defined;
|
|
|
|
|
* falls back to a no-op passthrough (VAD always open, PCM unmodified) otherwise.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef VOICECAT_AUDIO_APM_PROCESSOR_H
|
|
|
|
|
#define VOICECAT_AUDIO_APM_PROCESSOR_H
|
|
|
|
|
|
|
|
|
|
#include <cstdint>
|
|
|
|
|
#include <memory>
|
|
|
|
|
|
|
|
|
|
namespace voicecat::audio {
|
|
|
|
|
|
|
|
|
|
class ApmProcessor {
|
|
|
|
|
public:
|
|
|
|
|
virtual ~ApmProcessor() = default;
|
|
|
|
|
|
|
|
|
|
// Feed the most recent playback reference (for AEC). Call before process_capture().
|
|
|
|
|
virtual void process_render(const int16_t* pcm, int samples, int sample_rate) = 0;
|
|
|
|
|
|
|
|
|
|
// Process one capture frame in-place (AEC, NS, AGC).
|
|
|
|
|
// Returns true if VAD detects speech (or always true in passthrough mode).
|
|
|
|
|
// Returns false → caller should skip encode/send (silence gate).
|
|
|
|
|
virtual bool process_capture(int16_t* pcm, int samples, int sample_rate) = 0;
|
|
|
|
|
|
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
|
|
|
// Factory: returns a real APM if VOICECAT_HAS_APM is defined, else a passthrough. Used for
|
|
|
|
|
// recv-side per-stream noise reduction (docs/voice.md §10) — gating doesn't apply there, so
|
|
|
|
|
// this stays a passthrough until a real APM/NS backend exists (still inert; see
|
|
|
|
|
// PROGRESS.md). Do not use this for the send-side VAD gate — see create_vad() below.
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
static std::unique_ptr<ApmProcessor> create();
|
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
|
|
|
|
|
|
|
|
// Factory for the send-side input gate (docs/voice.md §11): a lightweight, dependency-free
|
|
|
|
|
// energy/RMS VAD with configurable threshold + hang-time. webrtc-audio-processing (the
|
|
|
|
|
// originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson build,
|
|
|
|
|
// unfinished MinGW support, hard abseil-cpp dependency — see PROGRESS.md), so this is the
|
|
|
|
|
// real v1 implementation behind the same ApmProcessor interface, not a passthrough. No AEC
|
|
|
|
|
// — process_render() is a no-op here; that's a real limitation versus the originally-planned
|
|
|
|
|
// APM, not just a deferred VAD.
|
|
|
|
|
// rms_threshold: normalized 0.0-1.0 RMS-of-int16-range; default ~0.025.
|
|
|
|
|
// hang_time_ms: how long the gate stays open after the last loud frame; default 300 ms
|
|
|
|
|
// (matches AudioEngine's kTalkHangoverMs so "talking" and "gate open" agree).
|
|
|
|
|
static std::unique_ptr<ApmProcessor> create_vad(float rms_threshold = 0.025f,
|
|
|
|
|
int64_t hang_time_ms = 300);
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
} // namespace voicecat::audio
|
|
|
|
|
|
|
|
|
|
#endif // VOICECAT_AUDIO_APM_PROCESSOR_H
|