scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
/*
|
|
|
|
|
|
* codec/opus_codec.h — Opus encode/decode (libopus 1.6).
|
|
|
|
|
|
*
|
|
|
|
|
|
* Design: docs/voice.md §3–4. Per-channel AudioConfig (mono/stereo, bitrate, frame size,
|
|
|
|
|
|
* FEC, DTX, complexity). The server relays Opus payloads unmodified (no transcode).
|
|
|
|
|
|
*/
|
|
|
|
|
|
#ifndef VOICECAT_CODEC_OPUS_CODEC_H
|
|
|
|
|
|
#define VOICECAT_CODEC_OPUS_CODEC_H
|
|
|
|
|
|
|
|
|
|
|
|
#include <cstdint>
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
#include <vector>
|
|
|
|
|
|
|
|
|
|
|
|
#ifdef VOICECAT_HAS_OPUS
|
|
|
|
|
|
#include <opus/opus.h>
|
|
|
|
|
|
#endif
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
|
|
|
|
|
|
namespace voicecat::codec {
|
|
|
|
|
|
|
feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).
Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
the same user silently overwrote the first in SessionRegistry::set_user_stream.
Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
and receive (sync_remote_streams) paths via a shared
opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
(total samples instead of samples-per-channel), which would have overflowed
the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
and from disconnect() on a different thread -- both could see
udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
same std::thread (intermittent std::system_error under ctest). Fixed with a
teardown_mu_ guard instead of carrying the flake forward.
New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
handle_stream_announce enforces the channel's config, clamping (not
overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
request_id-correlated announce/result handling (request_id already
round-tripped on the wire; just wasn't read before). on_capture_frame is
kind-aware and upmixes mono capture to stereo when a stream's config calls
for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
through set_remote_stream. New run_talk_timer() thread emits
VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
(inject_capture), stereo-to-mono downmix at the decode/mix boundary,
RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
and last_voice_ms/talking; new set_stream_noise_reduction() and
poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
vc_get_stream_audio_config (effective Opus config for any stream you own or
a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
(mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
concurrent local streams, independent gain/mute/NS control, per-channel
config divergence via vc_get_stream_audio_config, talk indicators.
Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).
ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
|
|
|
|
// Mirrors voicecat::v1::OpusApplication (core/proto/voicecat.proto) without depending on
|
|
|
|
|
|
// generated protobuf headers from this low-level codec module.
|
|
|
|
|
|
enum class OpusApplication {
|
|
|
|
|
|
Voip = 0, // speech, optimized for low-rate intelligibility
|
|
|
|
|
|
Audio = 1, // music/screen-audio, optimized for fidelity
|
|
|
|
|
|
LowDelay = 2, // monitoring, minimal algorithmic delay
|
|
|
|
|
|
};
|
|
|
|
|
|
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
struct OpusParams {
|
feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).
Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
the same user silently overwrote the first in SessionRegistry::set_user_stream.
Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
and receive (sync_remote_streams) paths via a shared
opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
(total samples instead of samples-per-channel), which would have overflowed
the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
and from disconnect() on a different thread -- both could see
udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
same std::thread (intermittent std::system_error under ctest). Fixed with a
teardown_mu_ guard instead of carrying the flake forward.
New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
handle_stream_announce enforces the channel's config, clamping (not
overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
request_id-correlated announce/result handling (request_id already
round-tripped on the wire; just wasn't read before). on_capture_frame is
kind-aware and upmixes mono capture to stereo when a stream's config calls
for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
through set_remote_stream. New run_talk_timer() thread emits
VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
(inject_capture), stereo-to-mono downmix at the decode/mix boundary,
RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
and last_voice_ms/talking; new set_stream_noise_reduction() and
poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
vc_get_stream_audio_config (effective Opus config for any stream you own or
a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
(mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
concurrent local streams, independent gain/mute/NS control, per-channel
config divergence via vc_get_stream_audio_config, talk indicators.
Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).
ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
|
|
|
|
uint32_t sample_rate = 48000;
|
|
|
|
|
|
uint32_t bitrate_bps = 24000;
|
|
|
|
|
|
uint32_t frame_ms = 20;
|
|
|
|
|
|
bool stereo = false;
|
|
|
|
|
|
bool fec = true;
|
|
|
|
|
|
bool dtx = false;
|
|
|
|
|
|
uint32_t complexity = 10;
|
|
|
|
|
|
uint32_t expected_packet_loss = 0; // % 0..100
|
|
|
|
|
|
OpusApplication application = OpusApplication::Voip;
|
2026-06-20 13:40:47 +02:00
|
|
|
|
bool dred = false;
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
};
|
|
|
|
|
|
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
// Returns frame_samples for a given sample_rate + frame_ms.
|
|
|
|
|
|
inline int opus_frame_samples(const OpusParams& p) {
|
|
|
|
|
|
return static_cast<int>(p.sample_rate / 1000 * p.frame_ms);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
class OpusEncoder {
|
|
|
|
|
|
public:
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
OpusEncoder() = default;
|
|
|
|
|
|
~OpusEncoder() { destroy(); }
|
|
|
|
|
|
|
|
|
|
|
|
OpusEncoder(const OpusEncoder&) = delete;
|
|
|
|
|
|
OpusEncoder& operator=(const OpusEncoder&) = delete;
|
|
|
|
|
|
|
|
|
|
|
|
// Initialise with the given params. Must be called before encode().
|
|
|
|
|
|
// Returns true on success; check error_string() on failure.
|
|
|
|
|
|
bool init(const OpusParams& p);
|
|
|
|
|
|
|
|
|
|
|
|
// Encode one frame of PCM (frame_ms ms @ sample_rate Hz, mono or stereo).
|
|
|
|
|
|
// pcm: interleaved int16 samples (frame_samples * channels samples).
|
|
|
|
|
|
// out_buf: caller-allocated output buffer (recommend >= 4000 bytes).
|
|
|
|
|
|
// Returns number of bytes written to out_buf, or -1 on error.
|
|
|
|
|
|
int encode(const int16_t* pcm, int frame_samples, uint8_t* out_buf, int out_cap);
|
|
|
|
|
|
|
|
|
|
|
|
void destroy();
|
|
|
|
|
|
|
|
|
|
|
|
bool valid() const { return enc_ != nullptr; }
|
|
|
|
|
|
int frame_samples()const { return frame_samples_; }
|
|
|
|
|
|
int channels() const { return channels_; }
|
|
|
|
|
|
const char* error_string() const { return err_; }
|
|
|
|
|
|
|
|
|
|
|
|
private:
|
|
|
|
|
|
#ifdef VOICECAT_HAS_OPUS
|
|
|
|
|
|
::OpusEncoder* enc_ = nullptr;
|
|
|
|
|
|
#else
|
|
|
|
|
|
void* enc_ = nullptr;
|
|
|
|
|
|
#endif
|
|
|
|
|
|
int frame_samples_ = 0;
|
|
|
|
|
|
int channels_ = 1;
|
|
|
|
|
|
const char* err_ = nullptr;
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
class OpusDecoder {
|
|
|
|
|
|
public:
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
OpusDecoder() = default;
|
|
|
|
|
|
~OpusDecoder() { destroy(); }
|
|
|
|
|
|
|
|
|
|
|
|
OpusDecoder(const OpusDecoder&) = delete;
|
|
|
|
|
|
OpusDecoder& operator=(const OpusDecoder&) = delete;
|
|
|
|
|
|
|
|
|
|
|
|
// Initialise. Must be called before decode().
|
|
|
|
|
|
bool init(const OpusParams& p);
|
|
|
|
|
|
|
|
|
|
|
|
// Decode one Opus packet into out_pcm (frame_samples * channels int16 samples).
|
|
|
|
|
|
// opus_data=nullptr, len=0 → PLC (free, always enabled by libopus).
|
|
|
|
|
|
// fec=true, next valid packet in opus_data → FEC recovery from previous loss.
|
|
|
|
|
|
// Returns number of samples decoded (= frame_samples), or -1 on error.
|
|
|
|
|
|
int decode(const uint8_t* opus_data, int len, int16_t* out_pcm, int max_samples,
|
|
|
|
|
|
bool fec = false);
|
|
|
|
|
|
|
2026-06-20 13:40:47 +02:00
|
|
|
|
// Decode a lost frame using pre-parsed DRED state from the next received packet.
|
|
|
|
|
|
// dred_offset=0 means the frame immediately before the next packet.
|
|
|
|
|
|
// Returns frame_samples on success, -1 if DRED unavailable or decode failed.
|
|
|
|
|
|
#ifdef VOICECAT_HAS_OPUS
|
|
|
|
|
|
int decode_dred(::OpusDRED* dred, int32_t dred_offset, int16_t* out_pcm, int max_samples);
|
|
|
|
|
|
::OpusDecoder* raw() const { return dec_; }
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
void destroy();
|
|
|
|
|
|
|
|
|
|
|
|
bool valid() const { return dec_ != nullptr; }
|
|
|
|
|
|
int frame_samples()const { return frame_samples_; }
|
|
|
|
|
|
int channels() const { return channels_; }
|
|
|
|
|
|
const char* error_string() const { return err_; }
|
|
|
|
|
|
|
|
|
|
|
|
private:
|
|
|
|
|
|
#ifdef VOICECAT_HAS_OPUS
|
|
|
|
|
|
::OpusDecoder* dec_ = nullptr;
|
|
|
|
|
|
#else
|
|
|
|
|
|
void* dec_ = nullptr;
|
|
|
|
|
|
#endif
|
|
|
|
|
|
int frame_samples_ = 0;
|
|
|
|
|
|
int channels_ = 1;
|
|
|
|
|
|
const char* err_ = nullptr;
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
} // namespace voicecat::codec
|
|
|
|
|
|
|
|
|
|
|
|
#endif // VOICECAT_CODEC_OPUS_CODEC_H
|