Files
voice-cat/core/src/codec/opus_codec.h
Talon 867557eda1 feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).

Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
  the same user silently overwrote the first in SessionRegistry::set_user_stream.
  Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
  validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
  the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
  applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
  and receive (sync_remote_streams) paths via a shared
  opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
  and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
  (total samples instead of samples-per-channel), which would have overflowed
  the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
  and from disconnect() on a different thread -- both could see
  udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
  same std::thread (intermittent std::system_error under ctest). Fixed with a
  teardown_mu_ guard instead of carrying the flake forward.

New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
  FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
  handle_stream_announce enforces the channel's config, clamping (not
  overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
  std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
  request_id-correlated announce/result handling (request_id already
  round-tripped on the wire; just wasn't read before). on_capture_frame is
  kind-aware and upmixes mono capture to stereo when a stream's config calls
  for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
  through set_remote_stream. New run_talk_timer() thread emits
  VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
  (inject_capture), stereo-to-mono downmix at the decode/mix boundary,
  RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
  and last_voice_ms/talking; new set_stream_noise_reduction() and
  poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
  not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
  vc_get_stream_audio_config (effective Opus config for any stream you own or
  a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
  clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
  (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
  concurrent local streams, independent gain/mute/NS control, per-channel
  config divergence via vc_get_stream_audio_config, talk indicators.

Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).

ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00

119 lines
4.0 KiB
C++
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

/*
* codec/opus_codec.h — Opus encode/decode (libopus 1.6).
*
* Design: docs/voice.md §34. Per-channel AudioConfig (mono/stereo, bitrate, frame size,
* FEC, DTX, complexity). The server relays Opus payloads unmodified (no transcode).
*/
#ifndef VOICECAT_CODEC_OPUS_CODEC_H
#define VOICECAT_CODEC_OPUS_CODEC_H
#include <cstdint>
#include <vector>
#ifdef VOICECAT_HAS_OPUS
#include <opus/opus.h>
#endif
namespace voicecat::codec {
// Mirrors voicecat::v1::OpusApplication (core/proto/voicecat.proto) without depending on
// generated protobuf headers from this low-level codec module.
enum class OpusApplication {
Voip = 0, // speech, optimized for low-rate intelligibility
Audio = 1, // music/screen-audio, optimized for fidelity
LowDelay = 2, // monitoring, minimal algorithmic delay
};
struct OpusParams {
uint32_t sample_rate = 48000;
uint32_t bitrate_bps = 24000;
uint32_t frame_ms = 20;
bool stereo = false;
bool fec = true;
bool dtx = false;
uint32_t complexity = 10;
uint32_t expected_packet_loss = 0; // % 0..100
OpusApplication application = OpusApplication::Voip;
};
// Returns frame_samples for a given sample_rate + frame_ms.
inline int opus_frame_samples(const OpusParams& p) {
return static_cast<int>(p.sample_rate / 1000 * p.frame_ms);
}
class OpusEncoder {
public:
OpusEncoder() = default;
~OpusEncoder() { destroy(); }
OpusEncoder(const OpusEncoder&) = delete;
OpusEncoder& operator=(const OpusEncoder&) = delete;
// Initialise with the given params. Must be called before encode().
// Returns true on success; check error_string() on failure.
bool init(const OpusParams& p);
// Encode one frame of PCM (frame_ms ms @ sample_rate Hz, mono or stereo).
// pcm: interleaved int16 samples (frame_samples * channels samples).
// out_buf: caller-allocated output buffer (recommend >= 4000 bytes).
// Returns number of bytes written to out_buf, or -1 on error.
int encode(const int16_t* pcm, int frame_samples, uint8_t* out_buf, int out_cap);
void destroy();
bool valid() const { return enc_ != nullptr; }
int frame_samples()const { return frame_samples_; }
int channels() const { return channels_; }
const char* error_string() const { return err_; }
private:
#ifdef VOICECAT_HAS_OPUS
::OpusEncoder* enc_ = nullptr;
#else
void* enc_ = nullptr;
#endif
int frame_samples_ = 0;
int channels_ = 1;
const char* err_ = nullptr;
};
class OpusDecoder {
public:
OpusDecoder() = default;
~OpusDecoder() { destroy(); }
OpusDecoder(const OpusDecoder&) = delete;
OpusDecoder& operator=(const OpusDecoder&) = delete;
// Initialise. Must be called before decode().
bool init(const OpusParams& p);
// Decode one Opus packet into out_pcm (frame_samples * channels int16 samples).
// opus_data=nullptr, len=0 → PLC (free, always enabled by libopus).
// fec=true, next valid packet in opus_data → FEC recovery from previous loss.
// Returns number of samples decoded (= frame_samples), or -1 on error.
int decode(const uint8_t* opus_data, int len, int16_t* out_pcm, int max_samples,
bool fec = false);
void destroy();
bool valid() const { return dec_ != nullptr; }
int frame_samples()const { return frame_samples_; }
int channels() const { return channels_; }
const char* error_string() const { return err_; }
private:
#ifdef VOICECAT_HAS_OPUS
::OpusDecoder* dec_ = nullptr;
#else
void* dec_ = nullptr;
#endif
int frame_samples_ = 0;
int channels_ = 1;
const char* err_ = nullptr;
};
} // namespace voicecat::codec
#endif // VOICECAT_CODEC_OPUS_CODEC_H