feat(M3): multi-stream & per-channel tuning

Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).

Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
  the same user silently overwrote the first in SessionRegistry::set_user_stream.
  Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
  validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
  the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
  applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
  and receive (sync_remote_streams) paths via a shared
  opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
  and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
  (total samples instead of samples-per-channel), which would have overflowed
  the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
  and from disconnect() on a different thread -- both could see
  udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
  same std::thread (intermittent std::system_error under ctest). Fixed with a
  teardown_mu_ guard instead of carrying the flake forward.

New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
  FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
  handle_stream_announce enforces the channel's config, clamping (not
  overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
  std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
  request_id-correlated announce/result handling (request_id already
  round-tripped on the wire; just wasn't read before). on_capture_frame is
  kind-aware and upmixes mono capture to stereo when a stream's config calls
  for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
  through set_remote_stream. New run_talk_timer() thread emits
  VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
  (inject_capture), stereo-to-mono downmix at the decode/mix boundary,
  RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
  and last_voice_ms/talking; new set_stream_noise_reduction() and
  poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
  not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
  vc_get_stream_audio_config (effective Opus config for any stream you own or
  a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
  clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
  (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
  concurrent local streams, independent gain/mute/NS control, per-channel
  config divergence via vc_get_stream_audio_config, talk indicators.

Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).

ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-16 14:12:37 +02:00
parent c693cab35c
commit 867557eda1
17 changed files with 1102 additions and 133 deletions

View File

@@ -158,6 +158,22 @@ typedef struct vc_stream_desc {
const char* label; /* human label, e.g. "Microphone" */
} vc_stream_desc;
/* The effective Opus configuration in use for a stream — for a stream you own, this is
* StreamAnnounceResult.effective_audio (channel-enforced, docs/voice.md §3); for a remote
* stream, it's the peer's broadcast StreamInfo.audio. See vc_get_stream_audio_config. */
typedef struct vc_audio_config {
uint32_t codec; /* 0 = OPUS */
uint32_t mode; /* 0 = mono, 1 = stereo */
uint32_t sample_rate;
uint32_t bitrate_bps;
uint32_t frame_ms;
uint32_t application; /* 0 = VOIP, 1 = AUDIO, 2 = LOWDELAY */
int fec; /* bool */
uint32_t expected_packet_loss; /* % 0..100 */
int dtx; /* bool */
uint32_t complexity; /* 0..10 */
} vc_audio_config;
typedef struct vc_device {
const char* id;
const char* name;
@@ -208,6 +224,18 @@ VC_API vc_result vc_set_self_mute(vc_client* c, int mic_muted, int deafened);
VC_API vc_result vc_set_remote_stream(vc_client* c, uint32_t user_id, uint32_t stream_id,
float gain, int muted, int noise_reduction);
/* Effective Opus config in use for (user_id, stream_id) — your own stream or a peer's.
* VC_ERR_INVALID_ARG if the user/stream isn't known. */
VC_API vc_result vc_get_stream_audio_config(vc_client* c, uint32_t user_id, uint32_t stream_id,
vc_audio_config* out);
/* TEST-ONLY — not for production use. Bypasses the real capture device, injecting raw PCM
* directly into the named local stream's encode pipeline (see AudioEngine::inject_capture).
* Exists so automated tests can drive the real vc_client/ABI path end-to-end without a
* microphone. `stream_id` is the id returned by vc_stream_start. */
VC_API vc_result vc_test_inject_capture(vc_client* c, uint32_t stream_id, const int16_t* pcm,
size_t samples);
/* ── Text ─────────────────────────────────────────────────────────────────── */
VC_API vc_result vc_send_text(vc_client* c, vc_text_scope scope, uint32_t target_id,
const char* utf8);