feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC + SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise- reduction, talk indicators, and enforced per-channel Opus configurability (mono/stereo, bitrate, frame size, FEC/DTX, application). Bugs fixed along the way (found while implementing, not pre-existing scope): - Server hard-coded stream_id=1 for every announce, so a second stream from the same user silently overwrote the first in SessionRegistry::set_user_stream. Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop validates against announced_stream_ids_ before clearing. - Client dropped mode/dtx/complexity/application from effective_audio even for the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever applied to OpusParams. Fixed on both the send (handle_stream_announce_result) and receive (sync_remote_streams) paths via a shared opus_params_from_audio_config() helper. - OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application and wired it through. - on_playback's per-stream decode passed the wrong frame_size to opus_decode (total samples instead of samples-per-channel), which would have overflowed the decode buffer for any stereo stream. - teardown_voice() raced when called concurrently from run_io()'s own cleanup and from disconnect() on a different thread -- both could see udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the same std::thread (intermittent std::system_error under ctest). Fixed with a teardown_mu_ guard instead of carrying the flake forward. New: - Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/ FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX); handle_stream_announce enforces the channel's config, clamping (not overriding) bitrate_bps to its ceiling. - core/src/core/client.h/.cpp: local-stream state is now a std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with request_id-correlated announce/result handling (request_id already round-tripped on the wire; just wasn't read before). on_capture_frame is kind-aware and upmixes mono capture to stereo when a stream's config calls for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired through set_remote_stream. New run_talk_timer() thread emits VC_EVENT_TALK_STATE from both remote and local edge detection. - core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps (inject_capture), stereo-to-mono downmix at the decode/mix boundary, RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled and last_voice_ms/talking; new set_stream_noise_reduction() and poll_talk_transitions(). - core/src/session/session.h/.cpp: Stream now carries the full AudioConfig, not just sample_rate/frame_ms. - New additive C ABI (core/include/voicecat.h): vc_audio_config + vc_get_stream_audio_config (effective Opus config for any stream you own or a peer's); vc_test_inject_capture (test-only synthetic PCM injection, clearly marked, mirrors AudioEngine::inject_capture). - tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two concurrent local streams, independent gain/mute/NS control, per-channel config divergence via vc_get_stream_audio_config, talk indicators. Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback capture for SCREEN_AUDIO (synthetic injection only); true stereo playback output (AudioEngine's mixer/output device stays mono -- Opus itself is fully stereo-correct on the wire). ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive full-suite runs plus 8 standalone runs of the new test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
137
PROGRESS.md
137
PROGRESS.md
@@ -10,25 +10,20 @@ up instantly. Newest status at the top.
|
||||
|
||||
## ▶ Where we left off / next action
|
||||
|
||||
- **Done:** **M2 — voice, single stream** ✓ complete (2026-06-16), now genuinely satisfied
|
||||
through the real client library, not just `test_m2_voice`'s raw-socket harness.
|
||||
`vc_stream_start/stop`, UDP binding, capture→encode→seal→send, and recv→open→decode→
|
||||
playback were all `VC_ERR_NOT_IMPLEMENTED` stubs in `core/src/core/client.cpp` even after
|
||||
`test_m2_voice` went green — meaning `vccli`/any GUI client still couldn't actually talk.
|
||||
Implemented for real this session, plus server-side `StreamInfo` broadcast
|
||||
(`SessionRegistry::set_user_stream/clear_user_stream` → `UserEvent::UPDATED`) so a second
|
||||
client's `sync_remote_streams()` learns about a peer's stream without polling.
|
||||
`ctest --test-dir build/m1-dev` — **10/10 tests** green, including the new
|
||||
`test_voice_client_abi` (two real `vc_client` instances, not raw sockets, drive the full
|
||||
UDP-binding → stream-announce → cross-client `STREAM_STARTED`/`STOPPED` event path).
|
||||
`vccli --voice` (new flag, alongside `--host/--port/--nick/--channel/--mute/--text`) was
|
||||
manually verified live: two instances see each other's mic-stream start/stop in real time.
|
||||
Note: `test_m1_integration` and `test_m2_voice` have a pre-existing intermittent flake on
|
||||
Windows in their cleanup paths (thread-join / SQLite-file-handle release race, unrelated to
|
||||
this session's changes) — rerun in isolation if one fails standalone in the full suite.
|
||||
- **Next:** **M3 — multi-stream & per-channel tuning** (screen audio, listener-side per-user
|
||||
NR, jitter buffer stats API — note `voicecat.h` currently exposes no stats getter beyond
|
||||
`on_level`'s RMS meter, so a stats API needs new C ABI surface). See `docs/roadmap.md §M3`.
|
||||
- **Done:** **M3 — multi-stream & per-channel tuning** ✓ complete (2026-06-16). See the M3
|
||||
section below for the full file-by-file change list. `ctest --test-dir build/m1-dev` —
|
||||
**11/11 tests** green (3 consecutive full-suite runs), including the new
|
||||
`test_m3_multistream` (real `vc_client`s, not raw sockets — same lesson as M2: ABI-level
|
||||
coverage is what proves the client library, not just the wire protocol).
|
||||
**Two items intentionally still open** (carried forward, not silently dropped):
|
||||
- `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices`
|
||||
(device enumeration + VAD/PTT input gate) remain `VC_ERR_NOT_IMPLEMENTED` — explicitly
|
||||
scoped out of this M3 pass; revisit in a future milestone.
|
||||
- Stereo Opus is now wire-correct end-to-end (a channel configured `MODE_STEREO` really
|
||||
encodes/decodes 2-channel Opus packets), but `AudioEngine`'s playback mixer/output device
|
||||
stays mono internally — stereo streams are downmixed (avg L/R) immediately after decode,
|
||||
before mixing. True stereo *playback output* is a follow-up, not part of M3.
|
||||
- **Next:** **M4 — native clients** (Windows C#, macOS/iOS Swift). See `docs/roadmap.md §M4`.
|
||||
|
||||
---
|
||||
|
||||
@@ -37,9 +32,8 @@ up instantly. Newest status at the top.
|
||||
- [x] **M0 — Scaffolding** ✓ complete
|
||||
- [x] **M1 — Control plane** ✓ complete (2026-06-15)
|
||||
- [x] **M2 — Voice, single stream** ✓ complete (2026-06-16)
|
||||
- [ ] **M3 — Multi-stream & per-channel tuning** ← next
|
||||
- [ ] **M3 — Multi-stream & per-channel tuning** (screen audio, listener-side per-user NR)
|
||||
- [ ] **M4 — Native clients** (Windows C#, macOS/iOS Swift)
|
||||
- [x] **M3 — Multi-stream & per-channel tuning** ✓ complete (2026-06-16)
|
||||
- [ ] **M4 — Native clients** (Windows C#, macOS/iOS Swift) ← next
|
||||
- [ ] **M5 — Moderation, polish, beyond** (perms, bans, DRED; then file transfer, E2EE, …)
|
||||
|
||||
---
|
||||
@@ -140,6 +134,105 @@ wasn't met. Closed the gap:
|
||||
|
||||
---
|
||||
|
||||
## M3 — Multi-stream & per-channel tuning ✓ (completed 2026-06-16)
|
||||
|
||||
**Exit criterion:** ✓ `test_m3_multistream` — a real `vc_client` (A) runs two concurrent local
|
||||
streams (MIC + SCREEN_AUDIO) with distinct stream ids; a second client (B) sees both as
|
||||
separate `STREAM_STARTED` events and a `VC_EVENT_TALK_STATE` talking edge for A's MIC stream;
|
||||
B independently sets gain/mute/noise-reduction on each of A's streams without one call
|
||||
affecting the other; A then joins "Music Room" (channel 2: stereo/128kbps/`OPUS_AUDIO`/no
|
||||
DTX) and announces a fresh MIC stream there, while B stays in "Lobby" (channel 1: mono/24kbps/
|
||||
`OPUS_VOIP`/DTX on) — `vc_get_stream_audio_config` shows their effective Opus config differs
|
||||
exactly as the server enforces per channel. Passes in ~2.4s; verified across 8 consecutive
|
||||
standalone runs + 3 consecutive full-suite runs with no flakes.
|
||||
|
||||
Exploration before implementing turned up several bugs/gaps where the wire format already
|
||||
supported this milestone but the client/server logic didn't — these were fixed as part of M3,
|
||||
not treated as pre-existing-and-out-of-scope:
|
||||
|
||||
- [x] **Server `stream_id` bug** — `handle_stream_announce` always wrote `stream_id=1`, so a
|
||||
second stream from the same user silently overwrote the first in
|
||||
`SessionRegistry::set_user_stream`'s replace-by-id logic. Fixed with a per-session counter
|
||||
(`ConnSession::next_stream_id_`) + `announced_stream_ids_` (also now validated in
|
||||
`handle_stream_stop`, rejecting stops for ids the session never announced).
|
||||
- [x] **Per-channel `AudioConfig` was modeled but never populated/enforced.**
|
||||
`SessionRegistry::init_default_channels()` now seeds Lobby (id=1: mono, 24kbps, `OPUS_VOIP`,
|
||||
FEC+DTX on) and a new "Music Room" (id=2: stereo, 128kbps, `OPUS_AUDIO`, FEC+DTX off) with
|
||||
real `AudioConfig`s; new `SessionRegistry::channel_audio_config(channel_id)` accessor (there
|
||||
was no per-id channel getter before, only `channel_snapshot()`). `handle_stream_announce`
|
||||
now treats the channel's config as authoritative (mode/frame_ms/application/fec/dtx/
|
||||
complexity), clamping (not overriding) `bitrate_bps` to the channel's ceiling.
|
||||
- [x] **Client silently dropped `mode`/`dtx`/`complexity`/`application` from `effective_audio`**
|
||||
even for the single M2 stream — `handle_stream_announce_result` and `sync_remote_streams`
|
||||
only copied `sample_rate`/`bitrate_bps`/`frame_ms`/`fec` into `OpusParams`. New shared
|
||||
`opus_params_from_audio_config()` helper (`client.cpp`) fixes both the send and receive
|
||||
paths.
|
||||
- [x] `core/src/codec/opus_codec.h/.cpp` — new `OpusApplication` enum + `OpusParams::application`
|
||||
field; `OpusEncoder::init` now honors it instead of hardcoding `OPUS_APPLICATION_VOIP`.
|
||||
- [x] `core/src/session/session.h/.cpp` — `Stream` struct extended with the full `AudioConfig`
|
||||
(mode/bitrate_bps/application/fec/expected_packet_loss/dtx/complexity), not just
|
||||
sample_rate/frame_ms; `copy_streams()` now copies all of it.
|
||||
- [x] `core/src/core/client.h/.cpp` — local-stream state is now a `std::unordered_map<int,
|
||||
LocalStream>` keyed by `vc_stream_kind` (one active stream per kind — MIC/SCREEN_AUDIO/
|
||||
AUX_DEVICE are each singletons for a client), replacing the M2 single-stream fields.
|
||||
`StreamAnnounce`/`StreamAnnounceResult` round-trips are now correlated by `request_id`
|
||||
(already round-tripped on the wire; just wasn't read) via `pending_announce_kind_`, so
|
||||
multiple concurrent announces from one client resolve to the right `LocalStream`.
|
||||
`on_capture_frame` takes a `kind` parameter and upmixes mono capture to stereo (duplicate
|
||||
L=R) when a stream's channel config calls for it. `vc_set_self_mute`'s `mic_muted` only
|
||||
gates the `MIC` kind — a concurrent `SCREEN_AUDIO` share keeps playing while muted.
|
||||
`set_remote_stream` now actually wires `noise_reduction` through (previously parsed and
|
||||
discarded). New `run_talk_timer()` (a small dedicated thread, started alongside the UDP
|
||||
media path, never the miniaudio callback thread) polls both remote talk-state edges
|
||||
(`AudioEngine::poll_talk_transitions()`) and local capture-activity edges, emitting
|
||||
`VC_EVENT_TALK_STATE`.
|
||||
- [x] **Fixed a thread-join race in `teardown_voice()`** — it's called both from `run_io()`'s
|
||||
own cleanup and from `disconnect()`, on different threads; without serialization both could
|
||||
see `udp_thread_`/`talk_timer_thread_` as `joinable()` simultaneously and race to `join()`
|
||||
the same `std::thread` (UB; surfaced as an intermittent `std::system_error: No such process`
|
||||
under `ctest`). Added a `teardown_mu_` guard around the whole function. This pre-existed for
|
||||
`udp_thread_` alone (likely the same root cause as the `test_m1_integration`/`test_m2_voice`
|
||||
cleanup-path flake noted in the M2 section above) — adding `talk_timer_thread_`'s join just
|
||||
made it surface more often, so it was fixed properly here rather than carried forward again.
|
||||
- [x] `core/src/audio/audio_engine.h/.cpp` — `CaptureCallback` gained a `kind` parameter
|
||||
(the real miniaudio capture device is always tagged `kind=0`/MIC; a second concurrent local
|
||||
stream is fed via its own `inject_capture(kind, ...)` ring buffer — `inject_taps_`, keyed by
|
||||
kind — since there is only one real hardware capture device in M3). Fixed a buffer-sizing
|
||||
bug in `on_playback`'s per-stream decode (`opus_decode`'s `frame_size` parameter is
|
||||
samples-*per-channel*, not total samples — the old code passed `frames * params_.channels`,
|
||||
which would have overflowed the decode buffer for any stereo stream). Stereo decoder output
|
||||
is downmixed (avg L/R) into the engine's mono mix accumulator immediately after decode.
|
||||
`RemoteStream` gained `recv_ns`/`noise_reduction_enabled` (lazy `ApmProcessor` instantiation
|
||||
— freed on disable, so no separate instance cap is needed per the roadmap's guidance) and
|
||||
`last_voice_ms`/`talking` (talk-indicator edge state, updated in `push_recv_frame`); new
|
||||
`set_stream_noise_reduction()` and `poll_talk_transitions()`. Note: until `VOICECAT_HAS_APM`
|
||||
is wired to a real WebRTC APM build, the NS toggle is plumbed end-to-end but behaviorally a
|
||||
passthrough no-op (`ApmPassthrough` doesn't touch PCM) — same situation send-side APM has
|
||||
been in since M2; M3's job was the plumbing, not the DSP backend.
|
||||
- [x] **New C ABI surface** (`core/include/voicecat.h`, additive only):
|
||||
`vc_audio_config` struct + `vc_get_stream_audio_config(c, user_id, stream_id, out)` — the
|
||||
effective Opus config for a stream you own or a peer's, reading from the (now richer)
|
||||
`LocalStream`/`session::Stream`. `vc_test_inject_capture(c, stream_id, pcm, samples)` —
|
||||
clearly-marked **test-only**, forwards to `AudioEngine::inject_capture`, so
|
||||
`test_m3_multistream` can drive two concurrent synthetic-audio streams through the real ABI
|
||||
without a microphone.
|
||||
- [x] `tests/test_m3_multistream.cpp` — the M3 exit criterion (ABI-level, mirrors
|
||||
`test_voice_client_abi.cpp`'s approach per the M2 lesson). Registered in `tests/CMakeLists.txt`.
|
||||
|
||||
**Explicitly out of scope for this pass** (confirmed with the user before implementing):
|
||||
- `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices` (device
|
||||
enumeration + VAD/PTT input gate) — still `VC_ERR_NOT_IMPLEMENTED`. These were mentioned as
|
||||
"scoped to M3" in the M2 follow-up notes above, but docs/roadmap.md's M3 bullets never
|
||||
actually listed them — deferred again, now tracked explicitly rather than implicitly.
|
||||
- Real WASAPI desktop-audio loopback capture for `SCREEN_AUDIO` — the engine now supports
|
||||
feeding a second concurrent local stream via `inject_capture`, but only synthetic PCM is
|
||||
wired up; a real loopback capture device is a follow-up.
|
||||
- True stereo *playback output* — `AudioEngine`'s mixer/output device stays mono; stereo
|
||||
streams are downmixed after decode (see above). The Opus wire format itself is fully
|
||||
stereo-correct.
|
||||
|
||||
---
|
||||
|
||||
## Decisions log
|
||||
|
||||
All architecture/scope decisions are settled and recorded in
|
||||
|
||||
Reference in New Issue
Block a user