Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(M3): multi-stream & per-channel tuning

Browse Source 
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).

Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
  the same user silently overwrote the first in SessionRegistry::set_user_stream.
  Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
  validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
  the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
  applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
  and receive (sync_remote_streams) paths via a shared
  opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
  and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
  (total samples instead of samples-per-channel), which would have overflowed
  the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
  and from disconnect() on a different thread -- both could see
  udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
  same std::thread (intermittent std::system_error under ctest). Fixed with a
  teardown_mu_ guard instead of carrying the flake forward.

New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
  FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
  handle_stream_announce enforces the channel's config, clamping (not
  overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
  std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
  request_id-correlated announce/result handling (request_id already
  round-tripped on the wire; just wasn't read before). on_capture_frame is
  kind-aware and upmixes mono capture to stereo when a stream's config calls
  for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
  through set_remote_stream. New run_talk_timer() thread emits
  VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
  (inject_capture), stereo-to-mono downmix at the decode/mix boundary,
  RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
  and last_voice_ms/talking; new set_stream_noise_reduction() and
  poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
  not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
  vc_get_stream_audio_config (effective Opus config for any stream you own or
  a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
  clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
  (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
  concurrent local streams, independent gain/mute/NS control, per-channel
  config divergence via vc_get_stream_audio_config, talk indicators.

Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).

ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Talon
2026-06-16 14:12:37 +02:00
parent c693cab35c
commit 867557eda1
17 changed files with 1102 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
tests/CMakeLists.txt
View File

@@ -76,4 +76,14 @@ if(VOICECAT_USE_VCPKG_DEPS)
target_include_directories(test_voice_client_abi PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME voice_client_abi COMMAND test_voice_client_abi)
set_tests_properties(voice_client_abi PROPERTIES TIMEOUT 60)
# M3 exit criterion: multi-stream (mic + desktop audio), independent per-stream
# gain/mute/NS, per-channel Opus configurability, talk indicators -- all through the
# real C ABI (vc_client), not raw sockets.
add_executable(test_m3_multistream test_m3_multistream.cpp)
target_link_libraries(test_m3_multistream PRIVATE voicecat::server)
target_compile_features(test_m3_multistream PRIVATE cxx_std_20)
target_include_directories(test_m3_multistream PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME m3_multistream COMMAND test_m3_multistream)
set_tests_properties(m3_multistream PROPERTIES TIMEOUT 60)
endif()

356
tests/test_m3_multistream.cpp Normal file
View File

@@ -0,0 +1,356 @@
/*
* test_m3_multistream — M3 exit criterion, exercised through the real C ABI.
*
* Mirrors test_voice_client_abi.cpp's approach (real vc_client instances, not raw sockets —
* the M2 lesson is that ABI-level coverage is what actually proves the client library works).
* Covers the whole M3 milestone in one flow:
*
* 1. A starts two concurrent local streams (MIC + SCREEN_AUDIO) -- distinct stream ids,
* both visible to B as separate STREAM_STARTED events for the same user.
* 2. Synthetic PCM (vc_test_inject_capture) flows into both of A's streams without crashing
* and without disrupting the control/voice plane; B observes a VC_EVENT_TALK_STATE
* talking=true edge for A's MIC stream while both are still in the same channel (voice
* only relays within a channel, so this must happen before step 4 moves A elsewhere).
* 3. B independently gains/mutes/NS-toggles A's two streams (vc_set_remote_stream) --
* one call doesn't clobber the other's routing; a bogus stream_id is rejected.
* 4. Per-channel Opus configurability: A joins "Music Room" (channel 2, stereo/128kbps/
* OPUS_AUDIO/no DTX) before announcing there, while B stays in "Lobby" (channel 1,
* mono/24kbps/OPUS_VOIP/DTX) -- vc_get_stream_audio_config shows the two streams'
* effective config differs exactly as the server enforces it.
*/
#include <cstdio>
#ifdef VOICECAT_HAS_NET
#include <atomic>
#include <chrono>
#include <cmath>
#include <condition_variable>
#include <filesystem>
#include <mutex>
#include <string>
#include <thread>
#include <vector>
#include "voicecat.h"
#include "server.h"
#include "db.h"
// ── Event tracking ────────────────────────────────────────────────────────────
struct StreamEvent {
bool started; // true = STARTED, false = STOPPED
uint32_t user_id;
uint32_t stream_id;
};
struct TalkEvent {
uint32_t user_id;
uint32_t stream_id;
bool talking;
};
struct EventStore {
std::mutex mu;
std::condition_variable cv;
bool auth_ok{false};
uint32_t self_user_id{0};
bool channel_list_received{false};
std::vector<StreamEvent> stream_events;
std::vector<TalkEvent> talk_events;
bool disconnected{false};
const char* label{nullptr};
};
static void on_event(void* user, const vc_event* ev) {
auto* s = static_cast<EventStore*>(user);
std::lock_guard lk(s->mu);
switch (ev->type) {
case VC_EVENT_AUTH_RESULT:
s->auth_ok = (ev->result == VC_OK);
s->self_user_id = ev->user_id;
if (!s->auth_ok) std::fprintf(stderr, "[%s] AUTH FAILED: %s\n",
s->label ? s->label : "?", ev->text ? ev->text : "(no msg)");
break;
case VC_EVENT_CHANNEL_LIST:
s->channel_list_received = true;
break;
case VC_EVENT_STREAM_STARTED:
s->stream_events.push_back({true, ev->user_id, ev->stream_id});
break;
case VC_EVENT_STREAM_STOPPED:
s->stream_events.push_back({false, ev->user_id, ev->stream_id});
break;
case VC_EVENT_TALK_STATE:
s->talk_events.push_back({ev->user_id, ev->stream_id, ev->u32a != 0});
break;
case VC_EVENT_ERROR:
std::fprintf(stderr, "[%s] ERROR rc=%d: %s\n",
s->label ? s->label : "?", ev->result, ev->text ? ev->text : "");
break;
case VC_EVENT_DISCONNECTED:
s->disconnected = true;
break;
default:
break;
}
s->cv.notify_all();
}
template <typename Pred>
static bool wait_for(EventStore& s, Pred pred, int timeout_ms) {
auto deadline = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout_ms);
std::unique_lock lk(s.mu);
return s.cv.wait_until(lk, deadline, [&] { return pred(s); });
}
static std::vector<int16_t> make_sine_frame(int frame_idx, float freq_hz,
int frame_samples = 960) {
std::vector<int16_t> pcm(frame_samples);
for (int i = 0; i < frame_samples; ++i) {
float t = static_cast<float>(frame_idx * frame_samples + i) / 48000.0f;
pcm[i] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * freq_hz * t) * 16000.0f);
}
return pcm;
}
// ── Test harness ──────────────────────────────────────────────────────────────
static int g_failures = 0;
#define CHECK(cond) \
do { \
if (!(cond)) { \
std::printf("FAIL: %s (%s:%d)\n", #cond, __FILE__, __LINE__); \
++g_failures; \
} \
} while (0)
int main() {
auto tmp = std::filesystem::temp_directory_path() /
("vctest_m3_" + std::to_string(
std::chrono::steady_clock::now().time_since_epoch().count()));
std::filesystem::create_directories(tmp);
std::string data_dir = tmp.string();
std::atomic<uint16_t> bound_port{0};
std::mutex ready_mu;
std::condition_variable ready_cv;
bool ready{false};
voicecat::server::Config cfg;
cfg.data_dir = data_dir;
cfg.bind_port = 0;
cfg.media_port = 0;
cfg.server_name = "VoiceCat-M3Test";
cfg.allow_guests = true;
cfg.on_ready = [&](uint16_t p) {
bound_port.store(p);
{ std::lock_guard lk(ready_mu); ready = true; }
ready_cv.notify_all();
};
voicecat::server::Server server(cfg);
std::thread server_thread([&] { server.run(); });
{
std::unique_lock lk(ready_mu);
bool ok = ready_cv.wait_for(lk, std::chrono::seconds(10), [&] { return ready; });
if (!ok) {
std::printf("FAIL: server did not become ready within 10s\n");
server.stop();
server_thread.join();
std::filesystem::remove_all(tmp);
return 1;
}
}
uint16_t port = bound_port.load();
std::printf("m3_multistream: server ready on :%u\n", port);
// ── Client A: guest "M3-A" ────────────────────────────────────────────────
EventStore evA;
evA.label = "clientA";
vc_callbacks cbA{on_event, nullptr, &evA};
vc_config cfgA{"test-clientA", "0.1", VC_LOG_OFF};
vc_client* clientA = vc_client_create(&cfgA, cbA);
CHECK(clientA != nullptr);
CHECK(vc_connect(clientA, "127.0.0.1", port) == VC_OK);
CHECK(vc_authenticate_guest(clientA, "M3-A") == VC_OK);
CHECK(wait_for(evA, [](EventStore& s) { return s.auth_ok; }, 8000));
CHECK(wait_for(evA, [](EventStore& s) { return s.channel_list_received; }, 3000));
// ── Client B: guest "M3-B" ────────────────────────────────────────────────
EventStore evB;
evB.label = "clientB";
vc_callbacks cbB{on_event, nullptr, &evB};
vc_config cfgB{"test-clientB", "0.1", VC_LOG_OFF};
vc_client* clientB = vc_client_create(&cfgB, cbB);
CHECK(clientB != nullptr);
CHECK(vc_connect(clientB, "127.0.0.1", port) == VC_OK);
CHECK(vc_authenticate_guest(clientB, "M3-B") == VC_OK);
CHECK(wait_for(evB, [](EventStore& s) { return s.auth_ok; }, 8000));
CHECK(wait_for(evB, [](EventStore& s) { return s.channel_list_received; }, 3000));
uint32_t a_uid = 0;
{ std::lock_guard lk(evA.mu); a_uid = evA.self_user_id; }
// Both guests land in channel 1 (Lobby) automatically; give the async UDP binding
// handshake a moment to complete on both clients before announcing streams.
std::this_thread::sleep_for(std::chrono::milliseconds(500));
// ── 1. A starts MIC + SCREEN_AUDIO concurrently ──────────────────────────
vc_stream_desc mic_desc{};
mic_desc.kind = VC_STREAM_MIC;
mic_desc.label = "mic";
uint32_t mic_sid = 0;
CHECK(vc_stream_start(clientA, &mic_desc, &mic_sid) == VC_OK);
vc_stream_desc screen_desc{};
screen_desc.kind = VC_STREAM_SCREEN_AUDIO;
screen_desc.label = "desktop audio";
uint32_t screen_sid = 0;
CHECK(vc_stream_start(clientA, &screen_desc, &screen_sid) == VC_OK);
CHECK(mic_sid != 0 && screen_sid != 0 && mic_sid != screen_sid);
// B observes two distinct STREAM_STARTED events for user A.
bool b_saw_both = wait_for(evB, [&](EventStore& s) {
bool saw_mic = false, saw_screen = false;
for (auto& e : s.stream_events) {
if (!e.started || e.user_id != a_uid) continue;
if (e.stream_id == mic_sid) saw_mic = true;
if (e.stream_id == screen_sid) saw_screen = true;
}
return saw_mic && saw_screen;
}, 5000);
CHECK(b_saw_both);
// Also wait for A's own view of both streams (vc_test_inject_capture requires the
// LocalStream to be active, which flips on A's io_thread_ independently of -- and not
// necessarily before -- the broadcast B observes above).
bool a_self_saw_both = wait_for(evA, [&](EventStore& s) {
bool saw_mic = false, saw_screen = false;
for (auto& e : s.stream_events) {
if (!e.started || e.user_id != a_uid) continue;
if (e.stream_id == mic_sid) saw_mic = true;
if (e.stream_id == screen_sid) saw_screen = true;
}
return saw_mic && saw_screen;
}, 5000);
CHECK(a_self_saw_both);
// ── 2. Inject synthetic PCM into both of A's local streams ──────────────
for (int i = 0; i < 25; ++i) {
auto mic_pcm = make_sine_frame(i, 440.0f);
auto screen_pcm = make_sine_frame(i, 880.0f);
CHECK(vc_test_inject_capture(clientA, mic_sid, mic_pcm.data(), mic_pcm.size()) == VC_OK);
CHECK(vc_test_inject_capture(clientA, screen_sid, screen_pcm.data(), screen_pcm.size()) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(20));
}
// No disconnects/errors should have resulted from the dual-stream PCM flow.
{ std::lock_guard lk(evA.mu); CHECK(!evA.disconnected); }
{ std::lock_guard lk(evB.mu); CHECK(!evB.disconnected); }
// ── 5. Talk indicators ────────────────────────────────────────────────────
// While A and B are still both in Lobby (voice actually relays between them here --
// the SFU forwards within a channel, so this must happen before A moves to Music Room
// in step 4 below), confirm B observed a talking=true edge for A's MIC stream.
bool b_saw_talking = wait_for(evB, [&](EventStore& s) {
for (auto& e : s.talk_events)
if (e.user_id == a_uid && e.stream_id == mic_sid && e.talking) return true;
return false;
}, 3000);
CHECK(b_saw_talking);
// ── 3. B independently controls gain/mute/NS on each of A's streams ─────
CHECK(vc_set_remote_stream(clientB, a_uid, mic_sid, 1.0f, 0, 0) == VC_OK);
CHECK(vc_set_remote_stream(clientB, a_uid, screen_sid, 0.3f, 1, 1) == VC_OK);
CHECK(vc_set_remote_stream(clientB, a_uid, 0xDEADBEEF, 1.0f, 0, 0) == VC_ERR_INVALID_ARG);
// Toggle NS on/off a few times -- plumbing should never fault or disrupt the stream.
for (int i = 0; i < 3; ++i) {
CHECK(vc_set_remote_stream(clientB, a_uid, mic_sid, 1.0f, 0, 1) == VC_OK);
CHECK(vc_set_remote_stream(clientB, a_uid, mic_sid, 1.0f, 0, 0) == VC_OK);
}
{ std::lock_guard lk(evB.mu); CHECK(!evB.disconnected); }
// ── 4. Per-channel Opus configurability ──────────────────────────────────
// A moves to "Music Room" (channel 2: stereo/128kbps/OPUS_AUDIO/no DTX) and announces a
// fresh MIC stream there; B stays in "Lobby" (channel 1: mono/24kbps/OPUS_VOIP/DTX) with
// its own MIC stream. Their effective_audio should differ exactly as configured server-side.
CHECK(vc_stream_stop(clientA, mic_sid) == VC_OK);
CHECK(vc_join_channel(clientA, 2, nullptr) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(300));
vc_stream_desc music_mic_desc{};
music_mic_desc.kind = VC_STREAM_MIC;
music_mic_desc.label = "music-mic";
uint32_t a_music_mic_sid = 0;
CHECK(vc_stream_start(clientA, &music_mic_desc, &a_music_mic_sid) == VC_OK);
CHECK(wait_for(evA, [&](EventStore& s) {
for (auto& e : s.stream_events)
if (e.started && e.user_id == a_uid && e.stream_id == a_music_mic_sid) return true;
return false;
}, 5000));
vc_stream_desc b_mic_desc{};
b_mic_desc.kind = VC_STREAM_MIC;
b_mic_desc.label = "lobby-mic";
uint32_t b_mic_sid = 0;
CHECK(vc_stream_start(clientB, &b_mic_desc, &b_mic_sid) == VC_OK);
CHECK(wait_for(evB, [&](EventStore& s) {
uint32_t self = s.self_user_id;
for (auto& e : s.stream_events)
if (e.started && e.user_id == self && e.stream_id == b_mic_sid) return true;
return false;
}, 5000));
vc_audio_config a_cfg{};
vc_audio_config b_cfg{};
CHECK(vc_get_stream_audio_config(clientA, a_uid, a_music_mic_sid, &a_cfg) == VC_OK);
uint32_t b_uid = 0;
{ std::lock_guard lk(evB.mu); b_uid = evB.self_user_id; }
CHECK(vc_get_stream_audio_config(clientB, b_uid, b_mic_sid, &b_cfg) == VC_OK);
// Music Room: stereo, 128kbps, OPUS_AUDIO, DTX off. Lobby: mono, 24kbps, OPUS_VOIP, DTX on.
CHECK(a_cfg.mode == 1 /* stereo */);
CHECK(b_cfg.mode == 0 /* mono */);
CHECK(a_cfg.bitrate_bps == 128000);
CHECK(b_cfg.bitrate_bps == 24000);
CHECK(a_cfg.application == 1 /* OPUS_AUDIO */);
CHECK(b_cfg.application == 0 /* OPUS_VOIP */);
CHECK(a_cfg.dtx == 0);
CHECK(b_cfg.dtx != 0);
// ── Cleanup ───────────────────────────────────────────────────────────────
vc_disconnect(clientA);
vc_disconnect(clientB);
vc_client_destroy(clientA);
vc_client_destroy(clientB);
server.stop();
server_thread.join();
std::filesystem::remove_all(tmp);
if (g_failures == 0) {
std::printf("m3_multistream: all checks passed\n");
return 0;
}
std::printf("m3_multistream: %d failure(s)\n", g_failures);
return 1;
}
#else // !VOICECAT_HAS_NET
int main() {
std::printf("m3_multistream: SKIP (VOICECAT_HAS_NET not defined)\n");
return 0;
}
#endif // VOICECAT_HAS_NET