diff --git a/PROGRESS.md b/PROGRESS.md index acd57b0..4f87138 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -10,6 +10,39 @@ up instantly. Newest status at the top. ## ▶ Where we left off / next action +- **Done:** **Stereo screen-audio loopback capture on Windows** (2026-06-17). The WASAPI + loopback path (`start_loopback_capture`) used to hardcode `cfg.capture.channels = 1`, + downmixing the system's stereo mix to mono before the encoder ever saw it — so even on a + stereo channel, `SCREEN_AUDIO` was effectively mono (the encoder then upmixed L=R to + produce a *fake* stereo bitstream). Now the loopback device opens in the channel's mode: + stereo (interleaved L/R) when the channel is configured stereo, mono when mono. Real stereo + flows end-to-end through loopback → Opus encode → decode → stereo playback mixer. + - `audio_engine.h` — `CaptureCallback` gained an `int channels` parameter (the encoder + needs to know whether the PCM is real stereo or mono to avoid upmixing real stereo). + `start_loopback_capture(int kind)` → `start_loopback_capture(int kind, int channels)`. + New `loopback_channels_` member; new `feed_loopback_for_test` test hook (self-contained, + works on headless CI where the real WASAPI device can't init). + - `audio_engine.cpp` — `on_loopback` accumulator is now channel-aware (sized to + `frame_samples_*loopback_channels_`); `start_loopback_capture` sizes the accumulator off + the RT thread before `ma_device_start`, opens the device with `channels`, and falls back + to mono if the render endpoint rejects stereo (mirrors the playback path's fallback). + `on_capture`/`inject_capture`/`feed_capture_for_test` forward `channels` through the + callback (mic path always 1; loopback path 1 or 2). + - `client.cpp` — `on_capture_frame` takes `channels`; for `channels==2` (real stereo + loopback PCM) it encodes directly with no upmix; for `channels==1` on a stereo channel it + keeps the existing L=R upmix (mic stays mono in v1). `handle_stream_announce_result` reads + `effective_params.stereo` under the lock and passes `2` or `1` to `start_loopback_capture`. + - `test_vad_ptt_devices.cpp` — new `test_loopback_stereo_capture` behavior test: feeds a + loud-L / silent-R stereo signal through `feed_loopback_for_test`, encodes (as + `on_capture_frame` now does for `channels==2`), decodes, mixes, and asserts L≠R across + the frame (total_diff ~8.2M, well above the 960k threshold). A mono-downmixed-then- + upmixed bitstream would have L==R. Existing test lambda updated for the 4-arg callback. + - `docs/voice.md §8/§9` — diagram + notes updated: mic stays mono; `SCREEN_AUDIO` loopback + captures stereo when the channel is stereo. + - `cmake --build --preset m2-dev` + `ctest --preset m2-dev --parallel 1` — **18/18 green**. + The `vad_ptt_devices` VAD-gate sub-test has a pre-existing parallel-run timing flake + (passes serially and in isolation); unrelated to this change (VAD gate logic is unchanged + for `channels==1`, the only path the MIC uses). - **Done:** **Screen-audio sharing wired into the Windows WinForms client** (2026-06-17). The core already fully supported `SCREEN_AUDIO` capture on Windows (post-M3 WASAPI loopback via `VOICECAT_HAS_LOOPBACK`, always on for the `windows-client` preset — @@ -337,8 +370,9 @@ not treated as pre-existing-and-out-of-scope: `StreamAnnounce`/`StreamAnnounceResult` round-trips are now correlated by `request_id` (already round-tripped on the wire; just wasn't read) via `pending_announce_kind_`, so multiple concurrent announces from one client resolve to the right `LocalStream`. - `on_capture_frame` takes a `kind` parameter and upmixes mono capture to stereo (duplicate - L=R) when a stream's channel config calls for it. `vc_set_self_mute`'s `mic_muted` only + `on_capture_frame` takes a `kind` and `channels` parameter; for mono capture (`channels==1`) + on a stereo channel it upmixes L=R, and for real stereo capture (`channels==2`, the + `SCREEN_AUDIO` loopback path on a stereo channel) it encodes directly with no upmix. `vc_set_self_mute`'s `mic_muted` only gates the `MIC` kind — a concurrent `SCREEN_AUDIO` share keeps playing while muted. `set_remote_stream` now actually wires `noise_reduction` through (previously parsed and discarded). New `run_talk_timer()` (a small dedicated thread, started alongside the UDP diff --git a/core/src/audio/audio_engine.cpp b/core/src/audio/audio_engine.cpp index 07053be..c92ed8b 100644 --- a/core/src/audio/audio_engine.cpp +++ b/core/src/audio/audio_engine.cpp @@ -140,7 +140,11 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) { #ifdef VOICECAT_HAS_AUDIO // Pre-allocate capture accumulators before the devices start so on_capture / on_loopback // never allocate on the RT thread. count=0 means "empty"; the buf is sized to exactly one - // encoder frame so a memcpy into it can never overrun. + // encoder frame so a memcpy into it can never overrun. The mic accumulator is mono + // (params_.capture_channels, always 1 in v1 — no stereo mic). The loopback accumulator is + // sized mono here as a safe default and re-sized to frame_samples_*channels in + // start_loopback_capture() once the screen stream's channel mode is known (off the RT + // thread, before the loopback device is started). capture_accum_.buf.assign(static_cast(frame_samples_), 0); capture_accum_.count = 0; loopback_accum_.buf.assign(static_cast(frame_samples_), 0); @@ -288,7 +292,7 @@ void AudioEngine::inject_capture(int kind, const int16_t* pcm, size_t n) { frame[i] = tap->ring[(r + i) % kInjectCapSamples]; tap->read.store(r + frame_samples_, std::memory_order_release); - if (capture_cb_) capture_cb_(kind, frame.data(), frame_samples_); + if (capture_cb_) capture_cb_(kind, frame.data(), frame_samples_, 1); } } @@ -391,7 +395,8 @@ void AudioEngine::on_capture(const int16_t* pcm, ma_uint32 frames) { src += copy; remaining -= copy; if (capture_accum_.count == frame_samples_) { - capture_cb_(0, capture_accum_.buf.data(), frame_samples_); + capture_cb_(0, capture_accum_.buf.data(), frame_samples_, + static_cast(params_.capture_channels)); capture_accum_.count = 0; } } @@ -507,32 +512,39 @@ void AudioEngine::loopback_data_cb(ma_device* dev, void* /*out*/, const void* in void AudioEngine::on_loopback(const int16_t* pcm, ma_uint32 frames) { // Same accumulation as on_capture — the WASAPI loopback render endpoint's callback - // period is also hardware-driven and may not match frame_samples_. + // period is also hardware-driven and may not match frame_samples_. PCM here is + // interleaved across loopback_channels_ (1 = mono downmix, 2 = stereo L/R) — the + // accumulator was sized to frame_samples_*loopback_channels_ in start_loopback_capture, + // so a memcpy into it can never overrun. capture_cb_ receives samples-per-channel + // (frame_samples_) and the channel count explicitly. if (!capture_cb_ || frame_samples_ <= 0) return; + const int ch = std::max(1, loopback_channels_); const int16_t* src = pcm; - auto remaining = static_cast(frames); + auto remaining = static_cast(frames) * ch; + const int full = frame_samples_ * ch; while (remaining > 0) { - int space = frame_samples_ - loopback_accum_.count; + int space = full - loopback_accum_.count; int copy = std::min(remaining, space); std::memcpy(loopback_accum_.buf.data() + loopback_accum_.count, src, static_cast(copy) * sizeof(int16_t)); loopback_accum_.count += copy; src += copy; remaining -= copy; - if (loopback_accum_.count == frame_samples_) { - capture_cb_(loopback_kind_, loopback_accum_.buf.data(), frame_samples_); + if (loopback_accum_.count == full) { + capture_cb_(loopback_kind_, loopback_accum_.buf.data(), frame_samples_, ch); loopback_accum_.count = 0; } } } -bool AudioEngine::start_loopback_capture(int kind) { +bool AudioEngine::start_loopback_capture(int kind, int channels) { if (loopback_started_) return false; // already running; stop_loopback_capture() first + int ch = std::max(1, channels); loopback_accum_.count = 0; // discard any partial frame from a previous loopback session ma_device_config cfg = ma_device_config_init(ma_device_type_loopback); cfg.capture.format = ma_format_s16; - cfg.capture.channels = 1; // let miniaudio's converter remix from the system's mix format + cfg.capture.channels = static_cast(ch); cfg.sampleRate = params_.sample_rate; cfg.dataCallback = loopback_data_cb; cfg.pUserData = this; @@ -543,14 +555,31 @@ bool AudioEngine::start_loopback_capture(int kind) { // along with everything else playing — an accepted self-echo-loop characteristic of // desktop-audio capture, not a bug. - if (ma_device_init(nullptr, &cfg, &loopback_device_) != MA_SUCCESS) return false; + if (ma_device_init(nullptr, &cfg, &loopback_device_) != MA_SUCCESS) { + // Fallback: some unusual render endpoints may reject channels=2 even though WASAPI + // shared mode normally remixes transparently. Retry once at mono (mirror the playback + // device's fallback in start()) rather than leaving screen-audio capture dead. + if (ch != 1) { + ch = 1; + cfg.capture.channels = 1; + if (ma_device_init(nullptr, &cfg, &loopback_device_) != MA_SUCCESS) return false; + } else { + return false; + } + } if (ma_device_start(&loopback_device_) != MA_SUCCESS) { ma_device_uninit(&loopback_device_); return false; } - loopback_kind_ = kind; - loopback_started_ = true; + // Size the accumulator to one full encoder frame at the actual opened channel count. + // Done here (off the RT thread) before any callback can fire — satisfies architecture.md + // §3 (no allocation on RT threads). ma_device_start() above is what arms the callback. + loopback_channels_ = ch; + loopback_accum_.buf.assign(static_cast(frame_samples_ * ch), 0); + loopback_accum_.count = 0; + loopback_kind_ = kind; + loopback_started_ = true; return true; } @@ -561,14 +590,14 @@ void AudioEngine::stop_loopback_capture() { loopback_started_ = false; } #else // !VOICECAT_HAS_LOOPBACK -bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; } +bool AudioEngine::start_loopback_capture(int /*kind*/, int /*channels*/) { return false; } void AudioEngine::stop_loopback_capture() {} #endif // VOICECAT_HAS_LOOPBACK #endif // VOICECAT_HAS_AUDIO #ifndef VOICECAT_HAS_AUDIO -bool AudioEngine::start_loopback_capture(int /*kind*/) { return false; } +bool AudioEngine::start_loopback_capture(int /*kind*/, int /*channels*/) { return false; } void AudioEngine::stop_loopback_capture() {} #endif diff --git a/core/src/audio/audio_engine.h b/core/src/audio/audio_engine.h index 2d05ff3..388f8f4 100644 --- a/core/src/audio/audio_engine.h +++ b/core/src/audio/audio_engine.h @@ -105,10 +105,14 @@ class AudioEngine { public: // Callback type for encoded capture frames ready to be sent. `kind` identifies which // local stream this PCM belongs to (a vc_stream_kind value; 0 = MIC for the real capture - // device, which is always the "primary" tap). M3: multiple concurrent local streams are - // possible (e.g. MIC + SCREEN_AUDIO), each fed via its own injection tap (see - // inject_capture) since there is only one real hardware capture device. - using CaptureCallback = std::function; + // device, which is always the "primary" tap). `channels` is the channel count of the PCM + // buffer (1 = mono, 2 = stereo interleaved) — the mic capture device is mono in v1, but + // the WASAPI loopback path (SCREEN_AUDIO) captures in the channel's mode when stereo, so + // the encoder sees real interleaved L/R PCM rather than a mono upmix. M3: multiple + // concurrent local streams are possible (e.g. MIC + SCREEN_AUDIO), each fed via its own + // injection tap (see inject_capture) since there is only one real hardware capture device. + using CaptureCallback = std::function; AudioEngine(); ~AudioEngine(); @@ -131,8 +135,11 @@ class AudioEngine { // Real desktop-audio loopback capture (Windows/WASAPI only, VOICECAT_HAS_LOOPBACK). Feeds // `kind`'s capture_cb_ directly, same pattern as the real mic capture device — NOT routed - // through inject_capture()'s test-only ring. No-op (returns false) when unsupported. - bool start_loopback_capture(int kind); + // through inject_capture()'s test-only ring. `channels` is the channel count to open the + // loopback device with (1 = mono downmix of the system mix, 2 = stereo capture when the + // channel is configured stereo); the accumulator and capture_cb_ invocation are shaped to + // match. No-op (returns false) when unsupported. + bool start_loopback_capture(int kind, int channels); void stop_loopback_capture(); // Inject synthetic PCM directly into the capture pipeline (bypasses real device). @@ -193,13 +200,52 @@ class AudioEngine { src += copy; remaining -= copy; if (capture_accum_.count == frame_samples_) { - cb(0, capture_accum_.buf.data(), frame_samples_); + cb(0, capture_accum_.buf.data(), frame_samples_, 1); capture_accum_.count = 0; } } } #endif +#ifdef VOICECAT_HAS_LOOPBACK + // TEST-ONLY — drives the loopback accumulator directly with an explicit callback, the + // loopback analogue of feed_capture_for_test. Self-contained: sizes the accumulator and + // sets loopback_channels_ itself, so it works on headless CI where start_loopback_capture + // can't init a real WASAPI device. `channels` selects mono (1) or interleaved stereo (2). + // PCM is interleaved L/R when channels==2. Invokes `cb` once per full frame_samples_ + // per-channel chunk, with the channel count passed through so the encoder branch in + // on_capture_frame sees real stereo (channels==2) rather than a mono upmix. + void feed_loopback_for_test(const int16_t* pcm, int frames_per_channel, int channels, + const CaptureCallback& cb) { + if (frame_samples_ <= 0 || !cb) return; + const int ch = std::max(1, channels); + const int full = frame_samples_ * ch; + // Size the accumulator for the requested channel count (off the RT thread; this is a + // test-only path). start_loopback_capture() does the same sizing when it opens a real + // device, but on headless CI that init fails — so do it here too. + if (static_cast(loopback_accum_.buf.size()) != full) { + loopback_accum_.buf.assign(static_cast(full), 0); + loopback_accum_.count = 0; + } + loopback_channels_ = ch; + const int16_t* src = pcm; + auto remaining = frames_per_channel * ch; + while (remaining > 0) { + int space = full - loopback_accum_.count; + int copy = std::min(remaining, space); + std::memcpy(loopback_accum_.buf.data() + loopback_accum_.count, src, + static_cast(copy) * sizeof(int16_t)); + loopback_accum_.count += copy; + src += copy; + remaining -= copy; + if (loopback_accum_.count == full) { + cb(loopback_kind_, loopback_accum_.buf.data(), frame_samples_, ch); + loopback_accum_.count = 0; + } + } + } +#endif + private: #ifdef VOICECAT_HAS_AUDIO static void capture_data_cb(ma_device*, void*, const void*, ma_uint32); @@ -222,6 +268,7 @@ class AudioEngine { ma_device loopback_device_{}; bool loopback_started_ = false; int loopback_kind_ = 0; + int loopback_channels_ = 1; // channel count the loopback device was opened with #endif #endif diff --git a/core/src/core/client.cpp b/core/src/core/client.cpp index 662a329..7cf5312 100644 --- a/core/src/core/client.cpp +++ b/core/src/core/client.cpp @@ -852,7 +852,7 @@ voicecat::v1::Channel channel_from_vc(const vc_channel_info& c) { } // namespace -void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples) { +void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples, int channels) { std::lock_guard lk(local_streams_mu_); auto it = local_streams_.find(kind); if (it == local_streams_.end() || !it->second.active.load(std::memory_order_acquire)) return; @@ -888,9 +888,15 @@ void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples) { uint8_t opus_buf[1500]; int opus_len; - if (ls.effective_params.stereo) { - // Capture is always mono in M3 (no stereo capture device); upmix L=R so a - // channel configured for stereo still gets a real, spec-correct stereo Opus stream. + if (channels == 2) { + // Real interleaved stereo PCM (SCREEN_AUDIO loopback on a stereo channel) — encode + // directly, no upmix. `samples` is samples-per-channel, as OpusEncoder::encode expects. + opus_len = ls.encoder.encode(pcm, samples, opus_buf, sizeof(opus_buf)); + } else if (ls.effective_params.stereo) { + // Mono capture (mic, or loopback on a mono channel, or test injection) on a channel + // configured for stereo — upmix L=R so the stream is still a spec-correct stereo Opus + // bitstream. (Mic stays mono in v1 — no stereo capture device — but a stereo channel + // requires a stereo bitstream, hence the upmix.) std::vector stereo_pcm(static_cast(samples) * 2); for (int i = 0; i < samples; ++i) { stereo_pcm[i * 2] = pcm[i]; @@ -942,8 +948,8 @@ void vc_client::ensure_audio_running() { auto it = local_streams_.find(static_cast(VC_STREAM_MIC)); if (it != local_streams_.end()) p.capture_device_id = it->second.capture_device_id; } - audio_engine_.start(p, [this](int kind, const int16_t* pcm, int samples) { - on_capture_frame(kind, pcm, samples); + audio_engine_.start(p, [this](int kind, const int16_t* pcm, int samples, int channels) { + on_capture_frame(kind, pcm, samples, channels); }); } @@ -1049,6 +1055,7 @@ void vc_client::handle_stream_announce_result(uint64_t req_id, uint32_t emit_stream_id; bool ok_to_emit = false; int kind; + int loopback_channels = 1; // only meaningful for SCREEN_AUDIO; set under the lock { std::lock_guard lk(local_streams_mu_); auto pit = pending_announce_kind_.find(req_id); @@ -1087,11 +1094,18 @@ void vc_client::handle_stream_announce_result(uint64_t req_id, mic_vad_ = voicecat::audio::ApmProcessor::create_vad( vad_threshold_.load(std::memory_order_relaxed)); } + + // SCREEN_AUDIO loopback opens the WASAPI device in the channel's mode: stereo capture + // when the channel is stereo (real L/R, no downmix), mono otherwise. Captured under + // the lock alongside the rest of the LocalStream setup; used below after unlock. + if (kind == static_cast(VC_STREAM_SCREEN_AUDIO)) { + loopback_channels = ls.effective_params.stereo ? 2 : 1; + } } ensure_audio_running(); if (kind == static_cast(VC_STREAM_SCREEN_AUDIO)) { - audio_engine_.start_loopback_capture(kind); + audio_engine_.start_loopback_capture(kind, loopback_channels); } if (ok_to_emit) { diff --git a/core/src/core/client.h b/core/src/core/client.h index 7816639..c967f8f 100644 --- a/core/src/core/client.h +++ b/core/src/core/client.h @@ -272,7 +272,7 @@ struct vc_client { void run_udp_recv(); // capture_cb passed to audio_engine_.start(): encode + seal + send one frame for the // given local stream `kind` (M3: multiple concurrent local streams are possible). - void on_capture_frame(int kind, const int16_t* pcm, int samples); + void on_capture_frame(int kind, const int16_t* pcm, int samples, int channels); // Inspect a User proto's streams and wire up any new remote ssrc into audio_engine_, // emitting VC_EVENT_STREAM_STARTED/STOPPED as streams appear/disappear. void sync_remote_streams(const voicecat::v1::User& user); diff --git a/docs/voice.md b/docs/voice.md index 69a2db0..0879fc6 100644 --- a/docs/voice.md +++ b/docs/voice.md @@ -159,17 +159,25 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**. ## 8. Capture/playback pipeline (inside the core) ``` - device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate + mic device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate → Opus encode → frame header → AEAD → UDP send + screen audio ─(WASAPI loopback, 48k, mono or stereo per channel mode)→ Opus encode + → frame header → AEAD → UDP send + UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode - → per-stream recv-side NS (optional, per user) → per-stream gain/mute - → mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback, - 48k, stereo) → device + → per-stream recv-side NS (optional, per user) → per-stream gain/mute + → mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback, + 48k, stereo) → device ``` - Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA). - Playback is genuinely stereo end-to-end; capture stays mono (no stereo mic in v1). + Playback is genuinely stereo end-to-end. **Mic capture stays mono** (no stereo mic in v1); + a mono mic frame on a stereo channel is upmixed L=R before encoding so the Opus bitstream + is still spec-correct stereo. **Screen-audio (`SCREEN_AUDIO`) loopback** captures in the + channel's mode — stereo when the channel is stereo (real interleaved L/R, no downmix), mono + when the channel is mono — so a stereo music/screen-share channel gets genuine stereo + end-to-end. See §9 for the platform-specific loopback mechanism. - **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC + VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream (confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard @@ -234,7 +242,7 @@ normal stream; only the *source* is platform-specific. | Platform | Mechanism | Notes | |----------|-----------|-------| -| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. | +| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. | | **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. | | **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). | diff --git a/tests/test_vad_ptt_devices.cpp b/tests/test_vad_ptt_devices.cpp index bf1ca2a..8b6ea39 100644 --- a/tests/test_vad_ptt_devices.cpp +++ b/tests/test_vad_ptt_devices.cpp @@ -230,6 +230,90 @@ static void test_stereo_mix() { std::printf("test_stereo_mix: ok (total_diff=%lld)\n", static_cast(total_diff)); } +// ── 4a-2. Stereo screen-audio loopback capture (white-box, no audio hardware needed) ── +// Regression for the mono-loopback bug: start_loopback_capture used to hardcode channels=1, +// downmixing the system's stereo mix to mono before the encoder ever saw it (and on_capture_frame +// then upmixed L=R to produce a fake-stereo bitstream). Now the loopback device opens in the +// channel's mode (stereo when the channel is stereo), so the encoder receives real interleaved +// L/R PCM and encodes it directly. This test drives feed_loopback_for_test with a loud-L / +// silent-R stereo signal, encodes it (as on_capture_frame now does for channels==2), decodes, +// and mixes — asserting L != R across the frame. A mono-downmixed-then-upmixed bitstream would +// have L == R. Mirrors test_stereo_mix but routes the encode side through the loopback +// accumulator path that the fix touches (feed_loopback_for_test → on_loopback's accumulator). +#if defined(VOICECAT_HAS_LOOPBACK) && defined(VOICECAT_HAS_OPUS) +static void test_loopback_stereo_capture() { + voicecat::audio::AudioEngine engine; + voicecat::audio::AudioParams p; + p.sample_rate = 48000; + p.capture_channels = 1; // mic path — irrelevant here; loopback has its own channel count + p.playback_channels = 2; // stereo mix output (for mix_for_test below) + p.frame_ms = 20; + CHECK(engine.start(p)); // no capture_cb — the real mic (if any) won't touch capture_accum_ + + voicecat::codec::OpusParams stereo_params; + stereo_params.stereo = true; + stereo_params.application = voicecat::codec::OpusApplication::Audio; // screen-audio channel + stereo_params.bitrate_bps = 128000; // music/screen-audio channel default + int frame_samples = voicecat::codec::opus_frame_samples(stereo_params); + + voicecat::codec::OpusEncoder enc; + CHECK(enc.init(stereo_params)); + + // Loud left channel, silent right — a real mono downmix would average them into a single + // audible-but-quieter centered sample; true stereo keeps them distinct. + std::vector interleaved(static_cast(frame_samples) * 2); + for (int i = 0; i < frame_samples; ++i) { + float t = static_cast(i) / 48000.0f; + interleaved[i * 2] = static_cast(std::sin(2.0f * 3.14159265f * 440.0f * t) * 20000.0f); + interleaved[i * 2 + 1] = 0; + } + + // Encode via the loopback accumulator path: feed_loopback_for_test drives on_loopback's + // accumulator and invokes the callback with channels=2 (the fix). The callback encodes + // exactly as on_capture_frame does for real-stereo SCREEN_AUDIO PCM — no upmix. + uint8_t opus_buf[1500]; + int opus_len = 0; + int seen_channels = 0; + auto cb = [&](int /*kind*/, const int16_t* pcm, int /*samples*/, int channels) { + seen_channels = channels; + if (channels == 2) { + // The loopback accumulator must have preserved L/R distinctness pre-encode. + int64_t pre_diff = 0; + for (int i = 0; i < frame_samples; ++i) + pre_diff += std::abs(static_cast(pcm[i * 2]) - static_cast(pcm[i * 2 + 1])); + CHECK(pre_diff > static_cast(frame_samples) * 1000); + } + opus_len = enc.encode(pcm, frame_samples, opus_buf, sizeof(opus_buf)); + }; + engine.feed_loopback_for_test(interleaved.data(), frame_samples, 2, cb); + CHECK(seen_channels == 2); // the loopback path reported stereo, not downmixed mono + CHECK(opus_len > 0); + + // Decode + mix — same recv path as test_stereo_mix. A real stereo bitstream should + // survive with L != R; a mono-downmixed-then-upmixed bitstream would have L == R. + engine.init_recv_stream(/*ssrc=*/3, stereo_params); + voicecat::audio::JitterBuffer::Frame f; + f.seq = 0; + f.timestamp = 0; + f.fec_present = false; + f.payload.assign(opus_buf, opus_buf + opus_len); + engine.push_recv_frame(3, std::move(f)); + + std::vector out(static_cast(frame_samples) * 2, 0); + engine.mix_for_test(out.data(), static_cast(frame_samples)); + + int64_t total_diff = 0; + for (int i = 0; i < frame_samples; ++i) + total_diff += std::abs(static_cast(out[i * 2]) - static_cast(out[i * 2 + 1])); + CHECK(total_diff > static_cast(frame_samples) * 1000); + + engine.remove_stream(3); + engine.stop(); + std::printf("test_loopback_stereo_capture: ok (total_diff=%lld, seen_channels=%d)\n", + static_cast(total_diff), seen_channels); +} +#endif + // ── 4b. Playout-clock re-sync after a late join / silence gap ──────────────────── // Regression for the "talk indicator lit, no audio" bug: the playout clock free-runs (it // advances every callback via PLC), while the sender's frame timestamps only advance while it @@ -316,7 +400,7 @@ static void test_capture_frame_accumulation() { // and drives the same accumulator directly with the explicit `cb` below — no races. CHECK(engine.start(p)); - auto cb = [&](int /*kind*/, const int16_t* /*pcm*/, int samples) { + auto cb = [&](int /*kind*/, const int16_t* /*pcm*/, int samples, int /*channels*/) { ++call_count; if (samples != kExpected) wrong_size.store(true); }; @@ -501,6 +585,9 @@ int main() { test_device_enumeration(); #if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS) test_stereo_mix(); +#if defined(VOICECAT_HAS_LOOPBACK) + test_loopback_stereo_capture(); +#endif test_playout_resync(); #endif #ifdef VOICECAT_HAS_AUDIO