Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(audio): stereo screen-audio loopback capture on Windows

Browse Source 
start_loopback_capture hardcoded channels=1, forcing miniaudio to downmix the system's stereo mix to mono before the encoder saw it -- on_capture_frame then upmixed L=R to produce fake stereo. Now the loopback device opens in the channel's mode (stereo when the channel is stereo), CaptureCallback carries an explicit channels param so the encoder gets real interleaved L/R, and a mono fallback covers unusual render endpoints. New test_loopback_stereo_capture asserts L!=R end-to-end; 18/18 ctest green.
This commit is contained in:
Talon
2026-06-17 23:27:59 +02:00
parent a88656f2fa
commit cccf085a87
7 changed files with 258 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/voice.md
View File

@@ -159,17 +159,25 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**.
## 8. Capture/playback pipeline (inside the core)
```
device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
mic device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
→ Opus encode → frame header → AEAD → UDP send
screen audio ─(WASAPI loopback, 48k, mono or stereo per channel mode)→ Opus encode
→ frame header → AEAD → UDP send
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
→ mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
48k, stereo) → device
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
→ mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
48k, stereo) → device
```
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
Playback is genuinely stereo end-to-end; capture stays mono (no stereo mic in v1).
Playback is genuinely stereo end-to-end. **Mic capture stays mono** (no stereo mic in v1);
a mono mic frame on a stereo channel is upmixed L=R before encoding so the Opus bitstream
is still spec-correct stereo. **Screen-audio (`SCREEN_AUDIO`) loopback** captures in the
channel's mode — stereo when the channel is stereo (real interleaved L/R, no downmix), mono
when the channel is mono — so a stereo music/screen-share channel gets genuine stereo
end-to-end. See §9 for the platform-specific loopback mechanism.
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
@@ -234,7 +242,7 @@ normal stream; only the *source* is platform-specific.
| Platform | Mechanism | Notes |
|----------|-----------|-------|
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |