Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(audio): apply receive-side NR to stereo mic streams
Some checks failed
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Details
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Details

Browse Source 
The per-listener noise-reduction toggle (vc_set_remote_stream) did nothing
on Windows/macOS/iOS. The decode loop gated the RNNoise pass on
dec_channels == 1 as a proxy for "this stream is voice" (assuming
stereo => screen-share). The stereo-mic capture commit broke that: a stereo
mic with send-side NR off transmits stereo Opus, so the receiver decoded
two channels and skipped NR entirely. gain/mute have no channel guard, which
is why only NR appeared broken.

Thread the stream kind through init_recv_stream into RemoteStream::is_voice
(set from si.kind() == STREAM_MIC), gate receive NR on is_voice instead of
channel count, and fold a stereo voice frame to mono -> denoise -> duplicate
back across both channels in place (symmetric with the send-side downmix;
RNNoise is mono-only). Screen-audio shares are never denoised.

New test test_recv_noise_reduction drives AudioEngine and asserts a stereo
voice stream's noise floor collapses with NR on (RMS 1046 -> 0.1) while a
screen-audio share stays unchanged. ctest --preset dev green 29/29.
Docs: voice.md section 10. Shared-core fix; clients need only a rebuild.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Talon
2026-06-23 21:11:03 +02:00
parent f72219ddf3
commit b44a200b95
12 changed files with 261 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
core/src/audio/audio_engine.cpp
View File

@@ -592,11 +592,12 @@ int32_t AudioEngine::stream_playout_depth_samples(uint32_t ssrc) const {
#ifdef VOICECAT_HAS_OPUS
void AudioEngine::init_recv_stream(uint32_t ssrc, const codec::OpusParams& p,
uint32_t user_id, uint32_t stream_id) {
uint32_t user_id, uint32_t stream_id, bool is_voice) {
std::lock_guard lk(streams_mu_);
auto& stream = streams_[ssrc];
stream.user_id = user_id;
stream.stream_id = stream_id;
stream.is_voice = is_voice;
stream.fec_enabled_ = p.fec;
stream.decoder.init(p);
// Ring must be sized for this decoder's actual channel/frame-size — see RemoteStream::ring
@@ -788,11 +789,25 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
// `n` is samples-per-channel (matches the frame_samples convention used by
// OpusEncoder::encode elsewhere in the codebase).
// RNNoise is mono-only; a stereo stream (screen-audio share) is never voice, so skip
// NR there rather than denoise a garbled deinterleave (docs/voice.md §10).
if (stream.recv_ns && dec_channels == 1)
stream.recv_ns->process_capture(stream.decode_scratch.data(), n,
static_cast<int>(params_.sample_rate));
// Receive-side NR runs only on VOICE (MIC) streams — a screen-audio share is music/
// video, never voice, so it's left untouched (gating on dec_channels would silently
// skip a now-stereo mic, see docs/voice.md §10). RNNoise is mono-only, so a stereo mic
// is folded to mono in place (symmetric with the send-side downmix), denoised, then
// duplicated back across both channels — no allocation on this RT path.
if (stream.recv_ns && stream.is_voice) {
int16_t* s = stream.decode_scratch.data();
if (dec_channels == 2) {
for (int i = 0; i < n; ++i) // L/R -> mono, packed into [0..n)
s[i] = static_cast<int16_t>(
(static_cast<int32_t>(s[2 * i]) + static_cast<int32_t>(s[2 * i + 1])) /
2);
stream.recv_ns->process_capture(s, n, static_cast<int>(params_.sample_rate));
for (int i = n - 1; i >= 0; --i) // mono -> both channels (back-to-front)
s[2 * i] = s[2 * i + 1] = s[i];
} else if (dec_channels == 1) {
stream.recv_ns->process_capture(s, n, static_cast<int>(params_.sample_rate));
}
}
// PCM sink: deliver decoded per-stream audio to external consumer (bots,
// transcription, recording) before it enters the hardware mix. Atomic relaxed-

11
core/src/audio/audio_engine.h
View File

@@ -225,9 +225,11 @@ class AudioEngine {
#ifdef VOICECAT_HAS_OPUS
// Configure the Opus decoder for an incoming ssrc (must be called before
// push_recv_frame for that ssrc). user_id/stream_id identify the source for the
// pcm_sink_ callback. Thread-safe.
// pcm_sink_ callback. is_voice marks a MIC stream so the receive-side NR pass knows it may
// denoise it (a stereo mic is folded to mono first); screen-audio shares are never voice.
// Thread-safe.
void init_recv_stream(uint32_t ssrc, const codec::OpusParams& p,
uint32_t user_id, uint32_t stream_id);
uint32_t user_id, uint32_t stream_id, bool is_voice);
#endif
// External PCM tap: callback fired once per decoded Opus frame per remote stream, on the
@@ -475,6 +477,11 @@ class AudioEngine {
bool noise_reduction_enabled = false;
std::unique_ptr<ApmProcessor> recv_ns;
// True for a MIC stream (voice). The receive-side NR pass only denoises voice — a stereo
// mic is folded to mono first (RNNoise is mono-only); a stereo screen-audio share is not
// voice and is left untouched. Set in init_recv_stream. (docs/voice.md §10.)
bool is_voice = false;
// DRED: pre-allocated scratch for loss recovery. dred_state_ is per-stream; see
// AudioEngine::dred_dec_ (shared). Allocated in init_recv_stream(); freed in remove_stream().
#ifdef VOICECAT_HAS_OPUS