Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(audio): wire in-band FEC into the decoder loss path
Some checks failed
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Details
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Details

Browse Source 
The encoder set OPUS_SET_INBAND_FEC, but the decoder never invoked FEC --
the loss path went DRED -> PLC, so FEC redundancy was emitted (and paid for
in bitrate) yet never consumed.

Wire FEC recovery into AudioEngine::on_playback between DRED and PLC: copy
the next buffered packet once, try DRED, else (if the stream negotiated FEC)
decode(next_pkt, ..., fec=true), else PLC. Recovery priority is now
DRED -> FEC -> PLC. Add per-stream RemoteStream::fec_enabled_, captured from
OpusParams in init_recv_stream. Docs (voice.md) updated to match.

ctest --preset dev: 27/27 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Talon
2026-06-22 20:18:29 +02:00
parent ce2035f271
commit 7547b8e140
4 changed files with 62 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
core/src/audio/audio_engine.cpp
View File

@@ -597,6 +597,7 @@ void AudioEngine::init_recv_stream(uint32_t ssrc, const codec::OpusParams& p,
auto& stream = streams_[ssrc];
stream.user_id = user_id;
stream.stream_id = stream_id;
stream.fec_enabled_ = p.fec;
stream.decoder.init(p);
// Ring must be sized for this decoder's actual channel/frame-size — see RemoteStream::ring
// comment in audio_engine.h for why this can't just be the playback callback's frame count.
@@ -740,36 +741,48 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
sizeof(int16_t));
n = frame_samples;
} else {
// Try DRED recovery: if the next packet is already in the jitter buffer,
// parse its DRED extension and reconstruct the lost frame with it — producing
// better quality than PLC comfort noise. Falls back to PLC on any failure.
// Loss recovery for a missing frame, best-quality first: DRED (Opus 1.6 ML
// reconstruction) → in-band FEC (the low-bitrate copy of this frame the encoder
// embeds in the next packet) → PLC comfort noise. DRED and FEC both need the
// *next* packet already buffered, so copy it once and try each in turn.
n = -1;
#ifdef VOICECAT_HAS_OPUS
if (dred_dec_ && stream.dred_state_) {
uint32_t next_ts = stream.playout_ts + static_cast<uint32_t>(frame_samples);
size_t psz = stream.jitter.try_copy_front_payload(
next_ts, stream.dred_payload_scratch_.data(),
stream.dred_payload_scratch_.size());
if (psz > 0) {
int dred_end = 0;
int ret = opus_dred_parse(
dred_dec_, stream.dred_state_,
stream.dred_payload_scratch_.data(),
static_cast<opus_int32>(psz),
frame_samples, static_cast<opus_int32>(params_.sample_rate),
&dred_end, 0);
if (ret > 0)
n = stream.decoder.decode_dred(stream.dred_state_, 0,
stream.decode_scratch.data(),
frame_samples);
}
uint32_t next_ts = stream.playout_ts + static_cast<uint32_t>(frame_samples);
size_t psz = stream.jitter.try_copy_front_payload(
next_ts, stream.dred_payload_scratch_.data(),
stream.dred_payload_scratch_.size());
// 1. DRED: parse the next packet's deep-redundancy extension and reconstruct.
if (psz > 0 && dred_dec_ && stream.dred_state_) {
int dred_end = 0;
int ret = opus_dred_parse(
dred_dec_, stream.dred_state_,
stream.dred_payload_scratch_.data(),
static_cast<opus_int32>(psz),
frame_samples, static_cast<opus_int32>(params_.sample_rate),
&dred_end, 0);
if (ret > 0)
n = stream.decoder.decode_dred(stream.dred_state_, 0,
stream.decode_scratch.data(),
frame_samples);
}
// 2. In-band FEC: reconstruct the lost frame from the redundant copy carried in
// the next packet (decode_fec=1). Only when FEC is negotiated and the next
// packet is present; if it carries no LBRR data libopus falls back to PLC,
// so this is at worst a no-op relative to the PLC path below.
if (n <= 0 && psz > 0 && stream.fec_enabled_) {
n = stream.decoder.decode(
stream.dred_payload_scratch_.data(), static_cast<int>(psz),
stream.decode_scratch.data(), frame_samples, /*fec=*/true);
}
#endif
// 3. PLC: synthesize a continuation when no redundancy is available.
if (n <= 0) {
n = stream.decoder.decode(nullptr, 0, stream.decode_scratch.data(),
frame_samples);
}
if (n > 0) stream.plc_samples_since_real += n; // track PLC streak
if (n > 0) stream.plc_samples_since_real += n; // track concealment streak
}
if (n <= 0) break; // decoder error/exhausted PLC; rest of this period stays silent

5
core/src/audio/audio_engine.h
View File

@@ -482,6 +482,11 @@ class AudioEngine {
#endif
std::vector<uint8_t> dred_payload_scratch_; // pre-sized to 4000 bytes
// In-band FEC: whether the sender negotiated OPUS_SET_INBAND_FEC for this stream.
// When set, on_playback attempts decode_fec=1 from the next buffered packet to recover a
// lost frame (between DRED and PLC). Captured from OpusParams at init_recv_stream().
bool fec_enabled_ = false;
// Source identity: stored at init_recv_stream() so the pcm_sink_ callback can receive
// (user_id, stream_id) without a separate map lookup from the RT playback thread.
uint32_t user_id = 0;