fix(audio): wire in-band FEC into the decoder loss path
Some checks failed
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Build Linux Binaries / linux/arm64 (push) Has been cancelled

The encoder set OPUS_SET_INBAND_FEC, but the decoder never invoked FEC --
the loss path went DRED -> PLC, so FEC redundancy was emitted (and paid for
in bitrate) yet never consumed.

Wire FEC recovery into AudioEngine::on_playback between DRED and PLC: copy
the next buffered packet once, try DRED, else (if the stream negotiated FEC)
decode(next_pkt, ..., fec=true), else PLC. Recovery priority is now
DRED -> FEC -> PLC. Add per-stream RemoteStream::fec_enabled_, captured from
OpusParams in init_recv_stream. Docs (voice.md) updated to match.

ctest --preset dev: 27/27 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-22 20:18:29 +02:00
parent ce2035f271
commit 7547b8e140
4 changed files with 62 additions and 29 deletions

View File

@@ -691,6 +691,14 @@ iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sh
reconstructs the lost frame — otherwise falls back to standard PLC. New test: reconstructs the lost frame — otherwise falls back to standard PLC. New test:
`test_dred_toggle` (ctest 22/22). Files: `voicecat.proto`, `voicecat.h`, `test_dred_toggle` (ctest 22/22). Files: `voicecat.proto`, `voicecat.h`,
`opus_codec.{h,cpp}`, `audio_engine.{h,cpp}`, `client.cpp`, `session.{h,cpp}`. `opus_codec.{h,cpp}`, `audio_engine.{h,cpp}`, `client.cpp`, `session.{h,cpp}`.
- [x] **In-band FEC decoder wiring** — done (2026-06-22). The encoder set `OPUS_SET_INBAND_FEC`
all along, but the decoder never invoked it — the loss path went DRED → PLC, so FEC redundancy
was emitted (and paid for in bitrate) but never consumed. Wired the FEC recovery into
`AudioEngine::on_playback`'s loss branch between DRED and PLC: copy the next buffered packet
once, try DRED, else (if the stream negotiated FEC) `decode(next_pkt, …, fec=true)`, else PLC.
Added per-stream `RemoteStream::fec_enabled_`, captured from `OpusParams` in
`init_recv_stream`. Recovery priority is now **DRED → FEC → PLC**. ctest 27/27 green. Files:
`audio_engine.{h,cpp}`, `docs/voice.md`.
- [ ] **DRED toggle in client UIs** — expose the `dred` flag in all three channel-config UIs - [ ] **DRED toggle in client UIs** — expose the `dred` flag in all three channel-config UIs
so admins can enable it per channel. Windows: `ChannelEditForm` / `vc_channel_info.audio.dred` so admins can enable it per channel. Windows: `ChannelEditForm` / `vc_channel_info.audio.dred`
checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs

View File

@@ -597,6 +597,7 @@ void AudioEngine::init_recv_stream(uint32_t ssrc, const codec::OpusParams& p,
auto& stream = streams_[ssrc]; auto& stream = streams_[ssrc];
stream.user_id = user_id; stream.user_id = user_id;
stream.stream_id = stream_id; stream.stream_id = stream_id;
stream.fec_enabled_ = p.fec;
stream.decoder.init(p); stream.decoder.init(p);
// Ring must be sized for this decoder's actual channel/frame-size — see RemoteStream::ring // Ring must be sized for this decoder's actual channel/frame-size — see RemoteStream::ring
// comment in audio_engine.h for why this can't just be the playback callback's frame count. // comment in audio_engine.h for why this can't just be the playback callback's frame count.
@@ -740,17 +741,19 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
sizeof(int16_t)); sizeof(int16_t));
n = frame_samples; n = frame_samples;
} else { } else {
// Try DRED recovery: if the next packet is already in the jitter buffer, // Loss recovery for a missing frame, best-quality first: DRED (Opus 1.6 ML
// parse its DRED extension and reconstruct the lost frame with it — producing // reconstruction) → in-band FEC (the low-bitrate copy of this frame the encoder
// better quality than PLC comfort noise. Falls back to PLC on any failure. // embeds in the next packet) → PLC comfort noise. DRED and FEC both need the
// *next* packet already buffered, so copy it once and try each in turn.
n = -1; n = -1;
#ifdef VOICECAT_HAS_OPUS #ifdef VOICECAT_HAS_OPUS
if (dred_dec_ && stream.dred_state_) {
uint32_t next_ts = stream.playout_ts + static_cast<uint32_t>(frame_samples); uint32_t next_ts = stream.playout_ts + static_cast<uint32_t>(frame_samples);
size_t psz = stream.jitter.try_copy_front_payload( size_t psz = stream.jitter.try_copy_front_payload(
next_ts, stream.dred_payload_scratch_.data(), next_ts, stream.dred_payload_scratch_.data(),
stream.dred_payload_scratch_.size()); stream.dred_payload_scratch_.size());
if (psz > 0) {
// 1. DRED: parse the next packet's deep-redundancy extension and reconstruct.
if (psz > 0 && dred_dec_ && stream.dred_state_) {
int dred_end = 0; int dred_end = 0;
int ret = opus_dred_parse( int ret = opus_dred_parse(
dred_dec_, stream.dred_state_, dred_dec_, stream.dred_state_,
@@ -763,13 +766,23 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
stream.decode_scratch.data(), stream.decode_scratch.data(),
frame_samples); frame_samples);
} }
// 2. In-band FEC: reconstruct the lost frame from the redundant copy carried in
// the next packet (decode_fec=1). Only when FEC is negotiated and the next
// packet is present; if it carries no LBRR data libopus falls back to PLC,
// so this is at worst a no-op relative to the PLC path below.
if (n <= 0 && psz > 0 && stream.fec_enabled_) {
n = stream.decoder.decode(
stream.dred_payload_scratch_.data(), static_cast<int>(psz),
stream.decode_scratch.data(), frame_samples, /*fec=*/true);
} }
#endif #endif
// 3. PLC: synthesize a continuation when no redundancy is available.
if (n <= 0) { if (n <= 0) {
n = stream.decoder.decode(nullptr, 0, stream.decode_scratch.data(), n = stream.decoder.decode(nullptr, 0, stream.decode_scratch.data(),
frame_samples); frame_samples);
} }
if (n > 0) stream.plc_samples_since_real += n; // track PLC streak if (n > 0) stream.plc_samples_since_real += n; // track concealment streak
} }
if (n <= 0) break; // decoder error/exhausted PLC; rest of this period stays silent if (n <= 0) break; // decoder error/exhausted PLC; rest of this period stays silent

View File

@@ -482,6 +482,11 @@ class AudioEngine {
#endif #endif
std::vector<uint8_t> dred_payload_scratch_; // pre-sized to 4000 bytes std::vector<uint8_t> dred_payload_scratch_; // pre-sized to 4000 bytes
// In-band FEC: whether the sender negotiated OPUS_SET_INBAND_FEC for this stream.
// When set, on_playback attempts decode_fec=1 from the next buffered packet to recover a
// lost frame (between DRED and PLC). Captured from OpusParams at init_recv_stream().
bool fec_enabled_ = false;
// Source identity: stored at init_recv_stream() so the pcm_sink_ callback can receive // Source identity: stored at init_recv_stream() so the pcm_sink_ callback can receive
// (user_id, stream_id) without a separate map lookup from the RT playback thread. // (user_id, stream_id) without a separate map lookup from the RT playback thread.
uint32_t user_id = 0; uint32_t user_id = 0;

View File

@@ -125,11 +125,14 @@ Guidance baked into defaults / docs:
Layered, all configurable per channel: Layered, all configurable per channel:
1. **In-band FEC**Opus embeds a low-bitrate copy of the previous frame; the decoder 1. **In-band FEC**the encoder embeds a low-bitrate copy of the current frame in the
recovers a lost packet from the *next* one (costs one frame of latency on recovery). *next* packet (`OPUS_SET_INBAND_FEC`, redundancy scaled by `expected_packet_loss`). On a
Tuned by `expected_packet_loss`. loss, the receiver decodes that copy out of the next already-buffered packet with
`opus_decode(..., decode_fec=1)` — costing one frame of latency on recovery. Gated on the
per-stream `fec` flag; if the next packet carries no redundancy libopus yields PLC output,
so it is at worst a no-op relative to (2).
2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an 2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an
unrecovered loss; always on, free. unrecovered loss; always on, free. The terminal fallback when neither DRED nor FEC applies.
3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise 3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise
updates; cuts bandwidth and is bandwidth-friendly on busy channels. updates; cuts bandwidth and is bandwidth-friendly on busy channels.
4. **DRED (Deep REDundancy, per-channel toggle)** — Opus 1.6's ML redundancy: the encoder 4. **DRED (Deep REDundancy, per-channel toggle)** — Opus 1.6's ML redundancy: the encoder
@@ -137,9 +140,13 @@ Layered, all configurable per channel:
default). When a packet is lost, the receiver peeks at the next already-buffered packet, default). When a packet is lost, the receiver peeks at the next already-buffered packet,
parses its DRED extension (`opus_dred_parse`), and reconstructs the lost frame with parses its DRED extension (`opus_dred_parse`), and reconstructs the lost frame with
`opus_decoder_dred_decode` — producing significantly better audio than PLC comfort noise `opus_decoder_dred_decode` — producing significantly better audio than PLC comfort noise
for single-frame gaps. Falls back silently to PLC if the next packet has not arrived yet for single-frame gaps. Heavier CPU on the encoder (~510 % at 24 kbps); minimal overhead
or if the sender did not embed DRED. Heavier CPU on the encoder (~510 % at 24 kbps); on the decoder (parse is a fast header check on non-DRED packets).
minimal overhead on the decoder (parse is a fast header check on non-DRED packets).
When a frame is lost, `AudioEngine::on_playback` tries these recovery paths in quality order,
falling through on failure: **DRED → in-band FEC → PLC**. DRED and FEC both need the next
packet already buffered (one frame of look-ahead); when it has not arrived yet, recovery falls
straight through to PLC.
## 5. Jitter buffer ## 5. Jitter buffer