Fix audio clock drift and adaptive jitter buffering
.NET port / test (macos-latest) (push) Canceled after 0s
.NET port / test (ubuntu-24.04) (push) Canceled after 0s
.NET port / test (windows-latest) (push) Canceled after 0s
.NET port / apple-client (push) Canceled after 0s

This commit is contained in:
2026-09-20 16:26:03 +02:00
parent 3ee0df11a9
commit dd811a0bb8
23 changed files with 401 additions and 61 deletions
+4 -2
View File
@@ -177,8 +177,10 @@ Noise reduction does not gate speech.
`TimeProvider` timestamps; tests inject a clock. Threshold changes are atomic; all
processing state otherwise has one owner. Codec/DSP processing methods allocate no
managed memory after initialization, verified across 1,000 combined cycles. They run
on a managed worker, never the native real-time device callback. Native device rings,
jitter, mixer, and audio scheduling remain later work.
on a managed worker, never the native real-time device callback. Managed capture/playback
rings compensate independent clock drift around a configurable 20/40/60 ms target. Receive
jitter is duration-aware, reserves one codec frame for DRED/FEC look-ahead, and decodes each
channel packet at its actual duration before the fixed 20 ms mixer stage.
## Initial managed server
+19 -7
View File
@@ -117,6 +117,9 @@ Guidance baked into defaults / docs:
(accumulating two 960-frames for a 40 ms channel, splitting each into two 480-frames for a
10 ms channel, etc.). This keeps the hardware/`vc_stream_feed_pcm` contract a single 48 kHz
clock regardless of the channel's window — see `vc_client::on_capture_frame`.
The device/mixer quantum is not the Opus packet duration: managed send streams reframe it
into the channel's 5/10/20/40/60 ms packets, and receive streams decode at that duration
before slicing decoded PCM back into 20 ms mixer blocks.
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 2432 kbps, DTX on, FEC on.
Music/screen-audio channels → `STEREO`, `AUDIO`, 96128 kbps, DTX off, FEC optional.
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
@@ -152,26 +155,30 @@ straight through to PLC.
## 5. Jitter buffer
Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth playout**
(`core/src/audio/audio_engine.cpp``JitterBuffer` + `AudioEngine::on_playback`).
(`dotnet/src/VoiceCat.Audio/ReceiveStream.cs`).
- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
- **The playout clock is always bounded against the stream's *leading edge* (newest buffered
frame), never re-synced to the oldest.** The clock free-runs at the playback hardware rate,
frame), never re-synced to the oldest.** The managed playout clock is isolated from hardware
drift by the adaptive PCM ring,
while the sender omits VAD/PTT/DTX silence from its timestamps, so the two diverge across gaps
and late joins. Two corrections keep latency bounded:
- **(Re)seed to the leading edge** on first frame, on a talkspurt `marker`, or when the clock
has run past the newest frame (starved after silence). No artificial prebuffer — latency
starts as low as possible; buffered frames still play oldest-first.
has run past the newest frame (starved after silence). Playout retains the adaptive target;
when DRED/FEC is enabled that includes one codec frame of recovery look-ahead.
- **Frame-skip catch-up:** when the backlog grows past `target + hysteresis` (clock drift,
bursty arrival, reordering), fast-forward the clock to leave `target` buffered and drop the
now-stale frames. This is the downward force that prevents latency from ratcheting upward.
- `target` is the adaptive jitter estimate (EWMA of inter-arrival gap vs. the per-frame gap),
floored; silence gaps and reordered stragglers are rejected as outliers so they don't inflate
it. The late-drop window tracks `target` (floored/capped at 500 ms).
- `target` is an EWMA of arrival-gap variation measured in 48 kHz sample time. DRED or FEC
reserves one complete channel Opus frame of look-ahead, variation can raise the target to
120 ms, and packet history is bounded to 500 ms. Limits are durations rather than packet
counts, so 5 ms and 60 ms channels receive the same policy.
- Late frames past the playout point are dropped; gaps are filled by DRED (if the next frame
arrived) or PLC.
- The `marker` flag (start of talkspurt) — set by the sender on the first frame after a
transmission gap — lets the buffer reseed cleanly after silence/DTX without accumulating drift.
- One late device callback does not end a talkspurt. Capture resets only after 200 ms of
continuous starvation, avoiding a marker/rebuffer cascade from an isolated scheduling miss.
- Diagnostics per stream: `packets_lost`, `duplicates`, `underruns`, `target_depth_ms`.
```
@@ -180,6 +187,11 @@ Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth
jitter estimate feeds depth
```
Capture and playback use an allocation-free adaptive PCM ring between the managed 20 ms clock
and the hardware clock. Linear-interpolation correction, limited to ±0.5%, holds the ring near
its target instead of periodically dropping a block or rendering silence as device clocks drift.
The local **Audio buffering** preset is 20, 40 (default), or 60 ms and is persisted per client.
## 6. UDP keepalive & NAT
- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to