Fix audio clock drift and adaptive jitter buffering
This commit is contained in:
+4
-2
@@ -177,8 +177,10 @@ Noise reduction does not gate speech.
|
||||
`TimeProvider` timestamps; tests inject a clock. Threshold changes are atomic; all
|
||||
processing state otherwise has one owner. Codec/DSP processing methods allocate no
|
||||
managed memory after initialization, verified across 1,000 combined cycles. They run
|
||||
on a managed worker, never the native real-time device callback. Native device rings,
|
||||
jitter, mixer, and audio scheduling remain later work.
|
||||
on a managed worker, never the native real-time device callback. Managed capture/playback
|
||||
rings compensate independent clock drift around a configurable 20/40/60 ms target. Receive
|
||||
jitter is duration-aware, reserves one codec frame for DRED/FEC look-ahead, and decodes each
|
||||
channel packet at its actual duration before the fixed 20 ms mixer stage.
|
||||
|
||||
## Initial managed server
|
||||
|
||||
|
||||
+19
-7
@@ -117,6 +117,9 @@ Guidance baked into defaults / docs:
|
||||
(accumulating two 960-frames for a 40 ms channel, splitting each into two 480-frames for a
|
||||
10 ms channel, etc.). This keeps the hardware/`vc_stream_feed_pcm` contract a single 48 kHz
|
||||
clock regardless of the channel's window — see `vc_client::on_capture_frame`.
|
||||
The device/mixer quantum is not the Opus packet duration: managed send streams reframe it
|
||||
into the channel's 5/10/20/40/60 ms packets, and receive streams decode at that duration
|
||||
before slicing decoded PCM back into 20 ms mixer blocks.
|
||||
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 24–32 kbps, DTX on, FEC on.
|
||||
Music/screen-audio channels → `STEREO`, `AUDIO`, 96–128 kbps, DTX off, FEC optional.
|
||||
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
|
||||
@@ -152,26 +155,30 @@ straight through to PLC.
|
||||
## 5. Jitter buffer
|
||||
|
||||
Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth playout**
|
||||
(`core/src/audio/audio_engine.cpp` — `JitterBuffer` + `AudioEngine::on_playback`).
|
||||
(`dotnet/src/VoiceCat.Audio/ReceiveStream.cs`).
|
||||
|
||||
- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
|
||||
- **The playout clock is always bounded against the stream's *leading edge* (newest buffered
|
||||
frame), never re-synced to the oldest.** The clock free-runs at the playback hardware rate,
|
||||
frame), never re-synced to the oldest.** The managed playout clock is isolated from hardware
|
||||
drift by the adaptive PCM ring,
|
||||
while the sender omits VAD/PTT/DTX silence from its timestamps, so the two diverge across gaps
|
||||
and late joins. Two corrections keep latency bounded:
|
||||
- **(Re)seed to the leading edge** on first frame, on a talkspurt `marker`, or when the clock
|
||||
has run past the newest frame (starved after silence). No artificial prebuffer — latency
|
||||
starts as low as possible; buffered frames still play oldest-first.
|
||||
has run past the newest frame (starved after silence). Playout retains the adaptive target;
|
||||
when DRED/FEC is enabled that includes one codec frame of recovery look-ahead.
|
||||
- **Frame-skip catch-up:** when the backlog grows past `target + hysteresis` (clock drift,
|
||||
bursty arrival, reordering), fast-forward the clock to leave `target` buffered and drop the
|
||||
now-stale frames. This is the downward force that prevents latency from ratcheting upward.
|
||||
- `target` is the adaptive jitter estimate (EWMA of inter-arrival gap vs. the per-frame gap),
|
||||
floored; silence gaps and reordered stragglers are rejected as outliers so they don't inflate
|
||||
it. The late-drop window tracks `target` (floored/capped at 500 ms).
|
||||
- `target` is an EWMA of arrival-gap variation measured in 48 kHz sample time. DRED or FEC
|
||||
reserves one complete channel Opus frame of look-ahead, variation can raise the target to
|
||||
120 ms, and packet history is bounded to 500 ms. Limits are durations rather than packet
|
||||
counts, so 5 ms and 60 ms channels receive the same policy.
|
||||
- Late frames past the playout point are dropped; gaps are filled by DRED (if the next frame
|
||||
arrived) or PLC.
|
||||
- The `marker` flag (start of talkspurt) — set by the sender on the first frame after a
|
||||
transmission gap — lets the buffer reseed cleanly after silence/DTX without accumulating drift.
|
||||
- One late device callback does not end a talkspurt. Capture resets only after 200 ms of
|
||||
continuous starvation, avoiding a marker/rebuffer cascade from an isolated scheduling miss.
|
||||
- Diagnostics per stream: `packets_lost`, `duplicates`, `underruns`, `target_depth_ms`.
|
||||
|
||||
```
|
||||
@@ -180,6 +187,11 @@ Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth
|
||||
jitter estimate feeds depth
|
||||
```
|
||||
|
||||
Capture and playback use an allocation-free adaptive PCM ring between the managed 20 ms clock
|
||||
and the hardware clock. Linear-interpolation correction, limited to ±0.5%, holds the ring near
|
||||
its target instead of periodically dropping a block or rendering silence as device clocks drift.
|
||||
The local **Audio buffering** preset is 20, 40 (default), or 60 ms and is persisted per client.
|
||||
|
||||
## 6. UDP keepalive & NAT
|
||||
|
||||
- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to
|
||||
|
||||
Reference in New Issue
Block a user