fix(ios): pace mic feed with a prebuffer cushion to stop flutter/crackle
The iOS mic was unusable — a consistent ~40-60ms flutter + volume fade
('slow fan') on every preset. The core sends each captured frame
synchronously (no send pacer), so packet cadence == capture cadence, and
the receiver's playout keeps near-zero buffering by design (its jitter
estimate keys off the regular sender timestamp, so it's blind to arrival
jitter). That's smooth only for a steady sender (desktop miniaudio =
steady 20ms); the iOS AVAudioEngine tap delivers ~2 frames per ~40ms
callback -> bursty -> receiver underruns -> PLC fade.
Fix (iOS-only): the mic tap writes converted 48kHz int16 to an SPSC ring;
a 20ms feed pump drains it and calls feedPcm at a steady cadence. The pump
primes a small prebuffer cushion (3 frames ~60ms, self-healing up to
~120ms on underrun) before releasing, so the tap's bursts can't drain it
to empty. Never reads a partial frame (read consumes what it returns ->
partials were the crackle), and rebuilds with the current channel count
each rebuild() (a frozen count fed mono-as-stereo = octave-up on a
Stereo->Voice Chat switch).
Trade-off: ~60-120ms added mic-send latency, unavoidable when de-bursting
for a near-zero-buffer receiver. PROGRESS.md notes the proper follow-up:
make the jitter buffer measure real RFC-3550 arrival jitter so the
receiver absorbs bursts itself.
Verified: xcodebuild Debug BUILD SUCCEEDED (iOS Simulator, arm64).
This commit is contained in:
40
PROGRESS.md
40
PROGRESS.md
@@ -10,6 +10,46 @@ up instantly. Newest status at the top.
|
|||||||
|
|
||||||
## ▶ Where we left off / next action
|
## ▶ Where we left off / next action
|
||||||
|
|
||||||
|
- **[ ] Soon — jitter buffer should measure REAL arrival jitter (RFC 3550), not sender
|
||||||
|
timestamps.** `JitterBuffer::push` (`core/src/audio/audio_engine.cpp:84-108`) estimates
|
||||||
|
jitter from `gap = ts - last_push_ts_`, where `ts` is the **sender's timestamp** — which is
|
||||||
|
perfectly regular (`ls.timestamp += samples` every frame, independent of when the packet is
|
||||||
|
actually sent). So `diff` is always ~0, `jitter_est_` stays 0, and `target_depth_ms_` is
|
||||||
|
pinned at its ~20 ms floor. The buffer is therefore **blind to real network/arrival jitter
|
||||||
|
and to bursty senders** — it never deepens. Combined with the playout deliberately seeding
|
||||||
|
to near-zero depth (`on_playback`, ~line 715), the receiver tolerates only a *steady*
|
||||||
|
sender. This is exactly why the iOS mic needed a send-side pacing cushion (below) and why
|
||||||
|
genuine network jitter would also cause underruns. **Fix:** measure inter-arrival jitter
|
||||||
|
the RFC 3550 way — `D = (arrival_j - arrival_i) - (ts_j - ts_i)` using a wall-clock arrival
|
||||||
|
stamp captured in `push()` — and drive `target_depth_ms_` off that EWMA (keep the existing
|
||||||
|
marker/silence-gap outlier rejection). Then the receiver absorbs bursts itself and the iOS
|
||||||
|
send cushion could be reduced or removed. Shared-core change → add a test and re-verify
|
||||||
|
desktop↔desktop stays low-latency (steady sender ⇒ ~0 arrival jitter ⇒ no regression).
|
||||||
|
|
||||||
|
- **Done (2026-06-23):** **Fixed iOS mic flutter / crackle / octave-up.** The iOS mic was
|
||||||
|
unusable: a consistent ~40–60 ms flutter with volume fade ("talking through a slow fan") on
|
||||||
|
every preset. Root cause: the core sends each captured frame **synchronously**
|
||||||
|
(`on_capture_frame` → `encode_and_send_frame`, no send pacer), so packet cadence == capture
|
||||||
|
cadence; and the receiver's playout keeps **near-zero buffering** by design and its jitter
|
||||||
|
estimate is blind to arrival timing (see RFC-3550 item above). That's smooth only for a
|
||||||
|
*steady* sender (desktop miniaudio = steady 20 ms), but the iOS `AVAudioEngine` input tap
|
||||||
|
delivers ~2 frames per ~40 ms callback (more under VPIO) → bursty → receiver underruns → PLC
|
||||||
|
fade.
|
||||||
|
- **Fix (iOS-only, `clients/apple/iOS/VoiceCatiOS/IOSVoiceProcessingEngine.swift`):** the
|
||||||
|
mic tap converts to 48 kHz int16 and writes a lock-free SPSC ring; a 20 ms feed pump
|
||||||
|
drains it and calls `feedPcm` at a **steady** cadence so packets leave the core every
|
||||||
|
20 ms (what the receiver expects). The pump **primes a small prebuffer cushion**
|
||||||
|
(`PumpState.targetFrames`, 3 frames ≈ 60 ms, self-healing up to ~120 ms on underrun)
|
||||||
|
before releasing, so the tap's bursts can't drain it to empty. Two correctness rules
|
||||||
|
(each had bit us): never read a partial frame (`read` consumes what it returns →
|
||||||
|
discarding partials caused crackle), and rebuild the pump with the current channel count
|
||||||
|
every `rebuild()` (a frozen channel count fed mono-as-stereo = octave-up on a Stereo→Voice
|
||||||
|
Chat switch). Trade-off: ~60–120 ms added mic-send latency — unavoidable when de-bursting
|
||||||
|
for a near-zero-buffer receiver; the RFC-3550 fix above would let us shrink it.
|
||||||
|
- **Verify:** `xcodebuild` Debug **BUILD SUCCEEDED** (iOS Simulator, arm64). Audible test
|
||||||
|
requires a real device (simulator has no real mic route): mic should be smooth on Voice
|
||||||
|
Chat / Mono Mic / Stereo Mic, including switching presets while live (no octave).
|
||||||
|
|
||||||
- **Done (2026-06-23):** **Fixed Apple client link failure (stale xcframework missing
|
- **Done (2026-06-23):** **Fixed Apple client link failure (stale xcframework missing
|
||||||
RNNoise).** Both `VoiceCatMac` and `VoiceCatiOS` failed to link with `Undefined symbols for
|
RNNoise).** Both `VoiceCatMac` and `VoiceCatiOS` failed to link with `Undefined symbols for
|
||||||
architecture arm64: _rnnoise_create / _rnnoise_destroy / _rnnoise_process_frame`. Root
|
architecture arm64: _rnnoise_create / _rnnoise_destroy / _rnnoise_process_frame`. Root
|
||||||
|
|||||||
@@ -73,6 +73,16 @@ final class PCMRing {
|
|||||||
return n
|
return n
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Consumer-side snapshot of how many interleaved int16 samples are currently buffered. Lets a
|
||||||
|
/// paced consumer check for a full frame *before* calling `read`, so it never reads (and thus
|
||||||
|
/// discards) a partial frame. Single consumer only (same thread that calls `read`).
|
||||||
|
var availableSamples: Int {
|
||||||
|
let r = readIdx
|
||||||
|
OSMemoryBarrier()
|
||||||
|
let w = writeIdx
|
||||||
|
return Int(w &- r)
|
||||||
|
}
|
||||||
|
|
||||||
/// Discard everything buffered — call before (re)starting so stale pre-roll isn't played.
|
/// Discard everything buffered — call before (re)starting so stale pre-roll isn't played.
|
||||||
func reset() { OSMemoryBarrier(); readIdx = writeIdx }
|
func reset() { OSMemoryBarrier(); readIdx = writeIdx }
|
||||||
|
|
||||||
@@ -94,7 +104,9 @@ final class PCMRing {
|
|||||||
/// from it and renders through the engine output. This runs the whole time we're connected,
|
/// from it and renders through the engine output. This runs the whole time we're connected,
|
||||||
/// so remote audio plays even before the user joins voice (no "can't hear anyone").
|
/// so remote audio plays even before the user joins voice (no "can't hear anyone").
|
||||||
/// - **mic → core:** when the mic is active a tap on the input node converts to 48 kHz int16 and
|
/// - **mic → core:** when the mic is active a tap on the input node converts to 48 kHz int16 and
|
||||||
/// calls `client.feedPcm(micStreamId)`.
|
/// writes to a pacing ring; a 20 ms timer releases steady 960-sample frames to
|
||||||
|
/// `client.feedPcm(micStreamId)`. The core sends each captured frame synchronously, so this
|
||||||
|
/// steady cadence is what keeps packets from bursting and fluttering the receiver's playout.
|
||||||
///
|
///
|
||||||
/// Echo cancellation / noise suppression / AGC come from Apple's Voice-Processing I/O unit (VPIO),
|
/// Echo cancellation / noise suppression / AGC come from Apple's Voice-Processing I/O unit (VPIO),
|
||||||
/// which `inputNode.setVoiceProcessingEnabled(true)` enables. VPIO forces mono, so it is engaged
|
/// which `inputNode.setVoiceProcessingEnabled(true)` enables. VPIO forces mono, so it is engaged
|
||||||
@@ -119,6 +131,45 @@ final class IOSAudioEngine {
|
|||||||
private var micStreamId: UInt32 = 0
|
private var micStreamId: UInt32 = 0
|
||||||
private var captureChannels: UInt32 = 1
|
private var captureChannels: UInt32 = 1
|
||||||
|
|
||||||
|
// Mic feed pacing. The core sends each captured frame SYNCHRONOUSLY as it arrives
|
||||||
|
// (on_capture_frame → encode → sendto, client.cpp) — there is no send pacer in the core. On
|
||||||
|
// desktop miniaudio capture fires one 960-sample frame every 20 ms, so packets leave at a
|
||||||
|
// steady 20 ms. On iOS the AVAudioEngine input tap fires at the hardware IO-buffer period
|
||||||
|
// (often ~40 ms under VPIO), delivering ~2 frames at once: feeding those straight to the core
|
||||||
|
// bursts 2 packets out then goes quiet for ~40 ms, and the receiver's ~40 ms jitter buffer
|
||||||
|
// underruns on every gap → PLC fade ("talking through a slow fan" + ~40–60 ms flutter).
|
||||||
|
//
|
||||||
|
// Fix: pace the feed to a steady 20 ms. The tap converts to int16 and writes to a lock-free
|
||||||
|
// SPSC ring (producer, audio clock); a 20 ms timer releases ONE 960-sample frame per tick to
|
||||||
|
// feedPcm (consumer). The producer's average rate is locked to 48 kHz = exactly one frame per
|
||||||
|
// 20 ms, so it matches the consumer; the ring just absorbs the tap's 2-at-a-time bursts.
|
||||||
|
//
|
||||||
|
// Two correctness rules learned the hard way (these caused the earlier crackle + octave):
|
||||||
|
// 1. NEVER read a partial frame — `read` consumes whatever it returns, so reading <960 would
|
||||||
|
// silently discard those samples (crackle). The timer checks `availableSamples` first and
|
||||||
|
// only reads when a full frame is present; an underrun just skips the tick (nothing lost).
|
||||||
|
// 2. NEVER freeze the channel count in the timer — mono↔stereo preset switches change it. The
|
||||||
|
// timer is torn down and recreated inside `rebuild()`, so it always captures the current
|
||||||
|
// `captureChannels`; the ring is reset while the timer is stopped (no cross-thread race).
|
||||||
|
private let micRing = PCMRing(capacitySamples: 48000 * 2) // ~1 s stereo — ample elastic slack
|
||||||
|
private var micTimer: DispatchSourceTimer?
|
||||||
|
private let micQueue = DispatchQueue(label: "cat.voice.mic.feedPump")
|
||||||
|
private let micDrainScratch: UnsafeMutablePointer<Int16>
|
||||||
|
private static let micFrameSamplesPerChannel = 960 // 20 ms @ 48 kHz — core's frame size
|
||||||
|
|
||||||
|
/// Feed-pump state, touched only on `micQueue` (the pump's serial queue). A reference type so
|
||||||
|
/// the timer closure mutates it without capturing `self` (which is @MainActor). `targetFrames`
|
||||||
|
/// is the prebuffer depth: the pump fills this many frames before it starts releasing, so the
|
||||||
|
/// tap's bursty delivery (~2 frames at once) can't drain it to empty between bursts. It persists
|
||||||
|
/// across rebuilds and self-heals upward (capped) on an underrun, so it tunes to whatever IO
|
||||||
|
/// buffer size the active route/VPIO actually uses without a hard-coded guess.
|
||||||
|
private final class PumpState {
|
||||||
|
var primed = false
|
||||||
|
var targetFrames = 3 // ~60 ms initial cushion; grows on underrun up to maxTargetFrames
|
||||||
|
static let maxTargetFrames = 6 // ~120 ms cap — bounds added latency
|
||||||
|
}
|
||||||
|
private let pumpState = PumpState()
|
||||||
|
|
||||||
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders. The core
|
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders. The core
|
||||||
// delivers 48 kHz stereo int16 via the mixed-output sink; mainMixerNode adapts to the route.
|
// delivers 48 kHz stereo int16 via the mixed-output sink; mainMixerNode adapts to the route.
|
||||||
private let outFormat = AVAudioFormat(
|
private let outFormat = AVAudioFormat(
|
||||||
@@ -134,6 +185,8 @@ final class IOSAudioEngine {
|
|||||||
private init() {
|
private init() {
|
||||||
renderScratch = UnsafeMutablePointer<Int16>.allocate(capacity: renderScratchFrames * 2)
|
renderScratch = UnsafeMutablePointer<Int16>.allocate(capacity: renderScratchFrames * 2)
|
||||||
renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2)
|
renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2)
|
||||||
|
micDrainScratch = UnsafeMutablePointer<Int16>.allocate(capacity: 960 * 2)
|
||||||
|
micDrainScratch.initialize(repeating: 0, count: 960 * 2)
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Lifecycle
|
// MARK: - Lifecycle
|
||||||
@@ -164,6 +217,7 @@ final class IOSAudioEngine {
|
|||||||
func stop() {
|
func stop() {
|
||||||
guard isConnected else { return }
|
guard isConnected else { return }
|
||||||
micActive = false
|
micActive = false
|
||||||
|
stopMicTimer()
|
||||||
isConnected = false
|
isConnected = false
|
||||||
client?.setMixedOutputSink(nil, user: nil)
|
client?.setMixedOutputSink(nil, user: nil)
|
||||||
engine.inputNode.removeTap(onBus: 0)
|
engine.inputNode.removeTap(onBus: 0)
|
||||||
@@ -175,6 +229,7 @@ final class IOSAudioEngine {
|
|||||||
sourceNode = nil
|
sourceNode = nil
|
||||||
}
|
}
|
||||||
ring.reset()
|
ring.reset()
|
||||||
|
micRing.reset()
|
||||||
client = nil
|
client = nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -219,6 +274,9 @@ final class IOSAudioEngine {
|
|||||||
/// AVAudioSession config (category/mode/route) first (`IOSAudioRouter.applyConfiguration`).
|
/// AVAudioSession config (category/mode/route) first (`IOSAudioRouter.applyConfiguration`).
|
||||||
private func rebuild() {
|
private func rebuild() {
|
||||||
guard isConnected else { return }
|
guard isConnected else { return }
|
||||||
|
// Stop the feed pump before touching the tap / ring so the timer (on micQueue) can't race
|
||||||
|
// the ring reset in installMicTap. It is restarted at the end with the current channel count.
|
||||||
|
stopMicTimer()
|
||||||
if engine.isRunning { engine.stop() }
|
if engine.isRunning { engine.stop() }
|
||||||
engine.inputNode.removeTap(onBus: 0)
|
engine.inputNode.removeTap(onBus: 0)
|
||||||
|
|
||||||
@@ -252,6 +310,10 @@ final class IOSAudioEngine {
|
|||||||
} catch {
|
} catch {
|
||||||
logger.error("engine start failed: \(error.localizedDescription)")
|
logger.error("engine start failed: \(error.localizedDescription)")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Start the feed pump last, with the current channel count, so it never carries a stale
|
||||||
|
// (frozen) channel count across a mono↔stereo switch.
|
||||||
|
if micActive { startMicTimer() }
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Detach any previous source node and attach a fresh one pulling mixed PCM from the ring.
|
/// Detach any previous source node and attach a fresh one pulling mixed PCM from the ring.
|
||||||
@@ -286,10 +348,17 @@ final class IOSAudioEngine {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Install the mic tap: convert the input node's native format to 48 kHz int16 (mono or
|
/// Install the mic tap: convert the input node's native format to 48 kHz int16 (mono or
|
||||||
/// stereo per `captureChannels`) and feed it to the core. Rebuilds the converter each time
|
/// stereo per `captureChannels`) and write it to the pacing ring. The 20 ms feed pump
|
||||||
/// because the input format depends on the VPIO state and the active route.
|
/// (`startMicTimer`) releases steady 960-sample frames to `feedPcm` — see the mic-feed comment
|
||||||
|
/// above for why the tap must NOT call feedPcm directly (it bursts packets → receiver flutter).
|
||||||
|
/// Rebuilds the converter each time because the input format depends on the VPIO state and the
|
||||||
|
/// active route.
|
||||||
private func installMicTap() {
|
private func installMicTap() {
|
||||||
guard let client else { return }
|
guard client != nil else { return }
|
||||||
|
// Fresh ring on every (re)install — a rebuild must not feed stale pre-roll into the new tap.
|
||||||
|
// Safe here: the feed pump was stopped at the top of rebuild(), so no consumer is running.
|
||||||
|
micRing.reset()
|
||||||
|
let ring = micRing // captured by the closure as a `let` — no self capture (see mic-feed comment)
|
||||||
let inFormat = engine.inputNode.outputFormat(forBus: 0)
|
let inFormat = engine.inputNode.outputFormat(forBus: 0)
|
||||||
guard inFormat.sampleRate > 0 else {
|
guard inFormat.sampleRate > 0 else {
|
||||||
logger.error("input format unavailable (\(inFormat)) — mic will not transmit")
|
logger.error("input format unavailable (\(inFormat)) — mic will not transmit")
|
||||||
@@ -303,8 +372,7 @@ final class IOSAudioEngine {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
let sid = micStreamId
|
let chInt = Int(targetCh)
|
||||||
let c = client
|
|
||||||
engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in
|
engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in
|
||||||
// Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample.
|
// Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample.
|
||||||
let ratio = target.sampleRate / buffer.format.sampleRate
|
let ratio = target.sampleRate / buffer.format.sampleRate
|
||||||
@@ -319,9 +387,68 @@ final class IOSAudioEngine {
|
|||||||
}
|
}
|
||||||
guard status != .error, outBuf.frameLength > 0,
|
guard status != .error, outBuf.frameLength > 0,
|
||||||
let chData = outBuf.int16ChannelData else { return }
|
let chData = outBuf.int16ChannelData else { return }
|
||||||
// int16 interleaved → channelData[0] is the interleaved buffer.
|
// int16 interleaved → channelData[0] is the interleaved buffer. Write the converter's
|
||||||
c.feedPcm(streamId: sid, pcm: chData[0],
|
// variable-length output to the pacing ring; the 20 ms feed pump releases steady
|
||||||
samplesPerChannel: Int(outBuf.frameLength), channels: targetCh)
|
// 960-sample frames to feedPcm so packets leave the core at a steady 20 ms cadence.
|
||||||
|
ring.write(chData[0], count: Int(outBuf.frameLength) * chInt)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// MARK: - Mic feed pump (paces feedPcm at a steady 20 ms cadence)
|
||||||
|
|
||||||
|
/// Start the 20 ms feed pump. After priming a small cushion (`pumpState.targetFrames`), it
|
||||||
|
/// releases ONE 960-sample frame per tick from `micRing` to `feedPcm`, so the core (which sends
|
||||||
|
/// synchronously per captured frame) emits packets at a steady 20 ms — the cadence its receivers
|
||||||
|
/// expect. The cushion is essential: the receiver's playout deliberately keeps near-zero
|
||||||
|
/// buffering (low latency), so it tolerates a steady stream but not bursts; the iOS tap delivers
|
||||||
|
/// ~2 frames at once, and without the cushion the pump runs at ~0 depth and underruns on every
|
||||||
|
/// tap/timer phase beat (crackle). Recreated on every `rebuild()` so `ch` always reflects the
|
||||||
|
/// current `captureChannels` (mono↔stereo switches). Captures only locals + the reference-type
|
||||||
|
/// ring/client/state (no `self`, which is @MainActor).
|
||||||
|
private func startMicTimer() {
|
||||||
|
stopMicTimer()
|
||||||
|
guard let client else { return }
|
||||||
|
let ring = micRing
|
||||||
|
let scratch = micDrainScratch
|
||||||
|
let state = pumpState
|
||||||
|
let sid = micStreamId
|
||||||
|
let ch = max(1, min(2, Int(captureChannels)))
|
||||||
|
let frameSamples = Self.micFrameSamplesPerChannel
|
||||||
|
let full = frameSamples * ch
|
||||||
|
let chU32 = UInt32(ch)
|
||||||
|
// The ring was just reset in installMicTap, so the cushion must be refilled before sending.
|
||||||
|
state.primed = false
|
||||||
|
let feed: () -> Void = {
|
||||||
|
_ = ring.read(into: scratch, count: full) // caller guarantees a full frame is present
|
||||||
|
_ = client.feedPcm(streamId: sid, pcm: scratch,
|
||||||
|
samplesPerChannel: frameSamples, channels: chU32)
|
||||||
|
}
|
||||||
|
let t = DispatchSource.makeTimerSource(queue: micQueue)
|
||||||
|
t.schedule(deadline: .now(), repeating: .milliseconds(20), leeway: .milliseconds(2))
|
||||||
|
t.setEventHandler {
|
||||||
|
let frames = ring.availableSamples / full // whole frames currently buffered
|
||||||
|
if !state.primed {
|
||||||
|
if frames < state.targetFrames { return } // still filling the cushion (into silence)
|
||||||
|
state.primed = true
|
||||||
|
} else if frames == 0 {
|
||||||
|
// Underrun: the cushion drained. Grow it (capped) so it won't recur, then re-prime.
|
||||||
|
// Never read a partial frame — `read` consumes what it returns, so that would
|
||||||
|
// discard samples (the old crackle bug); skipping loses nothing, the samples wait.
|
||||||
|
if state.targetFrames < PumpState.maxTargetFrames { state.targetFrames += 1 }
|
||||||
|
state.primed = false
|
||||||
|
return
|
||||||
|
}
|
||||||
|
feed() // one steady frame per tick (frames >= 1 here)
|
||||||
|
// Catch-up: if the backlog grew past the cushion (pump descheduled, or producer ran
|
||||||
|
// ahead via a burst), release one extra frame to drain it and keep latency bounded.
|
||||||
|
if frames - 1 > state.targetFrames + 1 { feed() }
|
||||||
|
}
|
||||||
|
t.resume()
|
||||||
|
micTimer = t
|
||||||
|
}
|
||||||
|
|
||||||
|
private func stopMicTimer() {
|
||||||
|
micTimer?.cancel()
|
||||||
|
micTimer = nil
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user