fix(ios-audio): unify iOS audio onto one always-external AVAudioEngine
The iOS audio path was a hybrid: Voice-Chat-class presets ran a native VPIO AVAudioEngine (core external) while Stereo/Studio/A2DP presets ran the core's miniaudio devices. Nearly every "no input / no output / both" bug lived in the seam between the two paths — the lingering miniaudio capture unit fighting VPIO, the audioRestart ordering dance, the route-change "glitching" loop, stereo<->mono stickiness, and "can't hear anyone". Switching presets/routes mid-call routinely dropped a direction. Drive ALL iOS audio through one AVAudioEngine with the core fully external at all times: setExternalPlayback(1) once at connect, every MIC stream external_feed=1, mic via vc_stream_feed_pcm, playback via vc_set_mixed_output_sink (drained by an always-on AVAudioSourceNode so remote audio plays before joining voice). VPIO + AGC toggle per preset. Every preset/route/interruption change funnels through one deterministic Swift-only reconfigure (stop -> apply session config -> rebuild -> start) — no second path to hand off to, so a change can't drop a direction. - IOSVoiceProcessingEngine.swift -> IOSAudioEngine: always-on source-node playback, conditional mic tap, VPIO/AGC; one rebuild() backing startListening/stop/startMic/stopMic/reconfigure/setCaptureChannels. - IOSAudioRouter: 7 presets -> 4 (Voice Chat / Stereo Mic / Mono Mic / Advanced); persisted voiceProcessingEnabled + agcEnabled; setters call IOSAudioEngine.reconfigure() instead of audioRestart/reconcileVoicePath. - AudioSessionManager slimmed; SessionState mic lifecycle collapsed; AppState wires external playback + listening at connect, stop at disconnect; SettingsView shows 4 presets + Advanced VPIO/AGC toggles. No core/ABI/test changes — relies on the already-shipped external API (test_external_pcm, test_external_playback). xcodebuild iOS device Debug BUILD SUCCEEDED. Updates docs/voice.md §8 and PROGRESS.md.
This commit is contained in:
@@ -3,9 +3,9 @@ import Darwin
|
||||
import os
|
||||
import VoiceCatCore
|
||||
|
||||
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSVoiceProcessingEngine")
|
||||
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioEngine")
|
||||
|
||||
/// In-process single-producer/single-consumer int16 PCM ring for the VPIO playback path.
|
||||
/// In-process single-producer/single-consumer int16 PCM ring for the playback path.
|
||||
///
|
||||
/// producer = the core's mixer-timer thread (the `vc_set_mixed_output_sink` callback)
|
||||
/// consumer = the `AVAudioSourceNode` render thread
|
||||
@@ -79,38 +79,48 @@ final class PCMRing {
|
||||
/// Diagnostics: monotonic total samples written / read since the ring was created. The
|
||||
/// indices are already cumulative, so these are free. Only read them when both threads are
|
||||
/// quiesced (e.g. at teardown after the engine + mixer sink are stopped) — they are not
|
||||
/// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" (Bug 1
|
||||
/// core path) apart from "PCM arrived but produced no sound" (AVAudioEngine output graph).
|
||||
/// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" apart
|
||||
/// from "PCM arrived but produced no sound" (the AVAudioEngine output graph).
|
||||
var debugTotalWritten: UInt64 { writeIdx }
|
||||
var debugTotalRead: UInt64 { readIdx }
|
||||
}
|
||||
|
||||
/// Native iOS voice-processing audio path (docs/voice.md §8 "iOS voice processing").
|
||||
/// The single iOS audio engine (docs/voice.md §8 "iOS audio engine").
|
||||
///
|
||||
/// Real iOS echo cancellation / noise suppression / AGC come ONLY from Apple's Voice-Processing
|
||||
/// I/O unit (VPIO), which `AVAudioEngine.setVoiceProcessingEnabled(true)` enables. For VPIO to
|
||||
/// cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
|
||||
/// played-back signal from the mic), so on the AEC presets this engine drives both directions and
|
||||
/// the core runs in external mode (no hardware devices):
|
||||
/// - **Mic → core:** a tap on the VPIO input node → 48 kHz int16 → `client.feedPcm(micStreamId)`.
|
||||
/// **One path, always external.** On iOS the core never opens a miniaudio device: a MIC stream is
|
||||
/// always started with `external_feed=1`, `vc_set_external_playback(1)` is set once at connect, and
|
||||
/// this engine drives *both* directions through one `AVAudioEngine`:
|
||||
/// - **core → speaker:** the core's mixed-output sink fills `ring`; an `AVAudioSourceNode` pulls
|
||||
/// from it and renders through the VPIO output, giving AEC its reference signal.
|
||||
/// from it and renders through the engine output. This runs the whole time we're connected,
|
||||
/// so remote audio plays even before the user joins voice (no "can't hear anyone").
|
||||
/// - **mic → core:** when the mic is active a tap on the input node converts to 48 kHz int16 and
|
||||
/// calls `client.feedPcm(micStreamId)`.
|
||||
///
|
||||
/// Lifecycle is driven by `SessionState` join/leave. The Stereo Mic / Studio / A2DP presets keep
|
||||
/// the core's miniaudio path instead (they want raw / stereo / no-AEC routing VPIO can't provide).
|
||||
/// Echo cancellation / noise suppression / AGC come from Apple's Voice-Processing I/O unit (VPIO),
|
||||
/// which `inputNode.setVoiceProcessingEnabled(true)` enables. VPIO forces mono, so it is engaged
|
||||
/// only when the active preset wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`) — the
|
||||
/// Stereo Mic / A2DP configs run the same engine with VPIO off.
|
||||
///
|
||||
/// Every preset / route / interruption change funnels through `reconfigure()`: a single
|
||||
/// deterministic stop → AVAudioSession reconfigure → rebuild graph → start. There is no second
|
||||
/// (miniaudio) audio path to hand off to, so a switch cannot leave one direction dropped.
|
||||
@MainActor
|
||||
final class IOSVoiceProcessingEngine {
|
||||
static let shared = IOSVoiceProcessingEngine()
|
||||
final class IOSAudioEngine {
|
||||
static let shared = IOSAudioEngine()
|
||||
|
||||
private(set) var isRunning = false
|
||||
/// True while connected (between `startListening` and `stop`) — the playback graph should run.
|
||||
private(set) var isConnected = false
|
||||
/// True while a local mic stream is active — the input tap should be installed.
|
||||
private(set) var micActive = false
|
||||
|
||||
private let engine = AVAudioEngine()
|
||||
private var sourceNode: AVAudioSourceNode?
|
||||
private weak var client: VoiceCatClient?
|
||||
private var micStreamId: UInt32 = 0
|
||||
private var captureChannels: UInt32 = 1
|
||||
|
||||
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders and the engine
|
||||
// processes in. The core delivers 48 kHz stereo int16 via the mixed-output sink.
|
||||
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders. The core
|
||||
// delivers 48 kHz stereo int16 via the mixed-output sink; mainMixerNode adapts to the route.
|
||||
private let outFormat = AVAudioFormat(
|
||||
commonFormat: .pcmFormatFloat32, sampleRate: 48000, channels: 2, interleaved: false)!
|
||||
|
||||
@@ -121,33 +131,138 @@ final class IOSVoiceProcessingEngine {
|
||||
private let renderScratchFrames = 8192
|
||||
private let renderScratch: UnsafeMutablePointer<Int16>
|
||||
|
||||
// Mic-feed converter (input-node format → 48 kHz int16) and its target buffer. Owned here so
|
||||
// the (background) tap block reuses them instead of allocating per callback.
|
||||
private var micConverter: AVAudioConverter?
|
||||
private var micTargetFormat: AVAudioFormat?
|
||||
|
||||
private init() {
|
||||
renderScratch = UnsafeMutablePointer<Int16>.allocate(capacity: renderScratchFrames * 2)
|
||||
renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2)
|
||||
}
|
||||
|
||||
/// Start the VPIO engine for an active mic stream. The caller must have already enabled
|
||||
/// external playback on the core (`client.setExternalPlayback(true)` + `audioRestart()`) and
|
||||
/// started the MIC stream with `externalFeed: true`.
|
||||
func start(client: VoiceCatClient, micStreamId: UInt32, captureChannels: UInt32) {
|
||||
guard !isRunning else { return }
|
||||
// MARK: - Lifecycle
|
||||
|
||||
/// Begin playback-only (listening) operation. Called once at connect, after
|
||||
/// `client.setExternalPlayback(true)` and `AudioSessionManager.ensureSessionActive()`. Attaches
|
||||
/// the source node, wires the core's mixed-output sink into the ring, and starts the engine so
|
||||
/// remote audio plays immediately.
|
||||
func startListening(client: VoiceCatClient) {
|
||||
self.client = client
|
||||
self.micStreamId = micStreamId
|
||||
guard !isConnected else { return }
|
||||
isConnected = true
|
||||
ring.reset()
|
||||
|
||||
// Enable the voice-processing I/O unit (AEC/NS/AGC) on the shared input+output unit.
|
||||
// Wire the core's mixed-output sink into the ring (C function pointer, no captures). Stays
|
||||
// registered for the whole connection; the ring is drained by the source-node render block.
|
||||
let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque()
|
||||
client.setMixedOutputSink({ user, pcm, spc, ch, _ in
|
||||
guard let user, let pcm else { return }
|
||||
let ring = Unmanaged<PCMRing>.fromOpaque(user).takeUnretainedValue()
|
||||
ring.write(pcm, count: spc * Int(ch))
|
||||
}, user: ringPtr)
|
||||
|
||||
rebuild()
|
||||
}
|
||||
|
||||
/// Tear down the engine and unhook the core sink. Called on disconnect.
|
||||
func stop() {
|
||||
guard isConnected else { return }
|
||||
micActive = false
|
||||
isConnected = false
|
||||
client?.setMixedOutputSink(nil, user: nil)
|
||||
engine.inputNode.removeTap(onBus: 0)
|
||||
if engine.isRunning { engine.stop() }
|
||||
logger.info("audio engine stopped — ring written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples")
|
||||
try? engine.inputNode.setVoiceProcessingEnabled(false)
|
||||
if let src = sourceNode {
|
||||
engine.detach(src)
|
||||
sourceNode = nil
|
||||
}
|
||||
ring.reset()
|
||||
client = nil
|
||||
}
|
||||
|
||||
// MARK: - Mic transitions
|
||||
|
||||
/// Engage the mic: install the input tap and (if the preset wants it) VPIO. Called when the
|
||||
/// user joins voice, after the MIC stream (external_feed) is started.
|
||||
func startMic(streamId: UInt32, channels: UInt32) {
|
||||
micStreamId = streamId
|
||||
captureChannels = channels
|
||||
micActive = true
|
||||
rebuild()
|
||||
}
|
||||
|
||||
/// Disengage the mic: remove the tap and VPIO, keep playback running for remaining remote audio.
|
||||
func stopMic() {
|
||||
guard micActive else { return }
|
||||
micActive = false
|
||||
rebuild()
|
||||
}
|
||||
|
||||
/// Update the capture channel count (mono↔stereo) for the active mic and rebuild.
|
||||
func setCaptureChannels(_ channels: UInt32) {
|
||||
captureChannels = channels
|
||||
if let client, micStreamId != 0 {
|
||||
client.setCaptureChannels(streamId: micStreamId, channels: channels)
|
||||
}
|
||||
if micActive { rebuild() }
|
||||
}
|
||||
|
||||
/// Re-apply the engine graph against the current AVAudioSession config (preset / route change).
|
||||
/// Safe to call when only listening — it just rebuilds the playback graph against the new route.
|
||||
func reconfigure() {
|
||||
guard isConnected else { return }
|
||||
rebuild()
|
||||
}
|
||||
|
||||
// MARK: - Graph (re)build
|
||||
|
||||
/// The single place that (re)builds and starts the engine graph. Deterministic: stop → set
|
||||
/// VPIO → (re)install the mic tap → start. The caller is responsible for having applied the
|
||||
/// AVAudioSession config (category/mode/route) first (`IOSAudioRouter.applyConfiguration`).
|
||||
private func rebuild() {
|
||||
guard isConnected else { return }
|
||||
if engine.isRunning { engine.stop() }
|
||||
engine.inputNode.removeTap(onBus: 0)
|
||||
|
||||
let useVPIO = micActive && IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
|
||||
do {
|
||||
try engine.inputNode.setVoiceProcessingEnabled(true)
|
||||
try engine.inputNode.setVoiceProcessingEnabled(useVPIO)
|
||||
} catch {
|
||||
logger.error("setVoiceProcessingEnabled failed: \(error.localizedDescription) — AEC unavailable")
|
||||
logger.error("setVoiceProcessingEnabled(\(useVPIO)) failed: \(error.localizedDescription)")
|
||||
}
|
||||
if useVPIO {
|
||||
// AGC is the one VPIO sub-stage iOS exposes; AEC+NS are bundled into the master switch.
|
||||
engine.inputNode.isVoiceProcessingAGCEnabled = IOSAudioRouter.shared.agcEnabled
|
||||
}
|
||||
|
||||
// ── Playback: source node pulls mixed PCM from the ring through the VPIO output. ──
|
||||
// (Re)build the playback source node AFTER the VPIO state is set, so it connects against the
|
||||
// correct (voice-processed or plain) output unit — mirrors the proven original ordering.
|
||||
rebuildSourceNode()
|
||||
if micActive { installMicTap() }
|
||||
|
||||
engine.prepare()
|
||||
do {
|
||||
try engine.start()
|
||||
let inFmt = engine.inputNode.outputFormat(forBus: 0)
|
||||
let outFmt = engine.outputNode.outputFormat(forBus: 0)
|
||||
let route = AVAudioSession.sharedInstance().currentRoute.outputs
|
||||
.map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ")
|
||||
logger.info("""
|
||||
engine started — mic=\(self.micActive) vpio=\(useVPIO) captureCh=\(self.captureChannels) \
|
||||
inFormat=\(inFmt) outputNode=\(outFmt) outputRoute=[\(route)]
|
||||
""")
|
||||
} catch {
|
||||
logger.error("engine start failed: \(error.localizedDescription)")
|
||||
}
|
||||
}
|
||||
|
||||
/// Detach any previous source node and attach a fresh one pulling mixed PCM from the ring.
|
||||
/// Rebuilt on every graph rebuild so it always connects against the current output unit (the
|
||||
/// VPIO state can change the output between rebuilds). Its format is route-independent —
|
||||
/// `mainMixerNode` adapts 48 kHz stereo to whatever the output route is.
|
||||
private func rebuildSourceNode() {
|
||||
if let old = sourceNode {
|
||||
engine.detach(old)
|
||||
sourceNode = nil
|
||||
}
|
||||
let ring = self.ring
|
||||
let scratch = self.renderScratch
|
||||
let scratchFrames = self.renderScratchFrames
|
||||
@@ -156,17 +271,11 @@ final class IOSVoiceProcessingEngine {
|
||||
let abl = UnsafeMutableAudioBufferListPointer(ablPtr)
|
||||
let n = min(frames, scratchFrames)
|
||||
let got = ring.read(into: scratch, count: n * 2) / 2 // interleaved stereo → frames
|
||||
// Deinterleave int16 → Float32 per channel; silence-fill any underrun tail.
|
||||
let scale: Float = 1.0 / 32768.0
|
||||
for ch in 0..<abl.count {
|
||||
guard let base = abl[ch].mData?.assumingMemoryBound(to: Float.self) else { continue }
|
||||
for i in 0..<frames {
|
||||
if i < got {
|
||||
let s = scratch[i * 2 + min(ch, 1)]
|
||||
base[i] = Float(s) * scale
|
||||
} else {
|
||||
base[i] = 0
|
||||
}
|
||||
base[i] = i < got ? Float(scratch[i * 2 + min(ch, 1)]) * scale : 0
|
||||
}
|
||||
}
|
||||
return noErr
|
||||
@@ -174,29 +283,33 @@ final class IOSVoiceProcessingEngine {
|
||||
sourceNode = src
|
||||
engine.attach(src)
|
||||
engine.connect(src, to: engine.mainMixerNode, format: outFormat)
|
||||
}
|
||||
|
||||
// ── Mic: tap the VPIO input node, convert to 48 kHz int16, feed the core. ──
|
||||
/// Install the mic tap: convert the input node's native format to 48 kHz int16 (mono or
|
||||
/// stereo per `captureChannels`) and feed it to the core. Rebuilds the converter each time
|
||||
/// because the input format depends on the VPIO state and the active route.
|
||||
private func installMicTap() {
|
||||
guard let client else { return }
|
||||
let inFormat = engine.inputNode.outputFormat(forBus: 0)
|
||||
guard inFormat.sampleRate > 0 else {
|
||||
logger.error("input format unavailable (\(inFormat)) — mic will not transmit")
|
||||
return
|
||||
}
|
||||
let targetCh = max(1, min(2, captureChannels))
|
||||
let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000,
|
||||
channels: AVAudioChannelCount(targetCh), interleaved: true)
|
||||
micTargetFormat = target
|
||||
micConverter = (target != nil && inFormat.sampleRate > 0)
|
||||
? AVAudioConverter(from: inFormat, to: target!) : nil
|
||||
if micConverter == nil {
|
||||
logger.error("mic converter unavailable (in=\(inFormat)) — mic will not transmit")
|
||||
guard let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000,
|
||||
channels: AVAudioChannelCount(targetCh), interleaved: true),
|
||||
let converter = AVAudioConverter(from: inFormat, to: target) else {
|
||||
logger.error("mic converter unavailable (in=\(inFormat), ch=\(targetCh)) — mic will not transmit")
|
||||
return
|
||||
}
|
||||
|
||||
let c = client
|
||||
let sid = micStreamId
|
||||
let converter = micConverter
|
||||
let tgt = micTargetFormat
|
||||
let c = client
|
||||
engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in
|
||||
guard let converter, let tgt else { return }
|
||||
// Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample.
|
||||
let ratio = tgt.sampleRate / buffer.format.sampleRate
|
||||
let ratio = target.sampleRate / buffer.format.sampleRate
|
||||
let outCap = AVAudioFrameCount(Double(buffer.frameLength) * ratio + 16)
|
||||
guard let outBuf = AVAudioPCMBuffer(pcmFormat: tgt, frameCapacity: outCap) else { return }
|
||||
guard let outBuf = AVAudioPCMBuffer(pcmFormat: target, frameCapacity: outCap) else { return }
|
||||
var fed = false
|
||||
let status = converter.convert(to: outBuf, error: nil) { _, outStatus in
|
||||
if fed { outStatus.pointee = .noDataNow; return nil }
|
||||
@@ -206,64 +319,9 @@ final class IOSVoiceProcessingEngine {
|
||||
}
|
||||
guard status != .error, outBuf.frameLength > 0,
|
||||
let chData = outBuf.int16ChannelData else { return }
|
||||
let spc = Int(outBuf.frameLength)
|
||||
// int16 interleaved → channelData[0] is the interleaved buffer for interleaved formats.
|
||||
c.feedPcm(streamId: sid, pcm: chData[0], samplesPerChannel: spc, channels: targetCh)
|
||||
// int16 interleaved → channelData[0] is the interleaved buffer.
|
||||
c.feedPcm(streamId: sid, pcm: chData[0],
|
||||
samplesPerChannel: Int(outBuf.frameLength), channels: targetCh)
|
||||
}
|
||||
|
||||
// ── Wire the core's mixed-output sink into the ring (C function pointer, no captures). ──
|
||||
let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque()
|
||||
client.setMixedOutputSink({ user, pcm, spc, ch, _ in
|
||||
guard let user, let pcm else { return }
|
||||
let ring = Unmanaged<PCMRing>.fromOpaque(user).takeUnretainedValue()
|
||||
ring.write(pcm, count: spc * Int(ch))
|
||||
}, user: ringPtr)
|
||||
|
||||
engine.prepare()
|
||||
do {
|
||||
try engine.start()
|
||||
isRunning = true
|
||||
// Diagnostics: capture the negotiated graph formats and the live output route so a
|
||||
// silent-playback report can be triaged (format/rate mismatch vs. routing vs. the
|
||||
// core not delivering PCM — see the ring stats logged in teardown()).
|
||||
let outFmt = engine.outputNode.outputFormat(forBus: 0)
|
||||
let mixFmt = engine.mainMixerNode.outputFormat(forBus: 0)
|
||||
let route = AVAudioSession.sharedInstance().currentRoute.outputs
|
||||
.map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ")
|
||||
logger.info("""
|
||||
VPIO engine started — inFormat=\(inFormat), captureCh=\(targetCh), \
|
||||
outputNode=\(outFmt), mainMixer=\(mixFmt), outputRoute=[\(route)]
|
||||
""")
|
||||
} catch {
|
||||
logger.error("VPIO engine start failed: \(error.localizedDescription)")
|
||||
teardown()
|
||||
}
|
||||
}
|
||||
|
||||
/// Stop the VPIO engine. The caller is responsible for restoring the core's hardware playback
|
||||
/// afterwards (`client.setExternalPlayback(false)` + `audioRestart()`).
|
||||
func stop() {
|
||||
guard isRunning else { return }
|
||||
teardown()
|
||||
logger.info("VPIO engine stopped")
|
||||
}
|
||||
|
||||
private func teardown() {
|
||||
client?.setMixedOutputSink(nil, user: nil)
|
||||
engine.inputNode.removeTap(onBus: 0)
|
||||
if engine.isRunning { engine.stop() }
|
||||
// Diagnostics (threads now quiesced): how much mixed PCM the core delivered into the ring
|
||||
// vs. how much the render thread consumed. written==0 ⇒ the core never delivered (Bug 1
|
||||
// core/lifecycle path); written>0 with no audible output ⇒ the AVAudioEngine output graph.
|
||||
logger.info("VPIO ring stats — written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples")
|
||||
try? engine.inputNode.setVoiceProcessingEnabled(false)
|
||||
if let src = sourceNode {
|
||||
engine.detach(src)
|
||||
sourceNode = nil
|
||||
}
|
||||
micConverter = nil
|
||||
micTargetFormat = nil
|
||||
ring.reset()
|
||||
isRunning = false
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user