Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ios-audio): unify iOS audio onto one always-external AVAudioEngine

Browse Source 
The iOS audio path was a hybrid: Voice-Chat-class presets ran a native
VPIO AVAudioEngine (core external) while Stereo/Studio/A2DP presets ran
the core's miniaudio devices. Nearly every "no input / no output / both"
bug lived in the seam between the two paths — the lingering miniaudio
capture unit fighting VPIO, the audioRestart ordering dance, the
route-change "glitching" loop, stereo<->mono stickiness, and
"can't hear anyone". Switching presets/routes mid-call routinely dropped
a direction.

Drive ALL iOS audio through one AVAudioEngine with the core fully
external at all times: setExternalPlayback(1) once at connect, every MIC
stream external_feed=1, mic via vc_stream_feed_pcm, playback via
vc_set_mixed_output_sink (drained by an always-on AVAudioSourceNode so
remote audio plays before joining voice). VPIO + AGC toggle per preset.
Every preset/route/interruption change funnels through one deterministic
Swift-only reconfigure (stop -> apply session config -> rebuild -> start)
— no second path to hand off to, so a change can't drop a direction.

- IOSVoiceProcessingEngine.swift -> IOSAudioEngine: always-on source-node
  playback, conditional mic tap, VPIO/AGC; one rebuild() backing
  startListening/stop/startMic/stopMic/reconfigure/setCaptureChannels.
- IOSAudioRouter: 7 presets -> 4 (Voice Chat / Stereo Mic / Mono Mic /
  Advanced); persisted voiceProcessingEnabled + agcEnabled; setters call
  IOSAudioEngine.reconfigure() instead of audioRestart/reconcileVoicePath.
- AudioSessionManager slimmed; SessionState mic lifecycle collapsed;
  AppState wires external playback + listening at connect, stop at
  disconnect; SettingsView shows 4 presets + Advanced VPIO/AGC toggles.

No core/ABI/test changes — relies on the already-shipped external API
(test_external_pcm, test_external_playback). xcodebuild iOS device Debug
BUILD SUCCEEDED. Updates docs/voice.md §8 and PROGRESS.md.
This commit is contained in:
Talon
2026-06-23 02:45:53 +02:00
parent 7547b8e140
commit d30c4ee2f5
8 changed files with 438 additions and 414 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
PROGRESS.md
View File

@@ -75,8 +75,42 @@ up instantly. Newest status at the top.
is now correct. If voice still fails, watch the server's rate-limited `[media] dropped frames —
unmapped-endpoint=…` line (NAT source-port rewrite would be the next suspect).
- **Awaiting on-device verification (2026-06-22):** **iOS real echo cancellation / noise suppression
via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS
- **Done (2026-06-23, Swift-only — no core/ABI change; awaiting on-device verification):** **iOS audio
stack unified — one always-external `AVAudioEngine`, miniaudio dropped on iOS.** The iOS audio path was
a fragile hybrid: Voice-Chat-class presets ran a native VPIO `AVAudioEngine` (core external) while
Stereo/Studio/A2DP presets ran the core's miniaudio devices. Nearly every bug lived in the seam
(lingering miniaudio capture unit fighting VPIO, the `audioRestart` ordering dance, the route-change
"glitching" loop, stereo↔mono stickiness, "can't hear anyone"), and switching presets/routes mid-call
routinely dropped input, output, or both. **Fix: drive *all* iOS audio through one `AVAudioEngine` with
the core fully external at all times** — `vc_set_external_playback(1)` once at connect, every MIC stream
`external_feed=1`, mic via `vc_stream_feed_pcm`, playback via `vc_set_mixed_output_sink`.
- `IOSVoiceProcessingEngine.swift` → **`IOSAudioEngine`** (same file): always-on `AVAudioSourceNode`
playback (runs whenever connected, so remote audio plays before you join voice); conditional mic tap;
VPIO + AGC toggled per config. One private `rebuild()` (stop → set VPIO → install tap → start) backs
`startListening`/`stop`/`startMic`/`stopMic`/`reconfigure`/`setCaptureChannels`. Kept the `PCMRing`
and ring-stats diagnostics.
- `IOSAudioRouter`: presets cut from seven to **four** — Voice Chat (VPIO mono, system output),
Stereo Mic / Mono Mic (internal built-in mic regardless of output, A2DP-capable, no VPIO), Advanced
(manual). New persisted `voiceProcessingEnabled` (master AEC+NS) + `agcEnabled`; setters now call
`IOSAudioEngine.reconfigure()` instead of `client.audioRestart()` + `reconcileVoicePath`. Kept the
proven AVAudioSession recipes (category/mode/options, stereo capsule, `applyA2dpSpeakerFallback`).
- `AudioSessionManager` slimmed (drops `client`/`activeMicStreamId`/`reconcileVoicePath`; adds
`isActive`); interruption-end & device-change now `reconfigure()` the engine. `SessionState`
`doStartMicStream`/`stopMicStream` collapsed to start-stream + `startMic`/`stopMic` (no
`setExternalPlayback`/`audioRestart` toggling); `reconcileVoicePath` deleted. `AppState` sets external
playback + `startListening` at connect, `stop()` at disconnect. `SettingsView` → four presets +
Advanced VPIO/AGC toggles.
- **No core/ABI/test change** — relies on the already-shipped `vc_set_external_playback` /
`external_feed` / `vc_set_mixed_output_sink` / `vc_stream_feed_pcm` path (`test_external_pcm`,
`test_external_playback`). `xcodebuild` iOS device Debug **BUILD SUCCEEDED**. **Rebuild the
xcframework is NOT required** (no new symbols).
- **Next (user, on device):** two iPhones in a channel — verify BOTH directions survive every
transition and are never silent unless intended: Voice Chat (no echo, NR), listen-only before joining,
join↔leave repeatedly, switch Voice Chat↔Stereo↔Mono↔Advanced *while in voice*, A2DP connect/unplug,
wired connect/unplug, phone-call interruption + resume, screen-audio share.
- **Superseded by the 2026-06-23 unification above (2026-06-22):** **iOS real echo cancellation / noise
suppression via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS
AEC/NS/AGC come only from Apple's Voice-Processing I/O unit (VPIO), but the core uses miniaudio's
plain RemoteIO units — so `.voiceChat` mode alone never engaged AEC. Fix moves both mic capture and
playback to a native Swift `AVAudioEngine` (`setVoiceProcessingEnabled`) on the AEC presets, with the

15
clients/apple/iOS/VoiceCatiOS/AppState.swift
View File

@@ -91,6 +91,7 @@ final class AppState {
func disconnect() {
session?.stopMicStream()
session?.client.disconnect()
IOSAudioEngine.shared.stop()
AudioSessionManager.shared.deactivateSession()
session = nil
connectingClient?.disconnect()
@@ -160,17 +161,18 @@ final class AppState {
self.session = newSession
EventFeedback.shared.play(.login)
EventFeedback.shared.speak("Connected")
// Activate the audio session now, while connected — NOT lazily when the first
// remote stream arrives. The core opens its miniaudio playback device the moment
// a remote stream starts and only THEN emits .streamStarted; if we waited for
// that event to activate, the playback device would open against an inactive
// AVAudioSession and produce no sound (the "can't hear anyone" bug). Activating
// here guarantees the session is live before any device opens.
// Put the core into external-playback mode ONCE, now, before the session is
// activated or any remote stream can arrive — so the core never opens a miniaudio
// device on iOS (the single ordering rule of the unified audio path). Then activate
// the session and start the engine in listening mode so remote audio plays the
// moment someone talks, even before we join voice (no "can't hear anyone").
client.setExternalPlayback(true)
do {
try AudioSessionManager.shared.ensureSessionActive()
} catch {
print("Audio session activate on connect failed: \(error)")
}
IOSAudioEngine.shared.startListening(client: client)
} else {
connectStatus = "Auth failed: \(ev.result.description)"
showPasswordPrompt = true
@@ -178,6 +180,7 @@ final class AppState {
case .disconnected:
if session == nil { cancelConnect() }
else {
IOSAudioEngine.shared.stop()
AudioSessionManager.shared.deactivateSession()
session = nil; isConnecting = false
}

32
clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift
View File

@@ -8,19 +8,6 @@ private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "Audio
final class AudioSessionManager {
static let shared = AudioSessionManager()
weak var client: VoiceCatClient?
/// The stream ID of the currently active local MIC stream, if any. Set by `SessionState`
/// when the user joins/leaves voice so `IOSAudioRouter` can reset the core's capture
/// channel count (e.g. when switching stereo → mono) without going through `SessionState`.
var activeMicStreamId: UInt32?
/// Set by `SessionState`. Invoked by `IOSAudioRouter` after an audio-config change so the
/// voice path (native VPIO vs the core's miniaudio path) can be restarted to match the new
/// preset/route when the mic is active. No-op when not in voice. See
/// `SessionState.reconcileVoicePath()` and `IOSVoiceProcessingEngine`.
var reconcileVoicePath: (() -> Void)?
/// Tracks whether WE activated the session. The session must be active whenever the
/// AudioEngine is running (for capture OR playback), so it is activated when any audio
/// needs to play (a remote stream started OR the user joins voice) and only deactivated
@@ -28,6 +15,10 @@ final class AudioSessionManager {
/// want to hear remote audio.
private var isSessionActive = false
/// Whether the AVAudioSession is currently active (we activated it). Read by `IOSAudioRouter`
/// to decide whether the post-activation A2DP speaker fallback can be applied.
var isActive: Bool { isSessionActive }
func configure() {
// Load stored audio routing preferences and apply them before any audio session
// activation. IOSAudioRouter drives all iOS audio route selection via AVAudioSession;
@@ -114,18 +105,22 @@ final class AudioSessionManager {
switch type {
case .began:
// The system stops our AVAudioEngine and deactivates the session. Nothing to tear
// down — `IOSAudioEngine` rebuilds on resume.
logger.info("interruption began — session suspended by system")
isSessionActive = false // system deactivated us
client?.audioSuspend()
isSessionActive = false
case .ended:
let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0
let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue)
if options.contains(.shouldResume) {
do {
IOSAudioRouter.shared.applyConfiguration()
try AVAudioSession.sharedInstance().setActive(true)
isSessionActive = true
logger.info("interruption ended — session reactivated")
client?.audioResume()
IOSAudioRouter.shared.applyA2dpSpeakerFallback()
// Rebuild the engine graph against the restored route (both directions).
IOSAudioEngine.shared.reconfigure()
logger.info("interruption ended — session reactivated, engine rebuilt")
} catch {
logger.error("interruption ended — reactivation failed: \(error.localizedDescription)")
}
@@ -162,6 +157,9 @@ final class AudioSessionManager {
// the loud speaker (not the earpiece), and a replug should hand output back to A2DP.
if isSessionActive {
IOSAudioRouter.shared.applyA2dpSpeakerFallback()
// Rebind the engine (both directions) to the new route. The engine owns the route
// now, so this is the single thing that re-establishes audio after a device change.
IOSAudioEngine.shared.reconfigure()
}
}

275
clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift
View File

@@ -4,17 +4,18 @@ import VoiceCatCore
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioRouter")
/// iOS audio routing layer — drives all iOS audio route selection via `AVAudioSession`
/// *before* the core (miniaudio) opens its device. This class is the sole owner of the
/// session: miniaudio does NOT touch `AVAudioSession` on iOS, because the core opens its
/// devices through a `ma_context` configured with `sessionCategory = none` +
/// `noAudioSessionActivate/Deactivate` (see `AudioEngine::make_context_config` in
/// `core/src/audio/audio_engine.cpp`). Without that, miniaudio's default path resets the
/// category to `Record`/`Playback` with no options on every device open, wiping
/// `.allowBluetoothA2DP`/`.playAndRecord` and killing headphone/A2DP output — so that
/// config must stay in place. All iOS audio routing (input port selection, mic
/// orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) must be
/// driven from here.
/// iOS audio routing layer — the sole owner of `AVAudioSession` on iOS. On iOS the core never
/// opens a hardware (miniaudio) device: a single `AVAudioEngine` (`IOSAudioEngine`) drives both
/// capture and playback and the core runs fully external (see docs/voice.md §8). This class just
/// configures the *route* — category / mode / options, preferred input, data source, polar
/// pattern, stereo capsule — and `IOSAudioEngine` binds to whatever route is established. After
/// any change here the engine is rebuilt via `IOSAudioEngine.reconfigure()` (a deterministic
/// Swift-only stop → reconfigure → start); there is no second (miniaudio) audio path to hand off
/// to, so a change cannot leave one direction dropped.
///
/// (The core's iOS `ma_context` is still configured with `sessionCategory = none` +
/// `noAudioSessionActivate/Deactivate` in `AudioEngine::make_context_config` so that, should the
/// core ever open a device, miniaudio would not reset the category — but on iOS it does not.)
///
/// The three user-facing choices:
/// 1. **Input port** — which physical input (built-in mic, Bluetooth HFP, headset,
@@ -64,77 +65,59 @@ final class IOSAudioRouter: ObservableObject {
@Published var selectedInputPortId: String?
@Published var selectedDataSourceId: String?
@Published var selectedPolarPattern: String?
/// Master voice-processing switch (Apple VPIO: AEC + noise suppression bundled together).
/// iOS exposes no per-stage toggle, so this is the finest "echo cancellation / noise
/// reduction" control available. Only takes effect on a VPIO-capable config (mono + standard
/// + not A2DP); stereo / A2DP configs can't use VPIO regardless. Default on.
@Published var voiceProcessingEnabled: Bool = true
/// VPIO automatic gain control — the one VPIO sub-stage iOS lets us toggle independently.
/// Only meaningful when voice processing is active. Default on.
@Published var agcEnabled: Bool = true
@Published var showsRawModeSpeakerWarning: Bool = false
@Published var showsA2dpNoAecWarning: Bool = false
@Published var hasBluetoothDevice: Bool = false
@Published var hasWiredHeadset: Bool = false
/// Audio presets — sensible combinations of settings for common scenarios.
/// The app is about choice: users can pick a preset for a quick start, then
/// fine-tune individual settings under "Advanced Audio".
/// Audio presets — the four scenarios from the product spec. Pick a preset for a quick start,
/// then fine-tune individual settings under "Advanced". HFP / wired headsets are not separate
/// presets: Voice Chat lets the system route to them, and Advanced exposes manual selection.
enum AudioPreset: String, CaseIterable, Identifiable {
/// Standard iOS VoIP experience: AEC/AGC/HPF on, mono, system picks best route
/// (BT HFP if connected, wired if connected, speaker if nothing). Always available.
/// Voice chat: Apple VPIO does real AEC + noise suppression + AGC. Mono. The system picks
/// the best route (Bluetooth HFP / wired / speaker / earpiece). Always available.
case voiceChat = "Voice Chat"
/// Stereo built-in mic capture (front+back capsules). A2DP output if BT is connected,
/// else built-in speaker / wired. Standard processing (no AEC — stereo needs a non-VPIO
/// mode). Always available.
/// Internal **stereo** built-in mic regardless of the output route. A2DP output when a
/// Bluetooth headset is connected, else built-in speaker / wired. No VPIO (stereo can't
/// use it). Always available.
case stereoMic = "Stereo Mic"
/// Maximum fidelity: stereo mic, no AEC/AGC/HPF (raw mode). A2DP output if BT connected,
/// else speaker/wired. Always available. Echo risk on speaker.
case studio = "Studio (No Processing)"
/// Bluetooth HFP: BT mic + BT output, AEC on, mono. Only when BT is connected.
case bluetoothHeadset = "Bluetooth Headset (HFP)"
/// A2DP stereo output + built-in mono mic, AEC off. Only when BT is connected. (For
/// A2DP output + stereo mic, use the Stereo Mic preset while BT is connected.)
case btHeadphonesMonoMic = "BT Headphones + Mono Mic"
/// Wired headset/earpods: wired output + wired mic (or built-in), AEC on, mono.
/// Only when a wired audio device is connected.
case wiredHeadset = "Wired Headset"
/// Settings don't match any preset — user has tweaked advanced controls.
case custom = "Custom"
/// Internal **mono** built-in mic regardless of the output route. A2DP output when a
/// Bluetooth headset is connected, else built-in speaker / wired. No VPIO. Always available.
case monoMic = "Mono Mic"
/// Everything manual — input port, mic orientation / polar pattern, mono/stereo, Bluetooth
/// mode, raw vs standard, and the VPIO / AGC toggles. Also the display state when the
/// individual settings don't match a named preset.
case advanced = "Advanced"
var id: String { rawValue }
var requiresBluetooth: Bool {
switch self {
case .bluetoothHeadset, .btHeadphonesMonoMic: return true
default: return false
}
}
var requiresWired: Bool {
self == .wiredHeadset
}
var bluetoothMode: BluetoothMode {
switch self {
case .voiceChat, .bluetoothHeadset: return .btHfpVoice
// A2DP output when BT is connected; falls back to speaker/wired when it isn't.
case .stereoMic, .studio, .btHeadphonesMonoMic: return .builtInMicBtA2dp
case .wiredHeadset: return .builtInMicSpeaker
case .custom: return .builtInMicSpeaker // placeholder
case .voiceChat: return .btHfpVoice
// Internal-mic presets: A2DP output when BT is connected; speaker/wired when not.
case .stereoMic, .monoMic: return .builtInMicBtA2dp
case .advanced: return .builtInMicSpeaker // placeholder; Advanced sets it manually
}
}
var captureChannels: CaptureChannels {
switch self {
case .stereoMic, .studio: return .stereo
default: return .mono
}
self == .stereoMic ? .stereo : .mono
}
var micMode: MicMode {
switch self {
case .studio: return .raw
default: return .standard
}
}
var micMode: MicMode { .standard }
/// Whether this preset explicitly selects the built-in mic port.
/// Whether this preset explicitly pins the built-in mic port (the internal-mic presets).
var usesBuiltInMic: Bool {
switch self {
case .stereoMic, .studio, .btHeadphonesMonoMic: return true
case .stereoMic, .monoMic: return true
default: return false
}
}
@@ -170,6 +153,8 @@ final class IOSAudioRouter: ObservableObject {
private let kPolarPattern = "cat.voice.audio.polarPattern"
private let kPreset = "cat.voice.audio.preset"
private let kForceSpeaker = "cat.voice.audio.forceSpeaker"
private let kVoiceProcessing = "cat.voice.audio.voiceProcessing"
private let kAgc = "cat.voice.audio.agc"
/// Re-entrancy guard: setCategory/setPreferredInput/etc. trigger route-change
/// notifications synchronously on the same thread. Without this guard,
@@ -271,55 +256,41 @@ final class IOSAudioRouter: ObservableObject {
}
}
/// The presets available given the current device connection state.
/// Always includes Voice Chat, Stereo Mic, Studio, and Custom. BT presets only when
/// a Bluetooth device is connected. Wired preset only when a wired device is connected.
var availablePresets: [AudioPreset] {
AudioPreset.allCases.filter { preset in
if preset == .custom { return true }
if preset.requiresBluetooth && !hasBluetoothDevice { return false }
if preset.requiresWired && !hasWiredHeadset { return false }
return true
}
}
/// The presets the user can pick. All four are always available — the named presets simply
/// describe what to do "regardless of the output route", and Advanced is always offered.
var availablePresets: [AudioPreset] { AudioPreset.allCases }
/// Which preset matches the current settings, or .custom if nothing matches.
/// Checks device-specific presets first (BT, wired) so that e.g. when BT is connected
/// and settings match "Bluetooth Headset", it returns that instead of the equivalent
/// "Voice Chat" (which has the same bluetoothMode/micMode/channels but is more general).
/// Which named preset matches the current settings, or `.advanced` if nothing matches.
var activePreset: AudioPreset {
// Check device-specific presets first (most specific → least specific)
let order: [AudioPreset] = [
.bluetoothHeadset, .btHeadphonesMonoMic,
.wiredHeadset,
.voiceChat, .stereoMic, .studio,
]
for preset in order {
for preset in [AudioPreset.voiceChat, .stereoMic, .monoMic] {
if bluetoothMode == preset.bluetoothMode
&& captureChannels == preset.captureChannels
&& micMode == preset.micMode {
// Don't match a BT preset if no BT is connected — fall through to Voice Chat
if preset.requiresBluetooth && !hasBluetoothDevice { continue }
if preset.requiresWired && !hasWiredHeadset { continue }
return preset
}
}
return .custom
return .advanced
}
/// Whether the current configuration should use the native iOS Voice-Processing path (VPIO:
/// real AEC/NS/AGC via `IOSVoiceProcessingEngine`). True exactly when `applyConfiguration`
/// selects the `.voiceChat` AVAudioSession mode — mono + standard processing + not A2DP
/// (A2DP / stereo / raw modes can't use VPIO, so they keep the core's miniaudio path).
/// Whether the current configuration should engage Apple's Voice-Processing I/O unit (VPIO:
/// real AEC + noise suppression + AGC, driven by `IOSAudioEngine`). VPIO forces mono and
/// can't run on an A2DP route, so it is available only for a mono + standard + non-A2DP
/// config, and then only when the user hasn't disabled it via the Advanced master toggle.
var currentConfigUsesVoiceProcessing: Bool {
voiceProcessingEnabled && voiceProcessingAvailable
}
/// Whether the current config *could* use VPIO (mono + standard + non-A2DP), independent of
/// the user's master toggle. Drives whether the Advanced "Voice Processing" switch is shown.
var voiceProcessingAvailable: Bool {
captureChannels == .mono && micMode == .standard && bluetoothMode != .builtInMicBtA2dp
}
// MARK: - Apply configuration
/// Apply the full audio configuration to AVAudioSession. Call this before the core
/// opens its capture device (i.e. before `startMicStream` → `activateForStreaming`).
/// Re-entrant-safe: if a route-change notification fires synchronously during a
/// Apply the full audio configuration to AVAudioSession. Call this before (re)building the
/// `IOSAudioEngine` graph so the engine binds to the intended route (`applyAndReconfigure`
/// does both). Re-entrant-safe: if a route-change notification fires synchronously during a
/// `setCategory`/`setPreferredInput` call, the guard prevents re-entry.
func applyConfiguration() {
guard !isApplyingConfiguration else {
@@ -401,8 +372,8 @@ final class IOSAudioRouter: ObservableObject {
// route explicitly via setPreferredInput + setInputDataSource. With HFP disabled
// the system routes input to the built-in mic, but without the explicit
// preferred-input anchor the route can collapse during the mode switch
// (.voiceChat → .default) and the output dies. The channel count is requested by
// miniaudio at the audio-unit level (vc_set_capture_channels), NOT via
// (.voiceChat → .default) and the output dies. The channel count is carried by the
// engine's mic tap + vc_set_capture_channels, NOT via
// setPreferredInputNumberOfChannels(2) — that call collapses the A2DP output route.
configureStereoCapture(session: session)
} else if let portId = selectedInputPortId, !portId.isEmpty,
@@ -431,9 +402,9 @@ final class IOSAudioRouter: ObservableObject {
/// 3. `setPreferredInput(builtIn)` — anchor the input route explicitly. Without this
/// anchor the route can collapse during the mode switch (.voiceChat → .default).
/// 4. `setInputDataSource(stereoSource)` — commit the data source at the session level
/// The channel count itself is requested by miniaudio at the audio-unit level via
/// `vc_set_capture_channels(2)`. We must NOT call `setPreferredInputNumberOfChannels(2)`
/// — that session-level call collapses the A2DP output route.
/// The channel count itself is carried by the engine's mic tap (which captures 2 channels)
/// plus `vc_set_capture_channels(2)` so the core encodes stereo. We must NOT call
/// `setPreferredInputNumberOfChannels(2)` — that session-level call collapses the A2DP route.
private func configureStereoCapture(session: AVAudioSession) {
guard let builtIn = session.availableInputs?.first(where: { $0.portType == .builtInMic })
else {
@@ -538,6 +509,9 @@ final class IOSAudioRouter: ObservableObject {
selectedDataSourceId = UserDefaults.standard.string(forKey: kDataSourceId)
selectedPolarPattern = UserDefaults.standard.string(forKey: kPolarPattern)
forceSpeaker = UserDefaults.standard.bool(forKey: kForceSpeaker)
// VPIO toggles default ON when never set (object(forKey:) is nil → use true).
voiceProcessingEnabled = (UserDefaults.standard.object(forKey: kVoiceProcessing) as? Bool) ?? true
agcEnabled = (UserDefaults.standard.object(forKey: kAgc) as? Bool) ?? true
}
/// Persist current selections to UserDefaults.
func savePreferences() {
@@ -548,86 +522,88 @@ final class IOSAudioRouter: ObservableObject {
UserDefaults.standard.set(selectedDataSourceId, forKey: kDataSourceId)
UserDefaults.standard.set(selectedPolarPattern, forKey: kPolarPattern)
UserDefaults.standard.set(forceSpeaker, forKey: kForceSpeaker)
UserDefaults.standard.set(voiceProcessingEnabled, forKey: kVoiceProcessing)
UserDefaults.standard.set(agcEnabled, forKey: kAgc)
}
// MARK: - Selection setters (called from SettingsView pickers)
/// Shared tail for every setting change: persist, re-apply the AVAudioSession config, refresh
/// the route lists, re-evaluate the A2DP speaker fallback, and rebind the live engine to the
/// new route. `IOSAudioEngine.reconfigure()` is a no-op when not connected, so this is safe to
/// call from Settings whether or not a session is in progress. There is no longer a second
/// (miniaudio) audio path to hand off to, so one engine rebuild is the whole story.
private func applyAndReconfigure() {
savePreferences()
applyConfiguration()
if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() }
refreshRoutes()
IOSAudioEngine.shared.reconfigure()
}
func selectInputPort(_ portId: String) {
selectedInputPortId = portId
selectedDataSourceId = nil
selectedPolarPattern = nil
savePreferences()
applyConfiguration()
refreshRoutes()
applyAndReconfigure()
}
func selectDataSource(_ dataSourceId: String) {
selectedDataSourceId = dataSourceId
selectedPolarPattern = nil
savePreferences()
applyConfiguration()
refreshRoutes()
applyAndReconfigure()
}
func selectPolarPattern(_ pattern: String) {
selectedPolarPattern = pattern
savePreferences()
applyConfiguration()
refreshRoutes()
applyAndReconfigure()
}
func selectBluetoothMode(_ mode: BluetoothMode) {
bluetoothMode = mode
savePreferences()
applyConfiguration()
refreshRoutes()
// VPIO class or route may have changed — restart the voice path if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
applyAndReconfigure()
}
func setForceSpeaker(_ on: Bool) {
forceSpeaker = on
savePreferences()
applyConfiguration()
refreshRoutes()
// Route changed under a possibly-running VPIO engine — reconcile if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
applyAndReconfigure()
}
func selectMicMode(_ mode: MicMode) {
micMode = mode
applyAndReconfigure()
}
func setVoiceProcessingEnabled(_ on: Bool) {
voiceProcessingEnabled = on
applyAndReconfigure()
}
func setAgcEnabled(_ on: Bool) {
agcEnabled = on
// No session reconfigure needed — just rebuild the engine so VPIO picks up the AGC flag.
savePreferences()
applyConfiguration()
updateWarnings()
// Standard↔Raw flips the VPIO class — reconcile if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
IOSAudioEngine.shared.reconfigure()
}
func selectCaptureChannels(_ channels: CaptureChannels) {
captureChannels = channels
savePreferences()
applyConfiguration()
// Update the core's stored capture channel count (does not restart the engine).
if let streamId = AudioSessionManager.shared.activeMicStreamId {
_ = AudioSessionManager.shared.client?.setCaptureChannels(
streamId: streamId, channels: channels.channelCount)
}
// Restart the engine AFTER AVAudioSession routing has settled and the channel
// count is stored. The engine reopens playback first (committing the A2DP/output
// route), then capture — avoiding the race where stereo capture activation drops
// A2DP before the playback device has a chance to claim the route.
_ = AudioSessionManager.shared.client?.audioRestart()
// Mono↔stereo flips the VPIO class (stereo can't use VPIO) — reconcile if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() }
refreshRoutes()
// Push the channel count into the core's MIC stream, then rebuild the engine graph so the
// mic tap captures the right number of channels. The engine owns the route now, so there's
// no stereo-vs-A2DP race to sequence around.
IOSAudioEngine.shared.setCaptureChannels(channels.channelCount)
}
// MARK: - Presets
/// Apply a preset — sets all individual audio settings to the preset's values, then
/// applies the configuration. For presets that use the built-in mic (A2DP presets),
/// finds the built-in mic port UID from availableInputs.
/// Apply a named preset — set all individual settings to the preset's values, then re-apply
/// the configuration and rebind the engine. The internal-mic presets pin the built-in mic.
func applyPreset(_ preset: AudioPreset) {
guard preset != .custom else { return } // can't "apply" custom — it's a display state
guard preset != .advanced else { return } // Advanced is a display state, not "applied"
bluetoothMode = preset.bluetoothMode
micMode = preset.micMode
@@ -638,19 +614,17 @@ final class IOSAudioRouter: ObservableObject {
if preset == .voiceChat { forceSpeaker = true }
if preset.usesBuiltInMic {
// Find the built-in mic port from available inputs and select it.
let session = AVAudioSession.sharedInstance()
if let builtInMic = (session.availableInputs ?? []).first(where: {
// Pin the built-in mic. In stereo, iOS uses multiple capsules automatically; in mono
// the default orientation is fine — so don't force a specific data source / pattern.
if let builtInMic = (AVAudioSession.sharedInstance().availableInputs ?? []).first(where: {
$0.portType == .builtInMic
}) {
selectedInputPortId = builtInMic.uid
}
// Don't set a specific data source — in stereo mode, iOS uses multiple mic
// capsules automatically. In mono, the default orientation is fine.
selectedDataSourceId = nil
selectedPolarPattern = nil
} else {
// For Default and Bluetooth Headset presets, let the system pick the input.
// Voice Chat: let the system pick the input (Bluetooth HFP / wired / built-in).
selectedInputPortId = nil
selectedDataSourceId = nil
selectedPolarPattern = nil
@@ -659,18 +633,11 @@ final class IOSAudioRouter: ObservableObject {
UserDefaults.standard.set(preset.rawValue, forKey: kPreset)
savePreferences()
applyConfiguration()
// Update the core's stored capture channel count (does not restart the engine).
if let streamId = AudioSessionManager.shared.activeMicStreamId {
_ = AudioSessionManager.shared.client?.setCaptureChannels(
streamId: streamId, channels: preset.captureChannels.channelCount)
}
// Restart the engine AFTER AVAudioSession routing has settled and the channel
// count is stored. Playback opens first (commits A2DP route), then capture.
_ = AudioSessionManager.shared.client?.audioRestart()
if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() }
refreshRoutes()
// The preset may have flipped the VPIO class (and/or the route) — restart the voice path
// if the mic is active so AEC/NS engage (or disengage) to match the new preset.
AudioSessionManager.shared.reconcileVoicePath?()
// Push the channel count to the core, then rebuild the engine graph (VPIO on/off + tap).
IOSAudioEngine.shared.setCaptureChannels(preset.captureChannels.channelCount)
IOSAudioEngine.shared.reconfigure()
logger.info("applyPreset — \(preset.rawValue)")
}

294
clients/apple/iOS/VoiceCatiOS/IOSVoiceProcessingEngine.swift
View File

@@ -3,9 +3,9 @@ import Darwin
import os
import VoiceCatCore
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSVoiceProcessingEngine")
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioEngine")
/// In-process single-producer/single-consumer int16 PCM ring for the VPIO playback path.
/// In-process single-producer/single-consumer int16 PCM ring for the playback path.
///
/// producer = the core's mixer-timer thread (the `vc_set_mixed_output_sink` callback)
/// consumer = the `AVAudioSourceNode` render thread
@@ -79,38 +79,48 @@ final class PCMRing {
/// Diagnostics: monotonic total samples written / read since the ring was created. The
/// indices are already cumulative, so these are free. Only read them when both threads are
/// quiesced (e.g. at teardown after the engine + mixer sink are stopped) — they are not
/// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" (Bug 1
/// core path) apart from "PCM arrived but produced no sound" (AVAudioEngine output graph).
/// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" apart
/// from "PCM arrived but produced no sound" (the AVAudioEngine output graph).
var debugTotalWritten: UInt64 { writeIdx }
var debugTotalRead: UInt64 { readIdx }
}
/// Native iOS voice-processing audio path (docs/voice.md §8 "iOS voice processing").
/// The single iOS audio engine (docs/voice.md §8 "iOS audio engine").
///
/// Real iOS echo cancellation / noise suppression / AGC come ONLY from Apple's Voice-Processing
/// I/O unit (VPIO), which `AVAudioEngine.setVoiceProcessingEnabled(true)` enables. For VPIO to
/// cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
/// played-back signal from the mic), so on the AEC presets this engine drives both directions and
/// the core runs in external mode (no hardware devices):
/// - **Mic → core:** a tap on the VPIO input node → 48 kHz int16 → `client.feedPcm(micStreamId)`.
/// **One path, always external.** On iOS the core never opens a miniaudio device: a MIC stream is
/// always started with `external_feed=1`, `vc_set_external_playback(1)` is set once at connect, and
/// this engine drives *both* directions through one `AVAudioEngine`:
/// - **core → speaker:** the core's mixed-output sink fills `ring`; an `AVAudioSourceNode` pulls
/// from it and renders through the VPIO output, giving AEC its reference signal.
/// from it and renders through the engine output. This runs the whole time we're connected,
/// so remote audio plays even before the user joins voice (no "can't hear anyone").
/// - **mic → core:** when the mic is active a tap on the input node converts to 48 kHz int16 and
/// calls `client.feedPcm(micStreamId)`.
///
/// Lifecycle is driven by `SessionState` join/leave. The Stereo Mic / Studio / A2DP presets keep
/// the core's miniaudio path instead (they want raw / stereo / no-AEC routing VPIO can't provide).
/// Echo cancellation / noise suppression / AGC come from Apple's Voice-Processing I/O unit (VPIO),
/// which `inputNode.setVoiceProcessingEnabled(true)` enables. VPIO forces mono, so it is engaged
/// only when the active preset wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`) — the
/// Stereo Mic / A2DP configs run the same engine with VPIO off.
///
/// Every preset / route / interruption change funnels through `reconfigure()`: a single
/// deterministic stop → AVAudioSession reconfigure → rebuild graph → start. There is no second
/// (miniaudio) audio path to hand off to, so a switch cannot leave one direction dropped.
@MainActor
final class IOSVoiceProcessingEngine {
static let shared = IOSVoiceProcessingEngine()
final class IOSAudioEngine {
static let shared = IOSAudioEngine()
private(set) var isRunning = false
/// True while connected (between `startListening` and `stop`) — the playback graph should run.
private(set) var isConnected = false
/// True while a local mic stream is active — the input tap should be installed.
private(set) var micActive = false
private let engine = AVAudioEngine()
private var sourceNode: AVAudioSourceNode?
private weak var client: VoiceCatClient?
private var micStreamId: UInt32 = 0
private var captureChannels: UInt32 = 1
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders and the engine
// processes in. The core delivers 48 kHz stereo int16 via the mixed-output sink.
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders. The core
// delivers 48 kHz stereo int16 via the mixed-output sink; mainMixerNode adapts to the route.
private let outFormat = AVAudioFormat(
commonFormat: .pcmFormatFloat32, sampleRate: 48000, channels: 2, interleaved: false)!
@@ -121,33 +131,138 @@ final class IOSVoiceProcessingEngine {
private let renderScratchFrames = 8192
private let renderScratch: UnsafeMutablePointer<Int16>
// Mic-feed converter (input-node format → 48 kHz int16) and its target buffer. Owned here so
// the (background) tap block reuses them instead of allocating per callback.
private var micConverter: AVAudioConverter?
private var micTargetFormat: AVAudioFormat?
private init() {
renderScratch = UnsafeMutablePointer<Int16>.allocate(capacity: renderScratchFrames * 2)
renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2)
}
/// Start the VPIO engine for an active mic stream. The caller must have already enabled
/// external playback on the core (`client.setExternalPlayback(true)` + `audioRestart()`) and
/// started the MIC stream with `externalFeed: true`.
func start(client: VoiceCatClient, micStreamId: UInt32, captureChannels: UInt32) {
guard !isRunning else { return }
// MARK: - Lifecycle
/// Begin playback-only (listening) operation. Called once at connect, after
/// `client.setExternalPlayback(true)` and `AudioSessionManager.ensureSessionActive()`. Attaches
/// the source node, wires the core's mixed-output sink into the ring, and starts the engine so
/// remote audio plays immediately.
func startListening(client: VoiceCatClient) {
self.client = client
self.micStreamId = micStreamId
guard !isConnected else { return }
isConnected = true
ring.reset()
// Enable the voice-processing I/O unit (AEC/NS/AGC) on the shared input+output unit.
do {
try engine.inputNode.setVoiceProcessingEnabled(true)
} catch {
logger.error("setVoiceProcessingEnabled failed: \(error.localizedDescription) — AEC unavailable")
// Wire the core's mixed-output sink into the ring (C function pointer, no captures). Stays
// registered for the whole connection; the ring is drained by the source-node render block.
let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque()
client.setMixedOutputSink({ user, pcm, spc, ch, _ in
guard let user, let pcm else { return }
let ring = Unmanaged<PCMRing>.fromOpaque(user).takeUnretainedValue()
ring.write(pcm, count: spc * Int(ch))
}, user: ringPtr)
rebuild()
}
// ── Playback: source node pulls mixed PCM from the ring through the VPIO output. ──
/// Tear down the engine and unhook the core sink. Called on disconnect.
func stop() {
guard isConnected else { return }
micActive = false
isConnected = false
client?.setMixedOutputSink(nil, user: nil)
engine.inputNode.removeTap(onBus: 0)
if engine.isRunning { engine.stop() }
logger.info("audio engine stopped — ring written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples")
try? engine.inputNode.setVoiceProcessingEnabled(false)
if let src = sourceNode {
engine.detach(src)
sourceNode = nil
}
ring.reset()
client = nil
}
// MARK: - Mic transitions
/// Engage the mic: install the input tap and (if the preset wants it) VPIO. Called when the
/// user joins voice, after the MIC stream (external_feed) is started.
func startMic(streamId: UInt32, channels: UInt32) {
micStreamId = streamId
captureChannels = channels
micActive = true
rebuild()
}
/// Disengage the mic: remove the tap and VPIO, keep playback running for remaining remote audio.
func stopMic() {
guard micActive else { return }
micActive = false
rebuild()
}
/// Update the capture channel count (mono↔stereo) for the active mic and rebuild.
func setCaptureChannels(_ channels: UInt32) {
captureChannels = channels
if let client, micStreamId != 0 {
client.setCaptureChannels(streamId: micStreamId, channels: channels)
}
if micActive { rebuild() }
}
/// Re-apply the engine graph against the current AVAudioSession config (preset / route change).
/// Safe to call when only listening — it just rebuilds the playback graph against the new route.
func reconfigure() {
guard isConnected else { return }
rebuild()
}
// MARK: - Graph (re)build
/// The single place that (re)builds and starts the engine graph. Deterministic: stop → set
/// VPIO → (re)install the mic tap → start. The caller is responsible for having applied the
/// AVAudioSession config (category/mode/route) first (`IOSAudioRouter.applyConfiguration`).
private func rebuild() {
guard isConnected else { return }
if engine.isRunning { engine.stop() }
engine.inputNode.removeTap(onBus: 0)
let useVPIO = micActive && IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
do {
try engine.inputNode.setVoiceProcessingEnabled(useVPIO)
} catch {
logger.error("setVoiceProcessingEnabled(\(useVPIO)) failed: \(error.localizedDescription)")
}
if useVPIO {
// AGC is the one VPIO sub-stage iOS exposes; AEC+NS are bundled into the master switch.
engine.inputNode.isVoiceProcessingAGCEnabled = IOSAudioRouter.shared.agcEnabled
}
// (Re)build the playback source node AFTER the VPIO state is set, so it connects against the
// correct (voice-processed or plain) output unit — mirrors the proven original ordering.
rebuildSourceNode()
if micActive { installMicTap() }
engine.prepare()
do {
try engine.start()
let inFmt = engine.inputNode.outputFormat(forBus: 0)
let outFmt = engine.outputNode.outputFormat(forBus: 0)
let route = AVAudioSession.sharedInstance().currentRoute.outputs
.map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ")
logger.info("""
engine started — mic=\(self.micActive) vpio=\(useVPIO) captureCh=\(self.captureChannels) \
inFormat=\(inFmt) outputNode=\(outFmt) outputRoute=[\(route)]
""")
} catch {
logger.error("engine start failed: \(error.localizedDescription)")
}
}
/// Detach any previous source node and attach a fresh one pulling mixed PCM from the ring.
/// Rebuilt on every graph rebuild so it always connects against the current output unit (the
/// VPIO state can change the output between rebuilds). Its format is route-independent —
/// `mainMixerNode` adapts 48 kHz stereo to whatever the output route is.
private func rebuildSourceNode() {
if let old = sourceNode {
engine.detach(old)
sourceNode = nil
}
let ring = self.ring
let scratch = self.renderScratch
let scratchFrames = self.renderScratchFrames
@@ -156,17 +271,11 @@ final class IOSVoiceProcessingEngine {
let abl = UnsafeMutableAudioBufferListPointer(ablPtr)
let n = min(frames, scratchFrames)
let got = ring.read(into: scratch, count: n * 2) / 2 // interleaved stereo → frames
// Deinterleave int16 → Float32 per channel; silence-fill any underrun tail.
let scale: Float = 1.0 / 32768.0
for ch in 0..<abl.count {
guard let base = abl[ch].mData?.assumingMemoryBound(to: Float.self) else { continue }
for i in 0..<frames {
if i < got {
let s = scratch[i * 2 + min(ch, 1)]
base[i] = Float(s) * scale
} else {
base[i] = 0
}
base[i] = i < got ? Float(scratch[i * 2 + min(ch, 1)]) * scale : 0
}
}
return noErr
@@ -174,29 +283,33 @@ final class IOSVoiceProcessingEngine {
sourceNode = src
engine.attach(src)
engine.connect(src, to: engine.mainMixerNode, format: outFormat)
// ── Mic: tap the VPIO input node, convert to 48 kHz int16, feed the core. ──
let inFormat = engine.inputNode.outputFormat(forBus: 0)
let targetCh = max(1, min(2, captureChannels))
let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000,
channels: AVAudioChannelCount(targetCh), interleaved: true)
micTargetFormat = target
micConverter = (target != nil && inFormat.sampleRate > 0)
? AVAudioConverter(from: inFormat, to: target!) : nil
if micConverter == nil {
logger.error("mic converter unavailable (in=\(inFormat)) — mic will not transmit")
}
let c = client
/// Install the mic tap: convert the input node's native format to 48 kHz int16 (mono or
/// stereo per `captureChannels`) and feed it to the core. Rebuilds the converter each time
/// because the input format depends on the VPIO state and the active route.
private func installMicTap() {
guard let client else { return }
let inFormat = engine.inputNode.outputFormat(forBus: 0)
guard inFormat.sampleRate > 0 else {
logger.error("input format unavailable (\(inFormat)) — mic will not transmit")
return
}
let targetCh = max(1, min(2, captureChannels))
guard let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000,
channels: AVAudioChannelCount(targetCh), interleaved: true),
let converter = AVAudioConverter(from: inFormat, to: target) else {
logger.error("mic converter unavailable (in=\(inFormat), ch=\(targetCh)) — mic will not transmit")
return
}
let sid = micStreamId
let converter = micConverter
let tgt = micTargetFormat
let c = client
engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in
guard let converter, let tgt else { return }
// Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample.
let ratio = tgt.sampleRate / buffer.format.sampleRate
let ratio = target.sampleRate / buffer.format.sampleRate
let outCap = AVAudioFrameCount(Double(buffer.frameLength) * ratio + 16)
guard let outBuf = AVAudioPCMBuffer(pcmFormat: tgt, frameCapacity: outCap) else { return }
guard let outBuf = AVAudioPCMBuffer(pcmFormat: target, frameCapacity: outCap) else { return }
var fed = false
let status = converter.convert(to: outBuf, error: nil) { _, outStatus in
if fed { outStatus.pointee = .noDataNow; return nil }
@@ -206,64 +319,9 @@ final class IOSVoiceProcessingEngine {
}
guard status != .error, outBuf.frameLength > 0,
let chData = outBuf.int16ChannelData else { return }
let spc = Int(outBuf.frameLength)
// int16 interleaved → channelData[0] is the interleaved buffer for interleaved formats.
c.feedPcm(streamId: sid, pcm: chData[0], samplesPerChannel: spc, channels: targetCh)
// int16 interleaved → channelData[0] is the interleaved buffer.
c.feedPcm(streamId: sid, pcm: chData[0],
samplesPerChannel: Int(outBuf.frameLength), channels: targetCh)
}
// ── Wire the core's mixed-output sink into the ring (C function pointer, no captures). ──
let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque()
client.setMixedOutputSink({ user, pcm, spc, ch, _ in
guard let user, let pcm else { return }
let ring = Unmanaged<PCMRing>.fromOpaque(user).takeUnretainedValue()
ring.write(pcm, count: spc * Int(ch))
}, user: ringPtr)
engine.prepare()
do {
try engine.start()
isRunning = true
// Diagnostics: capture the negotiated graph formats and the live output route so a
// silent-playback report can be triaged (format/rate mismatch vs. routing vs. the
// core not delivering PCM — see the ring stats logged in teardown()).
let outFmt = engine.outputNode.outputFormat(forBus: 0)
let mixFmt = engine.mainMixerNode.outputFormat(forBus: 0)
let route = AVAudioSession.sharedInstance().currentRoute.outputs
.map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ")
logger.info("""
VPIO engine started — inFormat=\(inFormat), captureCh=\(targetCh), \
outputNode=\(outFmt), mainMixer=\(mixFmt), outputRoute=[\(route)]
""")
} catch {
logger.error("VPIO engine start failed: \(error.localizedDescription)")
teardown()
}
}
/// Stop the VPIO engine. The caller is responsible for restoring the core's hardware playback
/// afterwards (`client.setExternalPlayback(false)` + `audioRestart()`).
func stop() {
guard isRunning else { return }
teardown()
logger.info("VPIO engine stopped")
}
private func teardown() {
client?.setMixedOutputSink(nil, user: nil)
engine.inputNode.removeTap(onBus: 0)
if engine.isRunning { engine.stop() }
// Diagnostics (threads now quiesced): how much mixed PCM the core delivered into the ring
// vs. how much the render thread consumed. written==0 ⇒ the core never delivered (Bug 1
// core/lifecycle path); written>0 with no audible output ⇒ the AVAudioEngine output graph.
logger.info("VPIO ring stats — written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples")
try? engine.inputNode.setVoiceProcessingEnabled(false)
if let src = sourceNode {
engine.detach(src)
sourceNode = nil
}
micConverter = nil
micTargetFormat = nil
ring.reset()
isRunning = false
}
}

91
clients/apple/iOS/VoiceCatiOS/SessionState.swift
View File

@@ -59,7 +59,6 @@ final class SessionState {
self.client = client
self.selfUserId = selfUserId
self.permissions = permissions
AudioSessionManager.shared.client = client
refreshChannels()
refreshUsers()
syncSelfChannel()
@@ -73,16 +72,10 @@ final class SessionState {
broadcastPump.onBroadcastStarted = { [weak self] in self?.startScreenShare() }
broadcastPump.onBroadcastFinished = { [weak self] in self?.stopScreenShare() }
broadcastPump.start()
// When IOSAudioRouter changes the audio config, restart the voice path if needed so the
// native VPIO engine (AEC/NS/AGC) engages or disengages to match the new preset/route.
AudioSessionManager.shared.reconcileVoicePath = { [weak self] in self?.reconcileVoicePath() }
}
deinit {
broadcastPump.stop()
MainActor.assumeIsolated {
AudioSessionManager.shared.client = nil
}
}
// MARK: - Event dispatch
@@ -259,78 +252,39 @@ final class SessionState {
return
}
// VPIO path: on the AEC presets, the native AVAudioEngine does AEC/NS/AGC and the core
// runs in external mode (no hardware mic/playback). The mic stream is started with
// externalFeed so the core skips the hardware capture device; setExternalPlayback makes
// it skip the hardware playback device and deliver the mix to IOSVoiceProcessingEngine.
//
// ORDER MATTERS: set the external-playback flag now, but defer audioRestart() until
// AFTER startStream (below) so the MIC LocalStream — which carries external_feed=true —
// already exists when ensure_audio_running() derives external_capture. Restarting before
// the stream exists makes the core reopen a hardware capture device that is never dropped
// (the announce-result restart early-returns because the engine is already running); that
// lingering miniaudio capture unit then fights the AVAudioEngine VPIO unit on the same
// .voiceChat session and silences VPIO playback.
let useVPIO = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
client.setExternalPlayback(useVPIO)
// Unified iOS path: the core is always external (set at connect via setExternalPlayback +
// every MIC stream external_feed), and `IOSAudioEngine` drives capture + playback. So the
// mic stream is just started with external_feed=true and the engine is told the mic is now
// active — no setExternalPlayback toggle, no audioRestart ordering, no VPIO/miniaudio fork.
let desc = StreamDescriptor(kind: .mic, deviceId: voiceState.currentDeviceId, label: "Mic",
externalFeed: useVPIO)
externalFeed: true)
let (result, streamId) = client.startStream(desc)
if result == .ok {
guard result == .ok else {
addActivity("Failed to start mic: \(result.description)")
return
}
voiceState.micActive = true
voiceState.localStreamId = streamId
EventFeedback.shared.play(.voiceOn)
// Publish the active mic stream ID so IOSAudioRouter can reset the core's capture
// channel count when the user switches mono↔stereo (selectCaptureChannels /
// applyPreset). Without this, switching stereo→mono leaves the LocalStream's
// capture_channels field at 2 and the next engine start still opens stereo.
AudioSessionManager.shared.activeMicStreamId = streamId
// Store the user's capture channel selection before the server acknowledges
// the stream. The engine hasn't started yet at this point (it starts when
// handle_stream_announce_result fires), so vc_set_capture_channels just
// stores the value — no restart. ensure_audio_running() picks it up when
// the stream is confirmed and opens the device with the right channel count.
let channels = IOSAudioRouter.shared.captureChannels.channelCount
if channels != 1 {
client.setCaptureChannels(streamId: streamId, channels: channels)
}
if useVPIO {
// The external-feed MIC stream now exists, so restart the core into full
// external mode (no hardware capture/playback, mixer-timer only) — mic and
// speaker are owned entirely by the VPIO engine, which we start right after.
client.audioRestart()
IOSVoiceProcessingEngine.shared.start(
client: client, micStreamId: streamId, captureChannels: channels)
}
} else {
addActivity("Failed to start mic: \(result.description)")
if useVPIO { // revert external-playback mode so remote audio still plays
client.setExternalPlayback(false)
client.audioRestart()
}
}
// Engage the mic: installs the input tap and (per preset) VPIO, in one engine rebuild.
IOSAudioEngine.shared.startMic(streamId: streamId, channels: channels)
}
func stopMicStream() {
// Tear down the VPIO engine first (removes the mic tap + unregisters the mixed sink),
// then stop the mic stream, then restore the core's hardware playback for any remaining
// remote audio. Order matters: the mic stream must be gone before audioRestart so the
// core opens a normal playback device (and no capture device — there's no mic stream).
let wasVPIO = IOSVoiceProcessingEngine.shared.isRunning
if wasVPIO {
IOSVoiceProcessingEngine.shared.stop()
}
// Disengage the mic (removes the tap + VPIO) but keep the engine running for any remaining
// remote audio. Then stop the core's MIC stream. The core stays external throughout — no
// setExternalPlayback toggle, no audioRestart.
IOSAudioEngine.shared.stopMic()
if voiceState.localStreamId != 0 {
client.stopStream(voiceState.localStreamId)
voiceState.localStreamId = 0
AudioSessionManager.shared.activeMicStreamId = nil
EventFeedback.shared.play(.voiceOff)
}
if wasVPIO {
client.setExternalPlayback(false)
client.audioRestart() // reopen hardware playback (no mic stream → no hw capture)
}
voiceState.micActive = false
voiceState.level = 0
// Do NOT deactivate the AVAudioSession here — the user may still want to hear
@@ -338,19 +292,6 @@ final class SessionState {
// disconnecting from the server (see AppState.disconnect / .disconnected event).
}
/// Restart the voice path when the audio config changes mid-call (driven by IOSAudioRouter).
/// If VPIO is involved on either the current or desired side, restart the mic so the native
/// voice-processing engine engages/disengages and re-binds to the new route. Pure miniaudio
/// config tweaks need no restart — the core's own audioRestart (already issued) handles them.
private func reconcileVoicePath() {
guard voiceState.micActive else { return }
let want = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
let have = IOSVoiceProcessingEngine.shared.isRunning
guard want || have else { return }
stopMicStream()
doStartMicStream()
}
// MARK: - Screen audio share
/// Called when the broadcast extension becomes active. Announces the SCREEN_AUDIO stream;

36
clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift
View File

@@ -38,9 +38,9 @@ struct SettingsView: View {
.accessibilityLabel("Speaker output")
.accessibilityHint("Routes audio to the speaker instead of the earpiece when no headphones are connected.")
// Surface the voice-processing state. On the AEC presets the native iOS
// Surface the voice-processing state. On Voice Chat the native iOS
// Voice-Processing unit (VPIO) does echo cancellation, noise suppression and
// automatic gain control; the other presets (stereo/studio/A2DP) can't use it.
// automatic gain control; the stereo / mono-mic / A2DP configs can't use it.
if router.currentConfigUsesVoiceProcessing {
Label("Echo cancellation & noise suppression on (iOS voice processing)",
systemImage: "waveform.badge.mic")
@@ -48,18 +48,11 @@ struct SettingsView: View {
.foregroundStyle(.secondary)
.accessibilityLabel("Echo cancellation and noise suppression are on")
} else {
Label("No echo cancellation in this preset (stereo / studio / A2DP)",
Label("No echo cancellation in this configuration (stereo / A2DP / off)",
systemImage: "waveform.slash")
.font(.caption)
.foregroundStyle(.secondary)
.accessibilityLabel("Echo cancellation is off in this preset")
}
if !router.hasBluetoothDevice && !router.hasWiredHeadset {
Text("Connect Bluetooth headphones or a wired headset for more presets.")
.font(.caption)
.foregroundStyle(.secondary)
.accessibilityLabel("No external audio device connected")
.accessibilityLabel("Echo cancellation is off in this configuration")
}
}
@@ -127,6 +120,27 @@ struct SettingsView: View {
}
.accessibilityLabel("Microphone processing mode")
// Voice-processing (VPIO) controls — only meaningful on a VPIO-capable
// config (mono + standard + non-A2DP). iOS bundles echo cancellation and
// noise suppression into one master switch (no per-stage toggle); AGC is
// the one sub-stage it lets us control independently.
if router.voiceProcessingAvailable {
Toggle("Voice Processing (AEC + noise suppression)", isOn: Binding(
get: { router.voiceProcessingEnabled },
set: { router.setVoiceProcessingEnabled($0) }
))
.accessibilityLabel("Voice processing")
.accessibilityHint("Echo cancellation and noise suppression, bundled together by iOS.")
if router.voiceProcessingEnabled {
Toggle("Automatic Gain Control", isOn: Binding(
get: { router.agcEnabled },
set: { router.setAgcEnabled($0) }
))
.accessibilityLabel("Automatic gain control")
}
}
if router.showsRawModeSpeakerWarning {
Label(
"Raw mode on speaker — echo risk (no AEC)",

71
docs/voice.md
View File

@@ -230,37 +230,46 @@ Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth
in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
- **iOS mic capture:** all iOS audio routing is driven from Swift via `AVAudioSession` by the
`IOSAudioRouter` singleton *before* the core (miniaudio) opens its device — miniaudio does
NOT touch `AVAudioSession` on iOS. Input port selection (`availableInputs`), built-in mic
orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns
(`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic processing mode
(`.voiceChat` = Standard with AEC/AGC/HPF, or `.measurement` = Raw/Studio with all processing
off), Bluetooth mode (`.allowBluetoothHFP` HFP voice vs `.allowBluetoothA2DP` stereo output
vs neither), and stereo capture (`.stereo` polar pattern + `setPreferredInput` +
`setInputDataSource` → `vc_set_capture_channels`) are all set from Swift. The core then
opens whatever route AVAudioSession has established. When the user changes audio settings
mid-session, `IOSAudioRouter` suspends the core's devices (`vc_audio_suspend`), reconfigures
`AVAudioSession`, then restarts the devices (`vc_audio_restart`) so they reopen against the
new route — mirroring TeamTalk5's `closeSoundDevices`/`initSoundInputDevice`/
`initSoundOutputDevice` pattern.
- **iOS voice processing (AEC/NS/AGC) — native VPIO path.** Real iOS echo cancellation, noise
suppression and AGC are provided ONLY by Apple's **Voice-Processing I/O audio unit (VPIO)**,
*not* by the `AVAudioSession` mode alone. The core uses miniaudio's plain `RemoteIO` audio
units, which never engage VPIO — so `.voiceChat` mode by itself yields no AEC. For VPIO to
cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
played-back signal from the mic), so on the AEC presets (Voice Chat / Bluetooth Headset HFP /
Wired Headset) the Swift layer runs a native `AVAudioEngine` with
`inputNode.setVoiceProcessingEnabled(true)` and the core runs in **external mode**:
- **Mic:** the MIC stream is started with `vc_stream_desc.external_feed=1`; the VPIO input tap
feeds processed mic PCM via `vc_stream_feed_pcm`. The core skips its hardware capture device
(`AudioParams.external_capture`).
- **Playback:** `vc_set_external_playback(1)` makes the core skip its hardware playback device;
a mixer-timer thread drives decode+mix on a ~20 ms cadence and delivers the FINAL mixed PCM
via `vc_set_mixed_output_sink`. The Swift engine renders that through the VPIO output, so
VPIO has its echo-cancellation reference signal.
The Stereo Mic / Studio / A2DP presets keep the miniaudio path (they want raw / stereo /
no-AEC routing that VPIO can't provide — VPIO forces mono).
- **iOS audio — one path, always external.** On iOS the core **never opens a miniaudio device**:
a single `AVAudioEngine` (`IOSAudioEngine`) drives *both* directions, and the core runs fully
external for the whole connection. This is the single most important property of the iOS audio
stack — there is no second (miniaudio) path to switch to, so a preset/route change cannot leave
one direction dropped. The single ordering rule is: `vc_set_external_playback(1)` is set **once
at connect** (before the session is activated or any remote stream arrives), and every MIC
stream is started with `vc_stream_desc.external_feed=1`.
- **core → speaker:** the core's mixer-timer thread decodes+mixes on a ~20 ms cadence and
delivers the FINAL mixed PCM via `vc_set_mixed_output_sink`; an `AVAudioSourceNode` pulls it
from a lock-free ring and renders it. This runs the whole time we are connected, so remote
audio plays even before the user joins voice (kills the "can't hear anyone" race).
- **mic → core:** when the mic is active a tap on the engine's input node converts to 48 kHz
int16 (`vc_set_capture_channels` decides mono/stereo) and calls `vc_stream_feed_pcm`.
- **iOS routing** is still driven from Swift via `AVAudioSession` by the `IOSAudioRouter`
singleton — miniaudio never touches `AVAudioSession` on iOS. Input port selection
(`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom),
polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic
processing mode (Standard vs `.measurement` Raw), Bluetooth mode (`.allowBluetoothHFP` HFP voice
vs `.allowBluetoothA2DP` stereo output vs neither), and stereo capture (`.stereo` polar pattern
+ `setPreferredInput` + `setInputDataSource` → `vc_set_capture_channels`) are all set from
Swift. Any preset / route / interruption change funnels through one deterministic, Swift-only
rebuild: `IOSAudioEngine` stops, `IOSAudioRouter.applyConfiguration()` re-applies the
`AVAudioSession`, the graph is rebuilt against the new route, and the engine restarts. No
`vc_audio_restart`/`vc_audio_suspend` dance is needed for routing (the core has no hardware
devices to reopen) — this is the spirit of TeamTalk5's "close then re-init sound devices", but
entirely inside the Swift engine.
- **iOS voice processing (AEC/NS/AGC) — native VPIO.** Real iOS echo cancellation, noise
suppression and AGC come ONLY from Apple's **Voice-Processing I/O audio unit (VPIO)**, which
`inputNode.setVoiceProcessingEnabled(true)` enables; for it to cancel echo it must own BOTH the
mic capture and the playback — which the unified engine already does. VPIO forces **mono**, so
it is engaged only when the active config wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`:
mono + standard + non-A2DP + the user's master toggle). iOS exposes no per-stage VPIO control,
so the Advanced UI offers exactly two switches: a master **Voice Processing** (AEC + NS bundled)
and **AGC** (`isVoiceProcessingAGCEnabled`).
- **iOS presets** (`IOSAudioRouter.AudioPreset`): **Voice Chat** (VPIO mono, system output incl.
HFP/wired), **Stereo Mic** (internal stereo built-in mic regardless of output, A2DP-capable, no
VPIO), **Mono Mic** (internal mono built-in mic regardless of output, A2DP-capable, no VPIO),
and **Advanced** (every knob manual). A2DP output requires an internal-mic preset (the Bluetooth
device is output-only); the Stereo/Mono Mic presets fall back to the built-in speaker when no
external output is connected (`applyA2dpSpeakerFallback`).
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard