diff --git a/PROGRESS.md b/PROGRESS.md index 38510df..b2faad1 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -75,8 +75,42 @@ up instantly. Newest status at the top. is now correct. If voice still fails, watch the server's rate-limited `[media] dropped frames — unmapped-endpoint=…` line (NAT source-port rewrite would be the next suspect). -- **Awaiting on-device verification (2026-06-22):** **iOS real echo cancellation / noise suppression - via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS +- **Done (2026-06-23, Swift-only — no core/ABI change; awaiting on-device verification):** **iOS audio + stack unified — one always-external `AVAudioEngine`, miniaudio dropped on iOS.** The iOS audio path was + a fragile hybrid: Voice-Chat-class presets ran a native VPIO `AVAudioEngine` (core external) while + Stereo/Studio/A2DP presets ran the core's miniaudio devices. Nearly every bug lived in the seam + (lingering miniaudio capture unit fighting VPIO, the `audioRestart` ordering dance, the route-change + "glitching" loop, stereo↔mono stickiness, "can't hear anyone"), and switching presets/routes mid-call + routinely dropped input, output, or both. **Fix: drive *all* iOS audio through one `AVAudioEngine` with + the core fully external at all times** — `vc_set_external_playback(1)` once at connect, every MIC stream + `external_feed=1`, mic via `vc_stream_feed_pcm`, playback via `vc_set_mixed_output_sink`. + - `IOSVoiceProcessingEngine.swift` → **`IOSAudioEngine`** (same file): always-on `AVAudioSourceNode` + playback (runs whenever connected, so remote audio plays before you join voice); conditional mic tap; + VPIO + AGC toggled per config. One private `rebuild()` (stop → set VPIO → install tap → start) backs + `startListening`/`stop`/`startMic`/`stopMic`/`reconfigure`/`setCaptureChannels`. Kept the `PCMRing` + and ring-stats diagnostics. + - `IOSAudioRouter`: presets cut from seven to **four** — Voice Chat (VPIO mono, system output), + Stereo Mic / Mono Mic (internal built-in mic regardless of output, A2DP-capable, no VPIO), Advanced + (manual). New persisted `voiceProcessingEnabled` (master AEC+NS) + `agcEnabled`; setters now call + `IOSAudioEngine.reconfigure()` instead of `client.audioRestart()` + `reconcileVoicePath`. Kept the + proven AVAudioSession recipes (category/mode/options, stereo capsule, `applyA2dpSpeakerFallback`). + - `AudioSessionManager` slimmed (drops `client`/`activeMicStreamId`/`reconcileVoicePath`; adds + `isActive`); interruption-end & device-change now `reconfigure()` the engine. `SessionState` + `doStartMicStream`/`stopMicStream` collapsed to start-stream + `startMic`/`stopMic` (no + `setExternalPlayback`/`audioRestart` toggling); `reconcileVoicePath` deleted. `AppState` sets external + playback + `startListening` at connect, `stop()` at disconnect. `SettingsView` → four presets + + Advanced VPIO/AGC toggles. + - **No core/ABI/test change** — relies on the already-shipped `vc_set_external_playback` / + `external_feed` / `vc_set_mixed_output_sink` / `vc_stream_feed_pcm` path (`test_external_pcm`, + `test_external_playback`). `xcodebuild` iOS device Debug **BUILD SUCCEEDED**. **Rebuild the + xcframework is NOT required** (no new symbols). + - **Next (user, on device):** two iPhones in a channel — verify BOTH directions survive every + transition and are never silent unless intended: Voice Chat (no echo, NR), listen-only before joining, + join↔leave repeatedly, switch Voice Chat↔Stereo↔Mono↔Advanced *while in voice*, A2DP connect/unplug, + wired connect/unplug, phone-call interruption + resume, screen-audio share. + +- **Superseded by the 2026-06-23 unification above (2026-06-22):** **iOS real echo cancellation / noise + suppression via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS AEC/NS/AGC come only from Apple's Voice-Processing I/O unit (VPIO), but the core uses miniaudio's plain RemoteIO units — so `.voiceChat` mode alone never engaged AEC. Fix moves both mic capture and playback to a native Swift `AVAudioEngine` (`setVoiceProcessingEnabled`) on the AEC presets, with the diff --git a/clients/apple/iOS/VoiceCatiOS/AppState.swift b/clients/apple/iOS/VoiceCatiOS/AppState.swift index daf58c2..46b936a 100644 --- a/clients/apple/iOS/VoiceCatiOS/AppState.swift +++ b/clients/apple/iOS/VoiceCatiOS/AppState.swift @@ -91,6 +91,7 @@ final class AppState { func disconnect() { session?.stopMicStream() session?.client.disconnect() + IOSAudioEngine.shared.stop() AudioSessionManager.shared.deactivateSession() session = nil connectingClient?.disconnect() @@ -160,17 +161,18 @@ final class AppState { self.session = newSession EventFeedback.shared.play(.login) EventFeedback.shared.speak("Connected") - // Activate the audio session now, while connected — NOT lazily when the first - // remote stream arrives. The core opens its miniaudio playback device the moment - // a remote stream starts and only THEN emits .streamStarted; if we waited for - // that event to activate, the playback device would open against an inactive - // AVAudioSession and produce no sound (the "can't hear anyone" bug). Activating - // here guarantees the session is live before any device opens. + // Put the core into external-playback mode ONCE, now, before the session is + // activated or any remote stream can arrive — so the core never opens a miniaudio + // device on iOS (the single ordering rule of the unified audio path). Then activate + // the session and start the engine in listening mode so remote audio plays the + // moment someone talks, even before we join voice (no "can't hear anyone"). + client.setExternalPlayback(true) do { try AudioSessionManager.shared.ensureSessionActive() } catch { print("Audio session activate on connect failed: \(error)") } + IOSAudioEngine.shared.startListening(client: client) } else { connectStatus = "Auth failed: \(ev.result.description)" showPasswordPrompt = true @@ -178,6 +180,7 @@ final class AppState { case .disconnected: if session == nil { cancelConnect() } else { + IOSAudioEngine.shared.stop() AudioSessionManager.shared.deactivateSession() session = nil; isConnecting = false } diff --git a/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift b/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift index 6296582..a1c4cbf 100644 --- a/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift +++ b/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift @@ -8,19 +8,6 @@ private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "Audio final class AudioSessionManager { static let shared = AudioSessionManager() - weak var client: VoiceCatClient? - - /// The stream ID of the currently active local MIC stream, if any. Set by `SessionState` - /// when the user joins/leaves voice so `IOSAudioRouter` can reset the core's capture - /// channel count (e.g. when switching stereo → mono) without going through `SessionState`. - var activeMicStreamId: UInt32? - - /// Set by `SessionState`. Invoked by `IOSAudioRouter` after an audio-config change so the - /// voice path (native VPIO vs the core's miniaudio path) can be restarted to match the new - /// preset/route when the mic is active. No-op when not in voice. See - /// `SessionState.reconcileVoicePath()` and `IOSVoiceProcessingEngine`. - var reconcileVoicePath: (() -> Void)? - /// Tracks whether WE activated the session. The session must be active whenever the /// AudioEngine is running (for capture OR playback), so it is activated when any audio /// needs to play (a remote stream started OR the user joins voice) and only deactivated @@ -28,6 +15,10 @@ final class AudioSessionManager { /// want to hear remote audio. private var isSessionActive = false + /// Whether the AVAudioSession is currently active (we activated it). Read by `IOSAudioRouter` + /// to decide whether the post-activation A2DP speaker fallback can be applied. + var isActive: Bool { isSessionActive } + func configure() { // Load stored audio routing preferences and apply them before any audio session // activation. IOSAudioRouter drives all iOS audio route selection via AVAudioSession; @@ -114,18 +105,22 @@ final class AudioSessionManager { switch type { case .began: + // The system stops our AVAudioEngine and deactivates the session. Nothing to tear + // down — `IOSAudioEngine` rebuilds on resume. logger.info("interruption began — session suspended by system") - isSessionActive = false // system deactivated us - client?.audioSuspend() + isSessionActive = false case .ended: let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0 let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue) if options.contains(.shouldResume) { do { + IOSAudioRouter.shared.applyConfiguration() try AVAudioSession.sharedInstance().setActive(true) isSessionActive = true - logger.info("interruption ended — session reactivated") - client?.audioResume() + IOSAudioRouter.shared.applyA2dpSpeakerFallback() + // Rebuild the engine graph against the restored route (both directions). + IOSAudioEngine.shared.reconfigure() + logger.info("interruption ended — session reactivated, engine rebuilt") } catch { logger.error("interruption ended — reactivation failed: \(error.localizedDescription)") } @@ -162,6 +157,9 @@ final class AudioSessionManager { // the loud speaker (not the earpiece), and a replug should hand output back to A2DP. if isSessionActive { IOSAudioRouter.shared.applyA2dpSpeakerFallback() + // Rebind the engine (both directions) to the new route. The engine owns the route + // now, so this is the single thing that re-establishes audio after a device change. + IOSAudioEngine.shared.reconfigure() } } diff --git a/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift b/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift index c9dc7d7..91d528d 100644 --- a/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift +++ b/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift @@ -4,17 +4,18 @@ import VoiceCatCore private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioRouter") -/// iOS audio routing layer — drives all iOS audio route selection via `AVAudioSession` -/// *before* the core (miniaudio) opens its device. This class is the sole owner of the -/// session: miniaudio does NOT touch `AVAudioSession` on iOS, because the core opens its -/// devices through a `ma_context` configured with `sessionCategory = none` + -/// `noAudioSessionActivate/Deactivate` (see `AudioEngine::make_context_config` in -/// `core/src/audio/audio_engine.cpp`). Without that, miniaudio's default path resets the -/// category to `Record`/`Playback` with no options on every device open, wiping -/// `.allowBluetoothA2DP`/`.playAndRecord` and killing headphone/A2DP output — so that -/// config must stay in place. All iOS audio routing (input port selection, mic -/// orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) must be -/// driven from here. +/// iOS audio routing layer — the sole owner of `AVAudioSession` on iOS. On iOS the core never +/// opens a hardware (miniaudio) device: a single `AVAudioEngine` (`IOSAudioEngine`) drives both +/// capture and playback and the core runs fully external (see docs/voice.md §8). This class just +/// configures the *route* — category / mode / options, preferred input, data source, polar +/// pattern, stereo capsule — and `IOSAudioEngine` binds to whatever route is established. After +/// any change here the engine is rebuilt via `IOSAudioEngine.reconfigure()` (a deterministic +/// Swift-only stop → reconfigure → start); there is no second (miniaudio) audio path to hand off +/// to, so a change cannot leave one direction dropped. +/// +/// (The core's iOS `ma_context` is still configured with `sessionCategory = none` + +/// `noAudioSessionActivate/Deactivate` in `AudioEngine::make_context_config` so that, should the +/// core ever open a device, miniaudio would not reset the category — but on iOS it does not.) /// /// The three user-facing choices: /// 1. **Input port** — which physical input (built-in mic, Bluetooth HFP, headset, @@ -64,77 +65,59 @@ final class IOSAudioRouter: ObservableObject { @Published var selectedInputPortId: String? @Published var selectedDataSourceId: String? @Published var selectedPolarPattern: String? + /// Master voice-processing switch (Apple VPIO: AEC + noise suppression bundled together). + /// iOS exposes no per-stage toggle, so this is the finest "echo cancellation / noise + /// reduction" control available. Only takes effect on a VPIO-capable config (mono + standard + /// + not A2DP); stereo / A2DP configs can't use VPIO regardless. Default on. + @Published var voiceProcessingEnabled: Bool = true + /// VPIO automatic gain control — the one VPIO sub-stage iOS lets us toggle independently. + /// Only meaningful when voice processing is active. Default on. + @Published var agcEnabled: Bool = true @Published var showsRawModeSpeakerWarning: Bool = false @Published var showsA2dpNoAecWarning: Bool = false @Published var hasBluetoothDevice: Bool = false @Published var hasWiredHeadset: Bool = false - /// Audio presets — sensible combinations of settings for common scenarios. - /// The app is about choice: users can pick a preset for a quick start, then - /// fine-tune individual settings under "Advanced Audio". + /// Audio presets — the four scenarios from the product spec. Pick a preset for a quick start, + /// then fine-tune individual settings under "Advanced". HFP / wired headsets are not separate + /// presets: Voice Chat lets the system route to them, and Advanced exposes manual selection. enum AudioPreset: String, CaseIterable, Identifiable { - /// Standard iOS VoIP experience: AEC/AGC/HPF on, mono, system picks best route - /// (BT HFP if connected, wired if connected, speaker if nothing). Always available. + /// Voice chat: Apple VPIO does real AEC + noise suppression + AGC. Mono. The system picks + /// the best route (Bluetooth HFP / wired / speaker / earpiece). Always available. case voiceChat = "Voice Chat" - /// Stereo built-in mic capture (front+back capsules). A2DP output if BT is connected, - /// else built-in speaker / wired. Standard processing (no AEC — stereo needs a non-VPIO - /// mode). Always available. + /// Internal **stereo** built-in mic regardless of the output route. A2DP output when a + /// Bluetooth headset is connected, else built-in speaker / wired. No VPIO (stereo can't + /// use it). Always available. case stereoMic = "Stereo Mic" - /// Maximum fidelity: stereo mic, no AEC/AGC/HPF (raw mode). A2DP output if BT connected, - /// else speaker/wired. Always available. Echo risk on speaker. - case studio = "Studio (No Processing)" - /// Bluetooth HFP: BT mic + BT output, AEC on, mono. Only when BT is connected. - case bluetoothHeadset = "Bluetooth Headset (HFP)" - /// A2DP stereo output + built-in mono mic, AEC off. Only when BT is connected. (For - /// A2DP output + stereo mic, use the Stereo Mic preset while BT is connected.) - case btHeadphonesMonoMic = "BT Headphones + Mono Mic" - /// Wired headset/earpods: wired output + wired mic (or built-in), AEC on, mono. - /// Only when a wired audio device is connected. - case wiredHeadset = "Wired Headset" - /// Settings don't match any preset — user has tweaked advanced controls. - case custom = "Custom" + /// Internal **mono** built-in mic regardless of the output route. A2DP output when a + /// Bluetooth headset is connected, else built-in speaker / wired. No VPIO. Always available. + case monoMic = "Mono Mic" + /// Everything manual — input port, mic orientation / polar pattern, mono/stereo, Bluetooth + /// mode, raw vs standard, and the VPIO / AGC toggles. Also the display state when the + /// individual settings don't match a named preset. + case advanced = "Advanced" var id: String { rawValue } - var requiresBluetooth: Bool { - switch self { - case .bluetoothHeadset, .btHeadphonesMonoMic: return true - default: return false - } - } - - var requiresWired: Bool { - self == .wiredHeadset - } - var bluetoothMode: BluetoothMode { switch self { - case .voiceChat, .bluetoothHeadset: return .btHfpVoice - // A2DP output when BT is connected; falls back to speaker/wired when it isn't. - case .stereoMic, .studio, .btHeadphonesMonoMic: return .builtInMicBtA2dp - case .wiredHeadset: return .builtInMicSpeaker - case .custom: return .builtInMicSpeaker // placeholder + case .voiceChat: return .btHfpVoice + // Internal-mic presets: A2DP output when BT is connected; speaker/wired when not. + case .stereoMic, .monoMic: return .builtInMicBtA2dp + case .advanced: return .builtInMicSpeaker // placeholder; Advanced sets it manually } } var captureChannels: CaptureChannels { - switch self { - case .stereoMic, .studio: return .stereo - default: return .mono - } + self == .stereoMic ? .stereo : .mono } - var micMode: MicMode { - switch self { - case .studio: return .raw - default: return .standard - } - } + var micMode: MicMode { .standard } - /// Whether this preset explicitly selects the built-in mic port. + /// Whether this preset explicitly pins the built-in mic port (the internal-mic presets). var usesBuiltInMic: Bool { switch self { - case .stereoMic, .studio, .btHeadphonesMonoMic: return true + case .stereoMic, .monoMic: return true default: return false } } @@ -170,6 +153,8 @@ final class IOSAudioRouter: ObservableObject { private let kPolarPattern = "cat.voice.audio.polarPattern" private let kPreset = "cat.voice.audio.preset" private let kForceSpeaker = "cat.voice.audio.forceSpeaker" + private let kVoiceProcessing = "cat.voice.audio.voiceProcessing" + private let kAgc = "cat.voice.audio.agc" /// Re-entrancy guard: setCategory/setPreferredInput/etc. trigger route-change /// notifications synchronously on the same thread. Without this guard, @@ -271,55 +256,41 @@ final class IOSAudioRouter: ObservableObject { } } - /// The presets available given the current device connection state. - /// Always includes Voice Chat, Stereo Mic, Studio, and Custom. BT presets only when - /// a Bluetooth device is connected. Wired preset only when a wired device is connected. - var availablePresets: [AudioPreset] { - AudioPreset.allCases.filter { preset in - if preset == .custom { return true } - if preset.requiresBluetooth && !hasBluetoothDevice { return false } - if preset.requiresWired && !hasWiredHeadset { return false } - return true - } - } + /// The presets the user can pick. All four are always available — the named presets simply + /// describe what to do "regardless of the output route", and Advanced is always offered. + var availablePresets: [AudioPreset] { AudioPreset.allCases } - /// Which preset matches the current settings, or .custom if nothing matches. - /// Checks device-specific presets first (BT, wired) so that e.g. when BT is connected - /// and settings match "Bluetooth Headset", it returns that instead of the equivalent - /// "Voice Chat" (which has the same bluetoothMode/micMode/channels but is more general). + /// Which named preset matches the current settings, or `.advanced` if nothing matches. var activePreset: AudioPreset { - // Check device-specific presets first (most specific → least specific) - let order: [AudioPreset] = [ - .bluetoothHeadset, .btHeadphonesMonoMic, - .wiredHeadset, - .voiceChat, .stereoMic, .studio, - ] - for preset in order { + for preset in [AudioPreset.voiceChat, .stereoMic, .monoMic] { if bluetoothMode == preset.bluetoothMode && captureChannels == preset.captureChannels && micMode == preset.micMode { - // Don't match a BT preset if no BT is connected — fall through to Voice Chat - if preset.requiresBluetooth && !hasBluetoothDevice { continue } - if preset.requiresWired && !hasWiredHeadset { continue } return preset } } - return .custom + return .advanced } - /// Whether the current configuration should use the native iOS Voice-Processing path (VPIO: - /// real AEC/NS/AGC via `IOSVoiceProcessingEngine`). True exactly when `applyConfiguration` - /// selects the `.voiceChat` AVAudioSession mode — mono + standard processing + not A2DP - /// (A2DP / stereo / raw modes can't use VPIO, so they keep the core's miniaudio path). + /// Whether the current configuration should engage Apple's Voice-Processing I/O unit (VPIO: + /// real AEC + noise suppression + AGC, driven by `IOSAudioEngine`). VPIO forces mono and + /// can't run on an A2DP route, so it is available only for a mono + standard + non-A2DP + /// config, and then only when the user hasn't disabled it via the Advanced master toggle. var currentConfigUsesVoiceProcessing: Bool { + voiceProcessingEnabled && voiceProcessingAvailable + } + + /// Whether the current config *could* use VPIO (mono + standard + non-A2DP), independent of + /// the user's master toggle. Drives whether the Advanced "Voice Processing" switch is shown. + var voiceProcessingAvailable: Bool { captureChannels == .mono && micMode == .standard && bluetoothMode != .builtInMicBtA2dp } // MARK: - Apply configuration - /// Apply the full audio configuration to AVAudioSession. Call this before the core - /// opens its capture device (i.e. before `startMicStream` → `activateForStreaming`). - /// Re-entrant-safe: if a route-change notification fires synchronously during a + /// Apply the full audio configuration to AVAudioSession. Call this before (re)building the + /// `IOSAudioEngine` graph so the engine binds to the intended route (`applyAndReconfigure` + /// does both). Re-entrant-safe: if a route-change notification fires synchronously during a /// `setCategory`/`setPreferredInput` call, the guard prevents re-entry. func applyConfiguration() { guard !isApplyingConfiguration else { @@ -401,8 +372,8 @@ final class IOSAudioRouter: ObservableObject { // route explicitly via setPreferredInput + setInputDataSource. With HFP disabled // the system routes input to the built-in mic, but without the explicit // preferred-input anchor the route can collapse during the mode switch - // (.voiceChat → .default) and the output dies. The channel count is requested by - // miniaudio at the audio-unit level (vc_set_capture_channels), NOT via + // (.voiceChat → .default) and the output dies. The channel count is carried by the + // engine's mic tap + vc_set_capture_channels, NOT via // setPreferredInputNumberOfChannels(2) — that call collapses the A2DP output route. configureStereoCapture(session: session) } else if let portId = selectedInputPortId, !portId.isEmpty, @@ -431,9 +402,9 @@ final class IOSAudioRouter: ObservableObject { /// 3. `setPreferredInput(builtIn)` — anchor the input route explicitly. Without this /// anchor the route can collapse during the mode switch (.voiceChat → .default). /// 4. `setInputDataSource(stereoSource)` — commit the data source at the session level - /// The channel count itself is requested by miniaudio at the audio-unit level via - /// `vc_set_capture_channels(2)`. We must NOT call `setPreferredInputNumberOfChannels(2)` - /// — that session-level call collapses the A2DP output route. + /// The channel count itself is carried by the engine's mic tap (which captures 2 channels) + /// plus `vc_set_capture_channels(2)` so the core encodes stereo. We must NOT call + /// `setPreferredInputNumberOfChannels(2)` — that session-level call collapses the A2DP route. private func configureStereoCapture(session: AVAudioSession) { guard let builtIn = session.availableInputs?.first(where: { $0.portType == .builtInMic }) else { @@ -538,6 +509,9 @@ final class IOSAudioRouter: ObservableObject { selectedDataSourceId = UserDefaults.standard.string(forKey: kDataSourceId) selectedPolarPattern = UserDefaults.standard.string(forKey: kPolarPattern) forceSpeaker = UserDefaults.standard.bool(forKey: kForceSpeaker) + // VPIO toggles default ON when never set (object(forKey:) is nil → use true). + voiceProcessingEnabled = (UserDefaults.standard.object(forKey: kVoiceProcessing) as? Bool) ?? true + agcEnabled = (UserDefaults.standard.object(forKey: kAgc) as? Bool) ?? true } /// Persist current selections to UserDefaults. func savePreferences() { @@ -548,86 +522,88 @@ final class IOSAudioRouter: ObservableObject { UserDefaults.standard.set(selectedDataSourceId, forKey: kDataSourceId) UserDefaults.standard.set(selectedPolarPattern, forKey: kPolarPattern) UserDefaults.standard.set(forceSpeaker, forKey: kForceSpeaker) + UserDefaults.standard.set(voiceProcessingEnabled, forKey: kVoiceProcessing) + UserDefaults.standard.set(agcEnabled, forKey: kAgc) } // MARK: - Selection setters (called from SettingsView pickers) + /// Shared tail for every setting change: persist, re-apply the AVAudioSession config, refresh + /// the route lists, re-evaluate the A2DP speaker fallback, and rebind the live engine to the + /// new route. `IOSAudioEngine.reconfigure()` is a no-op when not connected, so this is safe to + /// call from Settings whether or not a session is in progress. There is no longer a second + /// (miniaudio) audio path to hand off to, so one engine rebuild is the whole story. + private func applyAndReconfigure() { + savePreferences() + applyConfiguration() + if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() } + refreshRoutes() + IOSAudioEngine.shared.reconfigure() + } + func selectInputPort(_ portId: String) { selectedInputPortId = portId selectedDataSourceId = nil selectedPolarPattern = nil - savePreferences() - applyConfiguration() - refreshRoutes() + applyAndReconfigure() } func selectDataSource(_ dataSourceId: String) { selectedDataSourceId = dataSourceId selectedPolarPattern = nil - savePreferences() - applyConfiguration() - refreshRoutes() + applyAndReconfigure() } func selectPolarPattern(_ pattern: String) { selectedPolarPattern = pattern - savePreferences() - applyConfiguration() - refreshRoutes() + applyAndReconfigure() } func selectBluetoothMode(_ mode: BluetoothMode) { bluetoothMode = mode - savePreferences() - applyConfiguration() - refreshRoutes() - // VPIO class or route may have changed — restart the voice path if mic is active. - AudioSessionManager.shared.reconcileVoicePath?() + applyAndReconfigure() } func setForceSpeaker(_ on: Bool) { forceSpeaker = on - savePreferences() - applyConfiguration() - refreshRoutes() - // Route changed under a possibly-running VPIO engine — reconcile if mic is active. - AudioSessionManager.shared.reconcileVoicePath?() + applyAndReconfigure() } func selectMicMode(_ mode: MicMode) { micMode = mode + applyAndReconfigure() + } + + func setVoiceProcessingEnabled(_ on: Bool) { + voiceProcessingEnabled = on + applyAndReconfigure() + } + + func setAgcEnabled(_ on: Bool) { + agcEnabled = on + // No session reconfigure needed — just rebuild the engine so VPIO picks up the AGC flag. savePreferences() - applyConfiguration() - updateWarnings() - // Standard↔Raw flips the VPIO class — reconcile if mic is active. - AudioSessionManager.shared.reconcileVoicePath?() + IOSAudioEngine.shared.reconfigure() } func selectCaptureChannels(_ channels: CaptureChannels) { captureChannels = channels savePreferences() applyConfiguration() - // Update the core's stored capture channel count (does not restart the engine). - if let streamId = AudioSessionManager.shared.activeMicStreamId { - _ = AudioSessionManager.shared.client?.setCaptureChannels( - streamId: streamId, channels: channels.channelCount) - } - // Restart the engine AFTER AVAudioSession routing has settled and the channel - // count is stored. The engine reopens playback first (committing the A2DP/output - // route), then capture — avoiding the race where stereo capture activation drops - // A2DP before the playback device has a chance to claim the route. - _ = AudioSessionManager.shared.client?.audioRestart() - // Mono↔stereo flips the VPIO class (stereo can't use VPIO) — reconcile if mic is active. - AudioSessionManager.shared.reconcileVoicePath?() + if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() } + refreshRoutes() + // Push the channel count into the core's MIC stream, then rebuild the engine graph so the + // mic tap captures the right number of channels. The engine owns the route now, so there's + // no stereo-vs-A2DP race to sequence around. + IOSAudioEngine.shared.setCaptureChannels(channels.channelCount) } // MARK: - Presets - /// Apply a preset — sets all individual audio settings to the preset's values, then - /// applies the configuration. For presets that use the built-in mic (A2DP presets), - /// finds the built-in mic port UID from availableInputs. + /// Apply a named preset — set all individual settings to the preset's values, then re-apply + /// the configuration and rebind the engine. The internal-mic presets pin the built-in mic. func applyPreset(_ preset: AudioPreset) { - guard preset != .custom else { return } // can't "apply" custom — it's a display state + guard preset != .advanced else { return } // Advanced is a display state, not "applied" bluetoothMode = preset.bluetoothMode micMode = preset.micMode @@ -638,19 +614,17 @@ final class IOSAudioRouter: ObservableObject { if preset == .voiceChat { forceSpeaker = true } if preset.usesBuiltInMic { - // Find the built-in mic port from available inputs and select it. - let session = AVAudioSession.sharedInstance() - if let builtInMic = (session.availableInputs ?? []).first(where: { + // Pin the built-in mic. In stereo, iOS uses multiple capsules automatically; in mono + // the default orientation is fine — so don't force a specific data source / pattern. + if let builtInMic = (AVAudioSession.sharedInstance().availableInputs ?? []).first(where: { $0.portType == .builtInMic }) { selectedInputPortId = builtInMic.uid } - // Don't set a specific data source — in stereo mode, iOS uses multiple mic - // capsules automatically. In mono, the default orientation is fine. selectedDataSourceId = nil selectedPolarPattern = nil } else { - // For Default and Bluetooth Headset presets, let the system pick the input. + // Voice Chat: let the system pick the input (Bluetooth HFP / wired / built-in). selectedInputPortId = nil selectedDataSourceId = nil selectedPolarPattern = nil @@ -659,18 +633,11 @@ final class IOSAudioRouter: ObservableObject { UserDefaults.standard.set(preset.rawValue, forKey: kPreset) savePreferences() applyConfiguration() - // Update the core's stored capture channel count (does not restart the engine). - if let streamId = AudioSessionManager.shared.activeMicStreamId { - _ = AudioSessionManager.shared.client?.setCaptureChannels( - streamId: streamId, channels: preset.captureChannels.channelCount) - } - // Restart the engine AFTER AVAudioSession routing has settled and the channel - // count is stored. Playback opens first (commits A2DP route), then capture. - _ = AudioSessionManager.shared.client?.audioRestart() + if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() } refreshRoutes() - // The preset may have flipped the VPIO class (and/or the route) — restart the voice path - // if the mic is active so AEC/NS engage (or disengage) to match the new preset. - AudioSessionManager.shared.reconcileVoicePath?() + // Push the channel count to the core, then rebuild the engine graph (VPIO on/off + tap). + IOSAudioEngine.shared.setCaptureChannels(preset.captureChannels.channelCount) + IOSAudioEngine.shared.reconfigure() logger.info("applyPreset — \(preset.rawValue)") } diff --git a/clients/apple/iOS/VoiceCatiOS/IOSVoiceProcessingEngine.swift b/clients/apple/iOS/VoiceCatiOS/IOSVoiceProcessingEngine.swift index 7bc7471..9e7908d 100644 --- a/clients/apple/iOS/VoiceCatiOS/IOSVoiceProcessingEngine.swift +++ b/clients/apple/iOS/VoiceCatiOS/IOSVoiceProcessingEngine.swift @@ -3,9 +3,9 @@ import Darwin import os import VoiceCatCore -private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSVoiceProcessingEngine") +private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioEngine") -/// In-process single-producer/single-consumer int16 PCM ring for the VPIO playback path. +/// In-process single-producer/single-consumer int16 PCM ring for the playback path. /// /// producer = the core's mixer-timer thread (the `vc_set_mixed_output_sink` callback) /// consumer = the `AVAudioSourceNode` render thread @@ -79,38 +79,48 @@ final class PCMRing { /// Diagnostics: monotonic total samples written / read since the ring was created. The /// indices are already cumulative, so these are free. Only read them when both threads are /// quiesced (e.g. at teardown after the engine + mixer sink are stopped) — they are not - /// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" (Bug 1 - /// core path) apart from "PCM arrived but produced no sound" (AVAudioEngine output graph). + /// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" apart + /// from "PCM arrived but produced no sound" (the AVAudioEngine output graph). var debugTotalWritten: UInt64 { writeIdx } var debugTotalRead: UInt64 { readIdx } } -/// Native iOS voice-processing audio path (docs/voice.md §8 "iOS voice processing"). +/// The single iOS audio engine (docs/voice.md §8 "iOS audio engine"). /// -/// Real iOS echo cancellation / noise suppression / AGC come ONLY from Apple's Voice-Processing -/// I/O unit (VPIO), which `AVAudioEngine.setVoiceProcessingEnabled(true)` enables. For VPIO to -/// cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the -/// played-back signal from the mic), so on the AEC presets this engine drives both directions and -/// the core runs in external mode (no hardware devices): -/// - **Mic → core:** a tap on the VPIO input node → 48 kHz int16 → `client.feedPcm(micStreamId)`. +/// **One path, always external.** On iOS the core never opens a miniaudio device: a MIC stream is +/// always started with `external_feed=1`, `vc_set_external_playback(1)` is set once at connect, and +/// this engine drives *both* directions through one `AVAudioEngine`: /// - **core → speaker:** the core's mixed-output sink fills `ring`; an `AVAudioSourceNode` pulls -/// from it and renders through the VPIO output, giving AEC its reference signal. +/// from it and renders through the engine output. This runs the whole time we're connected, +/// so remote audio plays even before the user joins voice (no "can't hear anyone"). +/// - **mic → core:** when the mic is active a tap on the input node converts to 48 kHz int16 and +/// calls `client.feedPcm(micStreamId)`. /// -/// Lifecycle is driven by `SessionState` join/leave. The Stereo Mic / Studio / A2DP presets keep -/// the core's miniaudio path instead (they want raw / stereo / no-AEC routing VPIO can't provide). +/// Echo cancellation / noise suppression / AGC come from Apple's Voice-Processing I/O unit (VPIO), +/// which `inputNode.setVoiceProcessingEnabled(true)` enables. VPIO forces mono, so it is engaged +/// only when the active preset wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`) — the +/// Stereo Mic / A2DP configs run the same engine with VPIO off. +/// +/// Every preset / route / interruption change funnels through `reconfigure()`: a single +/// deterministic stop → AVAudioSession reconfigure → rebuild graph → start. There is no second +/// (miniaudio) audio path to hand off to, so a switch cannot leave one direction dropped. @MainActor -final class IOSVoiceProcessingEngine { - static let shared = IOSVoiceProcessingEngine() +final class IOSAudioEngine { + static let shared = IOSAudioEngine() - private(set) var isRunning = false + /// True while connected (between `startListening` and `stop`) — the playback graph should run. + private(set) var isConnected = false + /// True while a local mic stream is active — the input tap should be installed. + private(set) var micActive = false private let engine = AVAudioEngine() private var sourceNode: AVAudioSourceNode? private weak var client: VoiceCatClient? private var micStreamId: UInt32 = 0 + private var captureChannels: UInt32 = 1 - // 48 kHz stereo Float32 (deinterleaved) — the format the source node renders and the engine - // processes in. The core delivers 48 kHz stereo int16 via the mixed-output sink. + // 48 kHz stereo Float32 (deinterleaved) — the format the source node renders. The core + // delivers 48 kHz stereo int16 via the mixed-output sink; mainMixerNode adapts to the route. private let outFormat = AVAudioFormat( commonFormat: .pcmFormatFloat32, sampleRate: 48000, channels: 2, interleaved: false)! @@ -121,33 +131,138 @@ final class IOSVoiceProcessingEngine { private let renderScratchFrames = 8192 private let renderScratch: UnsafeMutablePointer - // Mic-feed converter (input-node format → 48 kHz int16) and its target buffer. Owned here so - // the (background) tap block reuses them instead of allocating per callback. - private var micConverter: AVAudioConverter? - private var micTargetFormat: AVAudioFormat? - private init() { renderScratch = UnsafeMutablePointer.allocate(capacity: renderScratchFrames * 2) renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2) } - /// Start the VPIO engine for an active mic stream. The caller must have already enabled - /// external playback on the core (`client.setExternalPlayback(true)` + `audioRestart()`) and - /// started the MIC stream with `externalFeed: true`. - func start(client: VoiceCatClient, micStreamId: UInt32, captureChannels: UInt32) { - guard !isRunning else { return } + // MARK: - Lifecycle + + /// Begin playback-only (listening) operation. Called once at connect, after + /// `client.setExternalPlayback(true)` and `AudioSessionManager.ensureSessionActive()`. Attaches + /// the source node, wires the core's mixed-output sink into the ring, and starts the engine so + /// remote audio plays immediately. + func startListening(client: VoiceCatClient) { self.client = client - self.micStreamId = micStreamId + guard !isConnected else { return } + isConnected = true ring.reset() - // Enable the voice-processing I/O unit (AEC/NS/AGC) on the shared input+output unit. + // Wire the core's mixed-output sink into the ring (C function pointer, no captures). Stays + // registered for the whole connection; the ring is drained by the source-node render block. + let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque() + client.setMixedOutputSink({ user, pcm, spc, ch, _ in + guard let user, let pcm else { return } + let ring = Unmanaged.fromOpaque(user).takeUnretainedValue() + ring.write(pcm, count: spc * Int(ch)) + }, user: ringPtr) + + rebuild() + } + + /// Tear down the engine and unhook the core sink. Called on disconnect. + func stop() { + guard isConnected else { return } + micActive = false + isConnected = false + client?.setMixedOutputSink(nil, user: nil) + engine.inputNode.removeTap(onBus: 0) + if engine.isRunning { engine.stop() } + logger.info("audio engine stopped — ring written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples") + try? engine.inputNode.setVoiceProcessingEnabled(false) + if let src = sourceNode { + engine.detach(src) + sourceNode = nil + } + ring.reset() + client = nil + } + + // MARK: - Mic transitions + + /// Engage the mic: install the input tap and (if the preset wants it) VPIO. Called when the + /// user joins voice, after the MIC stream (external_feed) is started. + func startMic(streamId: UInt32, channels: UInt32) { + micStreamId = streamId + captureChannels = channels + micActive = true + rebuild() + } + + /// Disengage the mic: remove the tap and VPIO, keep playback running for remaining remote audio. + func stopMic() { + guard micActive else { return } + micActive = false + rebuild() + } + + /// Update the capture channel count (mono↔stereo) for the active mic and rebuild. + func setCaptureChannels(_ channels: UInt32) { + captureChannels = channels + if let client, micStreamId != 0 { + client.setCaptureChannels(streamId: micStreamId, channels: channels) + } + if micActive { rebuild() } + } + + /// Re-apply the engine graph against the current AVAudioSession config (preset / route change). + /// Safe to call when only listening — it just rebuilds the playback graph against the new route. + func reconfigure() { + guard isConnected else { return } + rebuild() + } + + // MARK: - Graph (re)build + + /// The single place that (re)builds and starts the engine graph. Deterministic: stop → set + /// VPIO → (re)install the mic tap → start. The caller is responsible for having applied the + /// AVAudioSession config (category/mode/route) first (`IOSAudioRouter.applyConfiguration`). + private func rebuild() { + guard isConnected else { return } + if engine.isRunning { engine.stop() } + engine.inputNode.removeTap(onBus: 0) + + let useVPIO = micActive && IOSAudioRouter.shared.currentConfigUsesVoiceProcessing do { - try engine.inputNode.setVoiceProcessingEnabled(true) + try engine.inputNode.setVoiceProcessingEnabled(useVPIO) } catch { - logger.error("setVoiceProcessingEnabled failed: \(error.localizedDescription) — AEC unavailable") + logger.error("setVoiceProcessingEnabled(\(useVPIO)) failed: \(error.localizedDescription)") + } + if useVPIO { + // AGC is the one VPIO sub-stage iOS exposes; AEC+NS are bundled into the master switch. + engine.inputNode.isVoiceProcessingAGCEnabled = IOSAudioRouter.shared.agcEnabled } - // ── Playback: source node pulls mixed PCM from the ring through the VPIO output. ── + // (Re)build the playback source node AFTER the VPIO state is set, so it connects against the + // correct (voice-processed or plain) output unit — mirrors the proven original ordering. + rebuildSourceNode() + if micActive { installMicTap() } + + engine.prepare() + do { + try engine.start() + let inFmt = engine.inputNode.outputFormat(forBus: 0) + let outFmt = engine.outputNode.outputFormat(forBus: 0) + let route = AVAudioSession.sharedInstance().currentRoute.outputs + .map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ") + logger.info(""" + engine started — mic=\(self.micActive) vpio=\(useVPIO) captureCh=\(self.captureChannels) \ + inFormat=\(inFmt) outputNode=\(outFmt) outputRoute=[\(route)] + """) + } catch { + logger.error("engine start failed: \(error.localizedDescription)") + } + } + + /// Detach any previous source node and attach a fresh one pulling mixed PCM from the ring. + /// Rebuilt on every graph rebuild so it always connects against the current output unit (the + /// VPIO state can change the output between rebuilds). Its format is route-independent — + /// `mainMixerNode` adapts 48 kHz stereo to whatever the output route is. + private func rebuildSourceNode() { + if let old = sourceNode { + engine.detach(old) + sourceNode = nil + } let ring = self.ring let scratch = self.renderScratch let scratchFrames = self.renderScratchFrames @@ -156,17 +271,11 @@ final class IOSVoiceProcessingEngine { let abl = UnsafeMutableAudioBufferListPointer(ablPtr) let n = min(frames, scratchFrames) let got = ring.read(into: scratch, count: n * 2) / 2 // interleaved stereo → frames - // Deinterleave int16 → Float32 per channel; silence-fill any underrun tail. let scale: Float = 1.0 / 32768.0 for ch in 0.. 0 else { + logger.error("input format unavailable (\(inFormat)) — mic will not transmit") + return + } let targetCh = max(1, min(2, captureChannels)) - let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000, - channels: AVAudioChannelCount(targetCh), interleaved: true) - micTargetFormat = target - micConverter = (target != nil && inFormat.sampleRate > 0) - ? AVAudioConverter(from: inFormat, to: target!) : nil - if micConverter == nil { - logger.error("mic converter unavailable (in=\(inFormat)) — mic will not transmit") + guard let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000, + channels: AVAudioChannelCount(targetCh), interleaved: true), + let converter = AVAudioConverter(from: inFormat, to: target) else { + logger.error("mic converter unavailable (in=\(inFormat), ch=\(targetCh)) — mic will not transmit") + return } - let c = client let sid = micStreamId - let converter = micConverter - let tgt = micTargetFormat + let c = client engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in - guard let converter, let tgt else { return } // Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample. - let ratio = tgt.sampleRate / buffer.format.sampleRate + let ratio = target.sampleRate / buffer.format.sampleRate let outCap = AVAudioFrameCount(Double(buffer.frameLength) * ratio + 16) - guard let outBuf = AVAudioPCMBuffer(pcmFormat: tgt, frameCapacity: outCap) else { return } + guard let outBuf = AVAudioPCMBuffer(pcmFormat: target, frameCapacity: outCap) else { return } var fed = false let status = converter.convert(to: outBuf, error: nil) { _, outStatus in if fed { outStatus.pointee = .noDataNow; return nil } @@ -206,64 +319,9 @@ final class IOSVoiceProcessingEngine { } guard status != .error, outBuf.frameLength > 0, let chData = outBuf.int16ChannelData else { return } - let spc = Int(outBuf.frameLength) - // int16 interleaved → channelData[0] is the interleaved buffer for interleaved formats. - c.feedPcm(streamId: sid, pcm: chData[0], samplesPerChannel: spc, channels: targetCh) + // int16 interleaved → channelData[0] is the interleaved buffer. + c.feedPcm(streamId: sid, pcm: chData[0], + samplesPerChannel: Int(outBuf.frameLength), channels: targetCh) } - - // ── Wire the core's mixed-output sink into the ring (C function pointer, no captures). ── - let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque() - client.setMixedOutputSink({ user, pcm, spc, ch, _ in - guard let user, let pcm else { return } - let ring = Unmanaged.fromOpaque(user).takeUnretainedValue() - ring.write(pcm, count: spc * Int(ch)) - }, user: ringPtr) - - engine.prepare() - do { - try engine.start() - isRunning = true - // Diagnostics: capture the negotiated graph formats and the live output route so a - // silent-playback report can be triaged (format/rate mismatch vs. routing vs. the - // core not delivering PCM — see the ring stats logged in teardown()). - let outFmt = engine.outputNode.outputFormat(forBus: 0) - let mixFmt = engine.mainMixerNode.outputFormat(forBus: 0) - let route = AVAudioSession.sharedInstance().currentRoute.outputs - .map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ") - logger.info(""" - VPIO engine started — inFormat=\(inFormat), captureCh=\(targetCh), \ - outputNode=\(outFmt), mainMixer=\(mixFmt), outputRoute=[\(route)] - """) - } catch { - logger.error("VPIO engine start failed: \(error.localizedDescription)") - teardown() - } - } - - /// Stop the VPIO engine. The caller is responsible for restoring the core's hardware playback - /// afterwards (`client.setExternalPlayback(false)` + `audioRestart()`). - func stop() { - guard isRunning else { return } - teardown() - logger.info("VPIO engine stopped") - } - - private func teardown() { - client?.setMixedOutputSink(nil, user: nil) - engine.inputNode.removeTap(onBus: 0) - if engine.isRunning { engine.stop() } - // Diagnostics (threads now quiesced): how much mixed PCM the core delivered into the ring - // vs. how much the render thread consumed. written==0 ⇒ the core never delivered (Bug 1 - // core/lifecycle path); written>0 with no audible output ⇒ the AVAudioEngine output graph. - logger.info("VPIO ring stats — written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples") - try? engine.inputNode.setVoiceProcessingEnabled(false) - if let src = sourceNode { - engine.detach(src) - sourceNode = nil - } - micConverter = nil - micTargetFormat = nil - ring.reset() - isRunning = false } } diff --git a/clients/apple/iOS/VoiceCatiOS/SessionState.swift b/clients/apple/iOS/VoiceCatiOS/SessionState.swift index 116e7e0..0c7cb77 100644 --- a/clients/apple/iOS/VoiceCatiOS/SessionState.swift +++ b/clients/apple/iOS/VoiceCatiOS/SessionState.swift @@ -59,7 +59,6 @@ final class SessionState { self.client = client self.selfUserId = selfUserId self.permissions = permissions - AudioSessionManager.shared.client = client refreshChannels() refreshUsers() syncSelfChannel() @@ -73,16 +72,10 @@ final class SessionState { broadcastPump.onBroadcastStarted = { [weak self] in self?.startScreenShare() } broadcastPump.onBroadcastFinished = { [weak self] in self?.stopScreenShare() } broadcastPump.start() - // When IOSAudioRouter changes the audio config, restart the voice path if needed so the - // native VPIO engine (AEC/NS/AGC) engages or disengages to match the new preset/route. - AudioSessionManager.shared.reconcileVoicePath = { [weak self] in self?.reconcileVoicePath() } } deinit { broadcastPump.stop() - MainActor.assumeIsolated { - AudioSessionManager.shared.client = nil - } } // MARK: - Event dispatch @@ -259,78 +252,39 @@ final class SessionState { return } - // VPIO path: on the AEC presets, the native AVAudioEngine does AEC/NS/AGC and the core - // runs in external mode (no hardware mic/playback). The mic stream is started with - // externalFeed so the core skips the hardware capture device; setExternalPlayback makes - // it skip the hardware playback device and deliver the mix to IOSVoiceProcessingEngine. - // - // ORDER MATTERS: set the external-playback flag now, but defer audioRestart() until - // AFTER startStream (below) so the MIC LocalStream — which carries external_feed=true — - // already exists when ensure_audio_running() derives external_capture. Restarting before - // the stream exists makes the core reopen a hardware capture device that is never dropped - // (the announce-result restart early-returns because the engine is already running); that - // lingering miniaudio capture unit then fights the AVAudioEngine VPIO unit on the same - // .voiceChat session and silences VPIO playback. - let useVPIO = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing - client.setExternalPlayback(useVPIO) - + // Unified iOS path: the core is always external (set at connect via setExternalPlayback + + // every MIC stream external_feed), and `IOSAudioEngine` drives capture + playback. So the + // mic stream is just started with external_feed=true and the engine is told the mic is now + // active — no setExternalPlayback toggle, no audioRestart ordering, no VPIO/miniaudio fork. let desc = StreamDescriptor(kind: .mic, deviceId: voiceState.currentDeviceId, label: "Mic", - externalFeed: useVPIO) + externalFeed: true) let (result, streamId) = client.startStream(desc) - if result == .ok { - voiceState.micActive = true - voiceState.localStreamId = streamId - EventFeedback.shared.play(.voiceOn) - // Publish the active mic stream ID so IOSAudioRouter can reset the core's capture - // channel count when the user switches mono↔stereo (selectCaptureChannels / - // applyPreset). Without this, switching stereo→mono leaves the LocalStream's - // capture_channels field at 2 and the next engine start still opens stereo. - AudioSessionManager.shared.activeMicStreamId = streamId - // Store the user's capture channel selection before the server acknowledges - // the stream. The engine hasn't started yet at this point (it starts when - // handle_stream_announce_result fires), so vc_set_capture_channels just - // stores the value — no restart. ensure_audio_running() picks it up when - // the stream is confirmed and opens the device with the right channel count. - let channels = IOSAudioRouter.shared.captureChannels.channelCount - if channels != 1 { - client.setCaptureChannels(streamId: streamId, channels: channels) - } - if useVPIO { - // The external-feed MIC stream now exists, so restart the core into full - // external mode (no hardware capture/playback, mixer-timer only) — mic and - // speaker are owned entirely by the VPIO engine, which we start right after. - client.audioRestart() - IOSVoiceProcessingEngine.shared.start( - client: client, micStreamId: streamId, captureChannels: channels) - } - } else { + guard result == .ok else { addActivity("Failed to start mic: \(result.description)") - if useVPIO { // revert external-playback mode so remote audio still plays - client.setExternalPlayback(false) - client.audioRestart() - } + return } + voiceState.micActive = true + voiceState.localStreamId = streamId + EventFeedback.shared.play(.voiceOn) + + let channels = IOSAudioRouter.shared.captureChannels.channelCount + if channels != 1 { + client.setCaptureChannels(streamId: streamId, channels: channels) + } + // Engage the mic: installs the input tap and (per preset) VPIO, in one engine rebuild. + IOSAudioEngine.shared.startMic(streamId: streamId, channels: channels) } func stopMicStream() { - // Tear down the VPIO engine first (removes the mic tap + unregisters the mixed sink), - // then stop the mic stream, then restore the core's hardware playback for any remaining - // remote audio. Order matters: the mic stream must be gone before audioRestart so the - // core opens a normal playback device (and no capture device — there's no mic stream). - let wasVPIO = IOSVoiceProcessingEngine.shared.isRunning - if wasVPIO { - IOSVoiceProcessingEngine.shared.stop() - } + // Disengage the mic (removes the tap + VPIO) but keep the engine running for any remaining + // remote audio. Then stop the core's MIC stream. The core stays external throughout — no + // setExternalPlayback toggle, no audioRestart. + IOSAudioEngine.shared.stopMic() if voiceState.localStreamId != 0 { client.stopStream(voiceState.localStreamId) voiceState.localStreamId = 0 - AudioSessionManager.shared.activeMicStreamId = nil EventFeedback.shared.play(.voiceOff) } - if wasVPIO { - client.setExternalPlayback(false) - client.audioRestart() // reopen hardware playback (no mic stream → no hw capture) - } voiceState.micActive = false voiceState.level = 0 // Do NOT deactivate the AVAudioSession here — the user may still want to hear @@ -338,19 +292,6 @@ final class SessionState { // disconnecting from the server (see AppState.disconnect / .disconnected event). } - /// Restart the voice path when the audio config changes mid-call (driven by IOSAudioRouter). - /// If VPIO is involved on either the current or desired side, restart the mic so the native - /// voice-processing engine engages/disengages and re-binds to the new route. Pure miniaudio - /// config tweaks need no restart — the core's own audioRestart (already issued) handles them. - private func reconcileVoicePath() { - guard voiceState.micActive else { return } - let want = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing - let have = IOSVoiceProcessingEngine.shared.isRunning - guard want || have else { return } - stopMicStream() - doStartMicStream() - } - // MARK: - Screen audio share /// Called when the broadcast extension becomes active. Announces the SCREEN_AUDIO stream; diff --git a/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift b/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift index c8b0c19..2b42936 100644 --- a/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift +++ b/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift @@ -38,9 +38,9 @@ struct SettingsView: View { .accessibilityLabel("Speaker output") .accessibilityHint("Routes audio to the speaker instead of the earpiece when no headphones are connected.") - // Surface the voice-processing state. On the AEC presets the native iOS + // Surface the voice-processing state. On Voice Chat the native iOS // Voice-Processing unit (VPIO) does echo cancellation, noise suppression and - // automatic gain control; the other presets (stereo/studio/A2DP) can't use it. + // automatic gain control; the stereo / mono-mic / A2DP configs can't use it. if router.currentConfigUsesVoiceProcessing { Label("Echo cancellation & noise suppression on (iOS voice processing)", systemImage: "waveform.badge.mic") @@ -48,18 +48,11 @@ struct SettingsView: View { .foregroundStyle(.secondary) .accessibilityLabel("Echo cancellation and noise suppression are on") } else { - Label("No echo cancellation in this preset (stereo / studio / A2DP)", + Label("No echo cancellation in this configuration (stereo / A2DP / off)", systemImage: "waveform.slash") .font(.caption) .foregroundStyle(.secondary) - .accessibilityLabel("Echo cancellation is off in this preset") - } - - if !router.hasBluetoothDevice && !router.hasWiredHeadset { - Text("Connect Bluetooth headphones or a wired headset for more presets.") - .font(.caption) - .foregroundStyle(.secondary) - .accessibilityLabel("No external audio device connected") + .accessibilityLabel("Echo cancellation is off in this configuration") } } @@ -127,6 +120,27 @@ struct SettingsView: View { } .accessibilityLabel("Microphone processing mode") + // Voice-processing (VPIO) controls — only meaningful on a VPIO-capable + // config (mono + standard + non-A2DP). iOS bundles echo cancellation and + // noise suppression into one master switch (no per-stage toggle); AGC is + // the one sub-stage it lets us control independently. + if router.voiceProcessingAvailable { + Toggle("Voice Processing (AEC + noise suppression)", isOn: Binding( + get: { router.voiceProcessingEnabled }, + set: { router.setVoiceProcessingEnabled($0) } + )) + .accessibilityLabel("Voice processing") + .accessibilityHint("Echo cancellation and noise suppression, bundled together by iOS.") + + if router.voiceProcessingEnabled { + Toggle("Automatic Gain Control", isOn: Binding( + get: { router.agcEnabled }, + set: { router.setAgcEnabled($0) } + )) + .accessibilityLabel("Automatic gain control") + } + } + if router.showsRawModeSpeakerWarning { Label( "Raw mode on speaker — echo risk (no AEC)", diff --git a/docs/voice.md b/docs/voice.md index 9a7bfaf..200d52b 100644 --- a/docs/voice.md +++ b/docs/voice.md @@ -230,37 +230,46 @@ Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no downmix), mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism. -- **iOS mic capture:** all iOS audio routing is driven from Swift via `AVAudioSession` by the - `IOSAudioRouter` singleton *before* the core (miniaudio) opens its device — miniaudio does - NOT touch `AVAudioSession` on iOS. Input port selection (`availableInputs`), built-in mic - orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns - (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic processing mode - (`.voiceChat` = Standard with AEC/AGC/HPF, or `.measurement` = Raw/Studio with all processing - off), Bluetooth mode (`.allowBluetoothHFP` HFP voice vs `.allowBluetoothA2DP` stereo output - vs neither), and stereo capture (`.stereo` polar pattern + `setPreferredInput` + - `setInputDataSource` → `vc_set_capture_channels`) are all set from Swift. The core then - opens whatever route AVAudioSession has established. When the user changes audio settings - mid-session, `IOSAudioRouter` suspends the core's devices (`vc_audio_suspend`), reconfigures - `AVAudioSession`, then restarts the devices (`vc_audio_restart`) so they reopen against the - new route — mirroring TeamTalk5's `closeSoundDevices`/`initSoundInputDevice`/ - `initSoundOutputDevice` pattern. -- **iOS voice processing (AEC/NS/AGC) — native VPIO path.** Real iOS echo cancellation, noise - suppression and AGC are provided ONLY by Apple's **Voice-Processing I/O audio unit (VPIO)**, - *not* by the `AVAudioSession` mode alone. The core uses miniaudio's plain `RemoteIO` audio - units, which never engage VPIO — so `.voiceChat` mode by itself yields no AEC. For VPIO to - cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the - played-back signal from the mic), so on the AEC presets (Voice Chat / Bluetooth Headset HFP / - Wired Headset) the Swift layer runs a native `AVAudioEngine` with - `inputNode.setVoiceProcessingEnabled(true)` and the core runs in **external mode**: - - **Mic:** the MIC stream is started with `vc_stream_desc.external_feed=1`; the VPIO input tap - feeds processed mic PCM via `vc_stream_feed_pcm`. The core skips its hardware capture device - (`AudioParams.external_capture`). - - **Playback:** `vc_set_external_playback(1)` makes the core skip its hardware playback device; - a mixer-timer thread drives decode+mix on a ~20 ms cadence and delivers the FINAL mixed PCM - via `vc_set_mixed_output_sink`. The Swift engine renders that through the VPIO output, so - VPIO has its echo-cancellation reference signal. - The Stereo Mic / Studio / A2DP presets keep the miniaudio path (they want raw / stereo / - no-AEC routing that VPIO can't provide — VPIO forces mono). +- **iOS audio — one path, always external.** On iOS the core **never opens a miniaudio device**: + a single `AVAudioEngine` (`IOSAudioEngine`) drives *both* directions, and the core runs fully + external for the whole connection. This is the single most important property of the iOS audio + stack — there is no second (miniaudio) path to switch to, so a preset/route change cannot leave + one direction dropped. The single ordering rule is: `vc_set_external_playback(1)` is set **once + at connect** (before the session is activated or any remote stream arrives), and every MIC + stream is started with `vc_stream_desc.external_feed=1`. + - **core → speaker:** the core's mixer-timer thread decodes+mixes on a ~20 ms cadence and + delivers the FINAL mixed PCM via `vc_set_mixed_output_sink`; an `AVAudioSourceNode` pulls it + from a lock-free ring and renders it. This runs the whole time we are connected, so remote + audio plays even before the user joins voice (kills the "can't hear anyone" race). + - **mic → core:** when the mic is active a tap on the engine's input node converts to 48 kHz + int16 (`vc_set_capture_channels` decides mono/stereo) and calls `vc_stream_feed_pcm`. +- **iOS routing** is still driven from Swift via `AVAudioSession` by the `IOSAudioRouter` + singleton — miniaudio never touches `AVAudioSession` on iOS. Input port selection + (`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom), + polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic + processing mode (Standard vs `.measurement` Raw), Bluetooth mode (`.allowBluetoothHFP` HFP voice + vs `.allowBluetoothA2DP` stereo output vs neither), and stereo capture (`.stereo` polar pattern + + `setPreferredInput` + `setInputDataSource` → `vc_set_capture_channels`) are all set from + Swift. Any preset / route / interruption change funnels through one deterministic, Swift-only + rebuild: `IOSAudioEngine` stops, `IOSAudioRouter.applyConfiguration()` re-applies the + `AVAudioSession`, the graph is rebuilt against the new route, and the engine restarts. No + `vc_audio_restart`/`vc_audio_suspend` dance is needed for routing (the core has no hardware + devices to reopen) — this is the spirit of TeamTalk5's "close then re-init sound devices", but + entirely inside the Swift engine. +- **iOS voice processing (AEC/NS/AGC) — native VPIO.** Real iOS echo cancellation, noise + suppression and AGC come ONLY from Apple's **Voice-Processing I/O audio unit (VPIO)**, which + `inputNode.setVoiceProcessingEnabled(true)` enables; for it to cancel echo it must own BOTH the + mic capture and the playback — which the unified engine already does. VPIO forces **mono**, so + it is engaged only when the active config wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`: + mono + standard + non-A2DP + the user's master toggle). iOS exposes no per-stage VPIO control, + so the Advanced UI offers exactly two switches: a master **Voice Processing** (AEC + NS bundled) + and **AGC** (`isVoiceProcessingAGCEnabled`). +- **iOS presets** (`IOSAudioRouter.AudioPreset`): **Voice Chat** (VPIO mono, system output incl. + HFP/wired), **Stereo Mic** (internal stereo built-in mic regardless of output, A2DP-capable, no + VPIO), **Mono Mic** (internal mono built-in mic regardless of output, A2DP-capable, no VPIO), + and **Advanced** (every knob manual). A2DP output requires an internal-mic preset (the Bluetooth + device is output-only); the Stereo/Mono Mic presets fall back to the built-in speaker when no + external output is connected (`applyA2dpSpeakerFallback`). - **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC + VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream (confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard