fix(ios-audio): unify iOS audio onto one always-external AVAudioEngine
The iOS audio path was a hybrid: Voice-Chat-class presets ran a native VPIO AVAudioEngine (core external) while Stereo/Studio/A2DP presets ran the core's miniaudio devices. Nearly every "no input / no output / both" bug lived in the seam between the two paths — the lingering miniaudio capture unit fighting VPIO, the audioRestart ordering dance, the route-change "glitching" loop, stereo<->mono stickiness, and "can't hear anyone". Switching presets/routes mid-call routinely dropped a direction. Drive ALL iOS audio through one AVAudioEngine with the core fully external at all times: setExternalPlayback(1) once at connect, every MIC stream external_feed=1, mic via vc_stream_feed_pcm, playback via vc_set_mixed_output_sink (drained by an always-on AVAudioSourceNode so remote audio plays before joining voice). VPIO + AGC toggle per preset. Every preset/route/interruption change funnels through one deterministic Swift-only reconfigure (stop -> apply session config -> rebuild -> start) — no second path to hand off to, so a change can't drop a direction. - IOSVoiceProcessingEngine.swift -> IOSAudioEngine: always-on source-node playback, conditional mic tap, VPIO/AGC; one rebuild() backing startListening/stop/startMic/stopMic/reconfigure/setCaptureChannels. - IOSAudioRouter: 7 presets -> 4 (Voice Chat / Stereo Mic / Mono Mic / Advanced); persisted voiceProcessingEnabled + agcEnabled; setters call IOSAudioEngine.reconfigure() instead of audioRestart/reconcileVoicePath. - AudioSessionManager slimmed; SessionState mic lifecycle collapsed; AppState wires external playback + listening at connect, stop at disconnect; SettingsView shows 4 presets + Advanced VPIO/AGC toggles. No core/ABI/test changes — relies on the already-shipped external API (test_external_pcm, test_external_playback). xcodebuild iOS device Debug BUILD SUCCEEDED. Updates docs/voice.md §8 and PROGRESS.md.
This commit is contained in:
38
PROGRESS.md
38
PROGRESS.md
@@ -75,8 +75,42 @@ up instantly. Newest status at the top.
|
|||||||
is now correct. If voice still fails, watch the server's rate-limited `[media] dropped frames —
|
is now correct. If voice still fails, watch the server's rate-limited `[media] dropped frames —
|
||||||
unmapped-endpoint=…` line (NAT source-port rewrite would be the next suspect).
|
unmapped-endpoint=…` line (NAT source-port rewrite would be the next suspect).
|
||||||
|
|
||||||
- **Awaiting on-device verification (2026-06-22):** **iOS real echo cancellation / noise suppression
|
- **Done (2026-06-23, Swift-only — no core/ABI change; awaiting on-device verification):** **iOS audio
|
||||||
via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS
|
stack unified — one always-external `AVAudioEngine`, miniaudio dropped on iOS.** The iOS audio path was
|
||||||
|
a fragile hybrid: Voice-Chat-class presets ran a native VPIO `AVAudioEngine` (core external) while
|
||||||
|
Stereo/Studio/A2DP presets ran the core's miniaudio devices. Nearly every bug lived in the seam
|
||||||
|
(lingering miniaudio capture unit fighting VPIO, the `audioRestart` ordering dance, the route-change
|
||||||
|
"glitching" loop, stereo↔mono stickiness, "can't hear anyone"), and switching presets/routes mid-call
|
||||||
|
routinely dropped input, output, or both. **Fix: drive *all* iOS audio through one `AVAudioEngine` with
|
||||||
|
the core fully external at all times** — `vc_set_external_playback(1)` once at connect, every MIC stream
|
||||||
|
`external_feed=1`, mic via `vc_stream_feed_pcm`, playback via `vc_set_mixed_output_sink`.
|
||||||
|
- `IOSVoiceProcessingEngine.swift` → **`IOSAudioEngine`** (same file): always-on `AVAudioSourceNode`
|
||||||
|
playback (runs whenever connected, so remote audio plays before you join voice); conditional mic tap;
|
||||||
|
VPIO + AGC toggled per config. One private `rebuild()` (stop → set VPIO → install tap → start) backs
|
||||||
|
`startListening`/`stop`/`startMic`/`stopMic`/`reconfigure`/`setCaptureChannels`. Kept the `PCMRing`
|
||||||
|
and ring-stats diagnostics.
|
||||||
|
- `IOSAudioRouter`: presets cut from seven to **four** — Voice Chat (VPIO mono, system output),
|
||||||
|
Stereo Mic / Mono Mic (internal built-in mic regardless of output, A2DP-capable, no VPIO), Advanced
|
||||||
|
(manual). New persisted `voiceProcessingEnabled` (master AEC+NS) + `agcEnabled`; setters now call
|
||||||
|
`IOSAudioEngine.reconfigure()` instead of `client.audioRestart()` + `reconcileVoicePath`. Kept the
|
||||||
|
proven AVAudioSession recipes (category/mode/options, stereo capsule, `applyA2dpSpeakerFallback`).
|
||||||
|
- `AudioSessionManager` slimmed (drops `client`/`activeMicStreamId`/`reconcileVoicePath`; adds
|
||||||
|
`isActive`); interruption-end & device-change now `reconfigure()` the engine. `SessionState`
|
||||||
|
`doStartMicStream`/`stopMicStream` collapsed to start-stream + `startMic`/`stopMic` (no
|
||||||
|
`setExternalPlayback`/`audioRestart` toggling); `reconcileVoicePath` deleted. `AppState` sets external
|
||||||
|
playback + `startListening` at connect, `stop()` at disconnect. `SettingsView` → four presets +
|
||||||
|
Advanced VPIO/AGC toggles.
|
||||||
|
- **No core/ABI/test change** — relies on the already-shipped `vc_set_external_playback` /
|
||||||
|
`external_feed` / `vc_set_mixed_output_sink` / `vc_stream_feed_pcm` path (`test_external_pcm`,
|
||||||
|
`test_external_playback`). `xcodebuild` iOS device Debug **BUILD SUCCEEDED**. **Rebuild the
|
||||||
|
xcframework is NOT required** (no new symbols).
|
||||||
|
- **Next (user, on device):** two iPhones in a channel — verify BOTH directions survive every
|
||||||
|
transition and are never silent unless intended: Voice Chat (no echo, NR), listen-only before joining,
|
||||||
|
join↔leave repeatedly, switch Voice Chat↔Stereo↔Mono↔Advanced *while in voice*, A2DP connect/unplug,
|
||||||
|
wired connect/unplug, phone-call interruption + resume, screen-audio share.
|
||||||
|
|
||||||
|
- **Superseded by the 2026-06-23 unification above (2026-06-22):** **iOS real echo cancellation / noise
|
||||||
|
suppression via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS
|
||||||
AEC/NS/AGC come only from Apple's Voice-Processing I/O unit (VPIO), but the core uses miniaudio's
|
AEC/NS/AGC come only from Apple's Voice-Processing I/O unit (VPIO), but the core uses miniaudio's
|
||||||
plain RemoteIO units — so `.voiceChat` mode alone never engaged AEC. Fix moves both mic capture and
|
plain RemoteIO units — so `.voiceChat` mode alone never engaged AEC. Fix moves both mic capture and
|
||||||
playback to a native Swift `AVAudioEngine` (`setVoiceProcessingEnabled`) on the AEC presets, with the
|
playback to a native Swift `AVAudioEngine` (`setVoiceProcessingEnabled`) on the AEC presets, with the
|
||||||
|
|||||||
@@ -91,6 +91,7 @@ final class AppState {
|
|||||||
func disconnect() {
|
func disconnect() {
|
||||||
session?.stopMicStream()
|
session?.stopMicStream()
|
||||||
session?.client.disconnect()
|
session?.client.disconnect()
|
||||||
|
IOSAudioEngine.shared.stop()
|
||||||
AudioSessionManager.shared.deactivateSession()
|
AudioSessionManager.shared.deactivateSession()
|
||||||
session = nil
|
session = nil
|
||||||
connectingClient?.disconnect()
|
connectingClient?.disconnect()
|
||||||
@@ -160,17 +161,18 @@ final class AppState {
|
|||||||
self.session = newSession
|
self.session = newSession
|
||||||
EventFeedback.shared.play(.login)
|
EventFeedback.shared.play(.login)
|
||||||
EventFeedback.shared.speak("Connected")
|
EventFeedback.shared.speak("Connected")
|
||||||
// Activate the audio session now, while connected — NOT lazily when the first
|
// Put the core into external-playback mode ONCE, now, before the session is
|
||||||
// remote stream arrives. The core opens its miniaudio playback device the moment
|
// activated or any remote stream can arrive — so the core never opens a miniaudio
|
||||||
// a remote stream starts and only THEN emits .streamStarted; if we waited for
|
// device on iOS (the single ordering rule of the unified audio path). Then activate
|
||||||
// that event to activate, the playback device would open against an inactive
|
// the session and start the engine in listening mode so remote audio plays the
|
||||||
// AVAudioSession and produce no sound (the "can't hear anyone" bug). Activating
|
// moment someone talks, even before we join voice (no "can't hear anyone").
|
||||||
// here guarantees the session is live before any device opens.
|
client.setExternalPlayback(true)
|
||||||
do {
|
do {
|
||||||
try AudioSessionManager.shared.ensureSessionActive()
|
try AudioSessionManager.shared.ensureSessionActive()
|
||||||
} catch {
|
} catch {
|
||||||
print("Audio session activate on connect failed: \(error)")
|
print("Audio session activate on connect failed: \(error)")
|
||||||
}
|
}
|
||||||
|
IOSAudioEngine.shared.startListening(client: client)
|
||||||
} else {
|
} else {
|
||||||
connectStatus = "Auth failed: \(ev.result.description)"
|
connectStatus = "Auth failed: \(ev.result.description)"
|
||||||
showPasswordPrompt = true
|
showPasswordPrompt = true
|
||||||
@@ -178,6 +180,7 @@ final class AppState {
|
|||||||
case .disconnected:
|
case .disconnected:
|
||||||
if session == nil { cancelConnect() }
|
if session == nil { cancelConnect() }
|
||||||
else {
|
else {
|
||||||
|
IOSAudioEngine.shared.stop()
|
||||||
AudioSessionManager.shared.deactivateSession()
|
AudioSessionManager.shared.deactivateSession()
|
||||||
session = nil; isConnecting = false
|
session = nil; isConnecting = false
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,19 +8,6 @@ private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "Audio
|
|||||||
final class AudioSessionManager {
|
final class AudioSessionManager {
|
||||||
static let shared = AudioSessionManager()
|
static let shared = AudioSessionManager()
|
||||||
|
|
||||||
weak var client: VoiceCatClient?
|
|
||||||
|
|
||||||
/// The stream ID of the currently active local MIC stream, if any. Set by `SessionState`
|
|
||||||
/// when the user joins/leaves voice so `IOSAudioRouter` can reset the core's capture
|
|
||||||
/// channel count (e.g. when switching stereo → mono) without going through `SessionState`.
|
|
||||||
var activeMicStreamId: UInt32?
|
|
||||||
|
|
||||||
/// Set by `SessionState`. Invoked by `IOSAudioRouter` after an audio-config change so the
|
|
||||||
/// voice path (native VPIO vs the core's miniaudio path) can be restarted to match the new
|
|
||||||
/// preset/route when the mic is active. No-op when not in voice. See
|
|
||||||
/// `SessionState.reconcileVoicePath()` and `IOSVoiceProcessingEngine`.
|
|
||||||
var reconcileVoicePath: (() -> Void)?
|
|
||||||
|
|
||||||
/// Tracks whether WE activated the session. The session must be active whenever the
|
/// Tracks whether WE activated the session. The session must be active whenever the
|
||||||
/// AudioEngine is running (for capture OR playback), so it is activated when any audio
|
/// AudioEngine is running (for capture OR playback), so it is activated when any audio
|
||||||
/// needs to play (a remote stream started OR the user joins voice) and only deactivated
|
/// needs to play (a remote stream started OR the user joins voice) and only deactivated
|
||||||
@@ -28,6 +15,10 @@ final class AudioSessionManager {
|
|||||||
/// want to hear remote audio.
|
/// want to hear remote audio.
|
||||||
private var isSessionActive = false
|
private var isSessionActive = false
|
||||||
|
|
||||||
|
/// Whether the AVAudioSession is currently active (we activated it). Read by `IOSAudioRouter`
|
||||||
|
/// to decide whether the post-activation A2DP speaker fallback can be applied.
|
||||||
|
var isActive: Bool { isSessionActive }
|
||||||
|
|
||||||
func configure() {
|
func configure() {
|
||||||
// Load stored audio routing preferences and apply them before any audio session
|
// Load stored audio routing preferences and apply them before any audio session
|
||||||
// activation. IOSAudioRouter drives all iOS audio route selection via AVAudioSession;
|
// activation. IOSAudioRouter drives all iOS audio route selection via AVAudioSession;
|
||||||
@@ -114,18 +105,22 @@ final class AudioSessionManager {
|
|||||||
|
|
||||||
switch type {
|
switch type {
|
||||||
case .began:
|
case .began:
|
||||||
|
// The system stops our AVAudioEngine and deactivates the session. Nothing to tear
|
||||||
|
// down — `IOSAudioEngine` rebuilds on resume.
|
||||||
logger.info("interruption began — session suspended by system")
|
logger.info("interruption began — session suspended by system")
|
||||||
isSessionActive = false // system deactivated us
|
isSessionActive = false
|
||||||
client?.audioSuspend()
|
|
||||||
case .ended:
|
case .ended:
|
||||||
let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0
|
let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0
|
||||||
let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue)
|
let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue)
|
||||||
if options.contains(.shouldResume) {
|
if options.contains(.shouldResume) {
|
||||||
do {
|
do {
|
||||||
|
IOSAudioRouter.shared.applyConfiguration()
|
||||||
try AVAudioSession.sharedInstance().setActive(true)
|
try AVAudioSession.sharedInstance().setActive(true)
|
||||||
isSessionActive = true
|
isSessionActive = true
|
||||||
logger.info("interruption ended — session reactivated")
|
IOSAudioRouter.shared.applyA2dpSpeakerFallback()
|
||||||
client?.audioResume()
|
// Rebuild the engine graph against the restored route (both directions).
|
||||||
|
IOSAudioEngine.shared.reconfigure()
|
||||||
|
logger.info("interruption ended — session reactivated, engine rebuilt")
|
||||||
} catch {
|
} catch {
|
||||||
logger.error("interruption ended — reactivation failed: \(error.localizedDescription)")
|
logger.error("interruption ended — reactivation failed: \(error.localizedDescription)")
|
||||||
}
|
}
|
||||||
@@ -162,6 +157,9 @@ final class AudioSessionManager {
|
|||||||
// the loud speaker (not the earpiece), and a replug should hand output back to A2DP.
|
// the loud speaker (not the earpiece), and a replug should hand output back to A2DP.
|
||||||
if isSessionActive {
|
if isSessionActive {
|
||||||
IOSAudioRouter.shared.applyA2dpSpeakerFallback()
|
IOSAudioRouter.shared.applyA2dpSpeakerFallback()
|
||||||
|
// Rebind the engine (both directions) to the new route. The engine owns the route
|
||||||
|
// now, so this is the single thing that re-establishes audio after a device change.
|
||||||
|
IOSAudioEngine.shared.reconfigure()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -4,17 +4,18 @@ import VoiceCatCore
|
|||||||
|
|
||||||
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioRouter")
|
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioRouter")
|
||||||
|
|
||||||
/// iOS audio routing layer — drives all iOS audio route selection via `AVAudioSession`
|
/// iOS audio routing layer — the sole owner of `AVAudioSession` on iOS. On iOS the core never
|
||||||
/// *before* the core (miniaudio) opens its device. This class is the sole owner of the
|
/// opens a hardware (miniaudio) device: a single `AVAudioEngine` (`IOSAudioEngine`) drives both
|
||||||
/// session: miniaudio does NOT touch `AVAudioSession` on iOS, because the core opens its
|
/// capture and playback and the core runs fully external (see docs/voice.md §8). This class just
|
||||||
/// devices through a `ma_context` configured with `sessionCategory = none` +
|
/// configures the *route* — category / mode / options, preferred input, data source, polar
|
||||||
/// `noAudioSessionActivate/Deactivate` (see `AudioEngine::make_context_config` in
|
/// pattern, stereo capsule — and `IOSAudioEngine` binds to whatever route is established. After
|
||||||
/// `core/src/audio/audio_engine.cpp`). Without that, miniaudio's default path resets the
|
/// any change here the engine is rebuilt via `IOSAudioEngine.reconfigure()` (a deterministic
|
||||||
/// category to `Record`/`Playback` with no options on every device open, wiping
|
/// Swift-only stop → reconfigure → start); there is no second (miniaudio) audio path to hand off
|
||||||
/// `.allowBluetoothA2DP`/`.playAndRecord` and killing headphone/A2DP output — so that
|
/// to, so a change cannot leave one direction dropped.
|
||||||
/// config must stay in place. All iOS audio routing (input port selection, mic
|
///
|
||||||
/// orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) must be
|
/// (The core's iOS `ma_context` is still configured with `sessionCategory = none` +
|
||||||
/// driven from here.
|
/// `noAudioSessionActivate/Deactivate` in `AudioEngine::make_context_config` so that, should the
|
||||||
|
/// core ever open a device, miniaudio would not reset the category — but on iOS it does not.)
|
||||||
///
|
///
|
||||||
/// The three user-facing choices:
|
/// The three user-facing choices:
|
||||||
/// 1. **Input port** — which physical input (built-in mic, Bluetooth HFP, headset,
|
/// 1. **Input port** — which physical input (built-in mic, Bluetooth HFP, headset,
|
||||||
@@ -64,77 +65,59 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
@Published var selectedInputPortId: String?
|
@Published var selectedInputPortId: String?
|
||||||
@Published var selectedDataSourceId: String?
|
@Published var selectedDataSourceId: String?
|
||||||
@Published var selectedPolarPattern: String?
|
@Published var selectedPolarPattern: String?
|
||||||
|
/// Master voice-processing switch (Apple VPIO: AEC + noise suppression bundled together).
|
||||||
|
/// iOS exposes no per-stage toggle, so this is the finest "echo cancellation / noise
|
||||||
|
/// reduction" control available. Only takes effect on a VPIO-capable config (mono + standard
|
||||||
|
/// + not A2DP); stereo / A2DP configs can't use VPIO regardless. Default on.
|
||||||
|
@Published var voiceProcessingEnabled: Bool = true
|
||||||
|
/// VPIO automatic gain control — the one VPIO sub-stage iOS lets us toggle independently.
|
||||||
|
/// Only meaningful when voice processing is active. Default on.
|
||||||
|
@Published var agcEnabled: Bool = true
|
||||||
@Published var showsRawModeSpeakerWarning: Bool = false
|
@Published var showsRawModeSpeakerWarning: Bool = false
|
||||||
@Published var showsA2dpNoAecWarning: Bool = false
|
@Published var showsA2dpNoAecWarning: Bool = false
|
||||||
@Published var hasBluetoothDevice: Bool = false
|
@Published var hasBluetoothDevice: Bool = false
|
||||||
@Published var hasWiredHeadset: Bool = false
|
@Published var hasWiredHeadset: Bool = false
|
||||||
|
|
||||||
/// Audio presets — sensible combinations of settings for common scenarios.
|
/// Audio presets — the four scenarios from the product spec. Pick a preset for a quick start,
|
||||||
/// The app is about choice: users can pick a preset for a quick start, then
|
/// then fine-tune individual settings under "Advanced". HFP / wired headsets are not separate
|
||||||
/// fine-tune individual settings under "Advanced Audio".
|
/// presets: Voice Chat lets the system route to them, and Advanced exposes manual selection.
|
||||||
enum AudioPreset: String, CaseIterable, Identifiable {
|
enum AudioPreset: String, CaseIterable, Identifiable {
|
||||||
/// Standard iOS VoIP experience: AEC/AGC/HPF on, mono, system picks best route
|
/// Voice chat: Apple VPIO does real AEC + noise suppression + AGC. Mono. The system picks
|
||||||
/// (BT HFP if connected, wired if connected, speaker if nothing). Always available.
|
/// the best route (Bluetooth HFP / wired / speaker / earpiece). Always available.
|
||||||
case voiceChat = "Voice Chat"
|
case voiceChat = "Voice Chat"
|
||||||
/// Stereo built-in mic capture (front+back capsules). A2DP output if BT is connected,
|
/// Internal **stereo** built-in mic regardless of the output route. A2DP output when a
|
||||||
/// else built-in speaker / wired. Standard processing (no AEC — stereo needs a non-VPIO
|
/// Bluetooth headset is connected, else built-in speaker / wired. No VPIO (stereo can't
|
||||||
/// mode). Always available.
|
/// use it). Always available.
|
||||||
case stereoMic = "Stereo Mic"
|
case stereoMic = "Stereo Mic"
|
||||||
/// Maximum fidelity: stereo mic, no AEC/AGC/HPF (raw mode). A2DP output if BT connected,
|
/// Internal **mono** built-in mic regardless of the output route. A2DP output when a
|
||||||
/// else speaker/wired. Always available. Echo risk on speaker.
|
/// Bluetooth headset is connected, else built-in speaker / wired. No VPIO. Always available.
|
||||||
case studio = "Studio (No Processing)"
|
case monoMic = "Mono Mic"
|
||||||
/// Bluetooth HFP: BT mic + BT output, AEC on, mono. Only when BT is connected.
|
/// Everything manual — input port, mic orientation / polar pattern, mono/stereo, Bluetooth
|
||||||
case bluetoothHeadset = "Bluetooth Headset (HFP)"
|
/// mode, raw vs standard, and the VPIO / AGC toggles. Also the display state when the
|
||||||
/// A2DP stereo output + built-in mono mic, AEC off. Only when BT is connected. (For
|
/// individual settings don't match a named preset.
|
||||||
/// A2DP output + stereo mic, use the Stereo Mic preset while BT is connected.)
|
case advanced = "Advanced"
|
||||||
case btHeadphonesMonoMic = "BT Headphones + Mono Mic"
|
|
||||||
/// Wired headset/earpods: wired output + wired mic (or built-in), AEC on, mono.
|
|
||||||
/// Only when a wired audio device is connected.
|
|
||||||
case wiredHeadset = "Wired Headset"
|
|
||||||
/// Settings don't match any preset — user has tweaked advanced controls.
|
|
||||||
case custom = "Custom"
|
|
||||||
|
|
||||||
var id: String { rawValue }
|
var id: String { rawValue }
|
||||||
|
|
||||||
var requiresBluetooth: Bool {
|
|
||||||
switch self {
|
|
||||||
case .bluetoothHeadset, .btHeadphonesMonoMic: return true
|
|
||||||
default: return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
var requiresWired: Bool {
|
|
||||||
self == .wiredHeadset
|
|
||||||
}
|
|
||||||
|
|
||||||
var bluetoothMode: BluetoothMode {
|
var bluetoothMode: BluetoothMode {
|
||||||
switch self {
|
switch self {
|
||||||
case .voiceChat, .bluetoothHeadset: return .btHfpVoice
|
case .voiceChat: return .btHfpVoice
|
||||||
// A2DP output when BT is connected; falls back to speaker/wired when it isn't.
|
// Internal-mic presets: A2DP output when BT is connected; speaker/wired when not.
|
||||||
case .stereoMic, .studio, .btHeadphonesMonoMic: return .builtInMicBtA2dp
|
case .stereoMic, .monoMic: return .builtInMicBtA2dp
|
||||||
case .wiredHeadset: return .builtInMicSpeaker
|
case .advanced: return .builtInMicSpeaker // placeholder; Advanced sets it manually
|
||||||
case .custom: return .builtInMicSpeaker // placeholder
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var captureChannels: CaptureChannels {
|
var captureChannels: CaptureChannels {
|
||||||
switch self {
|
self == .stereoMic ? .stereo : .mono
|
||||||
case .stereoMic, .studio: return .stereo
|
|
||||||
default: return .mono
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var micMode: MicMode {
|
var micMode: MicMode { .standard }
|
||||||
switch self {
|
|
||||||
case .studio: return .raw
|
|
||||||
default: return .standard
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Whether this preset explicitly selects the built-in mic port.
|
/// Whether this preset explicitly pins the built-in mic port (the internal-mic presets).
|
||||||
var usesBuiltInMic: Bool {
|
var usesBuiltInMic: Bool {
|
||||||
switch self {
|
switch self {
|
||||||
case .stereoMic, .studio, .btHeadphonesMonoMic: return true
|
case .stereoMic, .monoMic: return true
|
||||||
default: return false
|
default: return false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -170,6 +153,8 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
private let kPolarPattern = "cat.voice.audio.polarPattern"
|
private let kPolarPattern = "cat.voice.audio.polarPattern"
|
||||||
private let kPreset = "cat.voice.audio.preset"
|
private let kPreset = "cat.voice.audio.preset"
|
||||||
private let kForceSpeaker = "cat.voice.audio.forceSpeaker"
|
private let kForceSpeaker = "cat.voice.audio.forceSpeaker"
|
||||||
|
private let kVoiceProcessing = "cat.voice.audio.voiceProcessing"
|
||||||
|
private let kAgc = "cat.voice.audio.agc"
|
||||||
|
|
||||||
/// Re-entrancy guard: setCategory/setPreferredInput/etc. trigger route-change
|
/// Re-entrancy guard: setCategory/setPreferredInput/etc. trigger route-change
|
||||||
/// notifications synchronously on the same thread. Without this guard,
|
/// notifications synchronously on the same thread. Without this guard,
|
||||||
@@ -271,55 +256,41 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The presets available given the current device connection state.
|
/// The presets the user can pick. All four are always available — the named presets simply
|
||||||
/// Always includes Voice Chat, Stereo Mic, Studio, and Custom. BT presets only when
|
/// describe what to do "regardless of the output route", and Advanced is always offered.
|
||||||
/// a Bluetooth device is connected. Wired preset only when a wired device is connected.
|
var availablePresets: [AudioPreset] { AudioPreset.allCases }
|
||||||
var availablePresets: [AudioPreset] {
|
|
||||||
AudioPreset.allCases.filter { preset in
|
|
||||||
if preset == .custom { return true }
|
|
||||||
if preset.requiresBluetooth && !hasBluetoothDevice { return false }
|
|
||||||
if preset.requiresWired && !hasWiredHeadset { return false }
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Which preset matches the current settings, or .custom if nothing matches.
|
/// Which named preset matches the current settings, or `.advanced` if nothing matches.
|
||||||
/// Checks device-specific presets first (BT, wired) so that e.g. when BT is connected
|
|
||||||
/// and settings match "Bluetooth Headset", it returns that instead of the equivalent
|
|
||||||
/// "Voice Chat" (which has the same bluetoothMode/micMode/channels but is more general).
|
|
||||||
var activePreset: AudioPreset {
|
var activePreset: AudioPreset {
|
||||||
// Check device-specific presets first (most specific → least specific)
|
for preset in [AudioPreset.voiceChat, .stereoMic, .monoMic] {
|
||||||
let order: [AudioPreset] = [
|
|
||||||
.bluetoothHeadset, .btHeadphonesMonoMic,
|
|
||||||
.wiredHeadset,
|
|
||||||
.voiceChat, .stereoMic, .studio,
|
|
||||||
]
|
|
||||||
for preset in order {
|
|
||||||
if bluetoothMode == preset.bluetoothMode
|
if bluetoothMode == preset.bluetoothMode
|
||||||
&& captureChannels == preset.captureChannels
|
&& captureChannels == preset.captureChannels
|
||||||
&& micMode == preset.micMode {
|
&& micMode == preset.micMode {
|
||||||
// Don't match a BT preset if no BT is connected — fall through to Voice Chat
|
|
||||||
if preset.requiresBluetooth && !hasBluetoothDevice { continue }
|
|
||||||
if preset.requiresWired && !hasWiredHeadset { continue }
|
|
||||||
return preset
|
return preset
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return .custom
|
return .advanced
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Whether the current configuration should use the native iOS Voice-Processing path (VPIO:
|
/// Whether the current configuration should engage Apple's Voice-Processing I/O unit (VPIO:
|
||||||
/// real AEC/NS/AGC via `IOSVoiceProcessingEngine`). True exactly when `applyConfiguration`
|
/// real AEC + noise suppression + AGC, driven by `IOSAudioEngine`). VPIO forces mono and
|
||||||
/// selects the `.voiceChat` AVAudioSession mode — mono + standard processing + not A2DP
|
/// can't run on an A2DP route, so it is available only for a mono + standard + non-A2DP
|
||||||
/// (A2DP / stereo / raw modes can't use VPIO, so they keep the core's miniaudio path).
|
/// config, and then only when the user hasn't disabled it via the Advanced master toggle.
|
||||||
var currentConfigUsesVoiceProcessing: Bool {
|
var currentConfigUsesVoiceProcessing: Bool {
|
||||||
|
voiceProcessingEnabled && voiceProcessingAvailable
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether the current config *could* use VPIO (mono + standard + non-A2DP), independent of
|
||||||
|
/// the user's master toggle. Drives whether the Advanced "Voice Processing" switch is shown.
|
||||||
|
var voiceProcessingAvailable: Bool {
|
||||||
captureChannels == .mono && micMode == .standard && bluetoothMode != .builtInMicBtA2dp
|
captureChannels == .mono && micMode == .standard && bluetoothMode != .builtInMicBtA2dp
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Apply configuration
|
// MARK: - Apply configuration
|
||||||
|
|
||||||
/// Apply the full audio configuration to AVAudioSession. Call this before the core
|
/// Apply the full audio configuration to AVAudioSession. Call this before (re)building the
|
||||||
/// opens its capture device (i.e. before `startMicStream` → `activateForStreaming`).
|
/// `IOSAudioEngine` graph so the engine binds to the intended route (`applyAndReconfigure`
|
||||||
/// Re-entrant-safe: if a route-change notification fires synchronously during a
|
/// does both). Re-entrant-safe: if a route-change notification fires synchronously during a
|
||||||
/// `setCategory`/`setPreferredInput` call, the guard prevents re-entry.
|
/// `setCategory`/`setPreferredInput` call, the guard prevents re-entry.
|
||||||
func applyConfiguration() {
|
func applyConfiguration() {
|
||||||
guard !isApplyingConfiguration else {
|
guard !isApplyingConfiguration else {
|
||||||
@@ -401,8 +372,8 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
// route explicitly via setPreferredInput + setInputDataSource. With HFP disabled
|
// route explicitly via setPreferredInput + setInputDataSource. With HFP disabled
|
||||||
// the system routes input to the built-in mic, but without the explicit
|
// the system routes input to the built-in mic, but without the explicit
|
||||||
// preferred-input anchor the route can collapse during the mode switch
|
// preferred-input anchor the route can collapse during the mode switch
|
||||||
// (.voiceChat → .default) and the output dies. The channel count is requested by
|
// (.voiceChat → .default) and the output dies. The channel count is carried by the
|
||||||
// miniaudio at the audio-unit level (vc_set_capture_channels), NOT via
|
// engine's mic tap + vc_set_capture_channels, NOT via
|
||||||
// setPreferredInputNumberOfChannels(2) — that call collapses the A2DP output route.
|
// setPreferredInputNumberOfChannels(2) — that call collapses the A2DP output route.
|
||||||
configureStereoCapture(session: session)
|
configureStereoCapture(session: session)
|
||||||
} else if let portId = selectedInputPortId, !portId.isEmpty,
|
} else if let portId = selectedInputPortId, !portId.isEmpty,
|
||||||
@@ -431,9 +402,9 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
/// 3. `setPreferredInput(builtIn)` — anchor the input route explicitly. Without this
|
/// 3. `setPreferredInput(builtIn)` — anchor the input route explicitly. Without this
|
||||||
/// anchor the route can collapse during the mode switch (.voiceChat → .default).
|
/// anchor the route can collapse during the mode switch (.voiceChat → .default).
|
||||||
/// 4. `setInputDataSource(stereoSource)` — commit the data source at the session level
|
/// 4. `setInputDataSource(stereoSource)` — commit the data source at the session level
|
||||||
/// The channel count itself is requested by miniaudio at the audio-unit level via
|
/// The channel count itself is carried by the engine's mic tap (which captures 2 channels)
|
||||||
/// `vc_set_capture_channels(2)`. We must NOT call `setPreferredInputNumberOfChannels(2)`
|
/// plus `vc_set_capture_channels(2)` so the core encodes stereo. We must NOT call
|
||||||
/// — that session-level call collapses the A2DP output route.
|
/// `setPreferredInputNumberOfChannels(2)` — that session-level call collapses the A2DP route.
|
||||||
private func configureStereoCapture(session: AVAudioSession) {
|
private func configureStereoCapture(session: AVAudioSession) {
|
||||||
guard let builtIn = session.availableInputs?.first(where: { $0.portType == .builtInMic })
|
guard let builtIn = session.availableInputs?.first(where: { $0.portType == .builtInMic })
|
||||||
else {
|
else {
|
||||||
@@ -538,6 +509,9 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
selectedDataSourceId = UserDefaults.standard.string(forKey: kDataSourceId)
|
selectedDataSourceId = UserDefaults.standard.string(forKey: kDataSourceId)
|
||||||
selectedPolarPattern = UserDefaults.standard.string(forKey: kPolarPattern)
|
selectedPolarPattern = UserDefaults.standard.string(forKey: kPolarPattern)
|
||||||
forceSpeaker = UserDefaults.standard.bool(forKey: kForceSpeaker)
|
forceSpeaker = UserDefaults.standard.bool(forKey: kForceSpeaker)
|
||||||
|
// VPIO toggles default ON when never set (object(forKey:) is nil → use true).
|
||||||
|
voiceProcessingEnabled = (UserDefaults.standard.object(forKey: kVoiceProcessing) as? Bool) ?? true
|
||||||
|
agcEnabled = (UserDefaults.standard.object(forKey: kAgc) as? Bool) ?? true
|
||||||
}
|
}
|
||||||
/// Persist current selections to UserDefaults.
|
/// Persist current selections to UserDefaults.
|
||||||
func savePreferences() {
|
func savePreferences() {
|
||||||
@@ -548,86 +522,88 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
UserDefaults.standard.set(selectedDataSourceId, forKey: kDataSourceId)
|
UserDefaults.standard.set(selectedDataSourceId, forKey: kDataSourceId)
|
||||||
UserDefaults.standard.set(selectedPolarPattern, forKey: kPolarPattern)
|
UserDefaults.standard.set(selectedPolarPattern, forKey: kPolarPattern)
|
||||||
UserDefaults.standard.set(forceSpeaker, forKey: kForceSpeaker)
|
UserDefaults.standard.set(forceSpeaker, forKey: kForceSpeaker)
|
||||||
|
UserDefaults.standard.set(voiceProcessingEnabled, forKey: kVoiceProcessing)
|
||||||
|
UserDefaults.standard.set(agcEnabled, forKey: kAgc)
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Selection setters (called from SettingsView pickers)
|
// MARK: - Selection setters (called from SettingsView pickers)
|
||||||
|
|
||||||
|
/// Shared tail for every setting change: persist, re-apply the AVAudioSession config, refresh
|
||||||
|
/// the route lists, re-evaluate the A2DP speaker fallback, and rebind the live engine to the
|
||||||
|
/// new route. `IOSAudioEngine.reconfigure()` is a no-op when not connected, so this is safe to
|
||||||
|
/// call from Settings whether or not a session is in progress. There is no longer a second
|
||||||
|
/// (miniaudio) audio path to hand off to, so one engine rebuild is the whole story.
|
||||||
|
private func applyAndReconfigure() {
|
||||||
|
savePreferences()
|
||||||
|
applyConfiguration()
|
||||||
|
if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() }
|
||||||
|
refreshRoutes()
|
||||||
|
IOSAudioEngine.shared.reconfigure()
|
||||||
|
}
|
||||||
|
|
||||||
func selectInputPort(_ portId: String) {
|
func selectInputPort(_ portId: String) {
|
||||||
selectedInputPortId = portId
|
selectedInputPortId = portId
|
||||||
selectedDataSourceId = nil
|
selectedDataSourceId = nil
|
||||||
selectedPolarPattern = nil
|
selectedPolarPattern = nil
|
||||||
savePreferences()
|
applyAndReconfigure()
|
||||||
applyConfiguration()
|
|
||||||
refreshRoutes()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func selectDataSource(_ dataSourceId: String) {
|
func selectDataSource(_ dataSourceId: String) {
|
||||||
selectedDataSourceId = dataSourceId
|
selectedDataSourceId = dataSourceId
|
||||||
selectedPolarPattern = nil
|
selectedPolarPattern = nil
|
||||||
savePreferences()
|
applyAndReconfigure()
|
||||||
applyConfiguration()
|
|
||||||
refreshRoutes()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func selectPolarPattern(_ pattern: String) {
|
func selectPolarPattern(_ pattern: String) {
|
||||||
selectedPolarPattern = pattern
|
selectedPolarPattern = pattern
|
||||||
savePreferences()
|
applyAndReconfigure()
|
||||||
applyConfiguration()
|
|
||||||
refreshRoutes()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func selectBluetoothMode(_ mode: BluetoothMode) {
|
func selectBluetoothMode(_ mode: BluetoothMode) {
|
||||||
bluetoothMode = mode
|
bluetoothMode = mode
|
||||||
savePreferences()
|
applyAndReconfigure()
|
||||||
applyConfiguration()
|
|
||||||
refreshRoutes()
|
|
||||||
// VPIO class or route may have changed — restart the voice path if mic is active.
|
|
||||||
AudioSessionManager.shared.reconcileVoicePath?()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func setForceSpeaker(_ on: Bool) {
|
func setForceSpeaker(_ on: Bool) {
|
||||||
forceSpeaker = on
|
forceSpeaker = on
|
||||||
savePreferences()
|
applyAndReconfigure()
|
||||||
applyConfiguration()
|
|
||||||
refreshRoutes()
|
|
||||||
// Route changed under a possibly-running VPIO engine — reconcile if mic is active.
|
|
||||||
AudioSessionManager.shared.reconcileVoicePath?()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func selectMicMode(_ mode: MicMode) {
|
func selectMicMode(_ mode: MicMode) {
|
||||||
micMode = mode
|
micMode = mode
|
||||||
|
applyAndReconfigure()
|
||||||
|
}
|
||||||
|
|
||||||
|
func setVoiceProcessingEnabled(_ on: Bool) {
|
||||||
|
voiceProcessingEnabled = on
|
||||||
|
applyAndReconfigure()
|
||||||
|
}
|
||||||
|
|
||||||
|
func setAgcEnabled(_ on: Bool) {
|
||||||
|
agcEnabled = on
|
||||||
|
// No session reconfigure needed — just rebuild the engine so VPIO picks up the AGC flag.
|
||||||
savePreferences()
|
savePreferences()
|
||||||
applyConfiguration()
|
IOSAudioEngine.shared.reconfigure()
|
||||||
updateWarnings()
|
|
||||||
// Standard↔Raw flips the VPIO class — reconcile if mic is active.
|
|
||||||
AudioSessionManager.shared.reconcileVoicePath?()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func selectCaptureChannels(_ channels: CaptureChannels) {
|
func selectCaptureChannels(_ channels: CaptureChannels) {
|
||||||
captureChannels = channels
|
captureChannels = channels
|
||||||
savePreferences()
|
savePreferences()
|
||||||
applyConfiguration()
|
applyConfiguration()
|
||||||
// Update the core's stored capture channel count (does not restart the engine).
|
if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() }
|
||||||
if let streamId = AudioSessionManager.shared.activeMicStreamId {
|
refreshRoutes()
|
||||||
_ = AudioSessionManager.shared.client?.setCaptureChannels(
|
// Push the channel count into the core's MIC stream, then rebuild the engine graph so the
|
||||||
streamId: streamId, channels: channels.channelCount)
|
// mic tap captures the right number of channels. The engine owns the route now, so there's
|
||||||
}
|
// no stereo-vs-A2DP race to sequence around.
|
||||||
// Restart the engine AFTER AVAudioSession routing has settled and the channel
|
IOSAudioEngine.shared.setCaptureChannels(channels.channelCount)
|
||||||
// count is stored. The engine reopens playback first (committing the A2DP/output
|
|
||||||
// route), then capture — avoiding the race where stereo capture activation drops
|
|
||||||
// A2DP before the playback device has a chance to claim the route.
|
|
||||||
_ = AudioSessionManager.shared.client?.audioRestart()
|
|
||||||
// Mono↔stereo flips the VPIO class (stereo can't use VPIO) — reconcile if mic is active.
|
|
||||||
AudioSessionManager.shared.reconcileVoicePath?()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Presets
|
// MARK: - Presets
|
||||||
|
|
||||||
/// Apply a preset — sets all individual audio settings to the preset's values, then
|
/// Apply a named preset — set all individual settings to the preset's values, then re-apply
|
||||||
/// applies the configuration. For presets that use the built-in mic (A2DP presets),
|
/// the configuration and rebind the engine. The internal-mic presets pin the built-in mic.
|
||||||
/// finds the built-in mic port UID from availableInputs.
|
|
||||||
func applyPreset(_ preset: AudioPreset) {
|
func applyPreset(_ preset: AudioPreset) {
|
||||||
guard preset != .custom else { return } // can't "apply" custom — it's a display state
|
guard preset != .advanced else { return } // Advanced is a display state, not "applied"
|
||||||
|
|
||||||
bluetoothMode = preset.bluetoothMode
|
bluetoothMode = preset.bluetoothMode
|
||||||
micMode = preset.micMode
|
micMode = preset.micMode
|
||||||
@@ -638,19 +614,17 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
if preset == .voiceChat { forceSpeaker = true }
|
if preset == .voiceChat { forceSpeaker = true }
|
||||||
|
|
||||||
if preset.usesBuiltInMic {
|
if preset.usesBuiltInMic {
|
||||||
// Find the built-in mic port from available inputs and select it.
|
// Pin the built-in mic. In stereo, iOS uses multiple capsules automatically; in mono
|
||||||
let session = AVAudioSession.sharedInstance()
|
// the default orientation is fine — so don't force a specific data source / pattern.
|
||||||
if let builtInMic = (session.availableInputs ?? []).first(where: {
|
if let builtInMic = (AVAudioSession.sharedInstance().availableInputs ?? []).first(where: {
|
||||||
$0.portType == .builtInMic
|
$0.portType == .builtInMic
|
||||||
}) {
|
}) {
|
||||||
selectedInputPortId = builtInMic.uid
|
selectedInputPortId = builtInMic.uid
|
||||||
}
|
}
|
||||||
// Don't set a specific data source — in stereo mode, iOS uses multiple mic
|
|
||||||
// capsules automatically. In mono, the default orientation is fine.
|
|
||||||
selectedDataSourceId = nil
|
selectedDataSourceId = nil
|
||||||
selectedPolarPattern = nil
|
selectedPolarPattern = nil
|
||||||
} else {
|
} else {
|
||||||
// For Default and Bluetooth Headset presets, let the system pick the input.
|
// Voice Chat: let the system pick the input (Bluetooth HFP / wired / built-in).
|
||||||
selectedInputPortId = nil
|
selectedInputPortId = nil
|
||||||
selectedDataSourceId = nil
|
selectedDataSourceId = nil
|
||||||
selectedPolarPattern = nil
|
selectedPolarPattern = nil
|
||||||
@@ -659,18 +633,11 @@ final class IOSAudioRouter: ObservableObject {
|
|||||||
UserDefaults.standard.set(preset.rawValue, forKey: kPreset)
|
UserDefaults.standard.set(preset.rawValue, forKey: kPreset)
|
||||||
savePreferences()
|
savePreferences()
|
||||||
applyConfiguration()
|
applyConfiguration()
|
||||||
// Update the core's stored capture channel count (does not restart the engine).
|
if AudioSessionManager.shared.isActive { applyA2dpSpeakerFallback() }
|
||||||
if let streamId = AudioSessionManager.shared.activeMicStreamId {
|
|
||||||
_ = AudioSessionManager.shared.client?.setCaptureChannels(
|
|
||||||
streamId: streamId, channels: preset.captureChannels.channelCount)
|
|
||||||
}
|
|
||||||
// Restart the engine AFTER AVAudioSession routing has settled and the channel
|
|
||||||
// count is stored. Playback opens first (commits A2DP route), then capture.
|
|
||||||
_ = AudioSessionManager.shared.client?.audioRestart()
|
|
||||||
refreshRoutes()
|
refreshRoutes()
|
||||||
// The preset may have flipped the VPIO class (and/or the route) — restart the voice path
|
// Push the channel count to the core, then rebuild the engine graph (VPIO on/off + tap).
|
||||||
// if the mic is active so AEC/NS engage (or disengage) to match the new preset.
|
IOSAudioEngine.shared.setCaptureChannels(preset.captureChannels.channelCount)
|
||||||
AudioSessionManager.shared.reconcileVoicePath?()
|
IOSAudioEngine.shared.reconfigure()
|
||||||
logger.info("applyPreset — \(preset.rawValue)")
|
logger.info("applyPreset — \(preset.rawValue)")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ import Darwin
|
|||||||
import os
|
import os
|
||||||
import VoiceCatCore
|
import VoiceCatCore
|
||||||
|
|
||||||
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSVoiceProcessingEngine")
|
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAudioEngine")
|
||||||
|
|
||||||
/// In-process single-producer/single-consumer int16 PCM ring for the VPIO playback path.
|
/// In-process single-producer/single-consumer int16 PCM ring for the playback path.
|
||||||
///
|
///
|
||||||
/// producer = the core's mixer-timer thread (the `vc_set_mixed_output_sink` callback)
|
/// producer = the core's mixer-timer thread (the `vc_set_mixed_output_sink` callback)
|
||||||
/// consumer = the `AVAudioSourceNode` render thread
|
/// consumer = the `AVAudioSourceNode` render thread
|
||||||
@@ -79,38 +79,48 @@ final class PCMRing {
|
|||||||
/// Diagnostics: monotonic total samples written / read since the ring was created. The
|
/// Diagnostics: monotonic total samples written / read since the ring was created. The
|
||||||
/// indices are already cumulative, so these are free. Only read them when both threads are
|
/// indices are already cumulative, so these are free. Only read them when both threads are
|
||||||
/// quiesced (e.g. at teardown after the engine + mixer sink are stopped) — they are not
|
/// quiesced (e.g. at teardown after the engine + mixer sink are stopped) — they are not
|
||||||
/// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" (Bug 1
|
/// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" apart
|
||||||
/// core path) apart from "PCM arrived but produced no sound" (AVAudioEngine output graph).
|
/// from "PCM arrived but produced no sound" (the AVAudioEngine output graph).
|
||||||
var debugTotalWritten: UInt64 { writeIdx }
|
var debugTotalWritten: UInt64 { writeIdx }
|
||||||
var debugTotalRead: UInt64 { readIdx }
|
var debugTotalRead: UInt64 { readIdx }
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Native iOS voice-processing audio path (docs/voice.md §8 "iOS voice processing").
|
/// The single iOS audio engine (docs/voice.md §8 "iOS audio engine").
|
||||||
///
|
///
|
||||||
/// Real iOS echo cancellation / noise suppression / AGC come ONLY from Apple's Voice-Processing
|
/// **One path, always external.** On iOS the core never opens a miniaudio device: a MIC stream is
|
||||||
/// I/O unit (VPIO), which `AVAudioEngine.setVoiceProcessingEnabled(true)` enables. For VPIO to
|
/// always started with `external_feed=1`, `vc_set_external_playback(1)` is set once at connect, and
|
||||||
/// cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
|
/// this engine drives *both* directions through one `AVAudioEngine`:
|
||||||
/// played-back signal from the mic), so on the AEC presets this engine drives both directions and
|
|
||||||
/// the core runs in external mode (no hardware devices):
|
|
||||||
/// - **Mic → core:** a tap on the VPIO input node → 48 kHz int16 → `client.feedPcm(micStreamId)`.
|
|
||||||
/// - **core → speaker:** the core's mixed-output sink fills `ring`; an `AVAudioSourceNode` pulls
|
/// - **core → speaker:** the core's mixed-output sink fills `ring`; an `AVAudioSourceNode` pulls
|
||||||
/// from it and renders through the VPIO output, giving AEC its reference signal.
|
/// from it and renders through the engine output. This runs the whole time we're connected,
|
||||||
|
/// so remote audio plays even before the user joins voice (no "can't hear anyone").
|
||||||
|
/// - **mic → core:** when the mic is active a tap on the input node converts to 48 kHz int16 and
|
||||||
|
/// calls `client.feedPcm(micStreamId)`.
|
||||||
///
|
///
|
||||||
/// Lifecycle is driven by `SessionState` join/leave. The Stereo Mic / Studio / A2DP presets keep
|
/// Echo cancellation / noise suppression / AGC come from Apple's Voice-Processing I/O unit (VPIO),
|
||||||
/// the core's miniaudio path instead (they want raw / stereo / no-AEC routing VPIO can't provide).
|
/// which `inputNode.setVoiceProcessingEnabled(true)` enables. VPIO forces mono, so it is engaged
|
||||||
|
/// only when the active preset wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`) — the
|
||||||
|
/// Stereo Mic / A2DP configs run the same engine with VPIO off.
|
||||||
|
///
|
||||||
|
/// Every preset / route / interruption change funnels through `reconfigure()`: a single
|
||||||
|
/// deterministic stop → AVAudioSession reconfigure → rebuild graph → start. There is no second
|
||||||
|
/// (miniaudio) audio path to hand off to, so a switch cannot leave one direction dropped.
|
||||||
@MainActor
|
@MainActor
|
||||||
final class IOSVoiceProcessingEngine {
|
final class IOSAudioEngine {
|
||||||
static let shared = IOSVoiceProcessingEngine()
|
static let shared = IOSAudioEngine()
|
||||||
|
|
||||||
private(set) var isRunning = false
|
/// True while connected (between `startListening` and `stop`) — the playback graph should run.
|
||||||
|
private(set) var isConnected = false
|
||||||
|
/// True while a local mic stream is active — the input tap should be installed.
|
||||||
|
private(set) var micActive = false
|
||||||
|
|
||||||
private let engine = AVAudioEngine()
|
private let engine = AVAudioEngine()
|
||||||
private var sourceNode: AVAudioSourceNode?
|
private var sourceNode: AVAudioSourceNode?
|
||||||
private weak var client: VoiceCatClient?
|
private weak var client: VoiceCatClient?
|
||||||
private var micStreamId: UInt32 = 0
|
private var micStreamId: UInt32 = 0
|
||||||
|
private var captureChannels: UInt32 = 1
|
||||||
|
|
||||||
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders and the engine
|
// 48 kHz stereo Float32 (deinterleaved) — the format the source node renders. The core
|
||||||
// processes in. The core delivers 48 kHz stereo int16 via the mixed-output sink.
|
// delivers 48 kHz stereo int16 via the mixed-output sink; mainMixerNode adapts to the route.
|
||||||
private let outFormat = AVAudioFormat(
|
private let outFormat = AVAudioFormat(
|
||||||
commonFormat: .pcmFormatFloat32, sampleRate: 48000, channels: 2, interleaved: false)!
|
commonFormat: .pcmFormatFloat32, sampleRate: 48000, channels: 2, interleaved: false)!
|
||||||
|
|
||||||
@@ -121,33 +131,138 @@ final class IOSVoiceProcessingEngine {
|
|||||||
private let renderScratchFrames = 8192
|
private let renderScratchFrames = 8192
|
||||||
private let renderScratch: UnsafeMutablePointer<Int16>
|
private let renderScratch: UnsafeMutablePointer<Int16>
|
||||||
|
|
||||||
// Mic-feed converter (input-node format → 48 kHz int16) and its target buffer. Owned here so
|
|
||||||
// the (background) tap block reuses them instead of allocating per callback.
|
|
||||||
private var micConverter: AVAudioConverter?
|
|
||||||
private var micTargetFormat: AVAudioFormat?
|
|
||||||
|
|
||||||
private init() {
|
private init() {
|
||||||
renderScratch = UnsafeMutablePointer<Int16>.allocate(capacity: renderScratchFrames * 2)
|
renderScratch = UnsafeMutablePointer<Int16>.allocate(capacity: renderScratchFrames * 2)
|
||||||
renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2)
|
renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Start the VPIO engine for an active mic stream. The caller must have already enabled
|
// MARK: - Lifecycle
|
||||||
/// external playback on the core (`client.setExternalPlayback(true)` + `audioRestart()`) and
|
|
||||||
/// started the MIC stream with `externalFeed: true`.
|
/// Begin playback-only (listening) operation. Called once at connect, after
|
||||||
func start(client: VoiceCatClient, micStreamId: UInt32, captureChannels: UInt32) {
|
/// `client.setExternalPlayback(true)` and `AudioSessionManager.ensureSessionActive()`. Attaches
|
||||||
guard !isRunning else { return }
|
/// the source node, wires the core's mixed-output sink into the ring, and starts the engine so
|
||||||
|
/// remote audio plays immediately.
|
||||||
|
func startListening(client: VoiceCatClient) {
|
||||||
self.client = client
|
self.client = client
|
||||||
self.micStreamId = micStreamId
|
guard !isConnected else { return }
|
||||||
|
isConnected = true
|
||||||
ring.reset()
|
ring.reset()
|
||||||
|
|
||||||
// Enable the voice-processing I/O unit (AEC/NS/AGC) on the shared input+output unit.
|
// Wire the core's mixed-output sink into the ring (C function pointer, no captures). Stays
|
||||||
do {
|
// registered for the whole connection; the ring is drained by the source-node render block.
|
||||||
try engine.inputNode.setVoiceProcessingEnabled(true)
|
let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque()
|
||||||
} catch {
|
client.setMixedOutputSink({ user, pcm, spc, ch, _ in
|
||||||
logger.error("setVoiceProcessingEnabled failed: \(error.localizedDescription) — AEC unavailable")
|
guard let user, let pcm else { return }
|
||||||
|
let ring = Unmanaged<PCMRing>.fromOpaque(user).takeUnretainedValue()
|
||||||
|
ring.write(pcm, count: spc * Int(ch))
|
||||||
|
}, user: ringPtr)
|
||||||
|
|
||||||
|
rebuild()
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Playback: source node pulls mixed PCM from the ring through the VPIO output. ──
|
/// Tear down the engine and unhook the core sink. Called on disconnect.
|
||||||
|
func stop() {
|
||||||
|
guard isConnected else { return }
|
||||||
|
micActive = false
|
||||||
|
isConnected = false
|
||||||
|
client?.setMixedOutputSink(nil, user: nil)
|
||||||
|
engine.inputNode.removeTap(onBus: 0)
|
||||||
|
if engine.isRunning { engine.stop() }
|
||||||
|
logger.info("audio engine stopped — ring written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples")
|
||||||
|
try? engine.inputNode.setVoiceProcessingEnabled(false)
|
||||||
|
if let src = sourceNode {
|
||||||
|
engine.detach(src)
|
||||||
|
sourceNode = nil
|
||||||
|
}
|
||||||
|
ring.reset()
|
||||||
|
client = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Mic transitions
|
||||||
|
|
||||||
|
/// Engage the mic: install the input tap and (if the preset wants it) VPIO. Called when the
|
||||||
|
/// user joins voice, after the MIC stream (external_feed) is started.
|
||||||
|
func startMic(streamId: UInt32, channels: UInt32) {
|
||||||
|
micStreamId = streamId
|
||||||
|
captureChannels = channels
|
||||||
|
micActive = true
|
||||||
|
rebuild()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Disengage the mic: remove the tap and VPIO, keep playback running for remaining remote audio.
|
||||||
|
func stopMic() {
|
||||||
|
guard micActive else { return }
|
||||||
|
micActive = false
|
||||||
|
rebuild()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Update the capture channel count (mono↔stereo) for the active mic and rebuild.
|
||||||
|
func setCaptureChannels(_ channels: UInt32) {
|
||||||
|
captureChannels = channels
|
||||||
|
if let client, micStreamId != 0 {
|
||||||
|
client.setCaptureChannels(streamId: micStreamId, channels: channels)
|
||||||
|
}
|
||||||
|
if micActive { rebuild() }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Re-apply the engine graph against the current AVAudioSession config (preset / route change).
|
||||||
|
/// Safe to call when only listening — it just rebuilds the playback graph against the new route.
|
||||||
|
func reconfigure() {
|
||||||
|
guard isConnected else { return }
|
||||||
|
rebuild()
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Graph (re)build
|
||||||
|
|
||||||
|
/// The single place that (re)builds and starts the engine graph. Deterministic: stop → set
|
||||||
|
/// VPIO → (re)install the mic tap → start. The caller is responsible for having applied the
|
||||||
|
/// AVAudioSession config (category/mode/route) first (`IOSAudioRouter.applyConfiguration`).
|
||||||
|
private func rebuild() {
|
||||||
|
guard isConnected else { return }
|
||||||
|
if engine.isRunning { engine.stop() }
|
||||||
|
engine.inputNode.removeTap(onBus: 0)
|
||||||
|
|
||||||
|
let useVPIO = micActive && IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
|
||||||
|
do {
|
||||||
|
try engine.inputNode.setVoiceProcessingEnabled(useVPIO)
|
||||||
|
} catch {
|
||||||
|
logger.error("setVoiceProcessingEnabled(\(useVPIO)) failed: \(error.localizedDescription)")
|
||||||
|
}
|
||||||
|
if useVPIO {
|
||||||
|
// AGC is the one VPIO sub-stage iOS exposes; AEC+NS are bundled into the master switch.
|
||||||
|
engine.inputNode.isVoiceProcessingAGCEnabled = IOSAudioRouter.shared.agcEnabled
|
||||||
|
}
|
||||||
|
|
||||||
|
// (Re)build the playback source node AFTER the VPIO state is set, so it connects against the
|
||||||
|
// correct (voice-processed or plain) output unit — mirrors the proven original ordering.
|
||||||
|
rebuildSourceNode()
|
||||||
|
if micActive { installMicTap() }
|
||||||
|
|
||||||
|
engine.prepare()
|
||||||
|
do {
|
||||||
|
try engine.start()
|
||||||
|
let inFmt = engine.inputNode.outputFormat(forBus: 0)
|
||||||
|
let outFmt = engine.outputNode.outputFormat(forBus: 0)
|
||||||
|
let route = AVAudioSession.sharedInstance().currentRoute.outputs
|
||||||
|
.map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ")
|
||||||
|
logger.info("""
|
||||||
|
engine started — mic=\(self.micActive) vpio=\(useVPIO) captureCh=\(self.captureChannels) \
|
||||||
|
inFormat=\(inFmt) outputNode=\(outFmt) outputRoute=[\(route)]
|
||||||
|
""")
|
||||||
|
} catch {
|
||||||
|
logger.error("engine start failed: \(error.localizedDescription)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Detach any previous source node and attach a fresh one pulling mixed PCM from the ring.
|
||||||
|
/// Rebuilt on every graph rebuild so it always connects against the current output unit (the
|
||||||
|
/// VPIO state can change the output between rebuilds). Its format is route-independent —
|
||||||
|
/// `mainMixerNode` adapts 48 kHz stereo to whatever the output route is.
|
||||||
|
private func rebuildSourceNode() {
|
||||||
|
if let old = sourceNode {
|
||||||
|
engine.detach(old)
|
||||||
|
sourceNode = nil
|
||||||
|
}
|
||||||
let ring = self.ring
|
let ring = self.ring
|
||||||
let scratch = self.renderScratch
|
let scratch = self.renderScratch
|
||||||
let scratchFrames = self.renderScratchFrames
|
let scratchFrames = self.renderScratchFrames
|
||||||
@@ -156,17 +271,11 @@ final class IOSVoiceProcessingEngine {
|
|||||||
let abl = UnsafeMutableAudioBufferListPointer(ablPtr)
|
let abl = UnsafeMutableAudioBufferListPointer(ablPtr)
|
||||||
let n = min(frames, scratchFrames)
|
let n = min(frames, scratchFrames)
|
||||||
let got = ring.read(into: scratch, count: n * 2) / 2 // interleaved stereo → frames
|
let got = ring.read(into: scratch, count: n * 2) / 2 // interleaved stereo → frames
|
||||||
// Deinterleave int16 → Float32 per channel; silence-fill any underrun tail.
|
|
||||||
let scale: Float = 1.0 / 32768.0
|
let scale: Float = 1.0 / 32768.0
|
||||||
for ch in 0..<abl.count {
|
for ch in 0..<abl.count {
|
||||||
guard let base = abl[ch].mData?.assumingMemoryBound(to: Float.self) else { continue }
|
guard let base = abl[ch].mData?.assumingMemoryBound(to: Float.self) else { continue }
|
||||||
for i in 0..<frames {
|
for i in 0..<frames {
|
||||||
if i < got {
|
base[i] = i < got ? Float(scratch[i * 2 + min(ch, 1)]) * scale : 0
|
||||||
let s = scratch[i * 2 + min(ch, 1)]
|
|
||||||
base[i] = Float(s) * scale
|
|
||||||
} else {
|
|
||||||
base[i] = 0
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return noErr
|
return noErr
|
||||||
@@ -174,29 +283,33 @@ final class IOSVoiceProcessingEngine {
|
|||||||
sourceNode = src
|
sourceNode = src
|
||||||
engine.attach(src)
|
engine.attach(src)
|
||||||
engine.connect(src, to: engine.mainMixerNode, format: outFormat)
|
engine.connect(src, to: engine.mainMixerNode, format: outFormat)
|
||||||
|
|
||||||
// ── Mic: tap the VPIO input node, convert to 48 kHz int16, feed the core. ──
|
|
||||||
let inFormat = engine.inputNode.outputFormat(forBus: 0)
|
|
||||||
let targetCh = max(1, min(2, captureChannels))
|
|
||||||
let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000,
|
|
||||||
channels: AVAudioChannelCount(targetCh), interleaved: true)
|
|
||||||
micTargetFormat = target
|
|
||||||
micConverter = (target != nil && inFormat.sampleRate > 0)
|
|
||||||
? AVAudioConverter(from: inFormat, to: target!) : nil
|
|
||||||
if micConverter == nil {
|
|
||||||
logger.error("mic converter unavailable (in=\(inFormat)) — mic will not transmit")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
let c = client
|
/// Install the mic tap: convert the input node's native format to 48 kHz int16 (mono or
|
||||||
|
/// stereo per `captureChannels`) and feed it to the core. Rebuilds the converter each time
|
||||||
|
/// because the input format depends on the VPIO state and the active route.
|
||||||
|
private func installMicTap() {
|
||||||
|
guard let client else { return }
|
||||||
|
let inFormat = engine.inputNode.outputFormat(forBus: 0)
|
||||||
|
guard inFormat.sampleRate > 0 else {
|
||||||
|
logger.error("input format unavailable (\(inFormat)) — mic will not transmit")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
let targetCh = max(1, min(2, captureChannels))
|
||||||
|
guard let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000,
|
||||||
|
channels: AVAudioChannelCount(targetCh), interleaved: true),
|
||||||
|
let converter = AVAudioConverter(from: inFormat, to: target) else {
|
||||||
|
logger.error("mic converter unavailable (in=\(inFormat), ch=\(targetCh)) — mic will not transmit")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
let sid = micStreamId
|
let sid = micStreamId
|
||||||
let converter = micConverter
|
let c = client
|
||||||
let tgt = micTargetFormat
|
|
||||||
engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in
|
engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in
|
||||||
guard let converter, let tgt else { return }
|
|
||||||
// Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample.
|
// Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample.
|
||||||
let ratio = tgt.sampleRate / buffer.format.sampleRate
|
let ratio = target.sampleRate / buffer.format.sampleRate
|
||||||
let outCap = AVAudioFrameCount(Double(buffer.frameLength) * ratio + 16)
|
let outCap = AVAudioFrameCount(Double(buffer.frameLength) * ratio + 16)
|
||||||
guard let outBuf = AVAudioPCMBuffer(pcmFormat: tgt, frameCapacity: outCap) else { return }
|
guard let outBuf = AVAudioPCMBuffer(pcmFormat: target, frameCapacity: outCap) else { return }
|
||||||
var fed = false
|
var fed = false
|
||||||
let status = converter.convert(to: outBuf, error: nil) { _, outStatus in
|
let status = converter.convert(to: outBuf, error: nil) { _, outStatus in
|
||||||
if fed { outStatus.pointee = .noDataNow; return nil }
|
if fed { outStatus.pointee = .noDataNow; return nil }
|
||||||
@@ -206,64 +319,9 @@ final class IOSVoiceProcessingEngine {
|
|||||||
}
|
}
|
||||||
guard status != .error, outBuf.frameLength > 0,
|
guard status != .error, outBuf.frameLength > 0,
|
||||||
let chData = outBuf.int16ChannelData else { return }
|
let chData = outBuf.int16ChannelData else { return }
|
||||||
let spc = Int(outBuf.frameLength)
|
// int16 interleaved → channelData[0] is the interleaved buffer.
|
||||||
// int16 interleaved → channelData[0] is the interleaved buffer for interleaved formats.
|
c.feedPcm(streamId: sid, pcm: chData[0],
|
||||||
c.feedPcm(streamId: sid, pcm: chData[0], samplesPerChannel: spc, channels: targetCh)
|
samplesPerChannel: Int(outBuf.frameLength), channels: targetCh)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Wire the core's mixed-output sink into the ring (C function pointer, no captures). ──
|
|
||||||
let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque()
|
|
||||||
client.setMixedOutputSink({ user, pcm, spc, ch, _ in
|
|
||||||
guard let user, let pcm else { return }
|
|
||||||
let ring = Unmanaged<PCMRing>.fromOpaque(user).takeUnretainedValue()
|
|
||||||
ring.write(pcm, count: spc * Int(ch))
|
|
||||||
}, user: ringPtr)
|
|
||||||
|
|
||||||
engine.prepare()
|
|
||||||
do {
|
|
||||||
try engine.start()
|
|
||||||
isRunning = true
|
|
||||||
// Diagnostics: capture the negotiated graph formats and the live output route so a
|
|
||||||
// silent-playback report can be triaged (format/rate mismatch vs. routing vs. the
|
|
||||||
// core not delivering PCM — see the ring stats logged in teardown()).
|
|
||||||
let outFmt = engine.outputNode.outputFormat(forBus: 0)
|
|
||||||
let mixFmt = engine.mainMixerNode.outputFormat(forBus: 0)
|
|
||||||
let route = AVAudioSession.sharedInstance().currentRoute.outputs
|
|
||||||
.map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ")
|
|
||||||
logger.info("""
|
|
||||||
VPIO engine started — inFormat=\(inFormat), captureCh=\(targetCh), \
|
|
||||||
outputNode=\(outFmt), mainMixer=\(mixFmt), outputRoute=[\(route)]
|
|
||||||
""")
|
|
||||||
} catch {
|
|
||||||
logger.error("VPIO engine start failed: \(error.localizedDescription)")
|
|
||||||
teardown()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Stop the VPIO engine. The caller is responsible for restoring the core's hardware playback
|
|
||||||
/// afterwards (`client.setExternalPlayback(false)` + `audioRestart()`).
|
|
||||||
func stop() {
|
|
||||||
guard isRunning else { return }
|
|
||||||
teardown()
|
|
||||||
logger.info("VPIO engine stopped")
|
|
||||||
}
|
|
||||||
|
|
||||||
private func teardown() {
|
|
||||||
client?.setMixedOutputSink(nil, user: nil)
|
|
||||||
engine.inputNode.removeTap(onBus: 0)
|
|
||||||
if engine.isRunning { engine.stop() }
|
|
||||||
// Diagnostics (threads now quiesced): how much mixed PCM the core delivered into the ring
|
|
||||||
// vs. how much the render thread consumed. written==0 ⇒ the core never delivered (Bug 1
|
|
||||||
// core/lifecycle path); written>0 with no audible output ⇒ the AVAudioEngine output graph.
|
|
||||||
logger.info("VPIO ring stats — written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples")
|
|
||||||
try? engine.inputNode.setVoiceProcessingEnabled(false)
|
|
||||||
if let src = sourceNode {
|
|
||||||
engine.detach(src)
|
|
||||||
sourceNode = nil
|
|
||||||
}
|
|
||||||
micConverter = nil
|
|
||||||
micTargetFormat = nil
|
|
||||||
ring.reset()
|
|
||||||
isRunning = false
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -59,7 +59,6 @@ final class SessionState {
|
|||||||
self.client = client
|
self.client = client
|
||||||
self.selfUserId = selfUserId
|
self.selfUserId = selfUserId
|
||||||
self.permissions = permissions
|
self.permissions = permissions
|
||||||
AudioSessionManager.shared.client = client
|
|
||||||
refreshChannels()
|
refreshChannels()
|
||||||
refreshUsers()
|
refreshUsers()
|
||||||
syncSelfChannel()
|
syncSelfChannel()
|
||||||
@@ -73,16 +72,10 @@ final class SessionState {
|
|||||||
broadcastPump.onBroadcastStarted = { [weak self] in self?.startScreenShare() }
|
broadcastPump.onBroadcastStarted = { [weak self] in self?.startScreenShare() }
|
||||||
broadcastPump.onBroadcastFinished = { [weak self] in self?.stopScreenShare() }
|
broadcastPump.onBroadcastFinished = { [weak self] in self?.stopScreenShare() }
|
||||||
broadcastPump.start()
|
broadcastPump.start()
|
||||||
// When IOSAudioRouter changes the audio config, restart the voice path if needed so the
|
|
||||||
// native VPIO engine (AEC/NS/AGC) engages or disengages to match the new preset/route.
|
|
||||||
AudioSessionManager.shared.reconcileVoicePath = { [weak self] in self?.reconcileVoicePath() }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
deinit {
|
deinit {
|
||||||
broadcastPump.stop()
|
broadcastPump.stop()
|
||||||
MainActor.assumeIsolated {
|
|
||||||
AudioSessionManager.shared.client = nil
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Event dispatch
|
// MARK: - Event dispatch
|
||||||
@@ -259,78 +252,39 @@ final class SessionState {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// VPIO path: on the AEC presets, the native AVAudioEngine does AEC/NS/AGC and the core
|
// Unified iOS path: the core is always external (set at connect via setExternalPlayback +
|
||||||
// runs in external mode (no hardware mic/playback). The mic stream is started with
|
// every MIC stream external_feed), and `IOSAudioEngine` drives capture + playback. So the
|
||||||
// externalFeed so the core skips the hardware capture device; setExternalPlayback makes
|
// mic stream is just started with external_feed=true and the engine is told the mic is now
|
||||||
// it skip the hardware playback device and deliver the mix to IOSVoiceProcessingEngine.
|
// active — no setExternalPlayback toggle, no audioRestart ordering, no VPIO/miniaudio fork.
|
||||||
//
|
|
||||||
// ORDER MATTERS: set the external-playback flag now, but defer audioRestart() until
|
|
||||||
// AFTER startStream (below) so the MIC LocalStream — which carries external_feed=true —
|
|
||||||
// already exists when ensure_audio_running() derives external_capture. Restarting before
|
|
||||||
// the stream exists makes the core reopen a hardware capture device that is never dropped
|
|
||||||
// (the announce-result restart early-returns because the engine is already running); that
|
|
||||||
// lingering miniaudio capture unit then fights the AVAudioEngine VPIO unit on the same
|
|
||||||
// .voiceChat session and silences VPIO playback.
|
|
||||||
let useVPIO = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
|
|
||||||
client.setExternalPlayback(useVPIO)
|
|
||||||
|
|
||||||
let desc = StreamDescriptor(kind: .mic, deviceId: voiceState.currentDeviceId, label: "Mic",
|
let desc = StreamDescriptor(kind: .mic, deviceId: voiceState.currentDeviceId, label: "Mic",
|
||||||
externalFeed: useVPIO)
|
externalFeed: true)
|
||||||
let (result, streamId) = client.startStream(desc)
|
let (result, streamId) = client.startStream(desc)
|
||||||
if result == .ok {
|
guard result == .ok else {
|
||||||
|
addActivity("Failed to start mic: \(result.description)")
|
||||||
|
return
|
||||||
|
}
|
||||||
voiceState.micActive = true
|
voiceState.micActive = true
|
||||||
voiceState.localStreamId = streamId
|
voiceState.localStreamId = streamId
|
||||||
EventFeedback.shared.play(.voiceOn)
|
EventFeedback.shared.play(.voiceOn)
|
||||||
// Publish the active mic stream ID so IOSAudioRouter can reset the core's capture
|
|
||||||
// channel count when the user switches mono↔stereo (selectCaptureChannels /
|
|
||||||
// applyPreset). Without this, switching stereo→mono leaves the LocalStream's
|
|
||||||
// capture_channels field at 2 and the next engine start still opens stereo.
|
|
||||||
AudioSessionManager.shared.activeMicStreamId = streamId
|
|
||||||
// Store the user's capture channel selection before the server acknowledges
|
|
||||||
// the stream. The engine hasn't started yet at this point (it starts when
|
|
||||||
// handle_stream_announce_result fires), so vc_set_capture_channels just
|
|
||||||
// stores the value — no restart. ensure_audio_running() picks it up when
|
|
||||||
// the stream is confirmed and opens the device with the right channel count.
|
|
||||||
let channels = IOSAudioRouter.shared.captureChannels.channelCount
|
let channels = IOSAudioRouter.shared.captureChannels.channelCount
|
||||||
if channels != 1 {
|
if channels != 1 {
|
||||||
client.setCaptureChannels(streamId: streamId, channels: channels)
|
client.setCaptureChannels(streamId: streamId, channels: channels)
|
||||||
}
|
}
|
||||||
if useVPIO {
|
// Engage the mic: installs the input tap and (per preset) VPIO, in one engine rebuild.
|
||||||
// The external-feed MIC stream now exists, so restart the core into full
|
IOSAudioEngine.shared.startMic(streamId: streamId, channels: channels)
|
||||||
// external mode (no hardware capture/playback, mixer-timer only) — mic and
|
|
||||||
// speaker are owned entirely by the VPIO engine, which we start right after.
|
|
||||||
client.audioRestart()
|
|
||||||
IOSVoiceProcessingEngine.shared.start(
|
|
||||||
client: client, micStreamId: streamId, captureChannels: channels)
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
addActivity("Failed to start mic: \(result.description)")
|
|
||||||
if useVPIO { // revert external-playback mode so remote audio still plays
|
|
||||||
client.setExternalPlayback(false)
|
|
||||||
client.audioRestart()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func stopMicStream() {
|
func stopMicStream() {
|
||||||
// Tear down the VPIO engine first (removes the mic tap + unregisters the mixed sink),
|
// Disengage the mic (removes the tap + VPIO) but keep the engine running for any remaining
|
||||||
// then stop the mic stream, then restore the core's hardware playback for any remaining
|
// remote audio. Then stop the core's MIC stream. The core stays external throughout — no
|
||||||
// remote audio. Order matters: the mic stream must be gone before audioRestart so the
|
// setExternalPlayback toggle, no audioRestart.
|
||||||
// core opens a normal playback device (and no capture device — there's no mic stream).
|
IOSAudioEngine.shared.stopMic()
|
||||||
let wasVPIO = IOSVoiceProcessingEngine.shared.isRunning
|
|
||||||
if wasVPIO {
|
|
||||||
IOSVoiceProcessingEngine.shared.stop()
|
|
||||||
}
|
|
||||||
if voiceState.localStreamId != 0 {
|
if voiceState.localStreamId != 0 {
|
||||||
client.stopStream(voiceState.localStreamId)
|
client.stopStream(voiceState.localStreamId)
|
||||||
voiceState.localStreamId = 0
|
voiceState.localStreamId = 0
|
||||||
AudioSessionManager.shared.activeMicStreamId = nil
|
|
||||||
EventFeedback.shared.play(.voiceOff)
|
EventFeedback.shared.play(.voiceOff)
|
||||||
}
|
}
|
||||||
if wasVPIO {
|
|
||||||
client.setExternalPlayback(false)
|
|
||||||
client.audioRestart() // reopen hardware playback (no mic stream → no hw capture)
|
|
||||||
}
|
|
||||||
voiceState.micActive = false
|
voiceState.micActive = false
|
||||||
voiceState.level = 0
|
voiceState.level = 0
|
||||||
// Do NOT deactivate the AVAudioSession here — the user may still want to hear
|
// Do NOT deactivate the AVAudioSession here — the user may still want to hear
|
||||||
@@ -338,19 +292,6 @@ final class SessionState {
|
|||||||
// disconnecting from the server (see AppState.disconnect / .disconnected event).
|
// disconnecting from the server (see AppState.disconnect / .disconnected event).
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Restart the voice path when the audio config changes mid-call (driven by IOSAudioRouter).
|
|
||||||
/// If VPIO is involved on either the current or desired side, restart the mic so the native
|
|
||||||
/// voice-processing engine engages/disengages and re-binds to the new route. Pure miniaudio
|
|
||||||
/// config tweaks need no restart — the core's own audioRestart (already issued) handles them.
|
|
||||||
private func reconcileVoicePath() {
|
|
||||||
guard voiceState.micActive else { return }
|
|
||||||
let want = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
|
|
||||||
let have = IOSVoiceProcessingEngine.shared.isRunning
|
|
||||||
guard want || have else { return }
|
|
||||||
stopMicStream()
|
|
||||||
doStartMicStream()
|
|
||||||
}
|
|
||||||
|
|
||||||
// MARK: - Screen audio share
|
// MARK: - Screen audio share
|
||||||
|
|
||||||
/// Called when the broadcast extension becomes active. Announces the SCREEN_AUDIO stream;
|
/// Called when the broadcast extension becomes active. Announces the SCREEN_AUDIO stream;
|
||||||
|
|||||||
@@ -38,9 +38,9 @@ struct SettingsView: View {
|
|||||||
.accessibilityLabel("Speaker output")
|
.accessibilityLabel("Speaker output")
|
||||||
.accessibilityHint("Routes audio to the speaker instead of the earpiece when no headphones are connected.")
|
.accessibilityHint("Routes audio to the speaker instead of the earpiece when no headphones are connected.")
|
||||||
|
|
||||||
// Surface the voice-processing state. On the AEC presets the native iOS
|
// Surface the voice-processing state. On Voice Chat the native iOS
|
||||||
// Voice-Processing unit (VPIO) does echo cancellation, noise suppression and
|
// Voice-Processing unit (VPIO) does echo cancellation, noise suppression and
|
||||||
// automatic gain control; the other presets (stereo/studio/A2DP) can't use it.
|
// automatic gain control; the stereo / mono-mic / A2DP configs can't use it.
|
||||||
if router.currentConfigUsesVoiceProcessing {
|
if router.currentConfigUsesVoiceProcessing {
|
||||||
Label("Echo cancellation & noise suppression on (iOS voice processing)",
|
Label("Echo cancellation & noise suppression on (iOS voice processing)",
|
||||||
systemImage: "waveform.badge.mic")
|
systemImage: "waveform.badge.mic")
|
||||||
@@ -48,18 +48,11 @@ struct SettingsView: View {
|
|||||||
.foregroundStyle(.secondary)
|
.foregroundStyle(.secondary)
|
||||||
.accessibilityLabel("Echo cancellation and noise suppression are on")
|
.accessibilityLabel("Echo cancellation and noise suppression are on")
|
||||||
} else {
|
} else {
|
||||||
Label("No echo cancellation in this preset (stereo / studio / A2DP)",
|
Label("No echo cancellation in this configuration (stereo / A2DP / off)",
|
||||||
systemImage: "waveform.slash")
|
systemImage: "waveform.slash")
|
||||||
.font(.caption)
|
.font(.caption)
|
||||||
.foregroundStyle(.secondary)
|
.foregroundStyle(.secondary)
|
||||||
.accessibilityLabel("Echo cancellation is off in this preset")
|
.accessibilityLabel("Echo cancellation is off in this configuration")
|
||||||
}
|
|
||||||
|
|
||||||
if !router.hasBluetoothDevice && !router.hasWiredHeadset {
|
|
||||||
Text("Connect Bluetooth headphones or a wired headset for more presets.")
|
|
||||||
.font(.caption)
|
|
||||||
.foregroundStyle(.secondary)
|
|
||||||
.accessibilityLabel("No external audio device connected")
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -127,6 +120,27 @@ struct SettingsView: View {
|
|||||||
}
|
}
|
||||||
.accessibilityLabel("Microphone processing mode")
|
.accessibilityLabel("Microphone processing mode")
|
||||||
|
|
||||||
|
// Voice-processing (VPIO) controls — only meaningful on a VPIO-capable
|
||||||
|
// config (mono + standard + non-A2DP). iOS bundles echo cancellation and
|
||||||
|
// noise suppression into one master switch (no per-stage toggle); AGC is
|
||||||
|
// the one sub-stage it lets us control independently.
|
||||||
|
if router.voiceProcessingAvailable {
|
||||||
|
Toggle("Voice Processing (AEC + noise suppression)", isOn: Binding(
|
||||||
|
get: { router.voiceProcessingEnabled },
|
||||||
|
set: { router.setVoiceProcessingEnabled($0) }
|
||||||
|
))
|
||||||
|
.accessibilityLabel("Voice processing")
|
||||||
|
.accessibilityHint("Echo cancellation and noise suppression, bundled together by iOS.")
|
||||||
|
|
||||||
|
if router.voiceProcessingEnabled {
|
||||||
|
Toggle("Automatic Gain Control", isOn: Binding(
|
||||||
|
get: { router.agcEnabled },
|
||||||
|
set: { router.setAgcEnabled($0) }
|
||||||
|
))
|
||||||
|
.accessibilityLabel("Automatic gain control")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if router.showsRawModeSpeakerWarning {
|
if router.showsRawModeSpeakerWarning {
|
||||||
Label(
|
Label(
|
||||||
"Raw mode on speaker — echo risk (no AEC)",
|
"Raw mode on speaker — echo risk (no AEC)",
|
||||||
|
|||||||
@@ -230,37 +230,46 @@ Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth
|
|||||||
in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
|
in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
|
||||||
downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
|
downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
|
||||||
genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
|
genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
|
||||||
- **iOS mic capture:** all iOS audio routing is driven from Swift via `AVAudioSession` by the
|
- **iOS audio — one path, always external.** On iOS the core **never opens a miniaudio device**:
|
||||||
`IOSAudioRouter` singleton *before* the core (miniaudio) opens its device — miniaudio does
|
a single `AVAudioEngine` (`IOSAudioEngine`) drives *both* directions, and the core runs fully
|
||||||
NOT touch `AVAudioSession` on iOS. Input port selection (`availableInputs`), built-in mic
|
external for the whole connection. This is the single most important property of the iOS audio
|
||||||
orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns
|
stack — there is no second (miniaudio) path to switch to, so a preset/route change cannot leave
|
||||||
(`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic processing mode
|
one direction dropped. The single ordering rule is: `vc_set_external_playback(1)` is set **once
|
||||||
(`.voiceChat` = Standard with AEC/AGC/HPF, or `.measurement` = Raw/Studio with all processing
|
at connect** (before the session is activated or any remote stream arrives), and every MIC
|
||||||
off), Bluetooth mode (`.allowBluetoothHFP` HFP voice vs `.allowBluetoothA2DP` stereo output
|
stream is started with `vc_stream_desc.external_feed=1`.
|
||||||
vs neither), and stereo capture (`.stereo` polar pattern + `setPreferredInput` +
|
- **core → speaker:** the core's mixer-timer thread decodes+mixes on a ~20 ms cadence and
|
||||||
`setInputDataSource` → `vc_set_capture_channels`) are all set from Swift. The core then
|
delivers the FINAL mixed PCM via `vc_set_mixed_output_sink`; an `AVAudioSourceNode` pulls it
|
||||||
opens whatever route AVAudioSession has established. When the user changes audio settings
|
from a lock-free ring and renders it. This runs the whole time we are connected, so remote
|
||||||
mid-session, `IOSAudioRouter` suspends the core's devices (`vc_audio_suspend`), reconfigures
|
audio plays even before the user joins voice (kills the "can't hear anyone" race).
|
||||||
`AVAudioSession`, then restarts the devices (`vc_audio_restart`) so they reopen against the
|
- **mic → core:** when the mic is active a tap on the engine's input node converts to 48 kHz
|
||||||
new route — mirroring TeamTalk5's `closeSoundDevices`/`initSoundInputDevice`/
|
int16 (`vc_set_capture_channels` decides mono/stereo) and calls `vc_stream_feed_pcm`.
|
||||||
`initSoundOutputDevice` pattern.
|
- **iOS routing** is still driven from Swift via `AVAudioSession` by the `IOSAudioRouter`
|
||||||
- **iOS voice processing (AEC/NS/AGC) — native VPIO path.** Real iOS echo cancellation, noise
|
singleton — miniaudio never touches `AVAudioSession` on iOS. Input port selection
|
||||||
suppression and AGC are provided ONLY by Apple's **Voice-Processing I/O audio unit (VPIO)**,
|
(`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom),
|
||||||
*not* by the `AVAudioSession` mode alone. The core uses miniaudio's plain `RemoteIO` audio
|
polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic
|
||||||
units, which never engage VPIO — so `.voiceChat` mode by itself yields no AEC. For VPIO to
|
processing mode (Standard vs `.measurement` Raw), Bluetooth mode (`.allowBluetoothHFP` HFP voice
|
||||||
cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
|
vs `.allowBluetoothA2DP` stereo output vs neither), and stereo capture (`.stereo` polar pattern
|
||||||
played-back signal from the mic), so on the AEC presets (Voice Chat / Bluetooth Headset HFP /
|
+ `setPreferredInput` + `setInputDataSource` → `vc_set_capture_channels`) are all set from
|
||||||
Wired Headset) the Swift layer runs a native `AVAudioEngine` with
|
Swift. Any preset / route / interruption change funnels through one deterministic, Swift-only
|
||||||
`inputNode.setVoiceProcessingEnabled(true)` and the core runs in **external mode**:
|
rebuild: `IOSAudioEngine` stops, `IOSAudioRouter.applyConfiguration()` re-applies the
|
||||||
- **Mic:** the MIC stream is started with `vc_stream_desc.external_feed=1`; the VPIO input tap
|
`AVAudioSession`, the graph is rebuilt against the new route, and the engine restarts. No
|
||||||
feeds processed mic PCM via `vc_stream_feed_pcm`. The core skips its hardware capture device
|
`vc_audio_restart`/`vc_audio_suspend` dance is needed for routing (the core has no hardware
|
||||||
(`AudioParams.external_capture`).
|
devices to reopen) — this is the spirit of TeamTalk5's "close then re-init sound devices", but
|
||||||
- **Playback:** `vc_set_external_playback(1)` makes the core skip its hardware playback device;
|
entirely inside the Swift engine.
|
||||||
a mixer-timer thread drives decode+mix on a ~20 ms cadence and delivers the FINAL mixed PCM
|
- **iOS voice processing (AEC/NS/AGC) — native VPIO.** Real iOS echo cancellation, noise
|
||||||
via `vc_set_mixed_output_sink`. The Swift engine renders that through the VPIO output, so
|
suppression and AGC come ONLY from Apple's **Voice-Processing I/O audio unit (VPIO)**, which
|
||||||
VPIO has its echo-cancellation reference signal.
|
`inputNode.setVoiceProcessingEnabled(true)` enables; for it to cancel echo it must own BOTH the
|
||||||
The Stereo Mic / Studio / A2DP presets keep the miniaudio path (they want raw / stereo /
|
mic capture and the playback — which the unified engine already does. VPIO forces **mono**, so
|
||||||
no-AEC routing that VPIO can't provide — VPIO forces mono).
|
it is engaged only when the active config wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`:
|
||||||
|
mono + standard + non-A2DP + the user's master toggle). iOS exposes no per-stage VPIO control,
|
||||||
|
so the Advanced UI offers exactly two switches: a master **Voice Processing** (AEC + NS bundled)
|
||||||
|
and **AGC** (`isVoiceProcessingAGCEnabled`).
|
||||||
|
- **iOS presets** (`IOSAudioRouter.AudioPreset`): **Voice Chat** (VPIO mono, system output incl.
|
||||||
|
HFP/wired), **Stereo Mic** (internal stereo built-in mic regardless of output, A2DP-capable, no
|
||||||
|
VPIO), **Mono Mic** (internal mono built-in mic regardless of output, A2DP-capable, no VPIO),
|
||||||
|
and **Advanced** (every knob manual). A2DP output requires an internal-mic preset (the Bluetooth
|
||||||
|
device is output-only); the Stereo/Mono Mic presets fall back to the built-in speaker when no
|
||||||
|
external output is connected (`applyA2dpSpeakerFallback`).
|
||||||
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
|
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
|
||||||
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
|
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
|
||||||
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
|
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
|
||||||
|
|||||||
Reference in New Issue
Block a user