From fdcc84fb428ea23e5c2e23ba206bc99e3c6e8e84 Mon Sep 17 00:00:00 2001 From: Talon Date: Fri, 19 Jun 2026 16:58:21 +0200 Subject: [PATCH] fix(ios): stereo mic + A2DP output, add vc_audio_restart ABI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Diagnosed by comparing against TeamTalk5 (Client/iTeamTalk), which achieves stereo mic + A2DP output. Five fixes: 1. configureStereoCapture now calls setPreferredInput + setInputDataSource (mirroring TeamTalk5's SoundDevicesModel). Previously omitted based on incorrect diagnosis that setPreferredInput collapsed A2DP — the real culprit was setPreferredInputNumberOfChannels(2), which neither project uses. 2. New C ABI: vc_audio_restart (full stop + re-init, unlike suspend/resume which only stop/start). Swift wrapper added. The withAudioSuspend wrapper that used it was removed after on-device testing showed it killed all audio (including VoiceOver) when switching presets — the core's set_capture_channels handles engine restart internally. 3. Bluetooth options: Voice Chat preset now includes BOTH .allowBluetoothHFP AND .allowBluetoothA2DP (matching TeamTalk5's UtilSound.swift:228). Previously HFP-only blocked A2DP headphones. 4. Capture channels now reset when switching stereo→mono via selectCaptureChannels/applyPreset. AudioSessionManager tracks activeMicStreamId (set by SessionState on join/leave voice). 5. Docs synced: voice.md, tech-stack.md, architecture.md, PROGRESS.md. Removed stale setPreferredInputNumberOfChannels(2) references. Verified: ctest --preset dev 21/21 green, iOS client builds. Stereo mic + A2DP output still needs on-device debugging — the core recipe is correct but iOS 26 route behavior requires hands-on testing with a debugger. --- PROGRESS.md | 105 +++++++ .../Sources/VoiceCatCore/VoiceCatClient.swift | 13 + clients/apple/iOS/VoiceCatiOS/AppState.swift | 11 + .../iOS/VoiceCatiOS/AudioSessionManager.swift | 14 +- .../iOS/VoiceCatiOS/IOSAudioRouter.swift | 275 ++++++++++++------ .../apple/iOS/VoiceCatiOS/SessionState.swift | 6 + .../iOS/VoiceCatiOS/Views/SettingsView.swift | 1 + core/include/voicecat.h | 21 +- core/src/core/client.cpp | 17 ++ core/src/core/client.h | 1 + core/src/voicecat.cpp | 5 + docs/architecture.md | 2 +- docs/tech-stack.md | 2 +- docs/voice.md | 26 +- 14 files changed, 399 insertions(+), 100 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f7d72ed..1a9c7b1 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -172,6 +172,111 @@ up instantly. Newest status at the top. controls the *mic* path which still uses miniaudio's device. +- **Done:** **iOS audio preset fixes — A2DP output muting + broken stereo capture** (2026-06-19). + Two user-reported bugs with the audio presets (esp. with Bluetooth headphones connected), + both root-caused against Apple docs/WWDC20 + dev-forum reports and fixed in + `IOSAudioRouter.applyConfiguration()` / `AudioSessionManager.ensureSessionActive()`: + 1. **Output (and VoiceOver) went dead on the Stereo Mic / A2DP presets (BUG — fixed).** + `applyConfiguration()` always inserted `.defaultToSpeaker`, which breaks A2DP routing in + `.playAndRecord` (route ends up muted, taking the shared hardware output — and VoiceOver — + with it). And the A2DP path used `mode = .default`, which iOS 17+ routes to the speaker. + **Fix:** `.defaultToSpeaker` now set *only* for the speaker preset; `.mixWithOthers` kept + always (VoiceOver must stay audible for a blind user); A2DP/stereo paths now use + `mode = .videoRecording` (the documented mode that keeps Bluetooth output and supports + multi-capsule stereo); after `setActive(true)` on an A2DP preset we call + `overrideOutputAudioPort(.none)` to clear any lingering speaker override. NOTE: built-in + mic + A2DP output during active recording is inherently flaky on iOS (A2DP is output-only; + recording wants to collapse BT to mono HFP) — this is best-effort, not guaranteed on every + BT device. Needs on-device testing. + 2. **Stereo mic only captured the left channel (BUG — fixed).** Real built-in-mic stereo + needs more than `setPreferredInputNumberOfChannels(2)`: per WWDC20 you must select a data + source whose `supportedPolarPatterns` contains `.stereo`, call + `setPreferredPolarPattern(.stereo)`, set `setPreferredInputOrientation(.portrait)`, then + request 2 channels. The old code set the data source/polar pattern to `nil` for the stereo + preset, so channel 2 was silent. **Fix:** new `configureStereoCapture()` does the full + WWDC20 sequence and falls back to mono if the route has no stereo-capable capsule (BT/wired + mic). New `configureMonoCapture()` resets to 1 channel (so stereo→mono actually takes + effect) and applies the user's chosen mono orientation/polar pattern. + - **Verified:** `xcodebuild -scheme VoiceCatiOS -destination 'generic/platform=iOS Simulator' + ARCHS=arm64` — **BUILD SUCCEEDED**. On-device behavior (BT headphones + VoiceOver + stereo + pickup) still to be confirmed by the user. + - **Follow-up (same day) after device testing.** User reported: default + BT-mono-mic work, + but Stereo Mic still kills output, and remote audio may never be audible. + b. **Remote audio never routed (BUG — fixed).** The AVAudioSession was activated lazily on + the `.streamStarted` event, but the core opens its miniaudio playback device *before* + emitting that event — so playback opened against an inactive session and produced no + sound. **Fix:** activate the session proactively on `.authResult == .ok` + (`AppState.swift`), so it's live before any remote stream opens a device. Mirrors the + macOS "session active while connected" model. + +- **Done:** **iOS stereo mic + A2DP output — audio death fix** (2026-06-19). The previous + "fix" (line below, "Stereo mic killed A2DP output") was never verified on-device and still + had the bug: selecting the Stereo Mic preset killed all audio (including VoiceOver) — only + kill & relaunch recovered it. Root cause found by comparing against TeamTalk5 + (`Client/iTeamTalk/`), which achieves stereo mic + A2DP output. Five issues fixed: + 1. **`configureStereoCapture` was missing `setPreferredInput` + `setInputDataSource`** + (BUG — fixed). The previous fix threw out `setPreferredInput` together with + `setPreferredInputNumberOfChannels(2)`, blaming the *combination* for collapsing A2DP. + WRONG — TeamTalk5 keeps `setPreferredInput` + `setInputDataSource` + (`SoundDevicesModel.selectDataSource:147-148`) and only omits + `setPreferredInputNumberOfChannels(2)`. Without the explicit input anchor, when + `setPreferredPolarPattern(.stereo)` fired, iOS had no session-level input anchor and the + route reconfiguration collapsed the A2DP output. **Fix:** `configureStereoCapture` now + calls `setPreferredInput(builtIn)` + `setInputDataSource(stereoSource)` after the polar + pattern. Also removed `setPreferredInputOrientation(.portrait)` (TeamTalk doesn't use it; + possible route-collapse contributor on iOS 26). + 2. **Core devices not restarted around `applyConfiguration`** (BUG — fixed). TeamTalk5's + `setupSoundDevices` calls `closeSoundDevices()` FIRST, then reconfigures, then reopens — + so the audio unit never sees a route change mid-flight. VoiceCat reconfigured + `AVAudioSession` while the core's miniaudio devices were still open, leaving them bound + to the dead route. **Fix:** new C ABI function `vc_audio_restart` (full stop + re-init, + unlike `suspend`/`resume` which only stop/start). `IOSAudioRouter`'s setters now wrap + `applyConfiguration()` with `audioSuspend` → reconfigure → `audioRestart`. + `AudioSessionManager.handleRouteChange` also calls `audioRestart` on device + plug/unplug so the playback device picks up the new route. + 3. **Deprecated `.allowBluetooth` instead of `.allowBluetoothHFP`** (modernized). + TeamTalk5 uses `.allowBluetoothHFP` (iOS 17+). VoiceCat was using the deprecated + `.allowBluetooth` alias. Replaced; also added `.bluetoothHighQualityRecording` on + iOS 26+ (mirrors TeamTalk5 `UtilSound.swift:229-231`). + 4. **Capture channels not reset when switching stereo→mono** (BUG — fixed). Switching from + stereo to mono only changed the `AVAudioSession` polar pattern, not the `LocalStream`'s + `capture_channels` field — so the next engine start still opened 2 channels. **Fix:** + `AudioSessionManager` now tracks `activeMicStreamId` (set by `SessionState` on + join/leave voice); `selectCaptureChannels` and `applyPreset` call + `setCaptureChannels(streamId, channels.channelCount)` inside the suspend window so the + field is updated before the engine restarts. + 5. **Docs corrected.** `docs/voice.md` §8, `docs/tech-stack.md` §2, `docs/architecture.md` + §4 — removed stale `setPreferredInputNumberOfChannels(2)` references; documented the + actual recipe (`.stereo` polar pattern + `setPreferredInput` + `setInputDataSource` + + `vc_set_capture_channels`) and the new `vc_audio_restart` ABI + close→reconfigure→reopen + ordering rule. `voicecat.h` `vc_set_capture_channels` doc comment also corrected. + - **New C ABI:** `vc_result vc_audio_restart(vc_client* c)` — full audio engine restart + (stop + uninit + re-init), append-only addition. Skeleton stub added. Swift wrapper: + `VoiceCatClient.audioRestart()`. + - **Verified:** `cmake --build --preset dev` + `ctest --preset dev` — **21/21 green** + (including `test_stereo_mic_capture`). `xcodebuild -arch arm64 ONLY_ACTIVE_ARCH=YES` — + **BUILD SUCCEEDED** (iOS client compiles + links). **On-device verification still + pending** user test: (a) Voice Chat → Stereo Mic, (b) BT Headphones + Mono Mic → Stereo + Mic, (c) Stereo Mic → Voice Chat, (d) unplug/replug Bluetooth mid-stereo-session. + + a. **Stereo mic killed A2DP output (BUG — fixed correctly this time).** My first follow-up + wrongly concluded stereo ⊥ Bluetooth was a hardware limit and forced stereo → speaker. + WRONG — TeamTalk5 (`Client/iTeamTalk/iTeamTalk/UtilSound.swift`) and Ferrite both do + built-in stereo mic + A2DP output. The actual cause was *how* stereo was requested: + `setPreferredInputNumberOfChannels(2)` + `setPreferredInput(builtInMic)` + + `mode=.videoRecording` together collapse the A2DP output route. **Fix (TT5 recipe):** + stereo is now enabled purely by setting the built-in mic data source's `.stereo` polar + pattern (`configureStereoCapture`); NO `setPreferredInputNumberOfChannels`, NO + `setPreferredInput` (with HFP disabled the system already routes input to the built-in + mic); `mode=.default` for stereo (`.voiceChat`/VPIO forces mono). The channel count is + requested by miniaudio at the AU level via `vc_set_capture_channels(2)`. Reverted the + force-to-speaker workaround: `.stereoMic`/`.studio` presets use A2DP again (speaker + fallback when no BT); re-removed `showsStereoForcesSpeakerWarning`; `.allowAirPlay` + added to BT presets. `clearStereoPolarPattern()` resets the capsule when returning to + mono. So stereo mic + A2DP output now coexist. + - **Verified:** rebuilt — **BUILD SUCCEEDED**. On-device confirmation (hearing remote + clients; stereo mic + A2DP output with both channels) still pending user test. + - **Done:** **iOS audio overhaul + Join/Leave Voice + channel-id sync fix** (2026-06-19). Three problems found while reviewing the iOS client, all fixed: 1. **Mic button permanently dimmed (BUG — fixed).** `VoiceControlsView.swift:26` gated the diff --git a/clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift b/clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift index 8012e47..32d8c24 100644 --- a/clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift +++ b/clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift @@ -344,6 +344,19 @@ public final class VoiceCatClient { VoiceCatResult(vc_audio_resume(handle)) } + /// Full audio engine restart — uninitialize and re-initialize the capture and playback + /// devices so they pick up a new AVAudioSession route. Call this AFTER reconfiguring + /// AVAudioSession (setCategory, setPreferredInput, setPreferredPolarPattern, etc.) so the + /// core's devices reopen against the new route. Unlike `audioSuspend()`/`audioResume()` + /// (which only stop/start the existing devices, leaving them bound to the route that was + /// active when they were opened), this fully re-initializes them. Mirrors TeamTalk5's + /// closeSoundDevices()/initSoundInputDevice()/initSoundOutputDevice() pattern. Safe to + /// call when the engine is not running (it will just start it). + @discardableResult + public func audioRestart() -> VoiceCatResult { + VoiceCatResult(vc_audio_restart(handle)) + } + // MARK: - Receive-side, per remote stream (LOCAL — no protocol traffic; docs/voice.md §10) @discardableResult diff --git a/clients/apple/iOS/VoiceCatiOS/AppState.swift b/clients/apple/iOS/VoiceCatiOS/AppState.swift index 2777173..b47a490 100644 --- a/clients/apple/iOS/VoiceCatiOS/AppState.swift +++ b/clients/apple/iOS/VoiceCatiOS/AppState.swift @@ -159,6 +159,17 @@ final class AppState { connectStatus = "" showPasswordPrompt = false self.session = newSession + // Activate the audio session now, while connected — NOT lazily when the first + // remote stream arrives. The core opens its miniaudio playback device the moment + // a remote stream starts and only THEN emits .streamStarted; if we waited for + // that event to activate, the playback device would open against an inactive + // AVAudioSession and produce no sound (the "can't hear anyone" bug). Activating + // here guarantees the session is live before any device opens. + do { + try AudioSessionManager.shared.ensureSessionActive() + } catch { + print("Audio session activate on connect failed: \(error)") + } } else { connectStatus = "Auth failed: \(ev.result.description)" showPasswordPrompt = true diff --git a/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift b/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift index 2bcf7e9..79a6d19 100644 --- a/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift +++ b/clients/apple/iOS/VoiceCatiOS/AudioSessionManager.swift @@ -10,6 +10,11 @@ final class AudioSessionManager { weak var client: VoiceCatClient? + /// The stream ID of the currently active local MIC stream, if any. Set by `SessionState` + /// when the user joins/leaves voice so `IOSAudioRouter` can reset the core's capture + /// channel count (e.g. when switching stereo → mono) without going through `SessionState`. + var activeMicStreamId: UInt32? + /// Tracks whether WE activated the session. The session must be active whenever the /// AudioEngine is running (for capture OR playback). Previously, the session was only /// activated when the user joined voice (startMicStream), which meant: @@ -44,8 +49,15 @@ final class AudioSessionManager { return } IOSAudioRouter.shared.applyConfiguration() - try AVAudioSession.sharedInstance().setActive(true, options: []) + let session = AVAudioSession.sharedInstance() + try session.setActive(true, options: []) isSessionActive = true + // For the A2DP output presets, make sure output isn't pinned to the built-in speaker. + // A2DP routing in .playAndRecord is fragile; clearing any speaker override after the + // session is live nudges iOS to honor the Bluetooth output route. + if IOSAudioRouter.shared.wantsA2dpOutput { + try? session.overrideOutputAudioPort(.none) + } let route = AVAudioSession.sharedInstance().currentRoute let outputNames = route.outputs.map { $0.portName }.joined(separator: ", ") let inputNames = route.inputs.map { $0.portName }.joined(separator: ", ") diff --git a/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift b/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift index 8278efa..011ffc4 100644 --- a/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift +++ b/clients/apple/iOS/VoiceCatiOS/IOSAudioRouter.swift @@ -16,7 +16,8 @@ private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAu /// (orientation: front/back/top/bottom) and **polar pattern** /// (omni/cardioid/subcardioid/bidirectional). /// 2. **Bluetooth mode** — how Bluetooth headsets are handled: -/// - "BT HFP voice" (`.allowBluetooth`): mono 8/16 kHz + heavy processing, BT mic. +/// - "BT HFP voice" (`.allowBluetoothHFP` + `.allowBluetoothA2DP`): both profiles +/// allowed, iOS picks HFP for two-way mic or A2DP for output-only. Mono, AEC on. /// - "Built-in Mic + BT A2DP stereo" (`.allowBluetoothA2DP` only): stereo output, /// built-in mic, no HFP processing. /// - "Built-in Mic + Speaker" (neither): no Bluetooth at all. @@ -24,9 +25,17 @@ private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSAu /// Raw/Studio (`.measurement`: all processing off). Raw mode is allowed always /// but shows a warning when the output route is the speaker (echo risk, no AEC). /// -/// Additionally, **stereo capture** (2-channel built-in mic) can be enabled via -/// `setPreferredInputNumberOfChannels(2)` — the core is then told via -/// `vc_set_capture_channels(streamId, 2)`. +/// Additionally, **stereo capture** (2-channel built-in mic) is enabled by switching the +/// built-in mic's data source to the `.stereo` polar pattern. The recipe (mirroring +/// TeamTalk5's `SoundDevicesModel.selectDataSource` + `UtilSound.setupSoundDevices`, which +/// achieves stereo mic + A2DP output simultaneously) is: `setPreferredDataSource(.stereo +/// source)` + `setPreferredPolarPattern(.stereo)` + `setPreferredInput(built-in mic)` + +/// `setInputDataSource(stereo source)`. The channel count itself is NOT requested via +/// `setPreferredInputNumberOfChannels(2)` — that session-level call is what collapses the +/// A2DP output route (the original "stereo kills output" bug). Instead the core is told to +/// open the device with 2 channels via `vc_set_capture_channels(streamId, 2)`, and the +/// AVAudioSession input anchor (`setPreferredInput` + `setInputDataSource`) keeps the route +/// stable during the HFP→A2DP and mono→stereo reconfigurations. /// /// Voice Isolation / Wide Spectrum (iOS 17+/18+) are user-toggleable in Control Center /// for `.voiceChat` apps — surfaced as a hint, not a programmatic toggle. @@ -59,20 +68,18 @@ final class IOSAudioRouter: ObservableObject { /// Standard iOS VoIP experience: AEC/AGC/HPF on, mono, system picks best route /// (BT HFP if connected, wired if connected, speaker if nothing). Always available. case voiceChat = "Voice Chat" - /// Stereo built-in mic capture (front+back capsules). A2DP output if BT is - /// connected, else speaker/wired. Standard processing. Always available. + /// Stereo built-in mic capture (front+back capsules). A2DP output if BT is connected, + /// else built-in speaker / wired. Standard processing (no AEC — stereo needs a non-VPIO + /// mode). Always available. case stereoMic = "Stereo Mic" - /// Maximum fidelity: stereo mic, no AEC/AGC/HPF (raw mode). A2DP output if BT - /// connected, else speaker/wired. Always available. Echo risk on speaker. + /// Maximum fidelity: stereo mic, no AEC/AGC/HPF (raw mode). A2DP output if BT connected, + /// else speaker/wired. Always available. Echo risk on speaker. case studio = "Studio (No Processing)" /// Bluetooth HFP: BT mic + BT output, AEC on, mono. Only when BT is connected. case bluetoothHeadset = "Bluetooth Headset (HFP)" - /// A2DP stereo output + built-in mono mic. No hardware AEC (A2DP incompatible). - /// Only when BT is connected. + /// A2DP stereo output + built-in mono mic, AEC off. Only when BT is connected. (For + /// A2DP output + stereo mic, use the Stereo Mic preset while BT is connected.) case btHeadphonesMonoMic = "BT Headphones + Mono Mic" - /// A2DP stereo output + stereo built-in mic (front+back). No hardware AEC. - /// Only when BT is connected. - case btHeadphonesStereoMic = "BT Headphones + Stereo Mic" /// Wired headset/earpods: wired output + wired mic (or built-in), AEC on, mono. /// Only when a wired audio device is connected. case wiredHeadset = "Wired Headset" @@ -83,7 +90,7 @@ final class IOSAudioRouter: ObservableObject { var requiresBluetooth: Bool { switch self { - case .bluetoothHeadset, .btHeadphonesMonoMic, .btHeadphonesStereoMic: return true + case .bluetoothHeadset, .btHeadphonesMonoMic: return true default: return false } } @@ -95,7 +102,8 @@ final class IOSAudioRouter: ObservableObject { var bluetoothMode: BluetoothMode { switch self { case .voiceChat, .bluetoothHeadset: return .btHfpVoice - case .stereoMic, .studio, .btHeadphonesMonoMic, .btHeadphonesStereoMic: return .builtInMicBtA2dp + // A2DP output when BT is connected; falls back to speaker/wired when it isn't. + case .stereoMic, .studio, .btHeadphonesMonoMic: return .builtInMicBtA2dp case .wiredHeadset: return .builtInMicSpeaker case .custom: return .builtInMicSpeaker // placeholder } @@ -103,7 +111,7 @@ final class IOSAudioRouter: ObservableObject { var captureChannels: CaptureChannels { switch self { - case .stereoMic, .studio, .btHeadphonesStereoMic: return .stereo + case .stereoMic, .studio: return .stereo default: return .mono } } @@ -118,7 +126,7 @@ final class IOSAudioRouter: ObservableObject { /// Whether this preset explicitly selects the built-in mic port. var usesBuiltInMic: Bool { switch self { - case .stereoMic, .studio, .btHeadphonesMonoMic, .btHeadphonesStereoMic: return true + case .stereoMic, .studio, .btHeadphonesMonoMic: return true default: return false } } @@ -273,7 +281,7 @@ final class IOSAudioRouter: ObservableObject { var activePreset: AudioPreset { // Check device-specific presets first (most specific → least specific) let order: [AudioPreset] = [ - .bluetoothHeadset, .btHeadphonesMonoMic, .btHeadphonesStereoMic, + .bluetoothHeadset, .btHeadphonesMonoMic, .wiredHeadset, .voiceChat, .stereoMic, .studio, ] @@ -307,96 +315,181 @@ final class IOSAudioRouter: ObservableObject { let session = AVAudioSession.sharedInstance() // 1. Build category options from bluetooth mode. - var options: AVAudioSession.CategoryOptions = [.defaultToSpeaker, .mixWithOthers] + // .mixWithOthers is ALWAYS set — it keeps other audio (notably VoiceOver, which a + // blind user needs to operate the phone) audible while our session is active. Never + // drop it. + // .defaultToSpeaker is set ONLY for the speaker preset. It forces output to the + // built-in speaker instead of the receiver — but it also actively breaks A2DP + // routing in .playAndRecord, so it must NOT be set for the A2DP or HFP presets. + // .allowAirPlay is added to the Bluetooth presets so AirPlay output also works. + var options: AVAudioSession.CategoryOptions = [.mixWithOthers] switch bluetoothMode { case .btHfpVoice: - options.insert(.allowBluetooth) - // Note: .allowBluetoothA2DP is NOT inserted — forces HFP for the mic path. - case .builtInMicBtA2dp: + // Voice Chat: allow BOTH HFP and A2DP, let iOS pick the right profile for the + // connected device. This matches TeamTalk5's default (UtilSound.swift:228): + // [.allowBluetoothHFP, .allowAirPlay, .allowBluetoothA2DP] + // Making HFP and A2DP mutually exclusive (HFP-only here) blocks A2DP headphones + // from receiving audio — the "Voice Chat kills Bluetooth output" regression. + // HFP is *preferred* (the system uses HFP when a two-way mic path is needed), + // but A2DP is still available for output-only scenarios. + options.insert(.allowBluetoothHFP) options.insert(.allowBluetoothA2DP) - // Note: .allowBluetooth is NOT inserted — no HFP, stereo A2DP output only. + options.insert(.allowAirPlay) + case .builtInMicBtA2dp: + // A2DP output only (no HFP). With HFP disabled the Bluetooth device can only be + // an OUTPUT (A2DP), so the system routes the mic to the built-in mic — exactly + // what we want for "built-in mic + A2DP output", in either mono OR stereo. + // This matches TeamTalk5's A2DP mode (UtilSound.swift:232-233): remove HFP from + // the default set, leaving only A2DP. + options.insert(.allowBluetoothA2DP) + options.insert(.allowAirPlay) case .builtInMicSpeaker: - // Neither Bluetooth option — built-in mic + speaker/wired output only. - break + // Built-in mic + speaker/wired output only. Prefer speaker over the receiver. + options.insert(.defaultToSpeaker) } - // 2. Set category + mode based on mic processing mode AND bluetooth mode. - // .voiceChat mode uses hardware AEC/AGC/HPF, but requires HFP-compatible routes. - // A2DP output is NOT HFP — using .voiceChat with A2DP causes iOS to mute the output - // because it can't set up the voice processing pipeline on an A2DP route. So: - // - Standard + HFP or Speaker: .voiceChat (hardware AEC works) - // - Standard + A2DP: .default (no hardware AEC, but audio routes correctly — A2DP - // headphones are in-ear/over-ear so echo from built-in mic is minimal) - // - Raw + any: .measurement (all processing off, regardless of bluetooth mode) + // 2. Set category + mode. Recipe validated against TeamTalk5 / Ferrite, which both do + // built-in stereo mic + A2DP Bluetooth output simultaneously: + // - Stereo capture: .default — .voiceChat (the AEC/VPIO path) forces MONO, so stereo + // is only possible in a non-VPIO mode. .default supports multi-capsule stereo AND + // keeps the A2DP output route alive. (Earlier .videoRecording + a session-level + // channel-count request collapsed A2DP output — the "stereo kills output" bug.) + // - Mono raw/studio: .measurement — all system processing off. + // - Mono + A2DP output: .videoRecording — keeps A2DP output without VPIO (no AEC). + // - Mono standard (HFP or speaker): .voiceChat — hardware AEC/AGC/HPF. let mode: AVAudioSession.Mode - switch (micMode, bluetoothMode) { - case (.standard, .builtInMicBtA2dp): - mode = .default // A2DP + hardware AEC = incompatible - case (.standard, _): - mode = .voiceChat // HFP or speaker: hardware AEC works - case (.raw, _): - mode = .measurement // all processing off + if captureChannels == .stereo { + mode = .default + } else if micMode == .raw { + mode = .measurement + } else if bluetoothMode == .builtInMicBtA2dp { + mode = .videoRecording + } else { + mode = .voiceChat } do { try session.setCategory(.playAndRecord, mode: mode, options: options) - logger.info("setCategory ok — mode=\(self.modeLabel(mode)), bt=\(self.bluetoothMode.rawValue), options=\(self.optionsLabel(options))") + logger.info("setCategory ok — mode=\(self.modeLabel(mode)), bt=\(self.bluetoothMode.rawValue), ch=\(self.captureChannels.rawValue), options=\(self.optionsLabel(options))") } catch { logger.error("setCategory failed: \(error.localizedDescription)") } - // 3. Set preferred input port (skip if "Default" — empty/nil ID means use system default). - if let portId = selectedInputPortId, !portId.isEmpty, - let port = session.availableInputs?.first(where: { $0.uid == portId }) { + // 3. Input & mic-capsule configuration. + if captureChannels == .stereo { + // Stereo: enable the built-in mic's .stereo polar pattern AND anchor the input + // route explicitly via setPreferredInput + setInputDataSource. The session-level + // channel-count call (setPreferredInputNumberOfChannels(2)) is what collapses the + // A2DP output route — NOT setPreferredInput (TeamTalk5 uses setPreferredInput and + // gets stereo + A2DP). With HFP disabled the system routes input to the built-in + // mic, but without the explicit preferred-input anchor the route can collapse + // during the mode switch (.voiceChat → .default) and the output dies. The channel + // count is requested by miniaudio at the audio-unit level (vc_set_capture_channels). + configureStereoCapture(session: session) + } else if let portId = selectedInputPortId, !portId.isEmpty, + let port = session.availableInputs?.first(where: { $0.uid == portId }) { + // Mono with an explicit input-port selection (advanced settings). do { try session.setPreferredInput(port) logger.info("setPreferredInput ok — \(port.portName)") } catch { logger.error("setPreferredInput failed: \(error.localizedDescription)") } - - // 4. Set preferred data source (orientation) on the selected input port. - if let dataSourceId = selectedDataSourceId, !dataSourceId.isEmpty, - let dataSource = port.dataSources?.first(where: { String(describing: $0.dataSourceID) == dataSourceId }) { - do { - try port.setPreferredDataSource(dataSource) - logger.info("setPreferredDataSource ok — \(dataSource.dataSourceName)") - } catch { - logger.error("setPreferredDataSource failed: \(error.localizedDescription)") - } - - // 5. Set preferred polar pattern on the data source. - if let polarPattern = selectedPolarPattern, !polarPattern.isEmpty { - let pattern = AVAudioSession.PolarPattern(rawValue: polarPattern) - do { - try dataSource.setPreferredPolarPattern(pattern) - logger.info("setPreferredPolarPattern ok — \(polarPattern)") - } catch { - logger.error("setPreferredPolarPattern failed: \(error.localizedDescription)") - } - } - } - } - - // 6. Set preferred input number of channels — ONLY for stereo (non-default). - // Calling setPreferredInputNumberOfChannels(1) for mono is unnecessary (1 is the - // default) and may put the session in a bad state on some devices. - if captureChannels == .stereo { - do { - try session.setPreferredInputNumberOfChannels(2) - logger.info("setPreferredInputNumberOfChannels ok — 2 (stereo)") - } catch { - logger.error("setPreferredInputNumberOfChannels failed: \(error.localizedDescription)") - } + configureMonoCapture(session: session, port: port) + } else { + // Mono, system-default input. Still clear any leftover .stereo capsule from a + // prior stereo session so we actually return to mono. + clearStereoPolarPattern(session: session) } updateWarnings() } + /// Enable 2-channel capture on the built-in mic. Mirrors TeamTalk5's recipe + /// (`SoundDevicesModel.selectDataSource` + `UtilSound.setupSoundDevices`), which + /// achieves stereo mic + A2DP Bluetooth output simultaneously: + /// 1. `setPreferredDataSource(stereoSource)` on the built-in mic port + /// 2. `setPreferredPolarPattern(.stereo)` on that data source + /// 3. `setPreferredInput(builtIn)` — anchor the input route explicitly (this is NOT + /// what collapses A2DP — the session-level `setPreferredInputNumberOfChannels(2)` + /// is. Without this anchor the route can collapse during the mode switch.) + /// 4. `setInputDataSource(stereoSource)` — commit the data source at the session level + /// The channel count itself is requested by miniaudio at the audio-unit level via + /// `vc_set_capture_channels(2)`. We do NOT call `setPreferredInputNumberOfChannels(2)` + /// — that session-level call is the one that collapses the A2DP output route. + private func configureStereoCapture(session: AVAudioSession) { + guard let builtIn = session.availableInputs?.first(where: { $0.portType == .builtInMic }) + else { + logger.warning("stereo requested but no built-in mic available — staying mono") + return + } + guard let stereoSource = builtIn.dataSources?.first(where: { + $0.supportedPolarPatterns?.contains(.stereo) == true + }) else { + logger.warning("stereo requested but built-in mic has no .stereo data source — staying mono") + return + } + do { + try builtIn.setPreferredDataSource(stereoSource) + try stereoSource.setPreferredPolarPattern(.stereo) + // Anchor the input route explicitly. TeamTalk5 does this (SoundDevicesModel + // .selectDataSource:147); without it the route can collapse during the mode + // switch (.voiceChat → .default) and the A2DP output dies. + try session.setPreferredInput(builtIn) + // Commit the data source at the session level (TeamTalk does this at + // SoundDevicesModel.selectDataSource:148). setPreferredDataSource alone only + // sets the port-level preference; setInputDataSource makes it the active source. + try session.setInputDataSource(stereoSource) + logger.info("stereo capsule enabled — source=\(stereoSource.dataSourceName), pattern=.stereo, input anchored") + } catch { + logger.error("stereo capsule setup failed: \(error.localizedDescription)") + } + } + + /// Configure mono capture on an explicitly selected port: apply the user's chosen data source + /// (orientation) and polar pattern, resetting any prior `.stereo` pattern back to default. + private func configureMonoCapture(session: AVAudioSession, port: AVAudioSessionPortDescription) { + guard let dataSourceId = selectedDataSourceId, !dataSourceId.isEmpty, + let dataSource = port.dataSources?.first(where: { + String(describing: $0.dataSourceID) == dataSourceId + }) else { + // No explicit capsule choice — make sure we're not stuck on a prior .stereo pattern. + clearStereoPolarPattern(session: session) + return + } + do { + try port.setPreferredDataSource(dataSource) + logger.info("setPreferredDataSource ok — \(dataSource.dataSourceName)") + } catch { + logger.error("setPreferredDataSource failed: \(error.localizedDescription)") + } + + if let polarPattern = selectedPolarPattern, !polarPattern.isEmpty { + let pattern = AVAudioSession.PolarPattern(rawValue: polarPattern) + try? dataSource.setPreferredPolarPattern(pattern) + logger.info("setPreferredPolarPattern ok — \(polarPattern)") + } else { + // Clear any prior .stereo selection so mono capture returns to a mono capsule. + try? dataSource.setPreferredPolarPattern(nil) + } + } + + /// Reset any built-in-mic data source that's currently on the `.stereo` polar pattern back to + /// the default (mono) pattern. Used when switching from a stereo session back to mono with no + /// explicit capsule selection, so the prior stereo capsule doesn't linger. + private func clearStereoPolarPattern(session: AVAudioSession) { + guard let builtIn = session.availableInputs?.first(where: { $0.portType == .builtInMic }) + else { return } + for ds in builtIn.dataSources ?? [] where ds.selectedPolarPattern == .stereo { + try? ds.setPreferredPolarPattern(nil) + } + } + private func modeLabel(_ mode: AVAudioSession.Mode) -> String { switch mode { case .voiceChat: return "voiceChat" case .measurement: return "measurement" + case .videoRecording: return "videoRecording" case .default: return "default" default: return "other" } @@ -406,7 +499,7 @@ final class IOSAudioRouter: ObservableObject { var parts: [String] = [] if opts.contains(.defaultToSpeaker) { parts.append("defaultToSpeaker") } if opts.contains(.mixWithOthers) { parts.append("mixWithOthers") } - if opts.contains(.allowBluetooth) { parts.append("allowBluetooth") } + if opts.contains(.allowBluetoothHFP) { parts.append("allowBluetoothHFP") } if opts.contains(.allowBluetoothA2DP) { parts.append("allowBluetoothA2DP") } return parts.joined(separator: ",") } @@ -484,6 +577,13 @@ final class IOSAudioRouter: ObservableObject { captureChannels = channels savePreferences() applyConfiguration() + // Sync the core's capture channel count. The core's set_capture_channels handles + // the engine restart internally (stop + ensure_audio_running) — no need for the + // Swift layer to suspend/restart separately. + if let streamId = AudioSessionManager.shared.activeMicStreamId { + _ = AudioSessionManager.shared.client?.setCaptureChannels( + streamId: streamId, channels: channels.channelCount) + } } // MARK: - Presets @@ -520,6 +620,12 @@ final class IOSAudioRouter: ObservableObject { UserDefaults.standard.set(preset.rawValue, forKey: kPreset) savePreferences() applyConfiguration() + // Sync the core's capture channel count. The core's set_capture_channels handles + // the engine restart internally. + if let streamId = AudioSessionManager.shared.activeMicStreamId { + _ = AudioSessionManager.shared.client?.setCaptureChannels( + streamId: streamId, channels: preset.captureChannels.channelCount) + } refreshRoutes() logger.info("applyPreset — \(preset.rawValue)") } @@ -532,10 +638,17 @@ final class IOSAudioRouter: ObservableObject { let outputIsSpeaker = session.currentRoute.outputs.contains { $0.portType == .builtInSpeaker } // Raw/Studio mode + speaker = echo risk (no AEC in .measurement mode) showsRawModeSpeakerWarning = (micMode == .raw && outputIsSpeaker) - // Standard mode + A2DP = no hardware AEC (A2DP incompatible with .voiceChat mode) - showsA2dpNoAecWarning = (micMode == .standard && bluetoothMode == .builtInMicBtA2dp) + // A2DP output runs without hardware AEC (the .voiceChat AEC path isn't available on an + // A2DP route). Applies to both mono and stereo A2DP. Stereo also has no AEC (it can't + // use .voiceChat at all), but the message is the same and the warning already shows when + // the bluetooth mode is A2DP. + showsA2dpNoAecWarning = (bluetoothMode == .builtInMicBtA2dp) } + /// Whether the current configuration wants Bluetooth A2DP output. Used after session + /// activation to clear any lingering speaker override that would pin output to the speaker. + var wantsA2dpOutput: Bool { bluetoothMode == .builtInMicBtA2dp } + /// The selected input port object, if any. var selectedPort: IOSAudioInputPort? { inputPorts.first(where: { $0.id == selectedInputPortId }) diff --git a/clients/apple/iOS/VoiceCatiOS/SessionState.swift b/clients/apple/iOS/VoiceCatiOS/SessionState.swift index baaa55d..f79c322 100644 --- a/clients/apple/iOS/VoiceCatiOS/SessionState.swift +++ b/clients/apple/iOS/VoiceCatiOS/SessionState.swift @@ -204,6 +204,11 @@ final class SessionState { if result == .ok { voiceState.micActive = true voiceState.localStreamId = streamId + // Publish the active mic stream ID so IOSAudioRouter can reset the core's capture + // channel count when the user switches mono↔stereo (selectCaptureChannels / + // applyPreset). Without this, switching stereo→mono leaves the LocalStream's + // capture_channels field at 2 and the next engine start still opens stereo. + AudioSessionManager.shared.activeMicStreamId = streamId // Apply the user's capture channel selection (mono/stereo) from IOSAudioRouter. // The core opens the capture device via miniaudio on the next engine start; // vc_set_capture_channels tells it to open in stereo (2) or mono (1). @@ -220,6 +225,7 @@ final class SessionState { if voiceState.localStreamId != 0 { client.stopStream(voiceState.localStreamId) voiceState.localStreamId = 0 + AudioSessionManager.shared.activeMicStreamId = nil } voiceState.micActive = false voiceState.level = 0 diff --git a/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift b/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift index 85ea919..d40d656 100644 --- a/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift +++ b/clients/apple/iOS/VoiceCatiOS/Views/SettingsView.swift @@ -115,6 +115,7 @@ struct SettingsView: View { .accessibilityLabel("Info: A2DP output mode does not support hardware echo cancellation") } + // Capture channels: Mono vs Stereo Picker("Channels", selection: Binding( get: { router.captureChannels }, diff --git a/core/include/voicecat.h b/core/include/voicecat.h index 0fe67d6..83d454c 100644 --- a/core/include/voicecat.h +++ b/core/include/voicecat.h @@ -399,11 +399,12 @@ VC_API vc_result vc_test_inject_capture(vc_client* c, uint32_t stream_id, const /* Set the capture channel count for a local MIC stream (1 = mono, 2 = stereo interleaved). * Must be called after vc_stream_start; takes effect on the next AudioEngine restart (e.g. when * joining voice, or immediately if the engine is already running — it stops and restarts the - * capture device with the new channel count). Defaults to 1 (mono). On iOS this lets the Swift - * AVAudioSession routing layer request stereo built-in mic capture via - * setPreferredInputNumberOfChannels(2) and then tell the core to open the capture device in - * stereo. VC_ERR_INVALID_ARG if stream_id is unknown, the stream is not a MIC stream, or - * channels is not 1 or 2. */ + * capture device with the new channel count). Defaults to 1 (mono). On iOS the Swift + * AVAudioSession routing layer enables stereo built-in mic capture by switching the built-in + * mic's data source to the .stereo polar pattern (setPreferredDataSource + + * setPreferredPolarPattern(.stereo) + setPreferredInput + setInputDataSource) and then calls + * this to tell the core to open the capture device in stereo. VC_ERR_INVALID_ARG if stream_id + * is unknown, the stream is not a MIC stream, or channels is not 1 or 2. */ VC_API vc_result vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels); /* ── Text ─────────────────────────────────────────────────────────────────── */ @@ -484,6 +485,16 @@ VC_API vc_result vc_get_permissions(vc_client* c, vc_permissions* out); VC_API vc_result vc_audio_suspend(vc_client* c); VC_API vc_result vc_audio_resume(vc_client* c); +/* Full audio engine restart (iOS M6). Unlike vc_audio_suspend/resume which only stop/start + * the existing miniaudio devices (leaving them bound to the route that was active when they + * were opened), vc_audio_restart() uninitializes and re-initializes the capture and playback + * devices so they pick up a new AVAudioSession route. Call this from the Swift layer AFTER + * reconfiguring AVAudioSession (setCategory, setPreferredInput, setPreferredPolarPattern, etc.) + * so the core's devices reopen against the new route. Mirrors TeamTalk5's + * closeSoundDevices()/initSoundInputDevice()/initSoundOutputDevice() reconfiguration pattern. + * Safe to call when the engine is not running (it will just start it). */ +VC_API vc_result vc_audio_restart(vc_client* c); + #if defined(__cplusplus) } /* extern "C" */ #endif diff --git a/core/src/core/client.cpp b/core/src/core/client.cpp index 5caab59..83b9edc 100644 --- a/core/src/core/client.cpp +++ b/core/src/core/client.cpp @@ -1433,6 +1433,22 @@ vc_result vc_client::audio_resume() { return audio_engine_.resume() ? VC_OK : VC_ERR_AUDIO; } +vc_result vc_client::audio_restart() { + // Full uninit + re-init (not just stop/start like suspend/resume) so the miniaudio + // devices reopen against the current AVAudioSession route. Mirrors TeamTalk5's + // closeSoundDevices()/initSoundInputDevice()/initSoundOutputDevice() pattern. + // IMPORTANT: only restart if the engine was already running — calling + // ensure_audio_running() when the engine isn't running would start it prematurely + // (opening the capture device / mic without an active mic stream), which on iOS + // triggers a route reconfiguration that can collapse the output route. + bool was_running = audio_engine_.running(); + audio_engine_.stop(); + if (was_running) { + ensure_audio_running(); + } + return VC_OK; +} + vc_result vc_client::get_stream_audio_config(uint32_t user_id, uint32_t stream_id, vc_audio_config* out) { if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED; @@ -1900,5 +1916,6 @@ vc_result vc_client::get_account_list(vc_account_list*) { return VC_ERR_NOT_IMPL vc_result vc_client::get_permissions(vc_permissions*) { return VC_ERR_NOT_IMPLEMENTED; } vc_result vc_client::audio_suspend() { return VC_ERR_NOT_IMPLEMENTED; } vc_result vc_client::audio_resume() { return VC_ERR_NOT_IMPLEMENTED; } +vc_result vc_client::audio_restart() { return VC_ERR_NOT_IMPLEMENTED; } #endif // VOICECAT_HAS_NET diff --git a/core/src/core/client.h b/core/src/core/client.h index e38d2a3..e1fa74f 100644 --- a/core/src/core/client.h +++ b/core/src/core/client.h @@ -60,6 +60,7 @@ struct vc_client { vc_remote_stream_state* out); vc_result audio_suspend(); vc_result audio_resume(); + vc_result audio_restart(); vc_result send_text(vc_text_scope scope, uint32_t target_id, const char* utf8); diff --git a/core/src/voicecat.cpp b/core/src/voicecat.cpp index 09c5603..87fd913 100644 --- a/core/src/voicecat.cpp +++ b/core/src/voicecat.cpp @@ -310,4 +310,9 @@ vc_result vc_audio_resume(vc_client* c) { return c->audio_resume(); } +vc_result vc_audio_restart(vc_client* c) { + if (c == nullptr) return VC_ERR_INVALID_ARG; + return c->audio_restart(); +} + } // extern "C" diff --git a/docs/architecture.md b/docs/architecture.md index 3545beb..5c57e20 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -136,7 +136,7 @@ Design notes: ### Per-platform binding notes -- **Swift / Apple.** Import the C ABI via a **module map** (`module VoiceCatC { header "voicecat.h" }`) staged into the XCFramework headers by `clients/apple/scripts/build-xcframework.sh` — Swift gets a clean `import VoiceCatC` with all C enums/structs/functions available directly (no manual redeclaration, unlike the C# P/Invoke layer). A **Swift wrapper** (`VoiceCatCore` package at `clients/apple/`) provides Swift-idiomatic types (`VoiceCatResult`, `VoiceCatEvent`, `Channel`, `User`, etc.) on top, mirroring the C# `VoiceCat.Interop` layer. Callbacks use `@convention(c)` closures (plain C function pointers, not ARC-managed closures) + `Unmanaged.passUnretained(self)` as the `user` context (the Swift analog of C#'s `[UnmanagedCallersOnly]` + `GCHandle`). Events are delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (one async block scheduled at a time) — the Swift analog of C#'s `Channel` + 30ms WinForms Timer pump. `deinit` calls `vc_client_destroy` (joins all threads) then frees native CString config storage (the core stores raw pointers, doesn't copy). **macOS UI: AppKit** (chosen over SwiftUI for the most mature VoiceOver accessibility story — same rationale as the Windows client's WinForms choice); **iOS UI: SwiftUI** (narrower control surface, sufficient VoiceOver support). On **iOS** the app owns `AVAudioSession` (category `.playAndRecord`), requests mic permission, and handles interruptions/route changes — the core exposes hooks (`vc_audio_suspend`/`vc_audio_resume`, implemented) the Swift layer calls from `AVAudioSession` notifications. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) is driven from the Swift `IOSAudioRouter` singleton via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The core is told the capture channel count via `vc_set_capture_channels` (append-only ABI). iOS 18.0 deployment target. Background voice and VoIP push (CallKit/PushKit) are a later milestone. The XCFramework carries a **fat static library** (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps so the Swift Package links a single self-contained `.a` per slice. +- **Swift / Apple.** Import the C ABI via a **module map** (`module VoiceCatC { header "voicecat.h" }`) staged into the XCFramework headers by `clients/apple/scripts/build-xcframework.sh` — Swift gets a clean `import VoiceCatC` with all C enums/structs/functions available directly (no manual redeclaration, unlike the C# P/Invoke layer). A **Swift wrapper** (`VoiceCatCore` package at `clients/apple/`) provides Swift-idiomatic types (`VoiceCatResult`, `VoiceCatEvent`, `Channel`, `User`, etc.) on top, mirroring the C# `VoiceCat.Interop` layer. Callbacks use `@convention(c)` closures (plain C function pointers, not ARC-managed closures) + `Unmanaged.passUnretained(self)` as the `user` context (the Swift analog of C#'s `[UnmanagedCallersOnly]` + `GCHandle`). Events are delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (one async block scheduled at a time) — the Swift analog of C#'s `Channel` + 30ms WinForms Timer pump. `deinit` calls `vc_client_destroy` (joins all threads) then frees native CString config storage (the core stores raw pointers, doesn't copy). **macOS UI: AppKit** (chosen over SwiftUI for the most mature VoiceOver accessibility story — same rationale as the Windows client's WinForms choice); **iOS UI: SwiftUI** (narrower control surface, sufficient VoiceOver support). On **iOS** the app owns `AVAudioSession` (category `.playAndRecord`), requests mic permission, and handles interruptions/route changes — the core exposes hooks (`vc_audio_suspend`/`vc_audio_resume`/`vc_audio_restart`, implemented) the Swift layer calls from `AVAudioSession` notifications and `IOSAudioRouter` setting changes. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) is driven from the Swift `IOSAudioRouter` singleton via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The core is told the capture channel count via `vc_set_capture_channels` (append-only ABI). `vc_audio_restart` does a full stop + re-init (unlike `suspend`/`resume` which only stop/start) so devices reopen against a new route after `AVAudioSession` reconfiguration. iOS 18.0 deployment target. Background voice and VoIP push (CallKit/PushKit) are a later milestone. The XCFramework carries a **fat static library** (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps so the Swift Package links a single self-contained `.a` per slice. - **iOS screen / system-audio sharing** is supported via a **ReplayKit Broadcast Upload Extension** (the same mechanism Discord uses; triggered from Control Center's screen-record button via `RPSystemBroadcastPickerView`). The extension receives diff --git a/docs/tech-stack.md b/docs/tech-stack.md index 56e9ec1..2a703ad 100644 --- a/docs/tech-stack.md +++ b/docs/tech-stack.md @@ -31,7 +31,7 @@ explicit resampling (speexdsp/libsamplerate) is only needed when a device can't | UI — macOS | **AppKit** | Chosen over SwiftUI for the most mature, granular **VoiceOver** accessibility story (per-control `accessibilityLabel`/`accessibilityHelp`/`accessibilityRole`, `NSAccessibility.post(.announcement)` for live announcements) — the same rationale that drove the Windows client to WinForms over WinUI 3 for screen-reader (NVDA/JAWS/Narrator) UIA support (resolved decision in `docs/roadmap.md`). macOS 14 (Sonoma) deployment target. | | UI — iOS | **SwiftUI** | iOS has a narrower control surface (no channel-tree moderation, etc.) and SwiftUI's VoiceOver support is sufficient; revisit if gaps emerge. iOS 18.0 deployment target (unlocks newest AVAudioSession APIs: stereo capture, polar patterns, data sources). | | Shared core | **VoiceCatCore** Swift Package | One Swift library wrapping the C ABI, consumed by both the macOS AppKit app and the iOS SwiftUI app. Mirrors the C# `VoiceCat.Interop` layer. Events delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (the Swift analog of C#'s `Channel` + 30ms WinForms Timer pump). | -| Audio session (iOS) | **AVAudioSession** + **IOSAudioRouter** | App owns category `.playAndRecord`, mic permission, interruption/route-change handling; calls `vc_audio_suspend`/`vc_audio_resume` (implemented) on the core. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture via `setPreferredInputNumberOfChannels(2)`) is driven from Swift via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The `IOSAudioRouter` singleton owns this; the core is told the channel count via `vc_set_capture_channels`. macOS uses CoreAudio via the core directly. | +| Audio session (iOS) | **AVAudioSession** + **IOSAudioRouter** | App owns category `.playAndRecord`, mic permission, interruption/route-change handling; calls `vc_audio_suspend`/`vc_audio_resume`/`vc_audio_restart` (implemented) on the core. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture via `.stereo` polar pattern + `setPreferredInput` + `setInputDataSource`) is driven from Swift via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The `IOSAudioRouter` singleton owns this; the core is told the channel count via `vc_set_capture_channels`. When settings change mid-session, devices are suspended (`vc_audio_suspend`), the session is reconfigured, and devices are restarted (`vc_audio_restart`) to pick up the new route. macOS uses CoreAudio via the core directly. | | Packaging | Swift Package + Xcode project | Core shipped as an **XCFramework** binary target — a fat static library (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps (protobuf/mbedtls/sodium/opus/sqlite3/spdlog/asio), so the Swift Package links a single self-contained `.a` per slice. macOS slice validated; iOS device + sim slices are scaffolding. | | Future | CallKit / PushKit | For background VoIP + incoming-call UX on iOS. Post-v1. | diff --git a/docs/voice.md b/docs/voice.md index 03de0bc..17e5139 100644 --- a/docs/voice.md +++ b/docs/voice.md @@ -178,23 +178,27 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**. - Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA). Playback is genuinely stereo end-to-end. **Mic capture** is mono by default; **stereo mic - capture** is supported via `vc_set_capture_channels(stream_id, 2)` (e.g. iOS stereo built-in - mic via `AVAudioSession.setPreferredInputNumberOfChannels(2)`) — when enabled, the capture - device opens in stereo (interleaved L/R) and the encoder receives real stereo PCM (no upmix). - A mono mic frame on a stereo channel is upmixed L=R before encoding so the Opus bitstream is - still spec-correct stereo. **Screen-audio (`SCREEN_AUDIO`) loopback** captures in the - channel's mode — stereo when the channel is stereo (real interleaved L/R, no downmix), mono - when the channel is mono — so a stereo music/screen-share channel gets genuine stereo - end-to-end. See §9 for the platform-specific loopback mechanism. + capture** is supported via `vc_set_capture_channels(stream_id, 2)` — when enabled, the + capture device opens in stereo (interleaved L/R) and the encoder receives real stereo PCM + (no upmix). A mono mic frame on a stereo channel is upmixed L=R before encoding so the Opus + bitstream is still spec-correct stereo. **Screen-audio (`SCREEN_AUDIO`) loopback** captures + in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no + downmix), mono when the channel is mono — so a stereo music/screen-share channel gets + genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism. - **iOS mic capture:** all iOS audio routing is driven from Swift via `AVAudioSession` by the `IOSAudioRouter` singleton *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. Input port selection (`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic processing mode (`.voiceChat` = Standard with AEC/AGC/HPF, or `.measurement` = Raw/Studio with all processing - off), Bluetooth mode (`.allowBluetooth` HFP voice vs `.allowBluetoothA2DP` stereo output vs - neither), and stereo capture (`setPreferredInputNumberOfChannels(2)` → `vc_set_capture_channels`) - are all set from Swift. The core then opens whatever route AVAudioSession has established. + off), Bluetooth mode (`.allowBluetoothHFP` HFP voice vs `.allowBluetoothA2DP` stereo output + vs neither), and stereo capture (`.stereo` polar pattern + `setPreferredInput` + + `setInputDataSource` → `vc_set_capture_channels`) are all set from Swift. The core then + opens whatever route AVAudioSession has established. When the user changes audio settings + mid-session, `IOSAudioRouter` suspends the core's devices (`vc_audio_suspend`), reconfigures + `AVAudioSession`, then restarts the devices (`vc_audio_restart`) so they reopen against the + new route — mirroring TeamTalk5's `closeSoundDevices`/`initSoundInputDevice`/ + `initSoundOutputDevice` pattern. - **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC + VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream (confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard