feat(ios): audio overhaul, Join/Leave Voice, channel-id sync fix, stereo mic capture
Three iOS client problems fixed plus a new core stereo-mic capture ABI: 1. Channel-id sync bug (mic button permanently dimmed): SessionState never synced currentChannelId from the self user's channelId on connect, so the mic button (gated on currentChannelId == 0) stayed dimmed. Added syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522); called from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult. Added applyServerMuteState() + serverMuted/serverDeafened to VoiceState. 2. Join/Leave Voice button: replaced icon-only mic toggle with explicit text button (parity with macOS). Mute/deafen disable when not in voice. 3. IOSAudioRouter.swift (new): full AVAudioSession routing layer — input port selection, built-in mic orientation/polar patterns, Bluetooth HFP/A2DP/Off modes, Standard/Raw mic processing, stereo capture, AirPlay, UserDefaults persistence. AudioSessionManager delegates to it. 4. Core stereo-mic capture (append-only ABI): vc_set_capture_channels() lets the core open the mic device in stereo (2-ch interleaved). LocalStream gains capture_channels; ensure_audio_running reads it; audio_engine.cpp capture_accum_ + on_capture updated to channel-aware accumulation. Test test_stereo_mic_capture (headless, L!=R stereo round-trip). Swift wrapper VoiceCatClient.setCaptureChannels. 5. Settings UI rework: AVAudioSession-derived input/output tree replaces miniaudio device picker. 6. iOS deployment target raised to 18.0 (Package.swift + project.pbxproj). swift-tools-version 6.0 with swiftLanguageModes .v5. Docs: tech-stack.md, architecture.md, voice.md, roadmap.md, building.md updated; stale 'vc_audio_suspend/resume deferred' claims corrected. Verified: ctest --preset dev 21/21 green; swift test 6/6 green; xcodebuild -target VoiceCatiOS -sdk iphonesimulator BUILD SUCCEEDED.
This commit is contained in:
@@ -136,7 +136,7 @@ Design notes:
|
||||
|
||||
### Per-platform binding notes
|
||||
|
||||
- **Swift / Apple.** Import the C ABI via a **module map** (`module VoiceCatC { header "voicecat.h" }`) staged into the XCFramework headers by `clients/apple/scripts/build-xcframework.sh` — Swift gets a clean `import VoiceCatC` with all C enums/structs/functions available directly (no manual redeclaration, unlike the C# P/Invoke layer). A **Swift wrapper** (`VoiceCatCore` package at `clients/apple/`) provides Swift-idiomatic types (`VoiceCatResult`, `VoiceCatEvent`, `Channel`, `User`, etc.) on top, mirroring the C# `VoiceCat.Interop` layer. Callbacks use `@convention(c)` closures (plain C function pointers, not ARC-managed closures) + `Unmanaged.passUnretained(self)` as the `user` context (the Swift analog of C#'s `[UnmanagedCallersOnly]` + `GCHandle`). Events are delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (one async block scheduled at a time) — the Swift analog of C#'s `Channel<VoiceCatEvent>` + 30ms WinForms Timer pump. `deinit` calls `vc_client_destroy` (joins all threads) then frees native CString config storage (the core stores raw pointers, doesn't copy). **macOS UI: AppKit** (chosen over SwiftUI for the most mature VoiceOver accessibility story — same rationale as the Windows client's WinForms choice); **iOS UI: SwiftUI** (narrower control surface, sufficient VoiceOver support). On **iOS** the app must still own `AVAudioSession` (category `.playAndRecord`, `.voiceChat` mode), request mic permission, and handle interruptions/route changes — the core exposes hooks (`vc_audio_suspend`/`vc_audio_resume`) the Swift layer calls from `AVAudioSession` notifications (these hooks are deferred until the iOS client milestone to keep the ABI stable). Background voice and VoIP push (CallKit/PushKit) are a later milestone. The XCFramework carries a **fat static library** (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps so the Swift Package links a single self-contained `.a` per slice.
|
||||
- **Swift / Apple.** Import the C ABI via a **module map** (`module VoiceCatC { header "voicecat.h" }`) staged into the XCFramework headers by `clients/apple/scripts/build-xcframework.sh` — Swift gets a clean `import VoiceCatC` with all C enums/structs/functions available directly (no manual redeclaration, unlike the C# P/Invoke layer). A **Swift wrapper** (`VoiceCatCore` package at `clients/apple/`) provides Swift-idiomatic types (`VoiceCatResult`, `VoiceCatEvent`, `Channel`, `User`, etc.) on top, mirroring the C# `VoiceCat.Interop` layer. Callbacks use `@convention(c)` closures (plain C function pointers, not ARC-managed closures) + `Unmanaged.passUnretained(self)` as the `user` context (the Swift analog of C#'s `[UnmanagedCallersOnly]` + `GCHandle`). Events are delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (one async block scheduled at a time) — the Swift analog of C#'s `Channel<VoiceCatEvent>` + 30ms WinForms Timer pump. `deinit` calls `vc_client_destroy` (joins all threads) then frees native CString config storage (the core stores raw pointers, doesn't copy). **macOS UI: AppKit** (chosen over SwiftUI for the most mature VoiceOver accessibility story — same rationale as the Windows client's WinForms choice); **iOS UI: SwiftUI** (narrower control surface, sufficient VoiceOver support). On **iOS** the app owns `AVAudioSession` (category `.playAndRecord`), requests mic permission, and handles interruptions/route changes — the core exposes hooks (`vc_audio_suspend`/`vc_audio_resume`, implemented) the Swift layer calls from `AVAudioSession` notifications. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) is driven from the Swift `IOSAudioRouter` singleton via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The core is told the capture channel count via `vc_set_capture_channels` (append-only ABI). iOS 18.0 deployment target. Background voice and VoIP push (CallKit/PushKit) are a later milestone. The XCFramework carries a **fat static library** (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps so the Swift Package links a single self-contained `.a` per slice.
|
||||
- **iOS screen / system-audio sharing** is supported via a **ReplayKit Broadcast Upload
|
||||
Extension** (the same mechanism Discord uses; triggered from Control Center's screen-record
|
||||
button via `RPSystemBroadcastPickerView`). The extension receives
|
||||
|
||||
@@ -370,7 +370,7 @@ links `libvoicecat` via the same `VoiceCatCore` Swift Package as the macOS clien
|
||||
Full details in [`clients/apple/README.md`](../clients/apple/README.md).
|
||||
|
||||
**Prerequisites:** Xcode, vcpkg (`VCPKG_ROOT` set), iOS Simulator runtime installed
|
||||
(Xcode > Settings > Platforms > iOS). iOS deployment target: 17.0.
|
||||
(Xcode > Settings > Platforms > iOS). iOS deployment target: 18.0.
|
||||
|
||||
### Build the XCFramework (all slices)
|
||||
|
||||
|
||||
@@ -66,9 +66,10 @@ exists from M1 so the protocol can be exercised long before any GUI.
|
||||
|
||||
**iOS (Swift/SwiftUI) — pending:**
|
||||
- SwiftUI app consuming the same `VoiceCatCore` package.
|
||||
- AVAudioSession, mic permission, foreground voice.
|
||||
- ~~AVAudioSession, mic permission, foreground voice.~~ ✓ Done — `IOSAudioRouter` drives
|
||||
all iOS audio routing (input ports, orientation/polar patterns, HFP/A2DP, Standard/Raw
|
||||
mic mode, stereo capture), `vc_audio_suspend`/`vc_audio_resume` for interruptions.
|
||||
- ReplayKit broadcast extension for `SCREEN_AUDIO`.
|
||||
- `vc_audio_suspend`/`vc_audio_resume` ABI hooks (deferred until this milestone).
|
||||
|
||||
**Exit:** non-technical user installs a client, saves a server, and joins.
|
||||
|
||||
|
||||
@@ -29,9 +29,9 @@ explicit resampling (speexdsp/libsamplerate) is only needed when a device can't
|
||||
|---------|--------|-------|
|
||||
| Language | **Swift 5.9+** | Direct **Swift↔C interop** — the C ABI (`voicecat.h`) is imported as a Clang module (`import VoiceCatC`) via a module map in the XCFramework headers; no manual struct/function redeclaration (unlike the C# P/Invoke layer). A Swift wrapper (`VoiceCatCore` package) provides Swift-idiomatic types on top. |
|
||||
| UI — macOS | **AppKit** | Chosen over SwiftUI for the most mature, granular **VoiceOver** accessibility story (per-control `accessibilityLabel`/`accessibilityHelp`/`accessibilityRole`, `NSAccessibility.post(.announcement)` for live announcements) — the same rationale that drove the Windows client to WinForms over WinUI 3 for screen-reader (NVDA/JAWS/Narrator) UIA support (resolved decision in `docs/roadmap.md`). macOS 14 (Sonoma) deployment target. |
|
||||
| UI — iOS | **SwiftUI** | iOS has a narrower control surface (no channel-tree moderation, etc.) and SwiftUI's VoiceOver support is sufficient; revisit if gaps emerge. |
|
||||
| UI — iOS | **SwiftUI** | iOS has a narrower control surface (no channel-tree moderation, etc.) and SwiftUI's VoiceOver support is sufficient; revisit if gaps emerge. iOS 18.0 deployment target (unlocks newest AVAudioSession APIs: stereo capture, polar patterns, data sources). |
|
||||
| Shared core | **VoiceCatCore** Swift Package | One Swift library wrapping the C ABI, consumed by both the macOS AppKit app and the iOS SwiftUI app. Mirrors the C# `VoiceCat.Interop` layer. Events delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (the Swift analog of C#'s `Channel<VoiceCatEvent>` + 30ms WinForms Timer pump). |
|
||||
| Audio session (iOS) | **AVAudioSession** | App owns category `.playAndRecord` + `.voiceChat` mode, mic permission, interruption/route-change handling; calls `vc_audio_suspend/resume` on the core. macOS uses CoreAudio via the core directly. (`vc_audio_suspend`/`vc_audio_resume` ABI hooks are deferred until the iOS client milestone — keep ABI stable.) |
|
||||
| Audio session (iOS) | **AVAudioSession** + **IOSAudioRouter** | App owns category `.playAndRecord`, mic permission, interruption/route-change handling; calls `vc_audio_suspend`/`vc_audio_resume` (implemented) on the core. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture via `setPreferredInputNumberOfChannels(2)`) is driven from Swift via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The `IOSAudioRouter` singleton owns this; the core is told the channel count via `vc_set_capture_channels`. macOS uses CoreAudio via the core directly. |
|
||||
| Packaging | Swift Package + Xcode project | Core shipped as an **XCFramework** binary target — a fat static library (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps (protobuf/mbedtls/sodium/opus/sqlite3/spdlog/asio), so the Swift Package links a single self-contained `.a` per slice. macOS slice validated; iOS device + sim slices are scaffolding. |
|
||||
| Future | CallKit / PushKit | For background VoIP + incoming-call UX on iOS. Post-v1. |
|
||||
|
||||
|
||||
@@ -177,12 +177,24 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**.
|
||||
```
|
||||
|
||||
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
|
||||
Playback is genuinely stereo end-to-end. **Mic capture stays mono** (no stereo mic in v1);
|
||||
a mono mic frame on a stereo channel is upmixed L=R before encoding so the Opus bitstream
|
||||
is still spec-correct stereo. **Screen-audio (`SCREEN_AUDIO`) loopback** captures in the
|
||||
Playback is genuinely stereo end-to-end. **Mic capture** is mono by default; **stereo mic
|
||||
capture** is supported via `vc_set_capture_channels(stream_id, 2)` (e.g. iOS stereo built-in
|
||||
mic via `AVAudioSession.setPreferredInputNumberOfChannels(2)`) — when enabled, the capture
|
||||
device opens in stereo (interleaved L/R) and the encoder receives real stereo PCM (no upmix).
|
||||
A mono mic frame on a stereo channel is upmixed L=R before encoding so the Opus bitstream is
|
||||
still spec-correct stereo. **Screen-audio (`SCREEN_AUDIO`) loopback** captures in the
|
||||
channel's mode — stereo when the channel is stereo (real interleaved L/R, no downmix), mono
|
||||
when the channel is mono — so a stereo music/screen-share channel gets genuine stereo
|
||||
end-to-end. See §9 for the platform-specific loopback mechanism.
|
||||
- **iOS mic capture:** all iOS audio routing is driven from Swift via `AVAudioSession` by the
|
||||
`IOSAudioRouter` singleton *before* the core (miniaudio) opens its device — miniaudio does
|
||||
NOT touch `AVAudioSession` on iOS. Input port selection (`availableInputs`), built-in mic
|
||||
orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns
|
||||
(`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic processing mode
|
||||
(`.voiceChat` = Standard with AEC/AGC/HPF, or `.measurement` = Raw/Studio with all processing
|
||||
off), Bluetooth mode (`.allowBluetooth` HFP voice vs `.allowBluetoothA2DP` stereo output vs
|
||||
neither), and stereo capture (`setPreferredInputNumberOfChannels(2)` → `vc_set_capture_channels`)
|
||||
are all set from Swift. The core then opens whatever route AVAudioSession has established.
|
||||
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
|
||||
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
|
||||
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
|
||||
|
||||
Reference in New Issue
Block a user