From cd530db02439182f7f5fb1f1e58a17b4af52448c Mon Sep 17 00:00:00 2001 From: Talon Date: Fri, 19 Jun 2026 03:39:12 +0200 Subject: [PATCH] docs(progress): plan iOS audio overhaul + Join/Leave Voice + channel-id sync fix --- PROGRESS.md | 103 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index 782f9b7..e432011 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -10,6 +10,109 @@ up instantly. Newest status at the top. ## ▶ Where we left off / next action +- **Planned (not started):** **iOS audio overhaul + Join/Leave Voice + channel-id sync fix** + (2026-06-19, plan written on Windows; implement on Mac). Three problems found while reviewing + the iOS client: + 1. **Mic button permanently dimmed (BUG — root cause).** `VoiceControlsView.swift:26` gates the + mic button on `session.currentChannelId == 0`, but `SessionState` never syncs + `currentChannelId` from the self user's `channelId` on connect. The server auto-places every + newly-authed user into the Lobby (channel 1, `server/src/session_registry.cpp:111`), but the + iOS client ignores it — `currentChannelId` is set in only two places: `.joinResult` + (`SessionState.swift:97`) and reset to 0 by `leaveChannel()` (`SessionState.swift:135`). The + `init` (`SessionState.swift:55-57`) calls `refreshUsers()` but throws away self's channelId; + the `.channelList`/`.userUpdated` handlers (`SessionState.swift:76-79`) likewise only re-fetch + the list. macOS does this sync correctly (`MainWindowController.swift:461,491,522`). Because + the button stays dimmed, `startMicStream()` (and thus the mic permission prompt) can never be + triggered — a chicken-and-egg that *looks* like a permission issue. **Not** a permission bug: + `Info.plist:23` declares `NSMicrophoneUsageDescription` and `SessionState.swift:139` calls + `AVAudioApplication.requestRecordPermission`. Fix = mirror macOS: add `syncSelfChannel()`, + call it from `init`/`.channelList`/`.userJoined`/`.userLeft`/`.userUpdated`/`.joinResult`; + also add `applyServerMuteState(muted:deafened:)` (iOS currently ignores server mute/deafen). + 2. **No Join/Leave Voice button (parity gap).** macOS (`MainWindowController.swift:303,767-797`) + and Windows have an explicit "Join Voice"/"Leave Voice" button; iOS has only an icon mic + toggle. Add an explicit button. **Screen-audio button deferred** to the ReplayKit Broadcast + Upload Extension milestone (PROGRESS.md line below) — needs the extension to actually function. + 3. **Limited audio input/output options (feature gap).** Current `SettingsView.swift:41-60` uses + miniaudio's `vc_list_devices` which on iOS returns ~2 entries ("iPhone mic"/"Default"). + miniaudio does NOT touch `AVAudioSession` on iOS — it opens the current default route via + CoreAudio (AudioUnit/AudioQueue) and that's it. All iOS audio routing (input port selection, + mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) must be + driven manually from Swift via `AVAudioSession` *before* the core opens its device. The + current `AudioSessionManager.swift:13-15` uses `.voiceChat` + `.allowBluetooth`, which forces + HFP (mono 8/16 kHz + heavy output processing) whenever a Bluetooth headset is connected — + this is why stereo output is degraded with BT headsets. + + **Plan (decisions locked with user 2026-06-19):** + - **Part A — Channel-id sync (bug fix, smallest, unblocks mic button):** `SessionState` gains + `syncSelfChannel()` (finds self in `users`, sets `currentChannelId = user.channelId`); called + from `init`, `.channelList`, `.userJoined`, `.userLeft`, `.userUpdated`, `.joinResult`. Add + `applyServerMuteState(muted:deafened:)` (mirror macOS lines 693-700); call from `.userUpdated`. + - **Part B — Join/Leave Voice button:** Replace the icon-only mic toggle in `VoiceControlsView` + with an explicit "Join Voice"/"Leave Voice" button (mirror macOS `micToggleButton`). Keep PTT + path, mute/deafen, disconnect, level meter. No screen-audio button (deferred). + - **Part C — iOS audio routing layer (Swift, new `IOSAudioRouter.swift`):** Drive selection via + `AVAudioSession` before the core opens its device. Input enumeration via + `availableInputs` → ports (builtInMic/bluetoothHFP/headsetMic/usbAudio/airPlay); for builtInMic + walk `port.dataSources` → expose **orientation** (front/back/top/bottom) + **polar patterns** + (`supportedPolarPatterns`: omni/cardioid/subcardioid/bidirectional). Apply via + `setPreferredInput`/`setPreferredDataSource`/`setPreferredPolarPattern`/ + `setPreferredInputNumberOfChannels(2)` for stereo. Bluetooth mode as category options: "BT HFP + voice" (`.allowBluetooth`, mono voice, BT mic) / "Built-in Mic + BT A2DP stereo" + (`.allowBluetoothA2DP` only — stereo output, built-in mic, **no HFP processing**) / "Built-in + Mic + Speaker" (neither). Mic processing mode: "Standard" (`.voiceChat`, AEC/AGC/HPF on) / + "Raw / Studio" (`.measurement`, all processing off — **allowed always with a warning** when + output route is the speaker, echo risk since no AEC). Output: read-only `currentRoute.outputs` + display + AirPlay via `AVRoutePickerView`. Persist choices in `UserDefaults`; re-apply on route + changes. Note: Voice Isolation / Wide Spectrum (iOS 17+/18+) are user-toggleable in Control + Center for `.voiceChat` apps — surface as a hint, not a programmatic toggle. + - **Part D — Core stereo-mic capture (C ABI change, append-only):** `AudioParams.capture_channels` + is hardcoded to 1 (`audio_engine.h:85`); encode path already stereo-capable (proven by WASAPI + loopback stereo work, 2026-06-17 entry below). New C ABI: `vc_result + vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels);` (1 or 2) — new + setter, not a struct change, keeps `vc_stream_desc` stable. `client.{h,cpp}`: `set_capture_channels` + → per-stream, flows into `AudioParams.capture_channels` before `AudioEngine::start` for MIC kind. + `client.cpp` `on_capture_frame`: mic path with `channels==2` reuses the existing stereo encode + branch. `audio_engine.cpp`: capture device already opens with `p.capture_channels` — just needs + the value to propagate. Swift `VoiceCatCore`: add `setCaptureChannels(streamId:channels:)`; + `IOSAudioRouter` calls it when user picks stereo built-in mic. + - **Part E — Settings UI rework:** Audio Input section (input port picker → if builtInMic, show + orientation + polar pattern sub-pickers + mic mode Standard/Raw + channels Mono/Stereo); Audio + Output section (bluetooth mode HFP/A2DP/Off + current route read-only + AirPlay button); Voice + section (existing: input mode, VAD threshold, PTT). Replace the current miniaudio-based device + picker (`SettingsView.swift:41-60`) with the AVAudioSession-derived tree. + - **Part F — Docs + deployment target:** Raise iOS deployment target to **18.0** (user decision — + unlocks newest audio APIs; `project.pbxproj` + `Package.swift` `.iOS(.v18)`). Update + `docs/tech-stack.md` §2 (iOS audio routing via AVAudioSession, stereo mic, deployment 18.0), + `docs/voice.md` (new iOS mic capture subsection), `docs/roadmap.md` (iOS pending list), + `docs/architecture.md` §4 (Swift binding notes — iOS audio routing in Swift layer, core + stereo-mic via new ABI setter), `PROGRESS.md` (this entry + next-action pointer). + - **Implementation order:** A (channel-id sync, testable immediately) → B (Join/Leave Voice) → + D (core stereo-mic + ABI, verify with `ctest --preset dev`) → C (Swift routing layer, depends + on D) → E (Settings UI, depends on C) → F (docs, same commit as work). + - **Verification:** `xcodebuild -target VoiceCatiOS -sdk iphonesimulator -configuration Debug + build` → BUILD SUCCEEDED; `swift test` green + stereo-mic capture round-trip test; `ctest + --preset dev` green (core changed in D); manual (simulator/device): join channel → Join Voice + works → input picker shows built-in mic sub-options (orientation/polar pattern) → toggle Raw + mode (warning appears on speaker) → toggle Bluetooth A2DP stereo output → toggle stereo capture. + - **Files to touch:** + - iOS Swift: `SessionState.swift`, `AudioSessionManager.swift` (+ new `IOSAudioRouter.swift`), + `Views/VoiceControlsView.swift`, `Views/SettingsView.swift`, `VoiceCatiOSApp.swift`, + `VoiceCatiOS.xcodeproj/project.pbxproj` (iOS 18.0). + - Swift core: `Sources/VoiceCatCore/VoiceCatClient.swift` (`setCaptureChannels`), + `Sources/VoiceCatCore/Models.swift` (input-port model), `Package.swift` (`.iOS(.v18)`). + - Core C++: `core/include/voicecat.h` (`vc_set_capture_channels`), `core/src/voicecat.cpp`, + `core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`. + - Docs: `docs/tech-stack.md`, `docs/voice.md`, `docs/roadmap.md`, `docs/architecture.md`, + `PROGRESS.md`. + - Tests: `clients/apple/Tests/VoiceCatCoreTests/`, `tests/` (stereo-mic encode test). + - **Why miniaudio can't do this alone:** On iOS, miniaudio uses CoreAudio (AudioUnit/AudioQueue) + to open the *current default route* for PCM I/O — it never touches `AVAudioSession`. + `ma_context_get_devices` on iOS returns a near-empty list (iOS doesn't expose a real CoreAudio + device list like macOS does). All iOS audio routing (input ports, data sources, polar patterns, + HFP/A2DP, measurement mode, `preferredInputNumberOfChannels`) must be driven from Swift via + `AVAudioSession` *before* miniaudio opens its device. miniaudio just reads whatever route + AVAudioSession has established. + - **Done:** **iOS SwiftUI client — `VoiceCatiOS`** (2026-06-19). Full SwiftUI app at `clients/apple/iOS/VoiceCatiOS.xcodeproj`. Mirrors the macOS AppKit and Windows WinForms clients feature-for-feature: saved server list (JSON + Keychain passwords via App Group