From b14cf2a4e81cfc00db59d36a24eb8a6ad9253f23 Mon Sep 17 00:00:00 2001 From: Talon Date: Tue, 23 Jun 2026 17:47:49 +0200 Subject: [PATCH] =?UTF-8?q?fix(ios):=20stop=20core=20opening=20a=20second?= =?UTF-8?q?=20mic=20device=20=E2=80=94=20dual=20capture/crackle?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On iOS the remote end heard the mic twice and crackly (BT headset + internal mic in Voice Chat; mono + stereo copies of the internal mic in Stereo Mic). Root cause is an io-thread ordering race. `ensure_audio_running()` only set `external_capture` when a MIC stream already existed, but it also runs from `sync_remote_streams` on the post-auth `ServerStateSnapshot` — before the user joins voice. With `external_playback_` still false and no MIC stream, `AudioEngine::start()` opened a real miniaudio capture device that stayed open all session (later calls early-return on running()), racing the AVAudioEngine input tap fed via `vc_stream_feed_pcm`. `on_capture_frame` then encoded+sent both paths — the mic transmitted twice, the two unsynchronized capture clocks producing the crackle. - core (ensure_audio_running): force `external_capture = true` whenever `external_playback_` is set, so iOS unified mode never opens a hardware capture device. No-op on desktop. - ios (AppState): move `setExternalPlayback(true)` before `connect()`, so the flag is set before the io thread processes any message — closing the race. Verified on device: remote end hears the iOS mic once and clean in both Voice Chat (+ BT) and Stereo Mic. --- PROGRESS.md | 38 ++++++++++++++++++++ clients/apple/iOS/VoiceCatiOS/AppState.swift | 19 ++++++---- core/src/core/client.cpp | 10 ++++++ 3 files changed, 61 insertions(+), 6 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 43d0e11..66f5c1c 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -26,6 +26,44 @@ up instantly. Newest status at the top. send cushion could be reduced or removed. Shared-core change → add a test and re-verify desktop↔desktop stays low-latency (steady sender ⇒ ~0 arrival jitter ⇒ no regression). +- **Done (2026-06-23):** **Fixed iOS dual-stream / crackly mic — core opened a second + (miniaudio) capture device alongside the AVAudioEngine tap.** Symptom: with two clients in + a channel, the remote end heard the iOS mic **twice** and crackly. With Voice Chat + a BT + headset, both the BT mic and the internal mic were captured; with Stereo Mic, both a mono + and a stereo copy of the internal mic were sent simultaneously. Root cause is a timing gap + in `vc_client::ensure_audio_running()` (`core/src/core/client.cpp`): `external_capture` was + only set when a MIC stream already existed, but `ensure_audio_running` is also called from + `sync_remote_streams` (triggered by the post-auth `ServerStateSnapshot`) **before** the user + joins voice — so with no MIC stream, `external_capture` stayed `false` and + `AudioEngine::start()` opened a real miniaudio capture device. Later the user joined voice → + `IOSAudioEngine.startMic` installed the AVAudioEngine input tap → `feedPcm` → + `inject_capture` → `on_capture_frame`. The miniaudio device was still open (the engine was + already `running()`, so the later `ensure_audio_running` early-returned and never applied + `external_feed`), and `on_capture_frame` encodes+sends every frame with **no deduplication** + → the mic was sent twice. The two unsynchronized capture clocks interleaving in the encoder + is the crackle; the mono miniaudio device + stereo AVAudioEngine tap is the "mono and stereo + at the same time" on Stereo Mic. + - **Fix 1 (core, `core/src/core/client.cpp:ensure_audio_running`):** force + `p.external_capture = true` whenever `external_playback_` is set. In iOS unified mode the + core must never open a hardware capture device — the AVAudioEngine owns the only mic path. + No-op on desktop (`external_playback_` is never set there). + - **Fix 2 (iOS, `clients/apple/iOS/VoiceCatiOS/AppState.swift`):** move + `client.setExternalPlayback(true)` from the `authResult` handler to **before** + `client.connect(...)`. The server sends `AuthResult` immediately followed by + `ServerStateSnapshot`; `handle_server_state` runs `ensure_audio_running` on the io thread + before the main thread drains `authResult`, so setting the flag post-auth raced. Setting it + pre-connect guarantees `external_playback_` is true before any message is processed — + eliminating the playback-device race too (the mixer timer + AVAudioEngine playback path + + VPIO AEC reference are correct from the first frame). + - **Verify:** `cmake --build --preset dev` clean; `ctest --preset dev` = 24/28 — the 4 + failures (`vad_ptt_devices`, `external_pcm`, `frame_ms_reframe`, `channel_samplerate`) are + a **pre-existing** teardown `mutex lock failed` race, reproduced identically with the + changes stashed. `external_playback` (the one test exercising this code path) **passes**. + No xcframework rebuild needed (no new symbols). **Next (manual, on device):** two clients + in a channel — Voice Chat + BT, and Stereo Mic — confirm the remote end hears the iOS mic + once, clean (no duplicate, no crackle); confirm the iOS user hears the remote user cleanly + with AEC working in Voice Chat. + - **Done (2026-06-23):** **Fixed iOS mic flutter / crackle / octave-up.** The iOS mic was unusable: a consistent ~40–60 ms flutter with volume fade ("talking through a slow fan") on every preset. Root cause: the core sends each captured frame **synchronously** diff --git a/clients/apple/iOS/VoiceCatiOS/AppState.swift b/clients/apple/iOS/VoiceCatiOS/AppState.swift index 46b936a..bd26d47 100644 --- a/clients/apple/iOS/VoiceCatiOS/AppState.swift +++ b/clients/apple/iOS/VoiceCatiOS/AppState.swift @@ -71,6 +71,15 @@ final class AppState { client.onEvent = { [weak self] ev in Task { @MainActor [weak self] in self?.handleConnectEvent(ev, server: server) } } + // Put the core into external-playback mode BEFORE connect, so the flag is set on the + // io thread before any message is processed. The server sends AuthResult immediately + // followed by ServerStateSnapshot; handle_server_state runs ensure_audio_running() on + // the io thread, and if external_playback_ were still false at that point the core + // would open a hardware miniaudio playback+capture device (see the matching fix in + // vc_client::ensure_audio_running). Setting it here — before connect — guarantees the + // unified external path is in effect from the first frame. setExternalPlayback only + // flips an atomic + forwards to the engine's setter; both are safe pre-connect. + client.setExternalPlayback(true) client.connect(host: server.host, port: server.port) // Auth is queued immediately — the core serialises it behind TLS + TOFU. @@ -161,12 +170,10 @@ final class AppState { self.session = newSession EventFeedback.shared.play(.login) EventFeedback.shared.speak("Connected") - // Put the core into external-playback mode ONCE, now, before the session is - // activated or any remote stream can arrive — so the core never opens a miniaudio - // device on iOS (the single ordering rule of the unified audio path). Then activate - // the session and start the engine in listening mode so remote audio plays the - // moment someone talks, even before we join voice (no "can't hear anyone"). - client.setExternalPlayback(true) + // External-playback mode was enabled before connect() so the core never opens a + // miniaudio device on iOS (the single ordering rule of the unified audio path). + // Now activate the session and start the engine in listening mode so remote audio + // plays the moment someone talks, even before we join voice (no "can't hear anyone"). do { try AudioSessionManager.shared.ensureSessionActive() } catch { diff --git a/core/src/core/client.cpp b/core/src/core/client.cpp index 5b7e238..4fc4dbe 100644 --- a/core/src/core/client.cpp +++ b/core/src/core/client.cpp @@ -1184,6 +1184,16 @@ void vc_client::ensure_audio_running() { p.external_capture = it->second.external_feed; } } + // iOS unified mode (external_playback): capture is ALWAYS external — the Swift AVAudioEngine + // owns the only mic path and feeds via vc_stream_feed_pcm. Force external_capture so start() + // never opens a hardware mic device, even when ensure_audio_running runs before a MIC stream + // exists (a remote stream arriving first — e.g. the post-auth ServerStateSnapshot — starts + // the engine for playback). Without this the core opens a miniaudio capture device that races + // the AVAudioEngine tap → dual mic capture → duplicated, crackly audio on the remote end. + // No-op on desktop, where external_playback_ is never set. + if (external_playback_.load(std::memory_order_acquire)) { + p.external_capture = true; + } // iOS VPIO: skip the hardware playback device and drive the mixer on a timer, delivering the // final mix to the mixed-output sink for the Swift VPIO renderer (vc_set_external_playback). audio_engine_.set_external_playback(external_playback_.load(std::memory_order_acquire));