fix(ios-audio): unify iOS audio onto one always-external AVAudioEngine
The iOS audio path was a hybrid: Voice-Chat-class presets ran a native VPIO AVAudioEngine (core external) while Stereo/Studio/A2DP presets ran the core's miniaudio devices. Nearly every "no input / no output / both" bug lived in the seam between the two paths — the lingering miniaudio capture unit fighting VPIO, the audioRestart ordering dance, the route-change "glitching" loop, stereo<->mono stickiness, and "can't hear anyone". Switching presets/routes mid-call routinely dropped a direction. Drive ALL iOS audio through one AVAudioEngine with the core fully external at all times: setExternalPlayback(1) once at connect, every MIC stream external_feed=1, mic via vc_stream_feed_pcm, playback via vc_set_mixed_output_sink (drained by an always-on AVAudioSourceNode so remote audio plays before joining voice). VPIO + AGC toggle per preset. Every preset/route/interruption change funnels through one deterministic Swift-only reconfigure (stop -> apply session config -> rebuild -> start) — no second path to hand off to, so a change can't drop a direction. - IOSVoiceProcessingEngine.swift -> IOSAudioEngine: always-on source-node playback, conditional mic tap, VPIO/AGC; one rebuild() backing startListening/stop/startMic/stopMic/reconfigure/setCaptureChannels. - IOSAudioRouter: 7 presets -> 4 (Voice Chat / Stereo Mic / Mono Mic / Advanced); persisted voiceProcessingEnabled + agcEnabled; setters call IOSAudioEngine.reconfigure() instead of audioRestart/reconcileVoicePath. - AudioSessionManager slimmed; SessionState mic lifecycle collapsed; AppState wires external playback + listening at connect, stop at disconnect; SettingsView shows 4 presets + Advanced VPIO/AGC toggles. No core/ABI/test changes — relies on the already-shipped external API (test_external_pcm, test_external_playback). xcodebuild iOS device Debug BUILD SUCCEEDED. Updates docs/voice.md §8 and PROGRESS.md.
This commit is contained in:
@@ -230,37 +230,46 @@ Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth
|
||||
in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
|
||||
downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
|
||||
genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
|
||||
- **iOS mic capture:** all iOS audio routing is driven from Swift via `AVAudioSession` by the
|
||||
`IOSAudioRouter` singleton *before* the core (miniaudio) opens its device — miniaudio does
|
||||
NOT touch `AVAudioSession` on iOS. Input port selection (`availableInputs`), built-in mic
|
||||
orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns
|
||||
(`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic processing mode
|
||||
(`.voiceChat` = Standard with AEC/AGC/HPF, or `.measurement` = Raw/Studio with all processing
|
||||
off), Bluetooth mode (`.allowBluetoothHFP` HFP voice vs `.allowBluetoothA2DP` stereo output
|
||||
vs neither), and stereo capture (`.stereo` polar pattern + `setPreferredInput` +
|
||||
`setInputDataSource` → `vc_set_capture_channels`) are all set from Swift. The core then
|
||||
opens whatever route AVAudioSession has established. When the user changes audio settings
|
||||
mid-session, `IOSAudioRouter` suspends the core's devices (`vc_audio_suspend`), reconfigures
|
||||
`AVAudioSession`, then restarts the devices (`vc_audio_restart`) so they reopen against the
|
||||
new route — mirroring TeamTalk5's `closeSoundDevices`/`initSoundInputDevice`/
|
||||
`initSoundOutputDevice` pattern.
|
||||
- **iOS voice processing (AEC/NS/AGC) — native VPIO path.** Real iOS echo cancellation, noise
|
||||
suppression and AGC are provided ONLY by Apple's **Voice-Processing I/O audio unit (VPIO)**,
|
||||
*not* by the `AVAudioSession` mode alone. The core uses miniaudio's plain `RemoteIO` audio
|
||||
units, which never engage VPIO — so `.voiceChat` mode by itself yields no AEC. For VPIO to
|
||||
cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
|
||||
played-back signal from the mic), so on the AEC presets (Voice Chat / Bluetooth Headset HFP /
|
||||
Wired Headset) the Swift layer runs a native `AVAudioEngine` with
|
||||
`inputNode.setVoiceProcessingEnabled(true)` and the core runs in **external mode**:
|
||||
- **Mic:** the MIC stream is started with `vc_stream_desc.external_feed=1`; the VPIO input tap
|
||||
feeds processed mic PCM via `vc_stream_feed_pcm`. The core skips its hardware capture device
|
||||
(`AudioParams.external_capture`).
|
||||
- **Playback:** `vc_set_external_playback(1)` makes the core skip its hardware playback device;
|
||||
a mixer-timer thread drives decode+mix on a ~20 ms cadence and delivers the FINAL mixed PCM
|
||||
via `vc_set_mixed_output_sink`. The Swift engine renders that through the VPIO output, so
|
||||
VPIO has its echo-cancellation reference signal.
|
||||
The Stereo Mic / Studio / A2DP presets keep the miniaudio path (they want raw / stereo /
|
||||
no-AEC routing that VPIO can't provide — VPIO forces mono).
|
||||
- **iOS audio — one path, always external.** On iOS the core **never opens a miniaudio device**:
|
||||
a single `AVAudioEngine` (`IOSAudioEngine`) drives *both* directions, and the core runs fully
|
||||
external for the whole connection. This is the single most important property of the iOS audio
|
||||
stack — there is no second (miniaudio) path to switch to, so a preset/route change cannot leave
|
||||
one direction dropped. The single ordering rule is: `vc_set_external_playback(1)` is set **once
|
||||
at connect** (before the session is activated or any remote stream arrives), and every MIC
|
||||
stream is started with `vc_stream_desc.external_feed=1`.
|
||||
- **core → speaker:** the core's mixer-timer thread decodes+mixes on a ~20 ms cadence and
|
||||
delivers the FINAL mixed PCM via `vc_set_mixed_output_sink`; an `AVAudioSourceNode` pulls it
|
||||
from a lock-free ring and renders it. This runs the whole time we are connected, so remote
|
||||
audio plays even before the user joins voice (kills the "can't hear anyone" race).
|
||||
- **mic → core:** when the mic is active a tap on the engine's input node converts to 48 kHz
|
||||
int16 (`vc_set_capture_channels` decides mono/stereo) and calls `vc_stream_feed_pcm`.
|
||||
- **iOS routing** is still driven from Swift via `AVAudioSession` by the `IOSAudioRouter`
|
||||
singleton — miniaudio never touches `AVAudioSession` on iOS. Input port selection
|
||||
(`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom),
|
||||
polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic
|
||||
processing mode (Standard vs `.measurement` Raw), Bluetooth mode (`.allowBluetoothHFP` HFP voice
|
||||
vs `.allowBluetoothA2DP` stereo output vs neither), and stereo capture (`.stereo` polar pattern
|
||||
+ `setPreferredInput` + `setInputDataSource` → `vc_set_capture_channels`) are all set from
|
||||
Swift. Any preset / route / interruption change funnels through one deterministic, Swift-only
|
||||
rebuild: `IOSAudioEngine` stops, `IOSAudioRouter.applyConfiguration()` re-applies the
|
||||
`AVAudioSession`, the graph is rebuilt against the new route, and the engine restarts. No
|
||||
`vc_audio_restart`/`vc_audio_suspend` dance is needed for routing (the core has no hardware
|
||||
devices to reopen) — this is the spirit of TeamTalk5's "close then re-init sound devices", but
|
||||
entirely inside the Swift engine.
|
||||
- **iOS voice processing (AEC/NS/AGC) — native VPIO.** Real iOS echo cancellation, noise
|
||||
suppression and AGC come ONLY from Apple's **Voice-Processing I/O audio unit (VPIO)**, which
|
||||
`inputNode.setVoiceProcessingEnabled(true)` enables; for it to cancel echo it must own BOTH the
|
||||
mic capture and the playback — which the unified engine already does. VPIO forces **mono**, so
|
||||
it is engaged only when the active config wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`:
|
||||
mono + standard + non-A2DP + the user's master toggle). iOS exposes no per-stage VPIO control,
|
||||
so the Advanced UI offers exactly two switches: a master **Voice Processing** (AEC + NS bundled)
|
||||
and **AGC** (`isVoiceProcessingAGCEnabled`).
|
||||
- **iOS presets** (`IOSAudioRouter.AudioPreset`): **Voice Chat** (VPIO mono, system output incl.
|
||||
HFP/wired), **Stereo Mic** (internal stereo built-in mic regardless of output, A2DP-capable, no
|
||||
VPIO), **Mono Mic** (internal mono built-in mic regardless of output, A2DP-capable, no VPIO),
|
||||
and **Advanced** (every knob manual). A2DP output requires an internal-mic preset (the Bluetooth
|
||||
device is output-only); the Stereo/Mono Mic presets fall back to the built-in speaker when no
|
||||
external output is connected (`applyA2dpSpeakerFallback`).
|
||||
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
|
||||
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
|
||||
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
|
||||
|
||||
Reference in New Issue
Block a user