Sleep-paced threads stall for hundreds of milliseconds when iOS coalesces a
backgrounded app's wakeups, and the deadline resets that discarded the deficit
left the capture backlog queued until its ring overflowed: regular dropouts
that worsen the longer the app stays backgrounded. Pace both 20 ms hands-offs
from the AVAudioSourceNode render callback instead (mix via Audio.RunCycle
with deviceClockedAudio, capture handoff from the ring), wake the UDP sender
on a queue signal instead of a 1 ms poll, and let stalled consumers drop their
backlog to the buffer target instead of ratcheting it.
Recovery from the 99937c9 audio-device-change commit broadened the
route-change recovery set to "everything except categoryChange /
routeConfigurationChange", which added .override. But .override is
fired by our own applyA2dpSpeakerFallback() -> overrideOutputAudioPort,
which recoverAudio() calls on every recovery. On an A2DP preset
(Stereo Mic / Mono Mic), disconnecting AirPods ping-ponged:
oldDeviceUnavailable -> recoverAudio -> applyA2dpSpeakerFallback
-> overrideOutputAudioPort(.speaker) -> .override routeChange
-> recoverAudio -> applyConfiguration (setCategory resets override)
-> applyA2dpSpeakerFallback -> overrideOutputAudioPort -> .override -> ...
Each iteration also rebuilt the AVAudioEngine via reconfigure() ->
rebuild() -- the audible reinitialize loop + CPU spin. Voice Chat and
Built-in Mic + Speaker were unaffected (applyA2dpSpeakerFallback
early-returns for non-A2DP modes, so no overrideOutputAudioPort call).
Two-part fix (pure Swift iOS-app target; no C ABI / proto / docs changes):
1. AudioSessionManager.handleRouteChange: added .override to the skip
list alongside .categoryChange / .routeConfigurationChange. .override
is only ever fired by our own overrideOutputAudioPort call, so
treating it as a recovery reason is the loop by definition. The
AVAudioEngineConfigurationChange observer in IOSVoiceProcessingEngine
remains as the backstop if an override ever actually stops the engine.
2. IOSAudioRouter.applyA2dpSpeakerFallback: made idempotent via a
lastAppliedOutputOverride tracker that skips the redundant
overrideOutputAudioPort call when the desired state (.none for
external output present, .speaker otherwise) already matches. Reset
to nil at the top of applyConfiguration() (setCategory can reset the
override) and on a failed call. Defense-in-depth on top of fix 1.
Build: xcodebuild -project clients/apple/iOS/VoiceCatiOS.xcodeproj
-scheme VoiceCatiOS -destination 'generic/platform=iOS' build green
(Xcode 26.5 / iOS 18.0).
Network drops (e.g. Wi-Fi -> cellular) and audio-device plug/unplug (wired
headphones, AirPods) used to leave the iOS client in a dead/zombie state:
the engine went silent, no reconnect was attempted, and a live-session
disconnect waited 30-60 s for the C core's TCP keepalive/reaper timeout.
Reconnect (AppState.swift, SessionState.swift):
- Two-layer reconcile. Once SessionState overwrites client.onEvent at auth
success, AppState.handleConnectEvent no longer sees live-session events.
Added a weak SessionState.appState; SessionState.handleEvent .disconnected
calls appState.onLiveSessionDisconnected after the cue -- the single path
AppState learns a live session dropped. Shared teardownLiveSessionAndReconnect
snapshots lastSession, stops audio, releases session/VoiceCatClient (io-
thread join via vc_client_destroy), resets the backoff, and arms
scheduleReconnect (exponential 1s -> 30s cap, indefinite, restored on auth
success via existing TOFU_MATCHED auto-confirm + idempotent join_channel).
- NWPathMonitor now runs WHILE CONNECTED (not only mid-reconnect). On a Wi-Fi
<-> cellular interface change or path .unsatisfied it calls
proactiveReconnect: tearing the session down BEFORE the C core notices the
dead socket collapses the 30-60 s reaper wait into ~1 s + first backoff
tick. Same-interface refreshes (BSSID roams) are ignored via pathSignature.
While mid-reconnect a .satisfied path resets the backoff for a fast retry.
- User-initiated disconnect()/cancelConnect() set userInitiatedDisconnect
and cancel all reconnect state (task + monitor + lastSession + connectedServer).
Audio recovery (AudioSessionManager.swift, IOSVoiceProcessingEngine.swift):
- Intent-gated recoverAudio() replaces the narrow .oldDeviceUnavailable/
.newDeviceAvailable route-change guard; fires on every externally-initiated
route change reason except the ones we cause ourselves (.categoryChange/
.routeConfigurationChange) to avoid a notification loop. Interruption-end
now always recovers instead of only when .shouldResume is set.
- Added AVAudioEngineConfigurationChange observer on the engine so a system
self-stop after our route-change handler wins the race is caught.
- IOSAudioEngine.rebuild() does a one-shot reactivation-retry on
engine.start() failure (iOS sometimes refuses until the session is
re-reactivated -- the silent-death case).
No C ABI / voicecat.h / proto / core changes. Swift-only. iOS sim build green
via scripts/build-ios-client.sh --no-configure (Xcode 26.5 / iOS 18.0 sim).
Three bugs fixed across the full stack (proto/server/core/ABI/Win/macOS/iOS):
1. Join/Leave Voice now truly subscribes/unsubscribes from the voice plane.
Previously the button only toggled the local mic — receiving was always on
(gated by channel membership alone). Added a protocol-level voice subscription
concept: new SubscribeVoiceRequest/UnsubscribeVoiceRequest/VoiceSubscriptionResult
proto messages, User.voice_subscribed field, vc_join_voice/vc_leave_voice C ABI
functions, VC_EVENT_VOICE_STATE event, server-side voice_subscribed flag checked
by the SFU relay recipient filter, and core-client gating of remote-stream
decoder setup. All three clients rewired to subscribe+mic on Join / unsubscribe
on Leave. Text chat works regardless of voice subscription.
2. Channel edit dialog now shows the channel's actual current settings. The read
struct vc_channel was missing sort_order and audio fields — only the write
struct vc_channel_info had them. Extended vc_channel with both (additive, no
ABI break), updated the session model and list_channels marshaling to populate
them, and updated all three clients' edit callers to use actual channel info
instead of hardcoded defaults.
3. Channel parameter updates now automatically restart everyone's streams.
Previously editing a channel's audio config persisted and broadcast a
ChannelEvent::UPDATED, but no layer restarted streams — encoders/decoders are
frozen at announce time. handle_channel_event now detects audio-config changes
on the user's current channel and stop->starts each active local stream. The
server reads the updated config on re-announce; peers wire up fresh decoders
at the new ssrc.
All 29 CTest tests pass; Windows DLL + C# client build clean. Apple clients not
yet compile-verified (Windows environment).
PTT was focus-scoped (WinForms KeyDown/KeyUp) so it died the moment the
window lost focus. Add an optional system-wide path using the Raw Input
API (RegisterRawInputDevices + WM_INPUT with RIDEV_INPUTSINK) instead of
a WH_KEYBOARD_LL low-level hook -- the latter is the keylogger pattern AV
heuristics flag, which is worse for our unsigned MinGW binary. Raw Input
involves no DLL injection or global hook and passes keys through.
- New VoiceCat.App/Native/RawInput.cs: P/Invoke + structs; register the
keyboard sink, decode WM_INPUT to vkey/up-down, GetAsyncKeyState helper.
- MainForm overrides OnHandleCreated/OnHandleDestroyed/WndProc to manage
the sink and route WM_INPUT to PTT; gates the focus-scoped KeyDown/KeyUp
off when system-wide is on; makes the Deactivate force-release
conditional; adds a GetAsyncKeyState watchdog on the pump timer so a
missed key-up (RDP/lock-screen) cannot leave PTT stuck.
- VoiceSettings.SystemWidePtt (default ON) + system-wide checkbox in the
Audio settings PTT section.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-listener noise-reduction toggle (vc_set_remote_stream) did nothing
on Windows/macOS/iOS. The decode loop gated the RNNoise pass on
dec_channels == 1 as a proxy for "this stream is voice" (assuming
stereo => screen-share). The stereo-mic capture commit broke that: a stereo
mic with send-side NR off transmits stereo Opus, so the receiver decoded
two channels and skipped NR entirely. gain/mute have no channel guard, which
is why only NR appeared broken.
Thread the stream kind through init_recv_stream into RemoteStream::is_voice
(set from si.kind() == STREAM_MIC), gate receive NR on is_voice instead of
channel count, and fold a stereo voice frame to mono -> denoise -> duplicate
back across both channels in place (symmetric with the send-side downmix;
RNNoise is mono-only). Screen-audio shares are never denoised.
New test test_recv_noise_reduction drives AudioEngine and asserts a stereo
voice stream's noise floor collapses with NR on (RMS 1046 -> 0.1) while a
screen-audio share stays unchanged. ctest --preset dev green 29/29.
Docs: voice.md section 10. Shared-core fix; clients need only a rebuild.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both desktop mics were hard-mono: the core defaults capture_channels=1 and
neither client ever called vc_set_capture_channels (only iOS did). Add a
persisted "Stereo microphone" toggle to each client's Audio settings, applied
when the mic stream starts and live via vc_set_capture_channels + vc_audio_restart.
Expose both ABI calls in the Windows interop; the macOS wrapper already had them.
Core fix: encode_and_send_frame now folds a stereo mic frame to mono on a mono
channel - previously the channels==2 branch encoded interleaved L/R directly even
on a mono channel, feeding a mono opus_encode 2x its samples (wrong pitch/garbage).
Real stereo still only reaches the wire on a stereo channel; on a mono channel the
mic is cleanly downmixed.
Test: test_stereo_mic_mono_channel. ctest --preset dev green (28/28). Docs: voice.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On iOS the remote end heard the mic twice and crackly (BT headset + internal
mic in Voice Chat; mono + stereo copies of the internal mic in Stereo Mic).
Root cause is an io-thread ordering race. `ensure_audio_running()` only set
`external_capture` when a MIC stream already existed, but it also runs from
`sync_remote_streams` on the post-auth `ServerStateSnapshot` — before the user
joins voice. With `external_playback_` still false and no MIC stream,
`AudioEngine::start()` opened a real miniaudio capture device that stayed open
all session (later calls early-return on running()), racing the AVAudioEngine
input tap fed via `vc_stream_feed_pcm`. `on_capture_frame` then encoded+sent
both paths — the mic transmitted twice, the two unsynchronized capture clocks
producing the crackle.
- core (ensure_audio_running): force `external_capture = true` whenever
`external_playback_` is set, so iOS unified mode never opens a hardware
capture device. No-op on desktop.
- ios (AppState): move `setExternalPlayback(true)` before `connect()`, so the
flag is set before the io thread processes any message — closing the race.
Verified on device: remote end hears the iOS mic once and clean in both Voice
Chat (+ BT) and Stereo Mic.
The iOS mic was unusable — a consistent ~40-60ms flutter + volume fade
('slow fan') on every preset. The core sends each captured frame
synchronously (no send pacer), so packet cadence == capture cadence, and
the receiver's playout keeps near-zero buffering by design (its jitter
estimate keys off the regular sender timestamp, so it's blind to arrival
jitter). That's smooth only for a steady sender (desktop miniaudio =
steady 20ms); the iOS AVAudioEngine tap delivers ~2 frames per ~40ms
callback -> bursty -> receiver underruns -> PLC fade.
Fix (iOS-only): the mic tap writes converted 48kHz int16 to an SPSC ring;
a 20ms feed pump drains it and calls feedPcm at a steady cadence. The pump
primes a small prebuffer cushion (3 frames ~60ms, self-healing up to
~120ms on underrun) before releasing, so the tap's bursts can't drain it
to empty. Never reads a partial frame (read consumes what it returns ->
partials were the crackle), and rebuilds with the current channel count
each rebuild() (a frozen count fed mono-as-stereo = octave-up on a
Stereo->Voice Chat switch).
Trade-off: ~60-120ms added mic-send latency, unavoidable when de-bursting
for a near-zero-buffer receiver. PROGRESS.md notes the proper follow-up:
make the jitter buffer measure real RFC-3550 arrival jitter so the
receiver absorbs bursts itself.
Verified: xcodebuild Debug BUILD SUCCEEDED (iOS Simulator, arm64).
Both VoiceCatMac and VoiceCatiOS failed to link with 'Undefined symbols
for architecture arm64: _rnnoise_create/_rnnoise_destroy/_rnnoise_process_frame'.
Root cause: build-xcframework.sh merged vcpkg deps into the fat static lib but
NOT the locally-built vendored librnnoise.a (a CMake target from
third_party/rnnoise/ linked privately into voicecat via VOICECAT_HAS_NS — not a
vcpkg dep). The xcframework had been rebuilt after the RNNoise commit but still
omitted the symbols, so every slice's libvoicecat-fat.a referenced _rnnoise_*
with no defining object. The iOS slices were also stale (pre-rnnoise) and
absent from the xcframework entirely.
Fix: build-xcframework.sh now also collects .a files from build/<preset>/lib/
(excluding libvoicecat*) so vendored CMake-target static libs like librnnoise.a
get merged in. Future-proof: any new vendored static-lib target landing in
build/<preset>/lib/ is picked up automatically. README 'Fat static library'
section updated.
Verify: rebuilt VoiceCatCore.xcframework --all → all 3 slices (macos-arm64,
ios-arm64, ios-arm64-simulator) now carry the 10 _rnnoise_* symbols; fat lib
~30 MB → ~33 MB. xcodebuild Debug BUILD SUCCEEDED for VoiceCatMac, VoiceCatiOS
(iphonesimulator arm64), and VoiceCatiOS (iphoneos arm64). No core/ABI/proto
changes — xcframework artifact + build script only.
The two-sided NR plumbing (RemoteStream::recv_ns + the per-listener
vc_set_remote_stream noise_reduction toggle) was wired but inert:
ApmProcessor::create() returned a no-op passthrough, because the
originally-planned webrtc-audio-processing has no working Windows/macOS
build. Drop in RNNoise as the real backend behind the same ApmProcessor
interface, lighting up both NR paths.
- Vendor RNNoise (BSD-3 + CC0) at third_party/rnnoise/ — the vcpkg port
is !windows !arm, so it can't cover our primary targets. Shrunk int8
model (78MB -> 11.7MB via upstream scripts/shrink_model.sh), built as a
standalone C static lib with no RTCD (portable scalar path on x86,
auto-NEON on arm64) under -DDISABLE_DEBUG_FLOAT. Model is baked in
(rnnoise_create(NULL)); no runtime file.
- New RnnoiseProcessor (core/src/audio/apm_processor.cpp) selected by
ApmProcessor::create() when VOICECAT_HAS_NS. Mono/48kHz/480-sample;
our clock is fixed 48kHz and Opus frame sizes are multiples of 480, so
no resampling. RT-safe: allocates at construction, lock-free in the
capture/playback callbacks.
- Receive-side: lit up via the factory; gated to mono streams (a stereo
stream is a screen-audio share, not voice).
- Send-side (new): vc_set_input_noise_reduction(client, enable) ABI +
vc_client::mic_ns_, run before input gain/VAD in on_capture_frame. A
stereo mic is downmixed to mono ONLY when NR is on — with NR off a
stereo mic keeps full stereo (never collapse mic quality unasked).
- Enable C as a project language for the vendored lib.
- New noise_suppression test: white noise through ApmProcessor::create()
drops ~99.9% RMS. ctest --preset dev green, 28/28. windows-client DLL
builds clean with vc_set_input_noise_reduction exported, system-only deps.
- Docs synced: voice.md §10, tech-stack.md §1/§5, third_party/README.md,
vcpkg.json note, PROGRESS.md, CLAUDE.md.
Client on/off UI toggles (Windows/macOS/iOS) are the remaining follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lets a user transmit a second hardware input device (e.g. line-in / aux)
alongside the mic, with its own device picker and volume, from Audio Settings.
No core/ABI/proto changes: the aux is a VC_STREAM_AUX_DEVICE stream started
with external_feed=1 and fed via vc_stream_feed_pcm (the same external-feed
pipeline screen-audio uses). Per-kind local_streams_ already allows mic +
screen + one aux to coexist; volume is a client-side gain multiply (the core's
vc_set_input_gain is mic-only/global). Aux is always-on (core never gates
AUX_DEVICE on VAD/PTT) and is tied to the voice session.
Windows: new Audio/InputDeviceCapture.cs (WASAPI shared-mode capture from a
real input endpoint + capture-endpoint enumeration); aux section in
AudioSettingsForm.cs; lifecycle in MainForm.cs; persistence in VoiceSettings.cs.
macOS: new Audio/InputDeviceCapture.swift (AVAudioEngine input-node tap pinned
to the chosen Core Audio device + device enumeration by stable UID); aux section
in SettingsWindowController.swift; lifecycle + UserDefaults persistence in
MainWindowController.swift; file registered in project.pbxproj.
Windows verified (C# solution builds clean; aux confirmed working). macOS build
+ E2E pending a Mac.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Input mode (VAD/PTT/Always-On), VAD threshold, and the new mic gain were
applied to the core + UI but never saved, so every relaunch reset to VAD
defaults. Each client now persists them and re-applies on connect:
- iOS: UserDefaults (SessionState.loadAndApplyVoiceSettings + setter writes)
- macOS: UserDefaults via MainWindowController didSet + loadPersistedAudioSettings
(settings window also restores the VAD slider from the stored threshold)
- Windows: new Models/VoiceSettings.cs (JSON at %AppData%\VoiceCat\voice.json,
mirrors FeedbackSettings) loaded/applied in MainForm
Add global send-side mic gain API vc_set_input_gain (applied to MIC PCM in
on_capture_frame before the VAD gate, clamped to int16) + Swift/C# bindings,
and a 0-300% (default 100%) mic-volume slider on all three clients.
Fix iOS chat: ChatView called sendText(scope:.channel) with no targetId (0),
so channel messages went nowhere; now passes session.currentChannelId.
Fix iOS per-user tuning for VoiceOver: the tuning sheet was long-press
.contextMenu only (invisible to VoiceOver); UserRow now also exposes the same
buttons via .accessibilityActions (no visual change).
Verified: core builds clean; ctest 24/27 (3 pre-existing teardown crashes,
reproduced with changes stashed); VoiceCatMac + VoiceCatiOS (arm64 sim) build
SUCCEEDED; VoiceCat.Interop dotnet build succeeded. Windows App not built
(WinForms can't build on macOS) — follows existing patterns.
The iOS audio path was a hybrid: Voice-Chat-class presets ran a native
VPIO AVAudioEngine (core external) while Stereo/Studio/A2DP presets ran
the core's miniaudio devices. Nearly every "no input / no output / both"
bug lived in the seam between the two paths — the lingering miniaudio
capture unit fighting VPIO, the audioRestart ordering dance, the
route-change "glitching" loop, stereo<->mono stickiness, and
"can't hear anyone". Switching presets/routes mid-call routinely dropped
a direction.
Drive ALL iOS audio through one AVAudioEngine with the core fully
external at all times: setExternalPlayback(1) once at connect, every MIC
stream external_feed=1, mic via vc_stream_feed_pcm, playback via
vc_set_mixed_output_sink (drained by an always-on AVAudioSourceNode so
remote audio plays before joining voice). VPIO + AGC toggle per preset.
Every preset/route/interruption change funnels through one deterministic
Swift-only reconfigure (stop -> apply session config -> rebuild -> start)
— no second path to hand off to, so a change can't drop a direction.
- IOSVoiceProcessingEngine.swift -> IOSAudioEngine: always-on source-node
playback, conditional mic tap, VPIO/AGC; one rebuild() backing
startListening/stop/startMic/stopMic/reconfigure/setCaptureChannels.
- IOSAudioRouter: 7 presets -> 4 (Voice Chat / Stereo Mic / Mono Mic /
Advanced); persisted voiceProcessingEnabled + agcEnabled; setters call
IOSAudioEngine.reconfigure() instead of audioRestart/reconcileVoicePath.
- AudioSessionManager slimmed; SessionState mic lifecycle collapsed;
AppState wires external playback + listening at connect, stop at
disconnect; SettingsView shows 4 presets + Advanced VPIO/AGC toggles.
No core/ABI/test changes — relies on the already-shipped external API
(test_external_pcm, test_external_playback). xcodebuild iOS device Debug
BUILD SUCCEEDED. Updates docs/voice.md §8 and PROGRESS.md.
The encoder set OPUS_SET_INBAND_FEC, but the decoder never invoked FEC --
the loss path went DRED -> PLC, so FEC redundancy was emitted (and paid for
in bitrate) yet never consumed.
Wire FEC recovery into AudioEngine::on_playback between DRED and PLC: copy
the next buffered packet once, try DRED, else (if the stream negotiated FEC)
decode(next_pkt, ..., fec=true), else PLC. Recovery priority is now
DRED -> FEC -> PLC. Add per-stream RemoteStream::fec_enabled_, captured from
OpusParams in init_recv_stream. Docs (voice.md) updated to match.
ctest --preset dev: 27/27 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Latency between speakers grew to multiple seconds and only reset on
rejoining voice. Root cause was the receiver playout logic, not the
codec settings: the playout clock free-ran in real time while the
sender omitted silence from its timestamps (and set no header flags),
and the only correction snapped the clock to the *oldest* buffered
frame — which could only ever add standing latency. target_depth_ms_
was computed but never enforced, so latency could only grow or reset.
Fix: bound playout against the stream's leading edge (newest frame).
(Re)seed to the leading edge on start/marker/starve (no prebuffer, so
latency stays low), and frame-skip catch-up trims any backlog beyond
target+hysteresis — the missing downward force.
Hardening: sender now stamps kFlagMarker (talkspurt start) and kFlagDtx,
consumed on recv for clean resync; adaptive late-drop window; EWMA
outlier rejection so silence gaps/stragglers don't poison the estimate;
duplicate counting and ring-underrun diagnostics.
New test_jitter_depth asserts depth stays bounded (<200ms) while
arrivals outrun playback. ctest --preset dev green (27/27).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-channel sample_rate field was inert after pinning the codec to
48 kHz. Make it meaningful without changing the 48 kHz clock: carry it as
OpusParams::max_bandwidth_hz and apply OPUS_SET_MAX_BANDWIDTH in
OpusEncoder::init (8000->narrowband, 16000->wideband, 24000->super-wideband,
48000->full). A low-bitrate room can now shed out-of-band content while
every endpoint keeps a single 48 kHz clock.
Make sample_rate channel-authoritative on the server: conn_session no
longer overrides effective sample_rate with the client's always-48000
request (it now behaves like frame_ms/mode). vc_get_stream_audio_config
reports the channel's configured rate for own streams too.
New ctest channel_samplerate: a 7 kHz tone is attenuated ~1000x on an
8 kHz (narrowband) channel vs a 48 kHz (full-band) channel, proving the cap
is in effect. ctest --preset dev 26/26.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The AudioEngine capture clock is fixed at 48 kHz / 20 ms (960-sample
frames), but a channel may set any Opus frame_ms (2.5..60 ms, voice.md
§3) and the server enforces it unclamped. on_capture_frame handed the
engine's 960-sample frame straight to an encoder configured for the
channel's window: frame_ms > 20 was silently ignored, and frame_ms < 20
broke entirely (receiver sized its decode buffer too small ->
OPUS_BUFFER_TOO_SMALL -> dead audio). Affected the hardware mic and
vc_stream_feed_pcm alike.
Reframe each captured/fed block to ls.frame_samples via a per-LocalStream
accumulator (pre-sized at announce, no RT-thread alloc) before
encode_and_send_frame; the 20 ms case stays a zero-copy fast path. Also
pin the codec to 48 kHz in opus_params_from_audio_config — it was honoring
a non-48k effective sample_rate against a 48 kHz PCM clock.
New ctest frame_ms_reframe covers 40 ms (accumulate) and 10 ms (split)
feed->encode->relay->decode->sink round trips. ctest --preset dev 25/25.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Talon
