Drop the M0 no-deps skeleton preset and all VOICECAT_HAS_NET/AUDIO/OPUS/NS
guards that it required. Every subsystem is fully implemented; the stub
#else paths were dead code that added noise to every header and source file.
- CMakePresets.json: remove skeleton configure/build/test entries
- CMakeLists.txt (root/core/tests): remove VOICECAT_USE_VCPKG_DEPS option
and guards; all targets now build unconditionally
- 17 C++ source files: unwrap HAS_* guards, delete stub #else blocks
- apm_processor.cpp: delete ApmPassthrough no-op class; create() always
returns RnnoiseProcessor
- 18 test files: remove HAS_* guards and stub int main() skip bodies
- docs/building.md: remove skeleton from preset table and prose
VOICECAT_HAS_LOOPBACK (Windows WASAPI loopback platform gate) unchanged.
29/29 ctest green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three bugs fixed across the full stack (proto/server/core/ABI/Win/macOS/iOS):
1. Join/Leave Voice now truly subscribes/unsubscribes from the voice plane.
Previously the button only toggled the local mic — receiving was always on
(gated by channel membership alone). Added a protocol-level voice subscription
concept: new SubscribeVoiceRequest/UnsubscribeVoiceRequest/VoiceSubscriptionResult
proto messages, User.voice_subscribed field, vc_join_voice/vc_leave_voice C ABI
functions, VC_EVENT_VOICE_STATE event, server-side voice_subscribed flag checked
by the SFU relay recipient filter, and core-client gating of remote-stream
decoder setup. All three clients rewired to subscribe+mic on Join / unsubscribe
on Leave. Text chat works regardless of voice subscription.
2. Channel edit dialog now shows the channel's actual current settings. The read
struct vc_channel was missing sort_order and audio fields — only the write
struct vc_channel_info had them. Extended vc_channel with both (additive, no
ABI break), updated the session model and list_channels marshaling to populate
them, and updated all three clients' edit callers to use actual channel info
instead of hardcoded defaults.
3. Channel parameter updates now automatically restart everyone's streams.
Previously editing a channel's audio config persisted and broadcast a
ChannelEvent::UPDATED, but no layer restarted streams — encoders/decoders are
frozen at announce time. handle_channel_event now detects audio-config changes
on the user's current channel and stop->starts each active local stream. The
server reads the updated config on re-announce; peers wire up fresh decoders
at the new ssrc.
All 29 CTest tests pass; Windows DLL + C# client build clean. Apple clients not
yet compile-verified (Windows environment).
OPUS_SET_DRED_DURATION was hardcoded to 2 (20 ms), meaning DRED only
covered 1/3 of a lost 60 ms frame and was useless above 20 ms channels.
Now computed as max(2, ceil(frame_ms/10)) so DRED always embeds enough
redundancy to reconstruct one full previous frame regardless of frame size.
The floor of 2 preserves two-frame burst-loss coverage at 10 ms channels.
Decoder side and server are unaffected (server relays payloads verbatim).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The per-listener noise-reduction toggle (vc_set_remote_stream) did nothing
on Windows/macOS/iOS. The decode loop gated the RNNoise pass on
dec_channels == 1 as a proxy for "this stream is voice" (assuming
stereo => screen-share). The stereo-mic capture commit broke that: a stereo
mic with send-side NR off transmits stereo Opus, so the receiver decoded
two channels and skipped NR entirely. gain/mute have no channel guard, which
is why only NR appeared broken.
Thread the stream kind through init_recv_stream into RemoteStream::is_voice
(set from si.kind() == STREAM_MIC), gate receive NR on is_voice instead of
channel count, and fold a stereo voice frame to mono -> denoise -> duplicate
back across both channels in place (symmetric with the send-side downmix;
RNNoise is mono-only). Screen-audio shares are never denoised.
New test test_recv_noise_reduction drives AudioEngine and asserts a stereo
voice stream's noise floor collapses with NR on (RMS 1046 -> 0.1) while a
screen-audio share stays unchanged. ctest --preset dev green 29/29.
Docs: voice.md section 10. Shared-core fix; clients need only a rebuild.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both desktop mics were hard-mono: the core defaults capture_channels=1 and
neither client ever called vc_set_capture_channels (only iOS did). Add a
persisted "Stereo microphone" toggle to each client's Audio settings, applied
when the mic stream starts and live via vc_set_capture_channels + vc_audio_restart.
Expose both ABI calls in the Windows interop; the macOS wrapper already had them.
Core fix: encode_and_send_frame now folds a stereo mic frame to mono on a mono
channel - previously the channels==2 branch encoded interleaved L/R directly even
on a mono channel, feeding a mono opus_encode 2x its samples (wrong pitch/garbage).
Real stereo still only reaches the wire on a stereo channel; on a mono channel the
mic is cleanly downmixed.
Test: test_stereo_mic_mono_channel. ctest --preset dev green (28/28). Docs: voice.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On iOS the remote end heard the mic twice and crackly (BT headset + internal
mic in Voice Chat; mono + stereo copies of the internal mic in Stereo Mic).
Root cause is an io-thread ordering race. `ensure_audio_running()` only set
`external_capture` when a MIC stream already existed, but it also runs from
`sync_remote_streams` on the post-auth `ServerStateSnapshot` — before the user
joins voice. With `external_playback_` still false and no MIC stream,
`AudioEngine::start()` opened a real miniaudio capture device that stayed open
all session (later calls early-return on running()), racing the AVAudioEngine
input tap fed via `vc_stream_feed_pcm`. `on_capture_frame` then encoded+sent
both paths — the mic transmitted twice, the two unsynchronized capture clocks
producing the crackle.
- core (ensure_audio_running): force `external_capture = true` whenever
`external_playback_` is set, so iOS unified mode never opens a hardware
capture device. No-op on desktop.
- ios (AppState): move `setExternalPlayback(true)` before `connect()`, so the
flag is set before the io thread processes any message — closing the race.
Verified on device: remote end hears the iOS mic once and clean in both Voice
Chat (+ BT) and Stereo Mic.
The two-sided NR plumbing (RemoteStream::recv_ns + the per-listener
vc_set_remote_stream noise_reduction toggle) was wired but inert:
ApmProcessor::create() returned a no-op passthrough, because the
originally-planned webrtc-audio-processing has no working Windows/macOS
build. Drop in RNNoise as the real backend behind the same ApmProcessor
interface, lighting up both NR paths.
- Vendor RNNoise (BSD-3 + CC0) at third_party/rnnoise/ — the vcpkg port
is !windows !arm, so it can't cover our primary targets. Shrunk int8
model (78MB -> 11.7MB via upstream scripts/shrink_model.sh), built as a
standalone C static lib with no RTCD (portable scalar path on x86,
auto-NEON on arm64) under -DDISABLE_DEBUG_FLOAT. Model is baked in
(rnnoise_create(NULL)); no runtime file.
- New RnnoiseProcessor (core/src/audio/apm_processor.cpp) selected by
ApmProcessor::create() when VOICECAT_HAS_NS. Mono/48kHz/480-sample;
our clock is fixed 48kHz and Opus frame sizes are multiples of 480, so
no resampling. RT-safe: allocates at construction, lock-free in the
capture/playback callbacks.
- Receive-side: lit up via the factory; gated to mono streams (a stereo
stream is a screen-audio share, not voice).
- Send-side (new): vc_set_input_noise_reduction(client, enable) ABI +
vc_client::mic_ns_, run before input gain/VAD in on_capture_frame. A
stereo mic is downmixed to mono ONLY when NR is on — with NR off a
stereo mic keeps full stereo (never collapse mic quality unasked).
- Enable C as a project language for the vendored lib.
- New noise_suppression test: white noise through ApmProcessor::create()
drops ~99.9% RMS. ctest --preset dev green, 28/28. windows-client DLL
builds clean with vc_set_input_noise_reduction exported, system-only deps.
- Docs synced: voice.md §10, tech-stack.md §1/§5, third_party/README.md,
vcpkg.json note, PROGRESS.md, CLAUDE.md.
Client on/off UI toggles (Windows/macOS/iOS) are the remaining follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Input mode (VAD/PTT/Always-On), VAD threshold, and the new mic gain were
applied to the core + UI but never saved, so every relaunch reset to VAD
defaults. Each client now persists them and re-applies on connect:
- iOS: UserDefaults (SessionState.loadAndApplyVoiceSettings + setter writes)
- macOS: UserDefaults via MainWindowController didSet + loadPersistedAudioSettings
(settings window also restores the VAD slider from the stored threshold)
- Windows: new Models/VoiceSettings.cs (JSON at %AppData%\VoiceCat\voice.json,
mirrors FeedbackSettings) loaded/applied in MainForm
Add global send-side mic gain API vc_set_input_gain (applied to MIC PCM in
on_capture_frame before the VAD gate, clamped to int16) + Swift/C# bindings,
and a 0-300% (default 100%) mic-volume slider on all three clients.
Fix iOS chat: ChatView called sendText(scope:.channel) with no targetId (0),
so channel messages went nowhere; now passes session.currentChannelId.
Fix iOS per-user tuning for VoiceOver: the tuning sheet was long-press
.contextMenu only (invisible to VoiceOver); UserRow now also exposes the same
buttons via .accessibilityActions (no visual change).
Verified: core builds clean; ctest 24/27 (3 pre-existing teardown crashes,
reproduced with changes stashed); VoiceCatMac + VoiceCatiOS (arm64 sim) build
SUCCEEDED; VoiceCat.Interop dotnet build succeeded. Windows App not built
(WinForms can't build on macOS) — follows existing patterns.
The encoder set OPUS_SET_INBAND_FEC, but the decoder never invoked FEC --
the loss path went DRED -> PLC, so FEC redundancy was emitted (and paid for
in bitrate) yet never consumed.
Wire FEC recovery into AudioEngine::on_playback between DRED and PLC: copy
the next buffered packet once, try DRED, else (if the stream negotiated FEC)
decode(next_pkt, ..., fec=true), else PLC. Recovery priority is now
DRED -> FEC -> PLC. Add per-stream RemoteStream::fec_enabled_, captured from
OpusParams in init_recv_stream. Docs (voice.md) updated to match.
ctest --preset dev: 27/27 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Latency between speakers grew to multiple seconds and only reset on
rejoining voice. Root cause was the receiver playout logic, not the
codec settings: the playout clock free-ran in real time while the
sender omitted silence from its timestamps (and set no header flags),
and the only correction snapped the clock to the *oldest* buffered
frame — which could only ever add standing latency. target_depth_ms_
was computed but never enforced, so latency could only grow or reset.
Fix: bound playout against the stream's leading edge (newest frame).
(Re)seed to the leading edge on start/marker/starve (no prebuffer, so
latency stays low), and frame-skip catch-up trims any backlog beyond
target+hysteresis — the missing downward force.
Hardening: sender now stamps kFlagMarker (talkspurt start) and kFlagDtx,
consumed on recv for clean resync; adaptive late-drop window; EWMA
outlier rejection so silence gaps/stragglers don't poison the estimate;
duplicate counting and ring-underrun diagnostics.
New test_jitter_depth asserts depth stays bounded (<200ms) while
arrivals outrun playback. ctest --preset dev green (27/27).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-channel sample_rate field was inert after pinning the codec to
48 kHz. Make it meaningful without changing the 48 kHz clock: carry it as
OpusParams::max_bandwidth_hz and apply OPUS_SET_MAX_BANDWIDTH in
OpusEncoder::init (8000->narrowband, 16000->wideband, 24000->super-wideband,
48000->full). A low-bitrate room can now shed out-of-band content while
every endpoint keeps a single 48 kHz clock.
Make sample_rate channel-authoritative on the server: conn_session no
longer overrides effective sample_rate with the client's always-48000
request (it now behaves like frame_ms/mode). vc_get_stream_audio_config
reports the channel's configured rate for own streams too.
New ctest channel_samplerate: a 7 kHz tone is attenuated ~1000x on an
8 kHz (narrowband) channel vs a 48 kHz (full-band) channel, proving the cap
is in effect. ctest --preset dev 26/26.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The AudioEngine capture clock is fixed at 48 kHz / 20 ms (960-sample
frames), but a channel may set any Opus frame_ms (2.5..60 ms, voice.md
§3) and the server enforces it unclamped. on_capture_frame handed the
engine's 960-sample frame straight to an encoder configured for the
channel's window: frame_ms > 20 was silently ignored, and frame_ms < 20
broke entirely (receiver sized its decode buffer too small ->
OPUS_BUFFER_TOO_SMALL -> dead audio). Affected the hardware mic and
vc_stream_feed_pcm alike.
Reframe each captured/fed block to ls.frame_samples via a per-LocalStream
accumulator (pre-sized at announce, no RT-thread alloc) before
encode_and_send_frame; the 20 ms case stays a zero-copy fast path. Also
pin the codec to 48 kHz in opus_params_from_audio_config — it was honoring
a non-48k effective sample_rate against a 48 kHz PCM clock.
New ctest frame_ms_reframe covers 40 ms (accumulate) and 10 ms (split)
feed->encode->relay->decode->sink round trips. ctest --preset dev 25/25.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
iOS "voice chat" had echo and no noise suppression: real iOS AEC/NS/AGC
come only from Apple's Voice-Processing I/O unit (VPIO), but the core
plays/captures via miniaudio's plain RemoteIO units, so .voiceChat mode
alone never engaged AEC.
Core (ABI PATCH 1->2):
- vc_set_mixed_output_sink + vc_set_external_playback. In external mode the
AudioEngine opens no hardware playback device; a mixer-timer thread drives
on_playback (decode+mix) on a ~20ms cadence and ships the final mix to the
sink. start() also skips the hardware capture device when the MIC stream is
external_feed (AudioParams.external_capture).
- New white-box test test_external_playback (drives the timer with no hw).
iOS/Swift:
- StreamDescriptor.externalFeed; VoiceCatClient.setMixedOutputSink /
setExternalPlayback wrappers.
- IOSVoiceProcessingEngine: AVAudioEngine + setVoiceProcessingEnabled; mic
tap -> feedPcm, mixed-sink lock-free ring -> AVAudioSourceNode (both share
the VPIO unit so AEC has its reference signal).
- IOSAudioRouter.currentConfigUsesVoiceProcessing scopes VPIO to the AEC
presets; SessionState join/leave + reconcileVoicePath() switch paths;
Voice Chat defaults to speaker; Settings surfaces AEC/NS state.
Known: pending on-device verification; a few bugs to fix afterward.
Adds "only selected apps" and "all apps except selected" screen-audio
modes to the Windows client, alongside the existing entire-desktop path.
Per-app capture uses WASAPI process loopback (AUDCLNT_ACTIVATIONTYPE_
PROCESS_LOOPBACK) via ProcessLoopbackCapture, mixed by ProcessAudioMixer
and fed to the core through vc_stream_feed_pcm (external_feed=1 so the
core skips its own loopback device).
Init must pass AUDCLNT_STREAMFLAGS_LOOPBACK | EVENTCALLBACK |
AUTOCONVERTPCM; the LOOPBACK flag is what makes the virtual endpoint
deliver rendered audio (without it every buffer is flagged SILENT) and
AUTOCONVERTPCM resamples the app's native format to 48k s16.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A bad UDP packet on a flaky link could permanently wedge the voice path,
unrecoverable even across app restarts. Three defects:
1. Anti-replay window was advanced from the UNAUTHENTICATED header seq
before the AEAD tag was checked, and not rolled back on failure. One
corrupted/forged frame shoved recv_highest_ far ahead, after which every
legitimate frame was rejected as "too old" forever. Reorder to
replay-check -> authenticate -> update (RFC 3711 3.3); the window now
moves only after a successful tag check.
2. The wire seq was only the low 16 bits of the nonce counter (zero-extended
on receive). After 65,536 frames the nonce desynced and all frames failed
auth. Widen the voice frame seq u16 -> u64 (header 14 -> 20 bytes). The
core owns all UDP framing, so Swift/C# clients need only a rebuild. This
is a versioned wire change: VOICECAT_PROTOCOL_VERSION 1 -> 2, handshake
rejects on mismatch.
3. Server leaked per-session UDP state on disconnect; unregister_session now
frees udp_endpoints_/udp_tokens_/ssrc_to_session_.
Also add rate-limited dropped-frame logging to MediaRelay so a wedged media
path is observable. New regression tests in test_media_aead.cpp cover the
poison (fails on old code) and the 16-bit wrap. ctest --preset dev
-E external_pcm: 22/22 pass (external_pcm aborts on a pre-existing CoreAudio
shutdown race, unrelated).
- Voice actions (Join Voice, Share Screen Audio) moved to a ToolStrip toolbar and
a new Voice menu in the menu bar; removed from the bottom voice panel
- Activity log and chat log collapsed into a single RichTextBox (rtbLog); activity
events appear in gray, chat messages in default color
- Private messaging reworked: each conversation opens in its own modeless
PrivateMessageForm instead of sharing the main chat log via a scope dropdown;
cboScope removed; main compose bar always sends to the current channel
- New "Messages -> New Private Message..." menu item (Ctrl+P) opens a UserPickerDialog
listing all connected server users (not just the current channel) so you can PM
anyone on the server
- Channel tree now shows live user counts, e.g. "General (3)" -- counts sourced from
the existing _users dictionary which already tracks all server users with channel IDs
- Global output volume slider (TrackBar, 0-100, default 80) added to the right panel;
wired to new vc_set_output_volume C ABI function that applies a master gain multiplier
in the audio engine playback callback after mixing all streams
- vc_set_output_volume added end-to-end: voicecat.h, audio_engine.h/.cpp,
client.h/.cpp, voicecat.cpp, NativeMethods.cs, VoiceCatClient.cs
- Documented Windows PowerShell ctest requirement in AGENTS.md and CLAUDE.md:
MinGW binaries exit 0xc0000139 in Git Bash; always run ctest/.exe via PowerShell
22/22 ctest green (PowerShell); dotnet build 0 warnings.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds Opus 1.6 DRED support end-to-end: encoder embeds 20 ms of ML
redundancy in every packet when enabled; decoder recovers lost frames
from the next buffered packet's DRED extension rather than falling back
to PLC comfort noise.
Protocol: bool dred = 11 added to AudioConfig (backward-compatible,
defaults false). C ABI: int dred added to vc_audio_config. Encoder:
OPUS_SET_DRED_DURATION(2) when dred=true. Decoder: OpusDREDDecoder +
per-stream OpusDRED scratch pre-allocated off the RT thread;
JitterBuffer::try_copy_front_payload peeks at the next packet without
popping on every PLC step; opus_decoder_dred_decode reconstructs the
lost frame if DRED data is present, otherwise falls back to PLC.
New test: test_dred_toggle (22/22 ctest green).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the stereo-mic/A2DP debugging settled, the iOS audio code carried
leftover TeamTalk5 comparison notes, source-line citations, TEMP DIAGNOSTIC
markers, and "this was the bug" narratives that no longer help. Reworded
those to state the current rules; kept the comments that document real
constraints (the setPreferredInputNumberOfChannels(2) trap, the re-entrancy
guard, the ma_context no-session-management config).
Comment-only — no behavior change. core builds, ctest --preset dev 21/21.
The real root cause of "selecting Stereo Mic kills headphone/A2DP output on Join
Voice." Every prior fix worked on the Swift IOSAudioRouter under the false premise
that "miniaudio does NOT touch AVAudioSession on iOS." It does: the core opened
devices via ma_device_init(nullptr, ...), and with a NULL context miniaudio 0.11.25
runs an iOS "hack" that sets the session category by device type, then
ma_context_init__coreaudio calls setCategory()+setActive() on every device open --
capture -> AVAudioSessionCategoryRecord with zero options. That wipes the
.playAndRecord category, the mode, and .allowBluetoothA2DP / .mixWithOthers /
.allowAirPlay that IOSAudioRouter had just configured, killing headphone/A2DP (and
even wired) output. Stereo presets break worst because they rely on the A2DP output
route the wipe removes. TeamTalk avoids this by opening RemoteIO/VPIO AudioUnits
directly and leaving the session entirely to the app.
Fix (core, cross-platform safe): AudioEngine now owns a ma_context built by
make_context_config() with coreaudio.sessionCategory = ma_ios_session_category_none
and noAudioSessionActivate/Deactivate = MA_TRUE, and routes all ma_device_init calls
(playback, capture, loopback) plus enumerate_devices through it. miniaudio no longer
touches AVAudioSession; IOSAudioRouter is the sole owner (the session is already
activated on connect in AppState before any device opens). Context is lazily inited
in start(), reused across restarts, uninited in ~AudioEngine.
Adds TEMP AudioSessionManager.logSessionState() diagnostics (after activate, on route
change, on .streamStarted) to verify on-device that the category stays
PlayAndRecord+allowBluetoothA2DP instead of flipping to Record. Remove once confirmed.
Windows: cmake --build --preset dev clean; ctest --preset dev 21/21.
iOS build + on-device verification pending on Mac.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three coordinated fixes for the bug where enabling stereo mic capture
caused all audio output (A2DP, speaker, wired) to go silent:
1. audio_engine.cpp — open playback before capture
On iOS, starting the stereo capture AudioUnit can trigger an audio
route reconfiguration that drops A2DP before the playback device has
a chance to claim the route. Opening and starting the playback device
first commits the output route (A2DP), so iOS is less likely to drop
it when stereo capture activates afterward.
2. client.cpp — decouple set_capture_channels from engine restart
Previously vc_set_capture_channels() stopped and restarted the engine
immediately, which opened capture first (old ordering) and raced
against the settling AVAudioSession route. Now it only stores the
channel count; the caller (Swift via vc_audio_restart) controls when
the engine restarts, after the route has settled.
3. IOSAudioRouter.swift — call audioRestart() after channel config
selectCaptureChannels() and applyPreset() now call audioRestart()
after applyConfiguration() + setCaptureChannels(). This is the
vc_audio_restart() path that was added to the ABI in fdcc84f but
never wired up in the Swift layer. The restart sees the stored
channel count and reopens devices in the correct order (playback
first, capture second).
The doStartMicStream path is unaffected: setCaptureChannels is called
before the server acknowledges the stream (engine not yet running), so
ensure_audio_running() picks up capture_channels=2 directly when the
stream is confirmed and opens with the right count from the start.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Diagnosed by comparing against TeamTalk5 (Client/iTeamTalk), which
achieves stereo mic + A2DP output. Five fixes:
1. configureStereoCapture now calls setPreferredInput +
setInputDataSource (mirroring TeamTalk5's SoundDevicesModel).
Previously omitted based on incorrect diagnosis that
setPreferredInput collapsed A2DP — the real culprit was
setPreferredInputNumberOfChannels(2), which neither project uses.
2. New C ABI: vc_audio_restart (full stop + re-init, unlike
suspend/resume which only stop/start). Swift wrapper added.
The withAudioSuspend wrapper that used it was removed after
on-device testing showed it killed all audio (including
VoiceOver) when switching presets — the core's
set_capture_channels handles engine restart internally.
3. Bluetooth options: Voice Chat preset now includes BOTH
.allowBluetoothHFP AND .allowBluetoothA2DP (matching TeamTalk5's
UtilSound.swift:228). Previously HFP-only blocked A2DP headphones.
4. Capture channels now reset when switching stereo→mono via
selectCaptureChannels/applyPreset. AudioSessionManager tracks
activeMicStreamId (set by SessionState on join/leave voice).
5. Docs synced: voice.md, tech-stack.md, architecture.md,
PROGRESS.md. Removed stale setPreferredInputNumberOfChannels(2)
references.
Verified: ctest --preset dev 21/21 green, iOS client builds.
Stereo mic + A2DP output still needs on-device debugging — the
core recipe is correct but iOS 26 route behavior requires
hands-on testing with a debugger.
Three iOS client problems fixed plus a new core stereo-mic capture ABI:
1. Channel-id sync bug (mic button permanently dimmed): SessionState never
synced currentChannelId from the self user's channelId on connect, so the
mic button (gated on currentChannelId == 0) stayed dimmed. Added
syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522);
called from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult.
Added applyServerMuteState() + serverMuted/serverDeafened to VoiceState.
2. Join/Leave Voice button: replaced icon-only mic toggle with explicit
text button (parity with macOS). Mute/deafen disable when not in voice.
3. IOSAudioRouter.swift (new): full AVAudioSession routing layer — input
port selection, built-in mic orientation/polar patterns, Bluetooth
HFP/A2DP/Off modes, Standard/Raw mic processing, stereo capture, AirPlay,
UserDefaults persistence. AudioSessionManager delegates to it.
4. Core stereo-mic capture (append-only ABI): vc_set_capture_channels()
lets the core open the mic device in stereo (2-ch interleaved). LocalStream
gains capture_channels; ensure_audio_running reads it; audio_engine.cpp
capture_accum_ + on_capture updated to channel-aware accumulation. Test
test_stereo_mic_capture (headless, L!=R stereo round-trip). Swift wrapper
VoiceCatClient.setCaptureChannels.
5. Settings UI rework: AVAudioSession-derived input/output tree replaces
miniaudio device picker.
6. iOS deployment target raised to 18.0 (Package.swift + project.pbxproj).
swift-tools-version 6.0 with swiftLanguageModes .v5.
Docs: tech-stack.md, architecture.md, voice.md, roadmap.md, building.md
updated; stale 'vc_audio_suspend/resume deferred' claims corrected.
Verified: ctest --preset dev 21/21 green; swift test 6/6 green;
xcodebuild -target VoiceCatiOS -sdk iphonesimulator BUILD SUCCEEDED.
Full SwiftUI app at clients/apple/iOS/VoiceCatiOS.xcodeproj:
- 24 Swift source files: AppState + SessionState (@Observable @MainActor),
AudioSessionManager (AVAudioSession owner + interruption/route handling),
ServerListStore/SavedServer (App Group container + Keychain sharing),
and 14 SwiftUI views covering the full feature set
- NavigationSplitView on iPad, TabView on iPhone (horizontalSizeClass)
- Channel tree via OutlineGroup, user list with context menu admin actions
- PTT via DragGesture(minimumDistance: 0) + @GestureState
- onEvent closures hop to MainActor via Task { @MainActor in ... }
- App Group: group.cat.voice.VoiceCat (shared with future ReplayKit extension)
C ABI: add vc_audio_suspend / vc_audio_resume (AudioEngine::suspend/resume)
called by AudioSessionManager on AVAudioSession interruption events.
XCFramework: add ios-arm64 and ios-arm64-simulator slices to build-xcframework.sh;
Package.swift gains .iOS(.v17) platform; CMakePresets.json adds apple-ios /
apple-ios-sim presets with arm64-ios / arm64-ios-simulator vcpkg triplets.
Verified: xcodebuild -target VoiceCatiOS -sdk iphonesimulator26.5 BUILD SUCCEEDED.
macOS port groundwork — core, server, tools, and tests now build and run on
macOS 26.5 / Apple Silicon. ctest --preset dev green 21/21 (2 consecutive runs).
apple-dev produces valid arm64 libvoicecat.a + XCFramework for the Swift Package.
Three real cross-platform bugs found and fixed (all latent on Windows/Linux):
1. test_m2_voice.cpp POSIX branch missing <netdb.h> — Linux glibc transitively
includes it, macOS doesn't. Would fail on any strict POSIX system.
2. SIGPIPE killing processes on macOS — writing to a closed TCP socket raises
SIGPIPE by default (doesn't exist on Windows, benign on Linux). Fixed by
ignoring SIGPIPE in both core client init and server startup (POSIX-only,
#ifndef _WIN32). Production fix, not just tests.
3. Use-after-free of Asio's kqueue reactor on server shutdown — the
deterministic test_tofu_flow segfault. TcpServerConn's tls_read_loop runs on
a blocking-I/O thread; when Server::run() returned, io_context was destroyed
while those threads were still running. On macOS kqueue the reactor pointer
is null'd immediately -> segfault in socket.close(). Latent on Windows IOCP
and Linux epoll. Fix: TcpAcceptor now tracks connections; new shutdown()
closes all + joins threads before io is destroyed; Server::stop() now closes
acceptor + media_relay too (was just io.stop()).
Verified: dev + apple-dev presets build green, 21/21 tests pass, server starts
+ two vccli text chat over TLS (M1 on Mac), vccli --voice starts MIC stream via
CoreAudio (M2 protocol-level), vccli --list-devices enumerates CoreAudio
devices, xcodebuild -create-xcframework produces valid VoiceCatCore.xcframework.
No ABI or proto changes. Docs updated: building.md, clients/apple/README.md,
PROGRESS.md, CLAUDE.md status line.
Rationalize the preset set to match the project's actual state (past M5):
- Rename dev->skeleton (no-deps stub smoke), m1-dev->dev (default dev preset)
- Drop m2-dev (cache-identical to m1-dev)
- Add release preset (optimized + tests on, symbols kept)
- Strip server-release binaries (-s linker flag)
- Add apple-dev/apple-ios/apple-ios-sim scaffolding presets for XCFramework
Add cmake/voicecat-toolchain.cmake wrapper that auto-resolves the vcpkg
triplet from the host platform (x64-mingw-static/x64-linux/arm64-osx) so
the main presets work on Windows/Linux/macOS without per-OS variants.
Update all docs (building.md, CLAUDE.md, README.md, AGENTS.md, deployment.md,
tech-stack.md, client READMEs) and stale preset-name references in code
comments. No C++ behavior changes — the core was already portable.
PerUserTuningDialog previously broadcast one gain/mute/NR set to *all* of a
user's streams, even though the core mixer (AudioEngine::RemoteStream) and
the C ABI (vc_set_remote_stream) were already per-stream. The UI had no
per-mix controls anywhere.
Reworks the dialog to enumerate ListUserStreams on open and render one row
per stream (kind + label + Gain + Mute + NR), each wiring only to its own
stream_id. Adds a read-back ABI counterpart, vc_get_remote_stream, so the
dialog opens at the listener's actual current per-stream settings (defaults
1.0/unmuted/NR-off) rather than always 100%. Additive ABI change only; no
existing symbols touched.
Tests: test_m3_multistream extended with getter round-trip assertions; new
C# smoke test exercises the full P/Invoke marshaling path with two clients.
Docs: voice.md §10 notes the getter. NR checkbox keeps its honest
'passthrough' label (NS DSP still unbuilt per §8).
Three reported bugs traced to one root cause plus two missing designed features:
1. Stale users + eternal PLC hiss (root cause): ConnSession::close() silently
erased dropped users without broadcasting UserEvent::LEFT, so peers never
learned the user left and their audio engines never called remove_stream —
Opus PLC synthesized comfort noise forever. Fix: broadcast_left() helper
+ close() broadcasts LEFT before erasing.
2. PLC cap (defense-in-depth): on_playback now caps pure PLC at ~2s, then
emits digital silence so a stale stream can never hiss forever even if
remove_stream is skipped. Resets automatically on fresh packets.
3. No timeout / no ping: client never sent Ping, server had no last_seen /
reaper, so half-open connections (NAT timeout, wifi loss, sleep) left
ghost users forever. Fix: client Ping every 15s with RTT measurement,
ConnSession::last_seen bumped on every inbound TCP/UDP frame, steady_timer
reaper sweeps every 15s and drops sessions older than 45s (configurable
via server::Config).
4. UDP KEEPALIVE: client sends plaintext kFrameKeepalive every 5s; server
bumps last_seen + echoes back. Keeps NAT bindings alive and lets media
activity defer the reaper independently of TCP.
5. Graceful client disconnect: vc_disconnect() sends Disconnect{code=0} via
a flag-based io-thread exit (no double-close race); server handles
client-sent Disconnect with immediate close() + LEFT broadcast.
3 new tests: disconnect_left, plc_cap, reaper_timeout. 21/21 ctest green.
Docs: protocol.md §6/§7, voice.md §6, architecture.md §5, PROGRESS.md.
start_loopback_capture hardcoded channels=1, forcing miniaudio to downmix the system's stereo mix to mono before the encoder saw it -- on_capture_frame then upmixed L=R to produce fake stereo. Now the loopback device opens in the channel's mode (stereo when the channel is stereo), CaptureCallback carries an explicit channels param so the encoder gets real interleaved L/R, and a mono fallback covers unusual render endpoints. New test_loopback_stereo_capture asserts L!=R end-to-end; 18/18 ctest green.
A connected Windows client would randomly snap from its joined channel
back to Lobby. Root cause was a state-sync inconsistency, not a drop:
the server delivered self-initiated state changes (channel join/leave,
stream announce/stop) only as a private *Result to the actor and
broadcast the authoritative UserEvent::UPDATED to everyone else. The
core never applied the result to its SessionModel, so vc_list_users()
kept self in the old channel; the Windows HandleUserUpdated rebuilds
_currentChannelId from vc_list_users() on any user's UPDATED event, so
the next unrelated event surfaced the stale self-channel.
Fix, per the response-vs-broadcast contract now documented in
docs/protocol.md §6: the *Result is pure ack/correlation/actor-private
payload; the resulting state change is broadcast to every client
INCLUDING the actor, and clients apply it to their local model rather
than re-deriving own state from a *Result.
- server: join/leave/stream announce+stop broadcast with exclude=0
- server: text fan-out includes the sender (channel + private echo)
- core: response handlers no longer mutate session_model_
- windows: drop optimistic text echo; render own message via the relay
- docs/protocol.md §6: document the response-vs-broadcast contract
Registry-level admin broadcasts (move/mute/kick/channel CRUD) already
used exclude=0 and were correct. ctest build/m1-dev 18/18 green;
VoiceCat.App builds 0 warnings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RemoteStream::playout_ts was seeded to 0 and only advanced inside the
decode loop (including on every PLC iteration), so it free-ran at ~1x
wall-clock regardless of whether the sender was transmitting. The
sender's frame timestamps only advance while it actually sends (the
VAD/PTT gate returns before ls.timestamp += samples). Across a late join
or any VAD/PTT silence gap the two clocks diverged without bound; once
past the jitter buffer's 500 ms late-drop window every real frame was
dropped-as-late (clock ahead) or never-due (clock behind) -> permanent
silence, while the talk indicator (driven by push_recv_frame, independent
of the jitter buffer) stayed lit.
Add JitterBuffer::peek_front_ts() (try-lock, RT-safe) and seed/re-sync
playout_ts to the earliest buffered frame on the first frame and whenever
it has drifted past +/-200/500 ms. This seeds startup and recovers after
every silence gap.
New regression test test_playout_resync free-runs the clock ~2 s past the
drop window, pushes a ts=0 frame, and asserts audible output: fails
(energy=0) without the fix, passes with it. ctest --preset m1-dev: 14/14.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AudioEngine::on_capture() was passing miniaudio's hardware callback period
(commonly 480 samples / 10 ms on WASAPI shared mode) directly to opus_encode(),
which requires exactly frame_samples_ (960 for 20 ms @ 48 kHz). The mismatch
returned OPUS_BAD_ARG and silently dropped every real mic frame, while screen
share and injected test frames happened to be correctly sized and worked fine.
Fix: accumulate PCM in a pre-allocated CaptureAccum buffer (mirroring the
existing RemoteStream::ring fix on the playback side) and only call capture_cb_
when a full frame_samples_ chunk is ready. Same pattern applied to on_loopback().
Add test_capture_frame_accumulation() to verify the accumulator fires exactly the
right number of callbacks for misaligned chunk sizes (480, 240+720, 1920 samples).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
on_playback() was passing miniaudio's hardware playback-callback frame
count to opus_decode()'s max_samples, instead of the decoder's fixed
frame size (960 samples @ 20ms/48kHz). Since real packets decode to
more samples than the (often smaller, e.g. ~480 on default low-latency
WASAPI) hardware period, opus_decode returned OPUS_BUFFER_TOO_SMALL on
nearly every callback -- packets were received/decrypted/jitter-buffered
correctly but never decoded into audible PCM. Result: control-plane
events and VAD worked, but zero audio in headphones.
mix_for_test()'s white-box test masked this since it always called
on_playback with frames == frame_samples, the one case where the bug
is invisible.
Fix: RemoteStream gained a small ring buffer (init_ring/push_ring/
pop_ring) that decouples decode cadence from playback-callback cadence.
on_playback now tops the ring up by decoding whole Opus frames (always
decoder.frame_samples(), never the hardware frame count) and drains
exactly what the callback asks for, silence-padding (PLC) on underrun.
Side effect: also fixes playout_ts, which was advancing by the wrong
unit (hardware frames instead of decoded samples) -- it now tracks
correctly against jitter-buffer timestamps.
ctest --test-dir build/m1-dev: 12/12 green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- docs/building.md: explains what each CMake preset (dev, m1-dev, m2-dev,
server-release) is actually for, and how to build voicecat-server + vccli
for manual testing. Linked from CLAUDE.md's doc index.
- core/include/voicecat.h, core/src/voicecat.cpp, core/src/protocol/protocol.h,
server/src/main.cpp: doc-header comments still claimed M0-skeleton/stub
behavior (VC_ERR_NOT_IMPLEMENTED everywhere, "prints what it would do",
protobuf codegen "commented") that M1-M3 made real. Updated to describe
current behavior, with the dev-preset stub fallback noted explicitly where
it still applies.
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).
Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
the same user silently overwrote the first in SessionRegistry::set_user_stream.
Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
and receive (sync_remote_streams) paths via a shared
opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
(total samples instead of samples-per-channel), which would have overflowed
the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
and from disconnect() on a different thread -- both could see
udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
same std::thread (intermittent std::system_error under ctest). Fixed with a
teardown_mu_ guard instead of carrying the flake forward.
New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
handle_stream_announce enforces the channel's config, clamping (not
overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
request_id-correlated announce/result handling (request_id already
round-tripped on the wire; just wasn't read before). on_capture_frame is
kind-aware and upmixes mono capture to stereo when a stream's config calls
for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
through set_remote_stream. New run_talk_timer() thread emits
VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
(inject_capture), stereo-to-mono downmix at the decode/mix boundary,
RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
and last_voice_ms/talking; new set_stream_noise_reduction() and
poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
vc_get_stream_audio_config (effective Opus config for any stream you own or
a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
(mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
concurrent local streams, independent gain/mute/NS control, per-channel
config divergence via vc_get_stream_audio_config, talk indicators.
Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).
ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
test_m2_voice passed against raw BSD sockets, but vc_client::stream_start/stop,
UDP binding, and capture/recv were still VC_ERR_NOT_IMPLEMENTED stubs -- meaning
vccli and any GUI client still couldn't actually talk. Implements the real
client-side UDP-binding handshake, media key derivation, capture->encode->seal->
send and recv->open->decode->playback paths, plus server-side StreamInfo
broadcast so peers learn about each other's streams via sync_remote_streams().
Adds test_voice_client_abi (two real vc_client instances, not raw sockets) and
vccli --voice/--mute/--text flags, manually verified live between two instances.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements the full M1 milestone. Two clients authenticate over TLS 1.3
(guest + Argon2id password) and exchange channel + private text messages
through a real server. All five ctest --preset m1-dev tests pass in ~1 s.
Key components added:
- vcpkg baseline + m1-dev preset (protobuf/mbedTLS/libsodium/asio/sqlite3)
- FrameCodec feed+emit, encode/decode_envelope, protobuf codegen
- TcpServerConn with blocking TLS handshake thread + tls_read_loop
- TlsContext (mbedTLS 1.3, ECDSA-P256 self-signed cert, TOFU on client)
- WorkerPool (3 threads, used for Argon2id)
- Database: SQLite + libsodium Argon2id, account lifecycle, bootstrap admin
- ServerIdentityManager: Ed25519 key + cert generate/persist/fingerprint
- ConnSession state machine: WaitingHello -> WaitingAuth -> Authenticated
- SessionRegistry: channel tree, user map, text routing, broadcast
- vc_client full M1 C ABI: connect/TLS/handshake/auth/text/disconnect
- voicecat-admin CLI: account add/reset/del/list
- test_m1_integration: M1 exit criterion, verified green
Bug fixed: double-framing in ConnSession::send_envelope -- encode_envelope
was adding the [4-byte len] prefix, then TcpServerConn::send_frame added
a second one, causing the client to parse [len][proto] as protobuf (silent
failure). Fixed by serializing raw protobuf bytes in send_envelope and
letting send_frame apply the single length prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
