voice-cat/PROGRESS.md

PROGRESS — VoiceCat

Living status. Update this file in the same commit as your work so the next agent picks up instantly. Newest status at the top.

• Date convention: ISO (YYYY-MM-DD).
• Statuses: [ ] not started · [~] in progress · [x] done.

---

▶ Where we left off / next action

• Planned (not started): External PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink) (2026-06-19, plan written on Windows; implement on Mac). A public, documented API for driving audio streams with externally-provided PCM instead of (or in addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our roadmap — that the current "miniaudio owns the device" model can't serve:

  1. ReplayKit Broadcast Upload Extension (iOS SCREEN_AUDIO) — the extension is a separate process with a ~50 MB memory cap and can't link the full AudioEngine (ma_device, capture/playback threads). It needs to feed CMSampleBuffer audio (system app audio) into the encode path without any audio hardware. The current plan in docs/voice.md §9 says the extension links "a minimal slice of the core (Opus encode + media send only)" — a public feed-PCM API is that minimal slice. The extension links Opus + the feed entry point, no ma_device needed.
  2. ScreenCaptureKit (macOS SCREEN_AUDIO) — SCStream delivers CMSampleBuffer in a callback; convert to int16 and feed. No need to route through miniaudio's device layer. This is how macOS screen-audio actually gets implemented — today it does NOT work: VOICECAT_HAS_LOOPBACK is Windows-only (core/CMakeLists.txt:88-95), so on macOS AudioEngine::start_loopback_capture() hits the #else stub (audio_engine.cpp:647-649) and returns false. The macOS client's "Share Screen Audio" button (MainWindowController.swift:800-816) calls startStream(.screenAudio) which announces the stream to peers but captures zero audio — peers hear silence. The button is left in place (not touched per user request); it'll work once this API + a ScreenCaptureKit tap ship on Mac.
  3. Bots — music bot, TTS bot, radio relay, transcription bot. They create a SCREEN_AUDIO/AUX_DEVICE stream and feed synthesized or decoded PCM via the feed API. No audio hardware required — runs headless on a server. Today the only way to feed external PCM is vc_test_inject_capture (TEST-ONLY, name signals "don't ship this") or re-implementing Opus encode + AEAD + UDP framing yourself (~500 lines of duplicated crypto/codec code per consumer).
  4. Custom clients / accessibility — soundboard, DAW integration, TTS of incoming chat, recording/transcription of remote audio. Need either feed (send) or tap (receive) or both.

  What we already have (input half, gated as test-only): vc_test_inject_capture (stream_id, pcm, samples) (voicecat.h, client.cpp:1452) feeds raw int16 PCM into the encode pipeline via AudioEngine::inject_capture(kind, pcm, n). It works for any stream kind, supports multiple concurrent injection taps (one ring buffer per local kind), and goes through the full encode → AEAD → UDP path. The encode path already handles channels == 1 || 2 (proven by the WASAPI stereo loopback work, 2026-06-17 entry below). The only problems: it's marked TEST-ONLY in the header, the name signals "don't use this in production," and it hardcodes mono (no channels parameter).

  What's missing (output half): today decoded remote audio is mixed and pushed to the miniaudio playback device (on_playback). There's no way for an external consumer to intercept the decoded PCM of a specific remote stream — it all goes to the hardware device. A bot that wants to record, transcribe, or re-broadcast remote audio has no hook.

  Plan (API design — clean, append-only, no struct changes, ABI-stable):

  • vc_stream_feed_pcm — promote vc_test_inject_capture to a public, documented API and add a channels parameter:

    /* External PCM feed — replaces the hardware capture device for this stream. Caller
       provides interleaved int16 PCM at the stream's sample rate. The core frames it,
       encodes (Opus), seals (AEAD), and sends (UDP). Works for any stream kind
       (MIC/SCREEN_AUDIO/AUX_DEVICE). The stream must be started first (vc_stream_start);
       this just replaces the capture source. channels = 1 (mono) or 2 (stereo interleaved).
       Thread-safe; may be called from any thread including audio callbacks. */
    vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id,
                                 const int16_t* pcm, size_t samples_per_channel,
                                 uint32_t channels);
    

  • vc_set_pcm_sink — symmetric output side: receive decoded remote audio as int16 PCM instead of (or in addition to) the hardware playback device:

    /* External PCM tap — receive decoded, mixed remote audio as int16 PCM. The callback
       fires on the audio thread with the mixed output for a specific remote stream. Pass
       cb=NULL to disable (default: disabled, hardware playback only). When enabled, PCM is
       delivered to the sink AND the hardware device (dual output) so a bot can record
       without disabling local monitoring. user_id+stream_id identify the source stream.
       The callback MUST NOT block — copy what you need and return (same contract as
       vc_callbacks.on_event). */
    typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
                                   const int16_t* pcm, size_t samples_per_channel,
                                   uint32_t channels, uint32_t sample_rate);
    vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
    

  • Core changes:
    • core/include/voicecat.h — add vc_pcm_sink_cb typedef + the two function declarations (append-only, after vc_test_inject_capture). Full doc comments on both (contract, thread-safety, lifetime, use cases).
    • core/src/voicecat.cpp — thin C trampolines → vc_client::stream_feed_pcm / set_pcm_sink.
    • core/src/core/client.{h,cpp} — stream_feed_pcm: validates stream_id, looks up the LocalStream's kind, calls audio_engine_.inject_capture(kind, pcm, n) (existing path) with the channel count forwarded. set_pcm_sink: stores the callback + user pointer; on_playback (or a new fan-out in the mixer) invokes it per remote stream alongside the existing hardware write. Keep vc_test_inject_capture as a deprecated alias calling stream_feed_pcm(..., channels=1) for source compatibility.
    • core/src/audio/audio_engine.{h,cpp} — inject_capture already exists per-kind; add a channels parameter to the ring-buffer write path (or a parallel stereo-aware variant). The encode path in client.cpp::on_capture_frame already handles channels==2 via the stereo encode branch — just plumb the value through. For the sink: add a pcm_sink_ member (callback + user); in on_playback after mixing, if the sink is set, copy the mixed PCM for the current stream and invoke the callback. The copy must stay off the RT-critical path — document the non-blocking contract.
  • Skeleton stub path: update client.cpp's #else (no-deps) stub section to add vc_stream_feed_pcm/vc_set_pcm_sink returning VC_ERR_NOT_IMPLEMENTED — keeps the skeleton preset green.
  • Swift VoiceCatCore: add feedPcm(streamId:pcm:samplesPerChannel:channels:) and setPcmSink(_:user:) (the Swift wrapper around vc_pcm_sink_cb — a @convention(c) closure + Unmanaged context, mirroring Callbacks.swift). Wraps both new ABI functions.
  • C# VoiceCat.Interop: add StreamFeedPcm(streamId, pcm, samples, channels) (with int16[] marshaling) and SetPcmSink (delegates via [UnmanagedCallersOnly] thunk, mirroring the event-callback pattern). Wraps both new ABI functions.
  • Tests:
    • tests/test_external_pcm.cpp (new) — test_feed_pcm_round_trip: two clients, A feeds a known mono sine wave via vc_stream_feed_pcm on a MIC stream, B receives via the normal decode path and asserts energy matches. test_feed_pcm_stereo: same with channels=2, assert L≠R end-to-end (mirrors the WASAPI loopback stereo test). test_pcm_sink: B sets a vc_pcm_sink_cb, A feeds PCM, assert the sink callback receives the decoded PCM with matching energy. All headless, no audio hardware.
    • clients/apple/Tests/VoiceCatCoreTests/ — Swift wrapper round-trip for feedPcm.
    • clients/windows/VoiceCat.Interop.Tests/ — C# wrapper round-trip.
  • Docs:
    • docs/architecture.md §4 — new subsection on external PCM feed/tap: the contract (caller provides interleaved int16 at the stream's sample rate; core frames/encodes/ seals/sends for feed; core decodes/mixes/delivers for sink; sink callback must not block), the use cases (ReplayKit, ScreenCaptureKit, bots, custom clients), and the relationship to vc_test_inject_capture (deprecated alias).
    • docs/voice.md §9 — update the iOS ReplayKit and macOS ScreenCaptureKit rows: both now consume vc_stream_feed_pcm instead of a "minimal slice of the core." Update the iOS detail bullets: the extension links Opus + vc_stream_feed_pcm (not a parallel media stack). Add a macOS ScreenCaptureKit note: convert CMSampleBuffer → int16, feed via vc_stream_feed_pcm — this is how macOS screen-audio actually ships.
    • docs/protocol.md — no protocol changes (the feed/sink are client-local; the wire format is identical whether PCM came from miniaudio or an external source). Note this explicitly.
    • docs/roadmap.md — add a milestone entry; update the iOS ReplayKit and macOS ScreenCaptureKit pending items to reference vc_stream_feed_pcm.
  • Implementation order:
    1. C ABI + core (voicecat.h, voicecat.cpp, client.{h,cpp}, audio_engine.{h,cpp}) + skeleton stub. Verify ctest --preset dev green.
    2. tests/test_external_pcm.cpp — the three behavior tests. Verify green.
    3. Swift VoiceCatCore wrapper + VoiceCatCoreTests round-trip.
    4. C# VoiceCat.Interop wrapper + VoiceCatClientSmokeTests round-trip.
    5. Docs (architecture.md, voice.md, protocol.md, roadmap.md, header comments).
    6. Then ReplayKit (iOS) and ScreenCaptureKit (macOS) become ~100-line consumers of this API instead of parallel media stacks.
  • Verification: ctest --preset dev green (3 new tests); swift test green; dotnet test green; xcodebuild (skeleton) green. The feed/sink tests are fully headless — no audio hardware, no simulator, no device — so they run in CI on every platform.
  • Files to touch:
    • Core C++: core/include/voicecat.h, core/src/voicecat.cpp, core/src/core/client.{h,cpp}, core/src/audio/audio_engine.{h,cpp}.
    • Tests: tests/test_external_pcm.cpp (new), tests/CMakeLists.txt.
    • Swift: clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift, clients/apple/Sources/VoiceCatCore/Callbacks.swift, clients/apple/Tests/VoiceCatCoreTests/ExternalPcmTests.swift (new).
    • C#: clients/windows/VoiceCat.Interop/VoiceCatClient.cs, clients/windows/VoiceCat.Interop/NativeMethods.cs, clients/windows/VoiceCat.Interop.Tests/ExternalPcmTests.cs (new).
    • Docs: docs/architecture.md, docs/voice.md, docs/protocol.md, docs/roadmap.md.
  • ABI stability: append-only — two new functions + one new typedef, no existing structs/enums changed. vc_test_inject_capture stays as a deprecated alias for source compatibility. Treat as a deliberate, versioned ABI event per docs/protocol.md §8.
  • Relationship to the iOS audio routing plan (entry below): orthogonal. That plan is about which hardware route miniaudio opens (AVAudioSession config in Swift). This plan is about bypassing miniaudio's hardware entirely (external PCM feed/tap). Both ship; they don't conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer controls the mic path which still uses miniaudio's device.

• Done: iOS audio preset fixes — A2DP output muting + broken stereo capture (2026-06-19). Two user-reported bugs with the audio presets (esp. with Bluetooth headphones connected), both root-caused against Apple docs/WWDC20 + dev-forum reports and fixed in IOSAudioRouter.applyConfiguration() / AudioSessionManager.ensureSessionActive():

  1. Output (and VoiceOver) went dead on the Stereo Mic / A2DP presets (BUG — fixed). applyConfiguration() always inserted .defaultToSpeaker, which breaks A2DP routing in .playAndRecord (route ends up muted, taking the shared hardware output — and VoiceOver — with it). And the A2DP path used mode = .default, which iOS 17+ routes to the speaker. Fix: .defaultToSpeaker now set only for the speaker preset; .mixWithOthers kept always (VoiceOver must stay audible for a blind user); A2DP/stereo paths now use mode = .videoRecording (the documented mode that keeps Bluetooth output and supports multi-capsule stereo); after setActive(true) on an A2DP preset we call overrideOutputAudioPort(.none) to clear any lingering speaker override. NOTE: built-in mic + A2DP output during active recording is inherently flaky on iOS (A2DP is output-only; recording wants to collapse BT to mono HFP) — this is best-effort, not guaranteed on every BT device. Needs on-device testing.
  2. Stereo mic only captured the left channel (BUG — fixed). Real built-in-mic stereo needs more than setPreferredInputNumberOfChannels(2): per WWDC20 you must select a data source whose supportedPolarPatterns contains .stereo, call setPreferredPolarPattern(.stereo), set setPreferredInputOrientation(.portrait), then request 2 channels. The old code set the data source/polar pattern to nil for the stereo preset, so channel 2 was silent. Fix: new configureStereoCapture() does the full WWDC20 sequence and falls back to mono if the route has no stereo-capable capsule (BT/wired mic). New configureMonoCapture() resets to 1 channel (so stereo→mono actually takes effect) and applies the user's chosen mono orientation/polar pattern.
  • Verified: xcodebuild -scheme VoiceCatiOS -destination 'generic/platform=iOS Simulator' ARCHS=arm64 — BUILD SUCCEEDED. On-device behavior (BT headphones + VoiceOver + stereo pickup) still to be confirmed by the user.
  • Follow-up (same day) after device testing. User reported: default + BT-mono-mic work, but Stereo Mic still kills output, and remote audio may never be audible. b. Remote audio never routed (BUG — fixed). The AVAudioSession was activated lazily on the .streamStarted event, but the core opens its miniaudio playback device before emitting that event — so playback opened against an inactive session and produced no sound. Fix: activate the session proactively on .authResult == .ok (AppState.swift), so it's live before any remote stream opens a device. Mirrors the macOS "session active while connected" model.

• Done: iOS stereo mic + A2DP output — audio death fix (2026-06-19). The previous "fix" (line below, "Stereo mic killed A2DP output") was never verified on-device and still had the bug: selecting the Stereo Mic preset killed all audio (including VoiceOver) — only kill & relaunch recovered it. Root cause found by comparing against TeamTalk5 (Client/iTeamTalk/), which achieves stereo mic + A2DP output. Five issues fixed:

  1. configureStereoCapture was missing setPreferredInput + setInputDataSource (BUG — fixed). The previous fix threw out setPreferredInput together with setPreferredInputNumberOfChannels(2), blaming the combination for collapsing A2DP. WRONG — TeamTalk5 keeps setPreferredInput + setInputDataSource (SoundDevicesModel.selectDataSource:147-148) and only omits setPreferredInputNumberOfChannels(2). Without the explicit input anchor, when setPreferredPolarPattern(.stereo) fired, iOS had no session-level input anchor and the route reconfiguration collapsed the A2DP output. Fix: configureStereoCapture now calls setPreferredInput(builtIn) + setInputDataSource(stereoSource) after the polar pattern. Also removed setPreferredInputOrientation(.portrait) (TeamTalk doesn't use it; possible route-collapse contributor on iOS 26).
  2. Core devices not restarted around applyConfiguration (BUG — fixed). TeamTalk5's setupSoundDevices calls closeSoundDevices() FIRST, then reconfigures, then reopens — so the audio unit never sees a route change mid-flight. VoiceCat reconfigured AVAudioSession while the core's miniaudio devices were still open, leaving them bound to the dead route. Fix: new C ABI function vc_audio_restart (full stop + re-init, unlike suspend/resume which only stop/start). IOSAudioRouter's setters now wrap applyConfiguration() with audioSuspend → reconfigure → audioRestart. AudioSessionManager.handleRouteChange also calls audioRestart on device plug/unplug so the playback device picks up the new route.
  3. Deprecated .allowBluetooth instead of .allowBluetoothHFP (modernized). TeamTalk5 uses .allowBluetoothHFP (iOS 17+). VoiceCat was using the deprecated .allowBluetooth alias. Replaced; also added .bluetoothHighQualityRecording on iOS 26+ (mirrors TeamTalk5 UtilSound.swift:229-231).
  4. Capture channels not reset when switching stereo→mono (BUG — fixed). Switching from stereo to mono only changed the AVAudioSession polar pattern, not the LocalStream's capture_channels field — so the next engine start still opened 2 channels. Fix: AudioSessionManager now tracks activeMicStreamId (set by SessionState on join/leave voice); selectCaptureChannels and applyPreset call setCaptureChannels(streamId, channels.channelCount) inside the suspend window so the field is updated before the engine restarts.
  5. Docs corrected. docs/voice.md §8, docs/tech-stack.md §2, docs/architecture.md §4 — removed stale setPreferredInputNumberOfChannels(2) references; documented the actual recipe (.stereo polar pattern + setPreferredInput + setInputDataSource + vc_set_capture_channels) and the new vc_audio_restart ABI + close→reconfigure→reopen ordering rule. voicecat.h vc_set_capture_channels doc comment also corrected.

  • New C ABI: vc_result vc_audio_restart(vc_client* c) — full audio engine restart (stop + uninit + re-init), append-only addition. Skeleton stub added. Swift wrapper: VoiceCatClient.audioRestart().

  • Verified: cmake --build --preset dev + ctest --preset dev — 21/21 green (including test_stereo_mic_capture). xcodebuild -arch arm64 ONLY_ACTIVE_ARCH=YES — BUILD SUCCEEDED (iOS client compiles + links). On-device verification still pending user test: (a) Voice Chat → Stereo Mic, (b) BT Headphones + Mono Mic → Stereo Mic, (c) Stereo Mic → Voice Chat, (d) unplug/replug Bluetooth mid-stereo-session.

    a. Stereo mic killed A2DP output (BUG — fixed correctly this time). My first follow-up wrongly concluded stereo ⊥ Bluetooth was a hardware limit and forced stereo → speaker. WRONG — TeamTalk5 (Client/iTeamTalk/iTeamTalk/UtilSound.swift) and Ferrite both do built-in stereo mic + A2DP output. The actual cause was how stereo was requested: setPreferredInputNumberOfChannels(2) + setPreferredInput(builtInMic) + mode=.videoRecording together collapse the A2DP output route. Fix (TT5 recipe): stereo is now enabled purely by setting the built-in mic data source's .stereo polar pattern (configureStereoCapture); NO setPreferredInputNumberOfChannels, NO setPreferredInput (with HFP disabled the system already routes input to the built-in mic); mode=.default for stereo (.voiceChat/VPIO forces mono). The channel count is requested by miniaudio at the AU level via vc_set_capture_channels(2). Reverted the force-to-speaker workaround: .stereoMic/.studio presets use A2DP again (speaker fallback when no BT); re-removed showsStereoForcesSpeakerWarning; .allowAirPlay added to BT presets. clearStereoPolarPattern() resets the capsule when returning to mono. So stereo mic + A2DP output now coexist.

  • Verified: rebuilt — BUILD SUCCEEDED. On-device confirmation (hearing remote clients; stereo mic + A2DP output with both channels) still pending user test.

• Done: iOS audio overhaul + Join/Leave Voice + channel-id sync fix (2026-06-19). Three problems found while reviewing the iOS client, all fixed:

  1. Mic button permanently dimmed (BUG — fixed). VoiceControlsView.swift:26 gated the mic button on session.currentChannelId == 0, but SessionState never synced currentChannelId from the self user's channelId on connect. The server auto-places every newly-authed user into the Lobby (channel 1, server/src/session_registry.cpp:111), but the iOS client ignored it. Fix: added syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522); called from init, .channelList, .userJoined/.userLeft/.userUpdated, .joinResult. Added applyServerMuteState(muted:deafened:) (mirrors macOS lines 693-700); called from .userUpdated. Added serverMuted/serverDeafened to VoiceState.
  2. No Join/Leave Voice button (fixed — parity with macOS). Replaced the icon-only mic toggle with an explicit "Join Voice"/"Leave Voice" text button (mirrors macOS micToggleButton). Mute/deafen buttons now disable when not in voice. PTT path kept.
  3. Limited audio input/output options (fixed — full IOSAudioRouter.swift). New IOSAudioRouter singleton drives all iOS audio routing via AVAudioSession before the core (miniaudio) opens its device: input port selection (availableInputs), built-in mic orientation (setPreferredDataSource: front/back/top/bottom), polar patterns (setPreferredPolarPattern: omni/cardioid/subcardioid/bidirectional), Bluetooth mode (.allowBluetooth HFP voice / .allowBluetoothA2DP stereo output / neither), mic processing mode (.voiceChat Standard with AEC/AGC/HPF / .measurement Raw with all processing off + speaker echo warning), stereo capture (setPreferredInputNumberOfChannels(2) → vc_set_capture_channels), AirPlay via AVRoutePickerView. All choices persisted in UserDefaults; re-applied on route changes. AudioSessionManager refactored to delegate routing to IOSAudioRouter.
  4. Core stereo-mic capture (new C ABI: vc_set_capture_channels). Append-only ABI addition: vc_result vc_set_capture_channels(vc_client*, uint32_t stream_id, uint32_t channels) (1=mono, 2=stereo). LocalStream gained a capture_channels field; ensure_audio_running() reads it into AudioParams.capture_channels before the device opens. audio_engine.cpp capture_accum_ sized to frame_samples_ * capture_channels; on_capture updated to the same channel-aware accumulation pattern as on_loopback. Skeleton stub added. Test test_stereo_mic_capture (headless, feeds L≠R stereo through the mic accumulator path, asserts L≠R end-to-end). ctest --preset dev — 21/21 green. Swift wrapper: VoiceCatClient.setCaptureChannels(streamId:channels:).
  5. Settings UI rework. SettingsView replaced the miniaudio-based device picker with the AVAudioSession-derived tree: Audio Input (port picker → built-in mic orientation/polar pattern sub-pickers + mic mode Standard/Raw + channels Mono/Stereo), Audio Output (bluetooth mode + current route read-only + AirPlay), Voice (input mode, VAD threshold).
  6. Deployment target raised to iOS 18.0. Package.swift + project.pbxproj (4 occurrences). Unlocks newest AVAudioSession APIs.
  7. Docs updated: docs/tech-stack.md §2 (iOS audio routing via IOSAudioRouter, stereo mic, deployment 18.0, fixed stale "deferred" claim about vc_audio_suspend/resume), docs/architecture.md §4 (Swift binding notes — IOSAudioRouter, fixed stale "deferred" claim), docs/voice.md (new iOS mic capture subsection), docs/roadmap.md (iOS pending list updated), docs/building.md §9 (deployment target 18.0), PROGRESS.md (this entry).
  • Verified: cmake --build --preset dev + ctest --preset dev — 21/21 green (including new test_stereo_mic_capture: total_diff=8433549, seen_channels=2).
  • Original plan (kept for reference): Three problems found while reviewing the iOS client:
  1. Mic button permanently dimmed (BUG — root cause). VoiceControlsView.swift:26 gates the mic button on session.currentChannelId == 0, but SessionState never syncs currentChannelId from the self user's channelId on connect. The server auto-places every newly-authed user into the Lobby (channel 1, server/src/session_registry.cpp:111), but the iOS client ignores it — currentChannelId is set in only two places: .joinResult (SessionState.swift:97) and reset to 0 by leaveChannel() (SessionState.swift:135). The init (SessionState.swift:55-57) calls refreshUsers() but throws away self's channelId; the .channelList/.userUpdated handlers (SessionState.swift:76-79) likewise only re-fetch the list. macOS does this sync correctly (MainWindowController.swift:461,491,522). Because the button stays dimmed, startMicStream() (and thus the mic permission prompt) can never be triggered — a chicken-and-egg that looks like a permission issue. Not a permission bug: Info.plist:23 declares NSMicrophoneUsageDescription and SessionState.swift:139 calls AVAudioApplication.requestRecordPermission. Fix = mirror macOS: add syncSelfChannel(), call it from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult; also add applyServerMuteState(muted:deafened:) (iOS currently ignores server mute/deafen).
  2. No Join/Leave Voice button (parity gap). macOS (MainWindowController.swift:303,767-797) and Windows have an explicit "Join Voice"/"Leave Voice" button; iOS has only an icon mic toggle. Add an explicit button. Screen-audio button deferred to the ReplayKit Broadcast Upload Extension milestone (PROGRESS.md line below) — needs the extension to actually function.
  3. Limited audio input/output options (feature gap). Current SettingsView.swift:41-60 uses miniaudio's vc_list_devices which on iOS returns ~2 entries ("iPhone mic"/"Default"). miniaudio does NOT touch AVAudioSession on iOS — it opens the current default route via CoreAudio (AudioUnit/AudioQueue) and that's it. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) must be driven manually from Swift via AVAudioSession before the core opens its device. The current AudioSessionManager.swift:13-15 uses .voiceChat + .allowBluetooth, which forces HFP (mono 8/16 kHz + heavy output processing) whenever a Bluetooth headset is connected — this is why stereo output is degraded with BT headsets.

  Plan (decisions locked with user 2026-06-19):

  • Part A — Channel-id sync (bug fix, smallest, unblocks mic button): SessionState gains syncSelfChannel() (finds self in users, sets currentChannelId = user.channelId); called from init, .channelList, .userJoined, .userLeft, .userUpdated, .joinResult. Add applyServerMuteState(muted:deafened:) (mirror macOS lines 693-700); call from .userUpdated.
  • Part B — Join/Leave Voice button: Replace the icon-only mic toggle in VoiceControlsView with an explicit "Join Voice"/"Leave Voice" button (mirror macOS micToggleButton). Keep PTT path, mute/deafen, disconnect, level meter. No screen-audio button (deferred).
  • Part C — iOS audio routing layer (Swift, new IOSAudioRouter.swift): Drive selection via AVAudioSession before the core opens its device. Input enumeration via availableInputs → ports (builtInMic/bluetoothHFP/headsetMic/usbAudio/airPlay); for builtInMic walk port.dataSources → expose orientation (front/back/top/bottom) + polar patterns (supportedPolarPatterns: omni/cardioid/subcardioid/bidirectional). Apply via setPreferredInput/setPreferredDataSource/setPreferredPolarPattern/ setPreferredInputNumberOfChannels(2) for stereo. Bluetooth mode as category options: "BT HFP voice" (.allowBluetooth, mono voice, BT mic) / "Built-in Mic + BT A2DP stereo" (.allowBluetoothA2DP only — stereo output, built-in mic, no HFP processing) / "Built-in Mic + Speaker" (neither). Mic processing mode: "Standard" (.voiceChat, AEC/AGC/HPF on) / "Raw / Studio" (.measurement, all processing off — allowed always with a warning when output route is the speaker, echo risk since no AEC). Output: read-only currentRoute.outputs display + AirPlay via AVRoutePickerView. Persist choices in UserDefaults; re-apply on route changes. Note: Voice Isolation / Wide Spectrum (iOS 17+/18+) are user-toggleable in Control Center for .voiceChat apps — surface as a hint, not a programmatic toggle.
  • Part D — Core stereo-mic capture (C ABI change, append-only): AudioParams.capture_channels is hardcoded to 1 (audio_engine.h:85); encode path already stereo-capable (proven by WASAPI loopback stereo work, 2026-06-17 entry below). New C ABI: vc_result vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels); (1 or 2) — new setter, not a struct change, keeps vc_stream_desc stable. client.{h,cpp}: set_capture_channels → per-stream, flows into AudioParams.capture_channels before AudioEngine::start for MIC kind. client.cpp on_capture_frame: mic path with channels==2 reuses the existing stereo encode branch. audio_engine.cpp: capture device already opens with p.capture_channels — just needs the value to propagate. Swift VoiceCatCore: add setCaptureChannels(streamId:channels:); IOSAudioRouter calls it when user picks stereo built-in mic.
  • Part E — Settings UI rework: Audio Input section (input port picker → if builtInMic, show orientation + polar pattern sub-pickers + mic mode Standard/Raw + channels Mono/Stereo); Audio Output section (bluetooth mode HFP/A2DP/Off + current route read-only + AirPlay button); Voice section (existing: input mode, VAD threshold, PTT). Replace the current miniaudio-based device picker (SettingsView.swift:41-60) with the AVAudioSession-derived tree.
  • Part F — Docs + deployment target: Raise iOS deployment target to 18.0 (user decision — unlocks newest audio APIs; project.pbxproj + Package.swift .iOS(.v18)). Update docs/tech-stack.md §2 (iOS audio routing via AVAudioSession, stereo mic, deployment 18.0), docs/voice.md (new iOS mic capture subsection), docs/roadmap.md (iOS pending list), docs/architecture.md §4 (Swift binding notes — iOS audio routing in Swift layer, core stereo-mic via new ABI setter), PROGRESS.md (this entry + next-action pointer).
  • Implementation order: A (channel-id sync, testable immediately) → B (Join/Leave Voice) → D (core stereo-mic + ABI, verify with ctest --preset dev) → C (Swift routing layer, depends on D) → E (Settings UI, depends on C) → F (docs, same commit as work).
  • Verification: xcodebuild -target VoiceCatiOS -sdk iphonesimulator -configuration Debug build → BUILD SUCCEEDED; swift test green + stereo-mic capture round-trip test; ctest --preset dev green (core changed in D); manual (simulator/device): join channel → Join Voice works → input picker shows built-in mic sub-options (orientation/polar pattern) → toggle Raw mode (warning appears on speaker) → toggle Bluetooth A2DP stereo output → toggle stereo capture.
  • Files to touch:
    • iOS Swift: SessionState.swift, AudioSessionManager.swift (+ new IOSAudioRouter.swift), Views/VoiceControlsView.swift, Views/SettingsView.swift, VoiceCatiOSApp.swift, VoiceCatiOS.xcodeproj/project.pbxproj (iOS 18.0).
    • Swift core: Sources/VoiceCatCore/VoiceCatClient.swift (setCaptureChannels), Sources/VoiceCatCore/Models.swift (input-port model), Package.swift (.iOS(.v18)).
    • Core C++: core/include/voicecat.h (vc_set_capture_channels), core/src/voicecat.cpp, core/src/core/client.{h,cpp}, core/src/audio/audio_engine.{h,cpp}.
    • Docs: docs/tech-stack.md, docs/voice.md, docs/roadmap.md, docs/architecture.md, PROGRESS.md.
    • Tests: clients/apple/Tests/VoiceCatCoreTests/, tests/ (stereo-mic encode test).
  • Why miniaudio can't do this alone: On iOS, miniaudio uses CoreAudio (AudioUnit/AudioQueue) to open the current default route for PCM I/O — it never touches AVAudioSession. ma_context_get_devices on iOS returns a near-empty list (iOS doesn't expose a real CoreAudio device list like macOS does). All iOS audio routing (input ports, data sources, polar patterns, HFP/A2DP, measurement mode, preferredInputNumberOfChannels) must be driven from Swift via AVAudioSession before miniaudio opens its device. miniaudio just reads whatever route AVAudioSession has established.

• Done: iOS SwiftUI client — VoiceCatiOS (2026-06-19). Full SwiftUI app at clients/apple/iOS/VoiceCatiOS.xcodeproj. Mirrors the macOS AppKit and Windows WinForms clients feature-for-feature: saved server list (JSON + Keychain passwords via App Group group.cat.voice.VoiceCat), TOFU server-identity sheet (first-connect and mismatch), connect flow (guest + account auth, connection state labels, password prompt), main window (NavigationSplitView on iPad, TabView on iPhone via horizontalSizeClass), channel tree (OutlineGroup recursive — children: nil for leaf nodes), user list with context menu (kick/ban/move/set-permissions/server-mute), chat (ScrollViewReader + LazyVStack), activity log, voice controls (mic toggle, PTT via DragGesture(minimumDistance: 0) + @GestureState, level meter Canvas), per-user tuning (gain slider, mute, NR toggles), admin sheets (channels CRUD, accounts CRUD, ban duration, move-to-channel), settings (input mode, VAD threshold, device picker, disconnect). All 24 Swift source files in place (10 model/state + 14 views). @Observable + @MainActor throughout — both AppState and SessionState are @MainActor. onEvent closures dispatch to main actor via Task { @MainActor in ... }. C ABI additions: vc_audio_suspend / vc_audio_resume (no-op when engine stopped) — called by AudioSessionManager on AVAudioSession interruption (phone calls, Siri). XCFramework: iOS arm64 + arm64-simulator slices added alongside existing macOS-arm64 slice; Package.swift updated with .iOS(.v17). Verified: xcodebuild -target VoiceCatiOS -sdk iphonesimulator26.5 -configuration Debug ... SYMROOT=... OBJROOT=... build → BUILD SUCCEEDED (clean build).

  • Next: ReplayKit Broadcast Upload Extension (VoiceCatBroadcast) for screen/system audio sharing on iOS — separate Xcode target, App Group credential sharing, SampleHandler.swift. Or: DRED/audio-quality polish (M5).

• Done: macOS AppKit client builds clean (Debug + Release) (2026-06-18). The previous "shipped" claim in the entry below was incorrect — xcodebuild actually failed on a fresh clone. Two real defect classes fixed, no source architecture changed:

  1. Swift compile errors in MainWindowController.swift (the previous agent wrote AppKit API calls from memory that didn't match the SDK):
     • NSAccessibility.post(notification:element:userInfo:) — wrong argument order. The Swift import (verified against AppKit.apinotes in the macOS 26.5 SDK) is NSAccessibility.post(element:notification:userInfo:). 3 call sites fixed.
     • NSAccessibilityPriorityMedium — not a Swift symbol. The C constants NSAccessibilityPriorityHigh/Medium/Low are imported by Swift as cases of the NSAccessibilityPriorityLevel enum (it's an NS_ENUM(NSInteger, ...) in NSAccessibilityConstants.h, no apinotes rename). Replaced with NSAccessibilityPriorityLevel.medium at 3 call sites.
     • streams.first(where: { $0.streamId == event.streamId }) — StreamSummary has no streamId property; the init parameter is named streamId but the stored property is id (per VoiceCatCore/Models.swift:102-110, Identifiable conformance). The bad property access made the closure fail to type-check, which made the compiler treat streams.first as a property (returning StreamSummary?) and then try to call it — hence "cannot call value of non-function type 'StreamSummary?'". Fixed to $0.id.
     • Removed a redundant fileprivate var description extension on VoiceCatResult in ConnectWindowController.swift — VoiceCatResult already has a public var description in VoiceCatCore/Enums.swift, so the redeclaration would have errored ("invalid redeclaration") once the compiler got past MainWindowController.swift.
  2. Linker error — libvoicecat-fat.a is C++ but the app target didn't link libc++ (the previous agent's project.pbxproj had OTHER_LDFLAGS empty). The pure-Swift app pulls in libvoicecat-fat.a (a static C++20 archive that references std::__1::*, __cxa_*, operator new/delete, etc.), but the linker has no reason to pull in libc++ on its own — there are no .cpp sources in the app target. The VoiceCatCore package's test target sidesteps this with linkerSettings: [.linkedLibrary("c++")] in Package.swift:54-56, which is why swift test was green but xcodebuild wasn't. Fixed by adding OTHER_LDFLAGS = ("$(inherited)", "-lc++") to BOTH the Debug and Release target configurations in VoiceCatMac.xcodeproj/project.pbxproj. otool -L on the produced dylib confirms /usr/lib/libc++.1.dylib is now linked.
  3. Release config tried to build x86_64 (XCFramework only has arm64) — the project-level Release config (AAAA…000C) lacked ONLY_ACTIVE_ARCH = YES, so Release built the standard ARCHS_STANDARD (arm64 + x86_64) and the x86_64 slice failed with Undefined symbols for architecture x86_64: _vc_authenticate_guest … because VoiceCatCore.xcframework/macos-arm64 only contains an arm64 slice. Added ONLY_ACTIVE_ARCH = YES to the project-level Release config to match the XCFramework. For distribution (App Store / universal binary), the right fix is to make build-xcframework.sh --all also produce an x86_64 macOS slice — left as a future enhancement; the current setup builds a working arm64 Release on Apple Silicon.
  • Verified: xcodebuild -project … -scheme VoiceCatMac -configuration Debug build → BUILD SUCCEEDED (clean build, not incremental). Release config → BUILD SUCCEEDED. otool -L on VoiceCatMac.debug.dylib shows libc++.1.dylib, Security.framework, AppKit, Foundation, CoreFoundation, swift runtime libs. nm shows _vc_client_create
    • _vc_version_string are present (the static libvoicecat-fat.a linked in). App launches and stays running (verified via background launch + kill -0).
  • Lesson: the previous "shipped" entry was written without ever running xcodebuild — a clean compile is the floor, not the goal (AGENTS.md). The swift test green status was real but tested only the VoiceCatCore package, not the macOS app target. The app target had never been built.

• Done: macOS AppKit UI — VoiceCatMac (2026-06-18). Full AppKit application at clients/apple/macOS/VoiceCatMac.xcodeproj. Mirrors the Windows WinForms client feature-for-feature: saved server list (JSON + Keychain passwords), TOFU server-identity sheet (first-connect and mismatch paths), connect flow (guest + account auth, connection state labels), main window (NSSplitView layout with NSOutlineView channel tree, NSTableView user list, NSTextView chat, activity log), voice controls panel (Join/Leave mic, screen audio, mute/deafen checkboxes, VAD/PTT/Always-On segmented control, VAD sensitivity slider, PTT key capture, input device picker, level meter), compose bar (scope picker for channel/private text, NSTextField + Send), right-click context menus on channels and users (join, create/edit/delete channels, kick/ban/move/server-mute/deafen/ set permissions), admin menu (server accounts sheet with CRUD), 10 sheet view controllers. Full VoiceOver accessibility: every control has setAccessibilityLabel; NSAccessibility.post (.announcementRequested) on join, talk-state, stream events. All 17 Swift source files in place; PttKeyCaptureSheet uses a KeyCaptureView: NSView subclass that becomes first responder and captures keyDown. Build prerequisite: run clients/apple/scripts/build-xcframework.sh first; then xcodebuild -project clients/apple/macOS/VoiceCatMac.xcodeproj -scheme VoiceCatMac build.

  • Next: iOS SwiftUI client (clients/apple/iOS/) — same VoiceCatCore Swift package, SwiftUI instead of AppKit. Or: DRED/audio-quality polish (M5).

• Done: macOS/iOS Swift core — VoiceCatCore package + tests (2026-06-18). The shared Swift core for the macOS (AppKit) and iOS (SwiftUI) clients is built and verified. This is the foundation that both platform UIs will build on — mirrors the Windows client's proven VoiceCat.Interop layer using Swift-native C interop.

  • Architecture decision: macOS UI = AppKit (not SwiftUI), for the most mature VoiceOver accessibility story — same rationale as the Windows client's WinForms-over- WinUI-3 decision. iOS stays SwiftUI. Recorded in docs/roadmap.md §2 and docs/tech-stack.md §2.
  • VoiceCatCore Swift Package (clients/apple/Package.swift): binary target (VoiceCatCoreXCF → VoiceCatCore.xcframework) + library target (VoiceCatCore — the Swift wrapper) + test target (VoiceCatCoreTests). 7 source files: Enums.swift (Swift mirrors of the 9 C enums), Config.swift, Event.swift (copies ev.text to String inside the callback — the #1 lifetime rule), Models.swift (Channel/User/Stream/Device/Permissions/Account/AudioConfig/…), Marshaling.swift (C arrays → Swift arrays + immediate vc_free_*), Callbacks.swift (@convention(c) + Unmanaged.passUnretained — the Swift analog of C#'s [UnmanagedCallersOnly] + GCHandle), VoiceCatClient.swift (owns vc_client*, all 38 C functions, deinit → vc_client_destroy then frees config CStrings, event delivery on @MainActor via coalesced DispatchQueue.main drain).
  • build-xcframework.sh (clients/apple/scripts/): runs cmake --preset apple-dev, merges libvoicecat.a + 107 vcpkg static deps into a single ~30 MB fat static library (libvoicecat-fat.a) via libtool -static, stages voicecat.h + a generated module.modulemap (module VoiceCatC { header "voicecat.h" }) into the XCFramework headers, runs xcodebuild -create-xcframework → clients/apple/VoiceCatCore.xcframework. The fat library is needed because SPM binary targets link ONE .a per slice — without it, the final executable gets undefined-symbol errors for protobuf/mbedtls/sodium/opus/… (the Apple equivalent of how the Windows client ships a single voicecat.dll with all deps statically linked).
  • Tests — 6/6 green: swift test against a real voicecat-server (built by cmake --preset dev). Tests mirror the C# VoiceCat.Interop.Tests: testVersionStringIsNonEmpty, testResultStringRoundTrips, testConnectTofuAuthListChannelsRoundTrips (connect → TOFU → confirm → guest auth → channels → permissions → guest ListAccounts rejected), testAdminChannelCrudAccountCrudRoundTrips (admin auth → channel CRUD → account CRUD), testScreenAudioStreamStartsAndStops (screen-audio stream lifecycle through Swift), testPerStreamRecvControlsRoundTrip (two clients, per-stream gain/mute/NR round-trip). Catches Swift-specific interop bugs (@convention(c) callback lifetime, Unmanaged pointer resolution, CString memory management, enum raw-value bridging, struct layout) that C++ ctest can't.
  • Key C interop discoveries: (1) Swift imports vc_client* (incomplete C struct) as OpaquePointer?, not a named type — private var handle: OpaquePointer?. (2) C enums are imported as UInt32-backed (not Int32) — all Swift enum mirrors use UInt32; vc_event.result is int32_t (signed), bridged via UInt32(bitPattern:). (3) Swift auto-marshals String to const char* for function params, but NOT for C struct fields — vc_stream_desc/vc_channel_info string fields need strdup + defer { free }. (4) selfPointer must be a computed property (not stored) to break the circular init dependency (Unmanaged.passUnretained(self) needs self fully initialized, but stored properties must be set first).
  • Docs updated: docs/tech-stack.md §2 (AppKit macOS / SwiftUI iOS / shared VoiceCatCore package / fat static lib), docs/architecture.md §4 (Swift binding notes), docs/roadmap.md M4 + §2 (AppKit decision recorded), clients/apple/README.md (full rewrite), PROGRESS.md (this entry), .gitignore (XCFramework + SPM artifacts).
  • Next: macOS AppKit app (clients/apple/macOS/) — connect dialog, saved-server list (Keychain), TOFU identity dialog, main window (NSOutlineView + NSTableView + NSTextView
    • activity log), voice controls, per-user tuning, full VoiceOver accessibility. Mirrors the Windows VoiceCat.App feature set.

• Done: macOS port — dev + apple-dev presets validated (2026-06-18). The core, server, tools, and tests now build and run on macOS (Apple Silicon, macOS 26.5, Apple clang 21). This lays the groundwork for the macOS/iOS Swift client. Three real bugs found and fixed (all were cross-platform issues that manifested on macOS but were latent on Windows/Linux):

  1. Missing <netdb.h> in test_m2_voice.cpp POSIX branch — the raw-socket test's #else branch included <arpa/inet.h>/<netinet/in.h>/<sys/socket.h>/<unistd.h> but not <netdb.h> (needed for addrinfo/getaddrinfo/freeaddrinfo). On Linux glibc these headers transitively include <netdb.h>; on macOS they don't. Fixed by adding # include <netdb.h> to the POSIX branch (mirrors core/src/core/client.cpp:16 which already had it). Real latent bug — would fail on any strict POSIX system.
  2. SIGPIPE killing processes on macOS — on macOS, writing to a closed TCP socket raises SIGPIPE by default (unlike Windows where it doesn't exist, or Linux where it's often benign). This killed test_tofu_flow (intermittent SIGPIPE/SEGFAULT) and would also kill voicecat-server and vccli in production when a peer dropped mid-write. Fixed by ignoring SIGPIPE (std::signal(SIGPIPE, SIG_IGN)) in both the core client init (core/src/core/client.cpp POSIX branch of the #ifdef _WIN32 WSAStartup block) and the server startup (server/src/server.cpp before the asio::signal_set). Both are POSIX-only (#ifndef _WIN32), process-global, and idempotent. The server's asio::signal_set(SIGINT, SIGTERM) is unaffected (independent signals).
  3. Use-after-free of Asio's kqueue reactor on server shutdown — the root cause of the deterministic test_tofu_flow segfault (EXC_BAD_ACCESS in kqueue_reactor::deregister_descriptor(this=0x0000000000000000)). TcpServerConn's tls_read_loop runs on a dedicated blocking-I/O thread (not async on io_context). When Server::stop() → io.stop() → Server::run() returned, the local asio::io_context was destroyed while tls_read_loop threads were still running. When a thread detected the disconnect and called TcpServerConn::close() → socket.close() → Asio tried to deregister from the kqueue reactor — but the reactor (owned by io_context) was already destroyed, and on macOS kqueue the reactor pointer is null'd immediately. Latent on Windows (IOCP) and Linux (epoll) where the timing is more forgiving. Fix: TcpAcceptor now tracks its connections (new conns_ vector + conns_mu_); TcpAcceptor::stop() closes all tracked connections while io_context is still alive; new TcpAcceptor::shutdown() method calls stop() then wait_closed() on each connection (new TcpServerConn::wait_closed() joins tls_thread_); Server::run() calls acceptor.shutdown() after io.run() returns and before io is destroyed; Server::stop()'s stop_fn_ now calls acceptor.stop() + media_relay->stop() + io.stop() (was just io.stop()). After acceptor.shutdown(), when tls_read_loop threads exit and call on_disconnected → ConnSession::close() → TcpServerConn::close(), the close() is a no-op (closing_.exchange(true) returns true) — no reactor access occurs after io is destroyed.
  • Environment setup: vcpkg cloned to ~/code/vcpkg + bootstrapped. VCPKG_ROOT must be set. Homebrew autoconf-archive is required (vcpkg's libsodium port needs it for autoreconf — brew install autoconf-archive). autoconf/automake/libtool were already installed; glibtoolize (Homebrew's macOS name for libtoolize) is handled by vcpkg automatically.
  • Verified: cmake --preset dev + cmake --build --preset dev green (21 binaries). ctest --preset dev --parallel 1 — 21/21 green (2 consecutive runs, 64s each). cmake --preset apple-dev + cmake --build --preset apple-dev green → valid 1.9 MB arm64 libvoicecat.a (167 exported C ABI symbols, correct visibility). xcodebuild -create-xcframework → valid VoiceCatCore.xcframework (macOS-arm64 slice with voicecat.h headers). Server runtime: voicecat-server starts, generates identity, SQLite, Lobby, binds TCP+UDP, clean SIGINT shutdown. Two vccli text chat over TLS (M1 exit criterion on Mac). vccli --voice --input-mode vad starts a MIC stream via CoreAudio (M2 protocol-level on Mac — ear test pending). vccli --list-devices enumerates 4 CoreAudio input + 3 output devices with correct defaults.
  • Apple framework linking: NOT needed — modern macOS ld (Xcode 26.5) auto-discovers CoreAudio/CoreFoundation frameworks in /System/Library/Frameworks without explicit -framework flags. miniaudio's MINIAUDIO_IMPLEMENTATION compiles the CoreAudio calls inline, and the linker resolves them automatically. No if(APPLE) CMake block was added.
  • Docs updated: docs/building.md (apple-dev row + §6 updated from "scaffolding" to "validated"), clients/apple/README.md (macOS slice build confirmed, iOS slices still scaffolding), PROGRESS.md (this entry).
  • Still deferred (per scope): macOS SCREEN_AUDIO loopback via ScreenCaptureKit (stub returns false — per docs/voice.md §9); iOS cross-compile presets (apple-ios/ apple-ios-sim — scaffolding); vc_audio_suspend/vc_audio_resume ABI hooks (defer to iOS client milestone, keep ABI stable).

• Done: CMake preset cleanup + cross-platform build config (2026-06-18). The preset set was a mess — dev (never used), m1-dev (the one everyone used), m2-dev (cache-identical to m1-dev, never used), no optimized+tests preset, no stripping. Cleaned up to a sensible set + added cross-platform triplet auto-resolution + Apple platform scaffolding:

  • Renames: dev→skeleton (no-deps stub smoke — accurately named now); m1-dev→dev (the default development preset — milestone-named presets were misleading since the project is past M5); m2-dev dropped (cache-identical to m1-dev).
  • New presets: release (optimized Release + tests on, symbols kept — run the suite against optimized code or profile); apple-dev / apple-ios / apple-ios-sim (scaffolding — static libvoicecat.a slices for the Swift Package / XCFramework; marked "not yet CI-validated, build on macOS to verify").
  • server-release now strips binaries (CMAKE_EXE_LINKER_FLAGS=-s + CMAKE_SHARED_LINKER_FLAGS=-s) — smaller executables for deployment.
  • Cross-platform triplet auto-resolution: new cmake/voicecat-toolchain.cmake wraps vcpkg's toolchain and resolves VCPKG_TARGET_TRIPLET / VCPKG_HOST_TRIPLET from CMAKE_HOST_SYSTEM_NAME + CMAKE_HOST_SYSTEM_PROCESSOR — x64-mingw-static on Windows, x64-linux on Linux, arm64-osx on Apple Silicon. The main presets (dev, release, server-release) now work on all three platforms without per-OS variants. Cross-compile presets (apple-ios, apple-ios-sim) override VCPKG_TARGET_TRIPLET explicitly. Hidden base preset renamed vcpkg-base→vcpkg-common (now points at the wrapper toolchain instead of vcpkg's toolchain directly).
  • No C++ source changes — the core was already portable (every #ifdef _WIN32 in client.cpp already had a POSIX #else; voicecat.h's export macro already handled GCC visibility; transport.cpp is pure Asio; miniaudio abstracts WASAPI/CoreAudio/ALSA).
  • Docs updated: docs/building.md (full rewrite — 8-preset table, platform matrix, preset history mapping old names to new, Apple scaffolding section), CLAUDE.md (build section reframed around dev as default, status line updated to M5-done), README.md (stale "M0 skeleton" framing replaced with current M5 reality), AGENTS.md (build section updated, stale "placeholder builtin-baseline" sentence deleted), docs/deployment.md (server-release now stripped + cross-platform note), docs/tech-stack.md §4 (triplet auto-resolution + Apple scaffolding note), clients/apple/README.md (new "Building the core for Apple platforms" section with XCFramework workflow), clients/windows/README.md (m1-dev→dev).
  • Code comments updated: core/include/voicecat.h, core/src/net/transport.h, core/src/voicecat.cpp, tests/CMakeLists.txt — preset name references updated.
  • Historical PROGRESS.md entries left intact — m1-dev/m2-dev mentions in older entries are a true record of what was run; rewriting them would falsify history. The preset history table in docs/building.md §1 maps old names to new.
  • Verified: cmake --list-presets shows all 8 presets. skeleton + dev configure and build green on Windows. Apple presets are scaffolding (won't build on Windows — expected; they require macOS).

• Done: Disconnect, timeout & keepalive system (2026-06-18). Three reported bugs traced to one root cause + two missing designed features, all fixed:

  1. Stale users after disconnect + eternal PLC hiss (root cause): ConnSession::close() silently erased dropped users from the registry without broadcasting UserEvent::LEFT. Peers never learned the user left → their user lists stayed stale AND their audio engines never called remove_stream → Opus PLC synthesized comfort noise forever (the "soft hissing that never goes away"). Fix: SessionRegistry::broadcast_left() helper (mirrors kick_user's first half); ConnSession::close() now broadcasts LEFT before erasing. Regression test test_disconnect_left (tests/).
  2. PLC cap (defense-in-depth): AudioEngine::on_playback now caps pure-PLC at ~2 s (kPlcCapSamples); after that it emits digital silence instead of more comfort noise, so a stale stream can never hiss forever even if remove_stream is never called. Resets automatically when fresh packets arrive. Test test_plc_cap (tests/).
  3. No timeout / no ping (missing feature): the client never sent Ping, the server had no last_seen / reaper, and half-open connections (NAT timeout, wifi loss, sleep) left ghost users forever. Fix: client sends Ping every 15 s from the io thread (RTT measured from Pong nonce); ConnSession::last_seen bumped on every inbound TCP/UDP frame; asio::steady_timer reaper sweeps every 15 s and drops sessions older than 45 s (configurable via server::Config::reaper_timeout_ms/reaper_sweep_ms). Each drop broadcasts LEFT via fix #1. Test test_reaper_timeout (tests/, 2 s timeout for fast turnaround).
  4. UDP KEEPALIVE (missing feature): client sends a plaintext KEEPALIVE frame every 5 s (voice_frame.h::kFrameKeepalive); server bumps last_seen + echoes back. Keeps NAT bindings alive and lets media activity defer the reaper independently of TCP.
  5. Graceful client disconnect: vc_disconnect() now sends Disconnect{code=0} when fully authenticated (VC_STATE_CONNECTED): it queues the envelope, sets a graceful_disconnect_pending_ flag, and joins the io thread — the io thread's drain_sends() sends the Disconnect, sees the flag, sets io_stop_, and exits naturally (its cleanup handles teardown_voice() + socket close). No main-thread socket close → no double-close race. Server handles client-sent Disconnect (handle_client_disconnect → close() → immediate LEFT broadcast, no reaper/EOF wait). In non-authenticated states, the original force-close path runs (with TOFU unblock).
  • Docs updated: docs/protocol.md §6/§7 (graceful disconnect, pinned N=3, reaper, last_seen), docs/voice.md §6 (KEEPALIVE plaintext + echo + last_seen), docs/architecture.md §5 (reaper timer), server::Config (reaper fields). ctest --preset m2-dev — 21/21 green (3 new tests: disconnect_left, plc_cap, reaper_timeout).

• Done: Stereo screen-audio loopback capture on Windows (2026-06-17). The WASAPI loopback path (start_loopback_capture) used to hardcode cfg.capture.channels = 1, downmixing the system's stereo mix to mono before the encoder ever saw it — so even on a stereo channel, SCREEN_AUDIO was effectively mono (the encoder then upmixed L=R to produce a fake stereo bitstream). Now the loopback device opens in the channel's mode: stereo (interleaved L/R) when the channel is configured stereo, mono when mono. Real stereo flows end-to-end through loopback → Opus encode → decode → stereo playback mixer.

  • audio_engine.h — CaptureCallback gained an int channels parameter (the encoder needs to know whether the PCM is real stereo or mono to avoid upmixing real stereo). start_loopback_capture(int kind) → start_loopback_capture(int kind, int channels). New loopback_channels_ member; new feed_loopback_for_test test hook (self-contained, works on headless CI where the real WASAPI device can't init).
  • audio_engine.cpp — on_loopback accumulator is now channel-aware (sized to frame_samples_*loopback_channels_); start_loopback_capture sizes the accumulator off the RT thread before ma_device_start, opens the device with channels, and falls back to mono if the render endpoint rejects stereo (mirrors the playback path's fallback). on_capture/inject_capture/feed_capture_for_test forward channels through the callback (mic path always 1; loopback path 1 or 2).
  • client.cpp — on_capture_frame takes channels; for channels==2 (real stereo loopback PCM) it encodes directly with no upmix; for channels==1 on a stereo channel it keeps the existing L=R upmix (mic stays mono in v1). handle_stream_announce_result reads effective_params.stereo under the lock and passes 2 or 1 to start_loopback_capture.
  • test_vad_ptt_devices.cpp — new test_loopback_stereo_capture behavior test: feeds a loud-L / silent-R stereo signal through feed_loopback_for_test, encodes (as on_capture_frame now does for channels==2), decodes, mixes, and asserts L≠R across the frame (total_diff ~8.2M, well above the 960k threshold). A mono-downmixed-then- upmixed bitstream would have L==R. Existing test lambda updated for the 4-arg callback.
  • docs/voice.md §8/§9 — diagram + notes updated: mic stays mono; SCREEN_AUDIO loopback captures stereo when the channel is stereo.
  • cmake --build --preset m2-dev + ctest --preset m2-dev --parallel 1 — 18/18 green. The vad_ptt_devices VAD-gate sub-test has a pre-existing parallel-run timing flake (passes serially and in isolation); unrelated to this change (VAD gate logic is unchanged for channels==1, the only path the MIC uses).

• Done: Screen-audio sharing wired into the Windows WinForms client (2026-06-17). The core already fully supported SCREEN_AUDIO capture on Windows (post-M3 WASAPI loopback via VOICECAT_HAS_LOOPBACK, always on for the windows-client preset — core/CMakeLists.txt:85; loopback start/stop at client.cpp:1093/audio_engine.cpp:529; VAD/PTT/self-mute/server-mute correctly bypassed for non-MIC kinds at client.cpp:862-879) and the C# Interop layer was already complete (VcStreamKind.ScreenAudio, StartStream/StopStream/SetRemoteStream/ListUserStreams all generic). The gap was purely UI wiring. No core, proto, or C ABI changes were needed — confirming the "the core should support it already" assessment.

  • MainForm.Designer.cs — new btnScreenShareToggle button in the voice panel top row (flpVoiceTop), right after btnMicToggle, with full AccessibleName/ AccessibleDescription per the existing accessibility convention.
  • MainForm.cs — new _screenStreamId field; BtnScreenShareToggle_Click handler mirroring BtnMicToggle_Click but with no device picker / VAD / PTT / mode / mute (screen audio bypasses all of those in the core). Independent of mic — can share without joining voice and vice versa. HandleDisconnected now resets _screenStreamId and disables the screen toggle. OnFormClosed now explicitly stops both mic and screen streams before Disconnect() (clean StreamStop messages go out before the control channel closes). HandleStreamStarted already labeled ScreenAudio => "screen audio"; PerUserTuningDialog.ApplySettings already iterates all of a peer's streams — both unchanged, peers can independently volume-tune a screen-audio stream vs that user's mic.
  • VoiceCatClientSmokeTests.cs — new ScreenAudioStream_Starts_And_Stops test: connects + TOFU + guest auth, StartStream(ScreenAudio), asserts VC_EVENT_STREAM_STARTED arrives with matching StreamId, StopStream, asserts VC_EVENT_STREAM_STOPPED. Passes headless (the StreamAnnounce succeeds regardless of whether the loopback device initializes on a CI box).
  • dotnet build — 0 warnings/errors. dotnet test — 4/4 tests green (3 existing + 1 new). Event trace confirms the full StreamStarted → UserUpdated → StreamStopped path through P/Invoke against a live voicecat-server.exe.
  • Not yet confirmed audible by ear — pending manual two-instance live test (one shares screen audio while something plays on the default render endpoint, the other hears it). This is the observable-behavior exit criterion per AGENTS.md.
  • Documented caveat (docs/voice.md §9, unchanged): whole-device WASAPI loopback inherently re-captures this app's own incoming voice mix (self-echo loop) — accepted characteristic, not a bug. Process-specific loopback (Windows 10 2004+ AUDIOCLIENT_ACTIVATION_PARAMS) is a future enhancement; miniaudio doesn't expose it.

• In progress: M5 — moderation & admin (2026-06-17). Server-side and C ABI are implemented and tested: permissions, kick/ban/move/server-mute, channel CRUD, in-app account management. Four new tests pass: test_m5_permissions, test_m5_kick_ban_move_mute, test_m5_admin_accounts, test_m5_channel_crud. vccli now exposes all M5 operations via CLI flags (--kick, --ban, --move, --server-mute/-unmute/-deafen/-undeafen, --set-permission, --create-channel, --edit-channel, --delete-channel, --create-account, --reset-password, --delete-account, --list-accounts) plus --username/--password for account auth and --self-mute/--self-deafen. Docs updated: docs/protocol.md (envelope tags for ServerMuteRequest/ListAccountsResult, User.server_deafened, GenericResult usage), docs/security.md (BLAKE2b channel passwords, bans schema). ctest --preset m1-dev — 18/18 green. Windows WinForms UI now exposes all M5 operations: channel CRUD with full per-channel Opus audio config, user moderation (kick/ban/move/server mute/server deafen/set permissions), and server account management. dotnet test of the Windows solution passes. Still to do: DRED/audio-quality polish.

• Done: Fixed multi-user voice — relayed frames failed AEAD decryption (nonce desync) (2026-06-17, reported live: with 2+ people in a channel, audio was one-directional — "I can hear them but they can't hear me" — and a 3rd joiner heard nobody). Root cause was in the SFU relay (server/src/media_relay.cpp). The media AEAD nonce is an implicit per-direction monotonic counter; open() reconstructs it from the 14-byte header's seq field (the AAD), so the wire contract is header.seq == the counter seal() used (the client honors this at client.cpp:907). The relay decrypted each inbound frame with the sender's key, then re-sealed with the recipient's send_crypto (its own counter) but forwarded the sender's header verbatim — so header.seq carried the sender's counter, not the recipient's. The recipient's open() rebuilt the wrong nonce → every relayed frame failed auth and was silently dropped. It only "worked" while the sender's counter coincidentally equalled the server→recipient counter (a single first-ever sender into a fresh recipient), which is exactly why the first/sole talker was heard but reverse/3rd-party audio was not. Fix: before re-sealing, the relay rewrites the outgoing header's seq (bytes [8..9]) to the recipient's peek_send_counter(), so each server→client direction is one contiguous monotonic counter and the nonce always matches (the anti-replay window also stops seeing false replays from interleaved senders). Safe because the jitter buffer orders by timestamp, not seq (audio_engine.h); seq exists only to carry the AEAD counter. No wire-format/proto/ABI change. Regression test added in tests/test_media_aead.cpp (test_relay_interleaved_reseal): two senders interleaved into one recipient all decrypt with the fix, and the verbatim-seq path is asserted to fail. ctest --preset m1-dev — 18/18 green (run via PowerShell; Git Bash can't resolve the runtime DLLs. vad_ptt_devices is timing-flaky over loopback — passes on re-run — unrelated to this fix). Latent, separate: the secondary "3rd joiner sometimes can't see other users" report is a control-plane (TCP snapshot/UserEvent) issue, not this AEAD bug — re-verify after live testing before investigating. Also still latent: the 16-bit seq wraps after 65536 frames per direction (faster on a busy relay) with no ROC, so the implicit counter desyncs on long continuous sessions (crypto.cpp open() TODO).

• Done: Fixed "randomly bumped to Lobby" in the Windows client — actors were excluded from their own state-change broadcasts (2026-06-17, reported live: a connected client would intermittently snap from its joined channel back to Lobby in the UI). Root cause was a design inconsistency, not a disconnect: the server delivered self-initiated state changes (channel join/leave, stream announce/stop) only as a private *Result to the actor and broadcast the authoritative UserEvent::UPDATED to everyone else. The core never applied the join result to its SessionModel, so vc_list_users() kept self in the old channel; the Windows HandleUserUpdated rebuilds _currentChannelId from vc_list_users() on any user's UPDATED event, so the next unrelated event (someone joining, announcing/stopping a stream, being muted) surfaced the stale self-channel → "bumped to Lobby." Flaky because it depended on other users' activity. Fix (broad, per the actor-sees-own-changes principle): the server now broadcasts these UserEvent::UPDATEDs to all clients including the actor (server/src/conn_session.cpp, broadcast(…, /*exclude*/ 0)), and text fan-out now includes the sender (resolve_text_targets), so every client converges via one authoritative path. The *Result is now purely ack/correlation/actor-private payload; response handlers no longer mutate the local model. Windows client drops its optimistic text echo (the relay comes back) and renders the sender's own message via HandleTextMessage. Documented the response-vs-broadcast contract in docs/protocol.md §6. Registry-level admin broadcasts (move/mute/kick/channel CRUD) already used exclude=0 and were correct. ctest --test-dir build/m1-dev — 18/18 green (PowerShell); Windows VoiceCat.App builds 0 warnings. Latent, not fixed: the client never sends the Ping keepalive that docs/protocol.md §7 describes (only the server answers pings) — unrelated to this bug, noted for later. Resolved (2026-06-18): full keepalive/timeout/disconnect system implemented — see "disconnect, timeout & keepalive" entry below.

• Done: Fixed a second silent-playback bug — the playout clock free-ran and drifted off the stream (2026-06-17, reported live: both vccli and the Windows client showed talking=1/0 correctly on VAD/PTT, mic + screen-share were recognized by peers, but nothing was audible). Root cause: RemoteStream::playout_ts was only ever seeded to 0 and then advanced one Opus frame per playback callback via the PLC path too (core/src/audio/audio_engine.cpp on_playback), so it free-ran at ~1× wall-clock regardless of whether the sender was transmitting. The sender's frame timestamps only advance while it actually sends (the VAD/PTT gate in core/src/core/client.cpp returns before ls.timestamp += samples). Across a late join or any VAD/PTT silence gap the two clocks diverged without bound; once past the jitter buffer's 500 ms late-drop window, every real frame was dropped-as-late (clock ahead) or never-due (clock behind) → permanent silence, while the talk indicator (driven by push_recv_frame, independent of the jitter buffer) stayed lit. The M3 E2E test missed it because clients there talked continuously right after joining, keeping the clocks aligned. Fix: JitterBuffer gained peek_front_ts() (try-lock, RT-safe); on_playback now seeds/re-syncs playout_ts to the earliest buffered frame on the first frame and whenever it has drifted past ±200/500 ms (kResyncAheadSamples/kResyncBehindSamples), which both seeds startup and recovers after every silence gap. New regression test test_playout_resync (tests/test_vad_ptt_devices.cpp): free-runs the clock ~2 s past the drop window, pushes a ts=0 frame, asserts audible output — verified to fail (energy=0) with the fix disabled, pass (energy≈15M) with it. ctest --test-dir build/m1-dev — 14/14 green (run via PowerShell; Git Bash exec gotcha for these binaries, see docs/building.md). Not yet confirmed audible by ear — pending the user re-running their live test.

• Done: Fixed silent-playback bug in AudioEngine::on_playback (2026-06-16, found via live manual test: two vccli --voice clients, control-plane events and VAD all correct, but zero audible output). Root cause: opus_decode()'s max_samples was being passed the hardware playback callback's frame count (miniaudio's own choice, frequently smaller than one Opus frame — e.g. ~480 samples on default low-latency WASAPI periods), instead of the decoder's fixed frame size (960 @ 20ms/48kHz). Since the real packet almost always decodes to more samples than that, opus_decode returned OPUS_BUFFER_TOO_SMALL on nearly every callback — frames were correctly received/decrypted/jitter-buffered, just never decoded into audible PCM. mix_for_test()'s white-box test masked this because it always called on_playback with frames == frame_samples, the one case where the bug is invisible. Fix: RemoteStream (core/src/audio/audio_engine.h) gained a small ring buffer (init_ring/push_ring/pop_ring) that decouples decode cadence from playback-callback cadence — on_playback (core/src/audio/audio_engine.cpp) now tops the ring up by decoding whole Opus frames (decoder.frame_samples(), never the hardware frames) and drains exactly frames samples-per-channel from it each callback, silence-padding (PLC) on underrun. Also fixes a latent playout_ts bug: it now advances by the actual decoded sample count per Opus frame, not by the hardware callback's (unrelated) frame count, which was the wrong unit for jitter-buffer timestamp comparisons. ctest --test-dir build/m1-dev — 12/12 green (run via PowerShell; Git Bash exec gotcha for these binaries, see docs/building.md). Not yet confirmed audible by ear — pending the user re-running their live two-vccli test.

• Done: Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ complete (2026-06-16). Closes all three items M3 explicitly carried forward as out of scope (see the dated section below for the full file-by-file change list). ctest --test-dir build/m1-dev — 12/12 tests green (3 consecutive full-suite runs), including the new test_vad_ptt_devices (real vc_clients against a real server, plus a white-box AudioEngine stereo-mix check — same ABI-level-coverage lesson as M2/M3). Manually verified live: vccli --list-devices against real hardware, and vccli --voice --input-mode vad connecting/streaming without incident. Still explicitly out of scope (carried forward, not silently dropped):

  • Real webrtc-audio-processing/AEC — no working Windows/MSVC build upstream; v1 ships a lightweight energy/RMS VAD instead (see docs/roadmap.md §2, docs/voice.md §8/§11). There is no AEC, NS, or AGC implementation at all, not just a deferred VAD. vc_set_remote_stream(..., noise_reduction)'s per-stream NS toggle is unaffected by this pass and stays exactly as inert as it was after M3 (ApmPassthrough, no PCM modification).
  • macOS/iOS SCREEN_AUDIO capture (ScreenCaptureKit / ReplayKit) — this pass is Windows-only for real loopback capture; other platforms keep vc_test_inject_capture as the only way to feed SCREEN_AUDIO.
  • Process-specific WASAPI loopback — miniaudio's loopback mode captures the whole render endpoint (including this app's own incoming voice mix), not a single process.
  • The pre-existing RT-thread rule violation in on_capture_frame/AudioEngine::on_capture (mutex lock, heap allocation, blocking sendto on the miniaudio real-time callback thread) — predates this work, documented but not fixed; fixing it needs the lock-free ring-buffer hand-off docs/architecture.md §3 specifies, a separate, larger refactor.

• Done: M4 — Windows WinForms C# client ✓ complete (2026-06-17). Full details in the M4 section below. ctest --preset m1-dev — 14/14 tests green. dotnet build — 0 warnings/errors across all three C# projects. Manually verified: saved servers, TOFU first-connect dialog, channel tree, join, voice (VAD/PTT/always-on + sensitivity slider + per-user gain/mute/NR), text chat (channel + private), device pickers, level meter.

• Next: macOS/iOS Swift client (M4 continued) and/or M5 (admin UI, moderation, kick/ ban). The C ABI is complete and stable through M4 — both directions are unblocked. See docs/roadmap.md §M4–M5.

---

Milestones (see docs/roadmap.md for full detail)

• M0 — Scaffolding ✓ complete
• M1 — Control plane ✓ complete (2026-06-15)
• M2 — Voice, single stream ✓ complete (2026-06-16)
• M3 — Multi-stream & per-channel tuning ✓ complete (2026-06-16)
• M4 — Native clients — Windows WinForms ✓ (2026-06-17); macOS AppKit ✓ (2026-06-18); iOS SwiftUI ✓ (2026-06-19)
• [~] M5 — Moderation, polish, beyond (perms, bans, DRED; then file transfer, E2EE, …)

---

M0 — Scaffolding ✓ (completed)

• Repo layout (core/ server/ tools/ clients/ tests/), CMake + presets, vcpkg manifest.
• C ABI header core/include/voicecat.h (full surface, stubbed).
• Protocol source-of-truth core/proto/voicecat.proto (matches docs/protocol.md).
• Core stubs for all six subsystems (net/crypto/codec/protocol/session/audio) + vc_client.
• voicecat-server (arg parsing, config, stub run) and vccli (drives the C ABI).
• CTest smoke test asserting the C ABI contract (not just "it compiles").
• .gitattributes (LF), .gitignore, .clang-format, onboarding docs.
• Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.

---

M1 — Control plane ✓ (completed 2026-06-15)

Exit criterion: ✓ test_m1_integration — two clients authenticate over TLS 1.3 (guest

• Argon2id password), exchange channel and private text messages. Passes in ~1 s.

• vcpkg baseline + m1-dev preset; find_package for protobuf/mbedTLS/libsodium/asio/sqlite3.
• FrameCodec feed + emit; encode_envelope / decode_envelope.
• Asio TCP acceptor + TcpServerConn (TLS path: blocking handshake thread + tls_read_loop).
• TlsContext (mbedTLS 1.3, server cert/identity, ECDSA-P256 self-signed, TOFU on client).
• WorkerPool (3 threads, used for Argon2id).
• Database — SQLite, Argon2id via libsodium, create_account / authenticate / bootstrap admin.
• voicecat-admin — account add/reset/del/list against live DB file.
• ServerIdentityManager — generate/persist Ed25519 key + cert; fingerprint display.
• ConnSession — WaitingHello → WaitingAuth → Authenticated state machine; full protocol relay.
• SessionRegistry — channel tree, user map, broadcast, text routing.
• vc_client (client.cpp) — full M1 C ABI: connect/TLS/ClientHello/AuthRequest/text/disconnect.
• Server::run() — io_context, acceptor, worker pool, signal handling, on_ready callback.
• test_m1_integration — M1 exit criterion. Verified green 2026-06-15.

Key bug fixed: double-framing in ConnSession::send_envelope — encode_envelope was pre-framing the protobuf, then TcpServerConn::send_frame re-framed it. Fixed by serializing raw protobuf bytes directly and letting send_frame add the single [4-byte len] prefix.

---

---

M2 — Voice, single stream ✓ (completed 2026-06-16)

Exit criterion: ✓ test_m2_voice — two headless clients authenticate over TLS, bind UDP, announce a MIC stream, send 50 encrypted Opus frames; server SFU relay re-encrypts + forwards to the second client; B receives ≥ 25 frames and all decrypt correctly. Passes in ~4 s.

• m2-dev preset (inherits vcpkg-base, binaryDir build/m2-dev); m1-dev also builds all M2 code.
• core/CMakeLists.txt — find_package(Opus), find_path(MINIAUDIO_INCLUDE_DIR).
• core/src/net/voice_frame.h — 14-byte UDP header (type/flags/codec/ssrc/seq/ts), serialize/parse, make_udp_binding_packet.
• SodiumMediaCrypto — ChaCha20-Poly1305 AEAD; counter-nonce; 64-bit sliding-window anti-replay; derive_send/recv from TLS RFC 5705 exporter.
• OpusEncoder / OpusDecoder — libopus 1.6, FEC, DTX, PLC (free; nullptr → decoder extrapolates).
• UdpMediaChannel — async UDP socket (asio); thread-safe send_to; async recv loop.
• JitterBuffer — per-ssrc, EWMA jitter estimation, adaptive depth 20–200 ms, late-drop at 500 ms.
• AudioEngine — miniaudio capture+playback; inject_capture() bypass for headless tests; per-ssrc RemoteStream with OpusDecoder + JitterBuffer.
• ApmProcessor — ApmPassthrough stub (VAD always open); WebRTC APM deferred until M3.
• on_tls_ready callback in TcpChannelCallbacks — server derives and stores media AEAD keys immediately after TLS handshake.
• ConnSession M2 — udp_token generated at construction; included in AuthResult; handle_udp_binding (verifies token, TCP ack); handle_stream_announce (assigns SSRC via registry); udp_media_port in ServerHello.
• SessionRegistry M2 — register_udp_token, find_by_udp_token, register_udp_endpoint, find_by_udp_endpoint, assign_ssrc, find_channel_sessions, user_channel.
• MediaRelay — SFU UDP relay; kFrameUdpBinding → endpoint binding; kFrameVoice → decrypt/re-encrypt/forward to channel members.
• Server::run() — creates and binds MediaRelay; passes media port to ConnSession; wires on_tls_ready to derive per-connection media AEAD keys.
• test_voice_frame — header round-trip, big-endian layout, binding packet format.
• test_media_aead — seal/open round-trip, anti-replay, tamper detection, multi-packet sequence.
• test_opus_codec — encode/decode round-trip energy check (within 3 dB), PLC, frame-samples helper.
• test_m2_voice — M2 exit criterion (raw-socket harness). Verified green 2026-06-16.

Follow-up (same day): the above made test_m2_voice pass, but vc_client's public voice methods were still stubs — the actual M2 exit criterion ("two vccli/early-GUI clients talk") wasn't met. Closed the gap:

• core/src/core/client.cpp — real stream_start/stream_stop/set_self_mute/ set_remote_stream; UDP-binding handshake (start_udp_binding/handle_udp_binding_ack/ finish_udp_binding); media key derivation from tls_ (RFC 5705 exporter); run_udp_recv (AEAD-open → JitterBuffer::Frame → audio_engine_.push_recv_frame); on_capture_frame (encode → seal → sendto); sync_remote_streams (diffs a User proto's streams against remote_streams_, wiring up OpusDecoders and emitting STREAM_STARTED/STOPPED). set_input_device/set_input_mode/set_push_to_talk/list_devices remain VC_ERR_NOT_IMPLEMENTED — no device-enumeration backend yet; scoped to M3 (VAD/PTT).
• core/src/session/session.cpp/h — SessionModel::find_user, find_user_by_ssrc, Stream{stream_id, ssrc, kind, label, sample_rate, frame_ms}.
• server/src/conn_session.cpp/h — handle_stream_announce/handle_stream_stop now broadcast via SessionRegistry::set_user_stream/clear_user_stream → UserEvent::UPDATED.
• server/src/session_registry.cpp/h — set_user_stream/clear_user_stream (mutate a user's StreamInfo list, return the updated User proto for broadcast).
• tests/test_voice_client_abi.cpp — drives two real vc_client instances through vc_connect/vc_authenticate_guest/vc_stream_start/vc_stream_stop; asserts client B observes client A's STREAM_STARTED/STOPPED events. Verified green 2026-06-16.
• tools/vccli/src/main.cpp — argv parsing (--host/--port/--nick/--channel/--voice/ --mute/--text); --voice starts a MIC stream and blocks on SIGINT, printing on_event callbacks live (unbuffered stdout — MinGW/MSVCRT treat _IOLBF as full buffering for non-console streams). Dropped the originally-planned --voice-loopback and the tx=N rx=M lost=K jitter=J stats line: voicecat.h exposes no PCM-injection hook or jitter/loss stats getter publicly, only on_event + on_level (RMS). Manually verified: two vccli --voice instances see each other's stream start in real time.

---

M3 — Multi-stream & per-channel tuning ✓ (completed 2026-06-16)

Exit criterion: ✓ test_m3_multistream — a real vc_client (A) runs two concurrent local streams (MIC + SCREEN_AUDIO) with distinct stream ids; a second client (B) sees both as separate STREAM_STARTED events and a VC_EVENT_TALK_STATE talking edge for A's MIC stream; B independently sets gain/mute/noise-reduction on each of A's streams without one call affecting the other; A then joins "Music Room" (channel 2: stereo/128kbps/OPUS_AUDIO/no DTX) and announces a fresh MIC stream there, while B stays in "Lobby" (channel 1: mono/24kbps/ OPUS_VOIP/DTX on) — vc_get_stream_audio_config shows their effective Opus config differs exactly as the server enforces per channel. Passes in ~2.4s; verified across 8 consecutive standalone runs + 3 consecutive full-suite runs with no flakes.

Exploration before implementing turned up several bugs/gaps where the wire format already supported this milestone but the client/server logic didn't — these were fixed as part of M3, not treated as pre-existing-and-out-of-scope:

• Server stream_id bug — handle_stream_announce always wrote stream_id=1, so a second stream from the same user silently overwrote the first in SessionRegistry::set_user_stream's replace-by-id logic. Fixed with a per-session counter (ConnSession::next_stream_id_) + announced_stream_ids_ (also now validated in handle_stream_stop, rejecting stops for ids the session never announced).
• Per-channel AudioConfig was modeled but never populated/enforced. SessionRegistry::init_default_channels() now seeds Lobby (id=1: mono, 24kbps, OPUS_VOIP, FEC+DTX on) and a new "Music Room" (id=2: stereo, 128kbps, OPUS_AUDIO, FEC+DTX off) with real AudioConfigs; new SessionRegistry::channel_audio_config(channel_id) accessor (there was no per-id channel getter before, only channel_snapshot()). handle_stream_announce now treats the channel's config as authoritative (mode/frame_ms/application/fec/dtx/ complexity), clamping (not overriding) bitrate_bps to the channel's ceiling.
• Client silently dropped mode/dtx/complexity/application from effective_audio even for the single M2 stream — handle_stream_announce_result and sync_remote_streams only copied sample_rate/bitrate_bps/frame_ms/fec into OpusParams. New shared opus_params_from_audio_config() helper (client.cpp) fixes both the send and receive paths.
• core/src/codec/opus_codec.h/.cpp — new OpusApplication enum + OpusParams::application field; OpusEncoder::init now honors it instead of hardcoding OPUS_APPLICATION_VOIP.
• core/src/session/session.h/.cpp — Stream struct extended with the full AudioConfig (mode/bitrate_bps/application/fec/expected_packet_loss/dtx/complexity), not just sample_rate/frame_ms; copy_streams() now copies all of it.
• core/src/core/client.h/.cpp — local-stream state is now a std::unordered_map<int, LocalStream> keyed by vc_stream_kind (one active stream per kind — MIC/SCREEN_AUDIO/ AUX_DEVICE are each singletons for a client), replacing the M2 single-stream fields. StreamAnnounce/StreamAnnounceResult round-trips are now correlated by request_id (already round-tripped on the wire; just wasn't read) via pending_announce_kind_, so multiple concurrent announces from one client resolve to the right LocalStream. on_capture_frame takes a kind and channels parameter; for mono capture (channels==1) on a stereo channel it upmixes L=R, and for real stereo capture (channels==2, the SCREEN_AUDIO loopback path on a stereo channel) it encodes directly with no upmix. vc_set_self_mute's mic_muted only gates the MIC kind — a concurrent SCREEN_AUDIO share keeps playing while muted. set_remote_stream now actually wires noise_reduction through (previously parsed and discarded). New run_talk_timer() (a small dedicated thread, started alongside the UDP media path, never the miniaudio callback thread) polls both remote talk-state edges (AudioEngine::poll_talk_transitions()) and local capture-activity edges, emitting VC_EVENT_TALK_STATE.
• Fixed a thread-join race in teardown_voice() — it's called both from run_io()'s own cleanup and from disconnect(), on different threads; without serialization both could see udp_thread_/talk_timer_thread_ as joinable() simultaneously and race to join() the same std::thread (UB; surfaced as an intermittent std::system_error: No such process under ctest). Added a teardown_mu_ guard around the whole function. This pre-existed for udp_thread_ alone (likely the same root cause as the test_m1_integration/test_m2_voice cleanup-path flake noted in the M2 section above) — adding talk_timer_thread_'s join just made it surface more often, so it was fixed properly here rather than carried forward again.
• core/src/audio/audio_engine.h/.cpp — CaptureCallback gained a kind parameter (the real miniaudio capture device is always tagged kind=0/MIC; a second concurrent local stream is fed via its own inject_capture(kind, ...) ring buffer — inject_taps_, keyed by kind — since there is only one real hardware capture device in M3). Fixed a buffer-sizing bug in on_playback's per-stream decode (opus_decode's frame_size parameter is samples-per-channel, not total samples — the old code passed frames * params_.channels, which would have overflowed the decode buffer for any stereo stream). Stereo decoder output is downmixed (avg L/R) into the engine's mono mix accumulator immediately after decode. RemoteStream gained recv_ns/noise_reduction_enabled (lazy ApmProcessor instantiation — freed on disable, so no separate instance cap is needed per the roadmap's guidance) and last_voice_ms/talking (talk-indicator edge state, updated in push_recv_frame); new set_stream_noise_reduction() and poll_talk_transitions(). Note: until VOICECAT_HAS_APM is wired to a real WebRTC APM build, the NS toggle is plumbed end-to-end but behaviorally a passthrough no-op (ApmPassthrough doesn't touch PCM) — same situation send-side APM has been in since M2; M3's job was the plumbing, not the DSP backend.
• New C ABI surface (core/include/voicecat.h, additive only): vc_audio_config struct + vc_get_stream_audio_config(c, user_id, stream_id, out) — the effective Opus config for a stream you own or a peer's, reading from the (now richer) LocalStream/session::Stream. vc_test_inject_capture(c, stream_id, pcm, samples) — clearly-marked test-only, forwards to AudioEngine::inject_capture, so test_m3_multistream can drive two concurrent synthetic-audio streams through the real ABI without a microphone.
• tests/test_m3_multistream.cpp — the M3 exit criterion (ABI-level, mirrors test_voice_client_abi.cpp's approach per the M2 lesson). Registered in tests/CMakeLists.txt.

Explicitly out of scope for this pass (confirmed with the user before implementing):

• vc_set_input_device/vc_set_input_mode/vc_set_push_to_talk/vc_list_devices (device enumeration + VAD/PTT input gate) — still VC_ERR_NOT_IMPLEMENTED. These were mentioned as "scoped to M3" in the M2 follow-up notes above, but docs/roadmap.md's M3 bullets never actually listed them — deferred again, now tracked explicitly rather than implicitly.
• Real WASAPI desktop-audio loopback capture for SCREEN_AUDIO — the engine now supports feeding a second concurrent local stream via inject_capture, but only synthetic PCM is wired up; a real loopback capture device is a follow-up.
• True stereo playback output — AudioEngine's mixer/output device stays mono; stereo streams are downmixed after decode (see above). The Opus wire format itself is fully stereo-correct.

---

Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ (completed 2026-06-16)

Closes all three items the M3 section above explicitly carried forward as out of scope.

Exit verification: ctest --test-dir build/m1-dev — 12/12 tests green (3 consecutive full-suite runs), including the new test_vad_ptt_devices (device enumeration + VAD/PTT gate through real vc_clients against a real server, plus a white-box AudioEngine stereo-mix check — no audio hardware needed for that last part). Also verified 5 consecutive standalone runs of the new test alone, no flakes. Manually verified live on Windows: vccli --list-devices against real hardware (3 input / 4 output devices, correct is_default flags), and vccli --voice --input-mode vad connecting + streaming without incident.

• Device enumeration (vc_list_devices) — AudioEngine::enumerate_devices(bool capture) (static, works without a running engine — inits a throwaway ma_context via ma_context_get_devices). device_id/vc_device.id is an opaque hex-encoded raw ma_device_id (not the device name — names aren't guaranteed unique); documented as an internal contract callers must round-trip, never construct by hand. vc_client::list_devices works in any connection state (no VC_STATE_CONNECTED gate) since device pickers need to populate pre-connect. vc_free_device_list is now a real free (was a no-op stub).
• Input device selection (vc_set_input_device) — stores the device id on the targeted LocalStream (new field); for the MIC stream, if the engine is already running, restarts it (stop() + ensure_audio_running()) to pick up the new device. Simplified: it restarts unconditionally rather than trying to detect whether the device id actually changed (AudioEngine has no getter for "current device").
• VAD/PTT input gate (vc_set_input_mode, vc_set_push_to_talk) — new EnergyVadProcessor (core/src/audio/apm_processor.cpp) implementing the existing ApmProcessor interface: energy/RMS threshold (default ~0.025 normalized) + hang-time (default 300 ms, matching kTalkHangoverMs). New factory ApmProcessor::create_vad(), kept separate from create() (which recv-side per-stream NS still uses, unaffected by this pass). vc_client gained current_input_mode_/ptt_active_/mic_vad_; the gate is inserted in on_capture_frame, MIC-only — SCREEN_AUDIO/AUX_DEVICE always bypass it (gating a desktop-audio share on the user's own voice activity would silently drop shared music/video audio). last_capture_ms (drives the talk indicator) is now updated after the gate check, not before, so a VAD/PTT-closed frame never shows as "talking". mic_vad_ is constructed once the MIC stream's StreamAnnounceResult lands (on io_thread_), not lazily inside the capture path.
• True stereo playback — AudioParams::channels split into capture_channels (stays
  1. and playback_channels (now 2, unconditionally). AudioEngine::on_playback no longer downmixes decoded stereo streams to mono before mixing — stereo decode output is mixed directly (L→L, R→R); mono decode output is upmixed (duplicated into both channels). Falls back to a 1-channel playback device once if the 2-channel ma_device_init fails (unusual hardware). New test-only AudioEngine::mix_for_test() exposes the mixer for white-box testing without a real ma_device.
• WASAPI loopback capture for SCREEN_AUDIO — new VOICECAT_HAS_LOOPBACK macro (core/CMakeLists.txt, Windows-only). AudioEngine gained a separate loopback_device_ (own lifecycle, decoupled from the mic capture/playback devices) with start_loopback_capture()/stop_loopback_capture(), using miniaudio's ma_device_type_loopback against the default render endpoint. Its callback feeds capture_cb_ directly (same pattern as the real mic capture device), not through inject_capture()'s test-only ring. Wired into vc_client::handle_stream_announce_result (start, alongside ensure_audio_running()) and stream_stop (stop) for VC_STREAM_SCREEN_AUDIO. Non-Windows builds keep vc_test_inject_capture as the only way to feed SCREEN_AUDIO.
• tools/vccli/src/main.cpp — new flags --list-devices, --input-device, --input-mode vad|ptt, --share-screen-audio; while --voice is running, a background stdin-reader thread accepts ptt on/ptt off/mode vad/mode ptt (the most portable way to drive PTT interactively from a headless CLI — no SIGUSR1 equivalent on Windows). Also prints VC_EVENT_TALK_STATE. Known minor caveat: on Windows the stdin-reader thread is detached (not joined) on exit, since std::getline can't be interrupted from another thread — a vc_client* use-after-free is theoretically possible if a command line arrives in the brief window between teardown and process exit; acceptable for a headless test/dev tool.
• tests/test_vad_ptt_devices.cpp — new test covering all four items above; registered in tests/CMakeLists.txt. tests/test_smoke.cpp's device-list assertion is now conditional on VOICECAT_HAS_AUDIO (was a hard VC_ERR_NOT_IMPLEMENTED assertion) — VC_OK only, never count > 0 (a headless CI build agent may legitimately report zero audio devices).

Still explicitly out of scope (carried forward, not silently dropped):

• Real webrtc-audio-processing/AEC — no working Windows/MSVC build upstream (see docs/roadmap.md §2's superseding decision-log entry). There is no AEC, NS, or AGC implementation at all, not just a deferred VAD. The per-stream NS toggle (vc_set_remote_stream(..., noise_reduction)) is unaffected by this pass and stays exactly as inert as it was after M3 (ApmPassthrough, no PCM modification) — don't mistake this pass for having fixed it.
• macOS/iOS SCREEN_AUDIO capture (ScreenCaptureKit / ReplayKit) — Windows-only loopback in this pass.
• Process-specific WASAPI loopback — whole-device capture only; inherently captures this app's own incoming voice mix along with everything else playing.
• The pre-existing RT-thread rule violation in on_capture_frame/AudioEngine::on_capture (mutex lock, heap allocation for the stereo-upmix path, blocking sendto, all on the miniaudio real-time callback thread) — predates this work (was already present in M2/M3); documented here explicitly rather than silently carried forward again. Fixing it properly needs the lock-free ring-buffer hand-off docs/architecture.md §3 specifies — a separate, larger refactor, out of scope for this pass.

---

M4 — Windows WinForms C# client ✓ (completed 2026-06-17)

Exit criterion: ✓ ctest --preset m1-dev — 14/14 tests green (existing 12 + 2 new C++ tests: test_channel_user_list_abi, test_tofu_flow). dotnet build — 0 warnings/errors. Manually verified: connect, TOFU first-connect dialog, channel tree, join, voice, text, device pickers, level meter. Accessibility: explicit AccessibleName/AccessibleDescription on every control, & mnemonics on every button, activity-log ListBox as screen-reader record.

New C++ ABI surface (all additive, backward-compatible):

• vc_list_channels / vc_list_users / vc_list_user_streams — pull-based snapshot getters for the channel tree + user list; session_model_mu_ added for cross-thread safety. SessionModel::apply_snapshot/apply_channel_event fixed to populate parent_id, password_protected, max_users (were permanently zeroed despite the struct declaring them).
• vc_join_channel(channel_id, password) — join with optional password; server replies via new VC_EVENT_JOIN_RESULT.
• VC_EVENT_SERVER_IDENTITY + vc_confirm_server_identity(accept) — TOFU gate that blocks io_thread_ until the UI approves or rejects. Pins the TLS leaf-cert SHA-256 fingerprint (verifiable directly at handshake), not the declared Ed25519 value (see docs/security.md §1.1 for why — the TLS cert and Ed25519 key are generated independently, no binding). vc_get_server_identity_display exposes the Ed25519 fingerprint for human-readable display.
• vc_config::tofu_store_path — optional per-user pin file path; defaults to a relative "./voicecat_tofu_pins.txt" so existing tests need no change.
• TcpAcceptor now dual-stacks (IPv6 + IPv4 fallback) so localhost → ::1 on Windows connects correctly without forcing users to type 127.0.0.1.
• VC_INPUT_ALWAYS_ON = 2 in vc_input_mode — transmit unconditionally (no VAD gate).
• vc_set_vad_threshold(float) — live VAD threshold update; EnergyVadProcessor stores it atomically so the RT capture path reads without a lock or allocation.

New C++ tests:

• test_channel_user_list_abi — snapshot getters, parent_id/password_protected/max_users regression, per-user stream list, invalid-user-id error, double-free idempotency.
• test_tofu_flow — first-connect blocks until confirmed; reject doesn't persist; reconnect to same identity reports MATCHED; rotated identity reports MISMATCH; vc_confirm_* with nothing pending returns an error.

windows-client CMake preset — Release, VOICECAT_BUILD_SHARED=ON, static MinGW runtime (-static-libgcc -static-libstdc++ -static -lwinpthread), no tools/tests. Outputs build/windows-client/bin/voicecat.dll with zero MinGW DLL dependencies (only Windows system DLLs remain — verified via objdump -p).

C# solution (clients/windows/, .NET 10 LTS net10.0-windows):

• VoiceCat.Interop — [LibraryImport] P/Invoke surface, [UnmanagedCallersOnly] callbacks, System.Threading.Channels.Channel<VoiceCatEvent> event delivery drained by 30ms WinForms Timer; VoiceCatClientHandle : SafeHandle guarantees vc_client_destroy.
• VoiceCat.App — WinForms UI:
  • ConnectDialog — saved-server ListBox, Add/Remove/Edit; servers persisted to %AppData%\VoiceCat\servers.json; passwords DPAPI-encrypted (ProtectedData, opt-in).
  • ServerIdentityDialog — shown only on FIRST_CONNECT/MISMATCH (never MATCHED); mismatch text and button ordering are starkly different ("WARNING" framing, Cancel default).
  • MainForm — TreeView channel tree, ListBox user list, RichTextBox chat, scope ComboBox (Channel/Private), activity-log ListBox, voice panel with mic toggle, mute/deafen checkboxes, VAD/PTT/Always-On radio group, VAD sensitivity TrackBar (1–100, hidden for non-VAD modes), device ComboBox + refresh, level ProgressBar.
  • PerUserTuningDialog — real-time gain TrackBar + mute/NR checkboxes; applied to all of a user's streams immediately (no OK/Cancel round-trip).
  • PttKeyCaptureDialog — focus-scoped PTT key capture.
• VoiceCat.Interop.Tests — xunit smoke test: connect → TOFU → guest auth → list channels purely via P/Invoke against a live voicecat-server.exe.

Explicitly out of scope for this pass:

• macOS/iOS Swift client — pending.
• Admin/moderation UI (kick/ban/permissions/account provisioning) — server-side dispatch for these messages is M5's job; building the UI now would require building the server side too.
• PTT hotkey is focus-scoped only (works while VoiceCat window has focus). A system-wide WH_KEYBOARD_LL hook would require escalated permissions and risk AV flagging — documented limitation, not silently omitted.
• Receive-side noise reduction (vc_set_remote_stream(..., noise_reduction)) is end-to-end plumbed but behaviorally a passthrough no-op (ApmPassthrough, no PCM modification) — same as before M4. The per-user NR checkbox in PerUserTuningDialog is labeled accordingly.

---

M5 — Moderation, polish, and beyond [~] (in progress 2026-06-17)

Exit criterion: four ABI-level tests green (test_m5_permissions, test_m5_kick_ban_move_mute, test_m5_admin_accounts, test_m5_channel_crud); vccli can drive all moderation/admin/channel operations against a live server.

• Server-side moderation & permissions:
  • server/src/session_registry.h/.cpp — per-session Permissions, permission helpers (can_kick, can_ban, etc.), kick/ban/move/server-mute, channel CRUD, DB-backed channel tree load/save, in-memory channel state.
  • server/src/conn_session.cpp — M5 dispatch handlers, permission checks, channel-password
    • max_users enforcement, UserEvent::UPDATED broadcast on join/leave.
  • core/proto/voicecat.proto — ServerMuteRequest, UserEvent.reason, User.server_deafened, ListAccountsResult, AccountEntry.
• C ABI / client-side:
  • core/include/voicecat.h — vc_permissions, vc_channel_info, vc_kick_user, vc_ban_user, vc_set_permission, vc_set_server_mute, vc_move_user, vc_create_channel, vc_edit_channel, vc_delete_channel, vc_create_account, vc_reset_password, vc_delete_account, vc_list_accounts, vc_get_permissions; new events VC_EVENT_GENERIC_RESULT and VC_EVENT_ACCOUNT_LIST.
  • core/src/voicecat.cpp, core/src/core/client.h/.cpp — implementations + server-mute/deafen gating on the client.
• Database: server/src/db.h/.cpp schema v2 (channels, bans), Argon2id accounts, BLAKE2b channel passwords, migrations.
• Tests: four new M5 tests registered in tests/CMakeLists.txt:
  • test_m5_permissions — grant/revoke permissions, verify enforcement.
  • test_m5_kick_ban_move_mute — kick, ban, move, server-mute/deafen.
  • test_m5_admin_accounts — create/reset/delete/list accounts.
  • test_m5_channel_crud — create/edit/delete channels, password + max_users enforcement.
• vccli (tools/vccli/src/main.cpp) — all M5 operations exposed via flags; account auth via --username/--password; async VC_EVENT_GENERIC_RESULT/VC_EVENT_ACCOUNT_LIST handling.
• Docs kept in sync: docs/protocol.md, docs/security.md, PROGRESS.md.

Key bug fixed: test_m5_channel_crud failed because SessionRegistry::create_channel broadcast ChannelEvent::CREATED from a moved-from entry.proto after channels_[id] = std::move(entry). Fixed by building the event before moving into the map.

Still to do:

• DRED/audio-quality polish.
• macOS/iOS Swift client (carried from M4).

---

Decisions log

All architecture/scope decisions are settled and recorded in docs/roadmap.md §2 "Resolved decisions" and reflected across docs/. If you make a new decision, record it there and link it here.

---

How to update this file

1. Check off tasks as you complete them; flip a milestone to [x] only when its exit criterion test passes.
2. Keep the "Where we left off / next action" block at the top accurate — it's the first thing the next agent reads.
3. When you start a milestone, copy its task list from docs/roadmap.md into a section here.