From de2c253199b56ec9173c80af21526ce4cb9aef12 Mon Sep 17 00:00:00 2001 From: Talon Date: Sat, 20 Jun 2026 13:14:39 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20condense=20PROGRESS.md=20=E2=80=94=20tr?= =?UTF-8?q?im=20completed=20history,=20keep=20pending=20items?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1398 → ~340 lines. All "Done" journal entries replaced by a compact "Recent completed work" summary; M0–M4 sections condensed to scan-able paragraphs. Pending items (iOS on-device verification, external PCM feed/tap API plan) kept at full length. M5 task list updated to mark macOS/iOS UI done and list remaining open items. Co-Authored-By: Claude Sonnet 4.6 --- PROGRESS.md | 1246 ++++++--------------------------------------------- 1 file changed, 135 insertions(+), 1111 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 4693f8d..de24c5c 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -197,778 +197,82 @@ up instantly. Newest status at the top. - **ABI stability:** append-only — two new functions + one new typedef, no existing structs/enums changed. `vc_test_inject_capture` stays as a deprecated alias for source compatibility. Treat as a deliberate, versioned ABI event per `docs/protocol.md §8`. - - **Relationship to the iOS audio routing plan (entry below):** orthogonal. That plan is - about *which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan - is about *bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship; - they don't conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer - controls the *mic* path which still uses miniaudio's device. + - **Relationship to the iOS audio routing plan:** orthogonal. The iOS routing layer controls + *which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan is about + *bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship; they don't + conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer controls the + *mic* path which still uses miniaudio's device. +--- -- **Done:** **iOS audio preset fixes — A2DP output muting + broken stereo capture** (2026-06-19). - Two user-reported bugs with the audio presets (esp. with Bluetooth headphones connected), - both root-caused against Apple docs/WWDC20 + dev-forum reports and fixed in - `IOSAudioRouter.applyConfiguration()` / `AudioSessionManager.ensureSessionActive()`: - 1. **Output (and VoiceOver) went dead on the Stereo Mic / A2DP presets (BUG — fixed).** - `applyConfiguration()` always inserted `.defaultToSpeaker`, which breaks A2DP routing in - `.playAndRecord` (route ends up muted, taking the shared hardware output — and VoiceOver — - with it). And the A2DP path used `mode = .default`, which iOS 17+ routes to the speaker. - **Fix:** `.defaultToSpeaker` now set *only* for the speaker preset; `.mixWithOthers` kept - always (VoiceOver must stay audible for a blind user); A2DP/stereo paths now use - `mode = .videoRecording` (the documented mode that keeps Bluetooth output and supports - multi-capsule stereo); after `setActive(true)` on an A2DP preset we call - `overrideOutputAudioPort(.none)` to clear any lingering speaker override. NOTE: built-in - mic + A2DP output during active recording is inherently flaky on iOS (A2DP is output-only; - recording wants to collapse BT to mono HFP) — this is best-effort, not guaranteed on every - BT device. Needs on-device testing. - 2. **Stereo mic only captured the left channel (BUG — fixed).** Real built-in-mic stereo - needs more than `setPreferredInputNumberOfChannels(2)`: per WWDC20 you must select a data - source whose `supportedPolarPatterns` contains `.stereo`, call - `setPreferredPolarPattern(.stereo)`, set `setPreferredInputOrientation(.portrait)`, then - request 2 channels. The old code set the data source/polar pattern to `nil` for the stereo - preset, so channel 2 was silent. **Fix:** new `configureStereoCapture()` does the full - WWDC20 sequence and falls back to mono if the route has no stereo-capable capsule (BT/wired - mic). New `configureMonoCapture()` resets to 1 channel (so stereo→mono actually takes - effect) and applies the user's chosen mono orientation/polar pattern. - - **Verified:** `xcodebuild -scheme VoiceCatiOS -destination 'generic/platform=iOS Simulator' - ARCHS=arm64` — **BUILD SUCCEEDED**. On-device behavior (BT headphones + VoiceOver + stereo - pickup) still to be confirmed by the user. - - **Follow-up (same day) after device testing.** User reported: default + BT-mono-mic work, - but Stereo Mic still kills output, and remote audio may never be audible. - b. **Remote audio never routed (BUG — fixed).** The AVAudioSession was activated lazily on - the `.streamStarted` event, but the core opens its miniaudio playback device *before* - emitting that event — so playback opened against an inactive session and produced no - sound. **Fix:** activate the session proactively on `.authResult == .ok` - (`AppState.swift`), so it's live before any remote stream opens a device. Mirrors the - macOS "session active while connected" model. +## Recent completed work -- **Done:** **iOS stereo mic + A2DP output — audio death fix** (2026-06-19). The previous - "fix" (line below, "Stereo mic killed A2DP output") was never verified on-device and still - had the bug: selecting the Stereo Mic preset killed all audio (including VoiceOver) — only - kill & relaunch recovered it. Root cause found by comparing against TeamTalk5 - (`Client/iTeamTalk/`), which achieves stereo mic + A2DP output. Five issues fixed: - 1. **`configureStereoCapture` was missing `setPreferredInput` + `setInputDataSource`** - (BUG — fixed). The previous fix threw out `setPreferredInput` together with - `setPreferredInputNumberOfChannels(2)`, blaming the *combination* for collapsing A2DP. - WRONG — TeamTalk5 keeps `setPreferredInput` + `setInputDataSource` - (`SoundDevicesModel.selectDataSource:147-148`) and only omits - `setPreferredInputNumberOfChannels(2)`. Without the explicit input anchor, when - `setPreferredPolarPattern(.stereo)` fired, iOS had no session-level input anchor and the - route reconfiguration collapsed the A2DP output. **Fix:** `configureStereoCapture` now - calls `setPreferredInput(builtIn)` + `setInputDataSource(stereoSource)` after the polar - pattern. Also removed `setPreferredInputOrientation(.portrait)` (TeamTalk doesn't use it; - possible route-collapse contributor on iOS 26). - 2. **Core devices not restarted around `applyConfiguration`** (BUG — fixed). TeamTalk5's - `setupSoundDevices` calls `closeSoundDevices()` FIRST, then reconfigures, then reopens — - so the audio unit never sees a route change mid-flight. VoiceCat reconfigured - `AVAudioSession` while the core's miniaudio devices were still open, leaving them bound - to the dead route. **Fix:** new C ABI function `vc_audio_restart` (full stop + re-init, - unlike `suspend`/`resume` which only stop/start). `IOSAudioRouter`'s setters now wrap - `applyConfiguration()` with `audioSuspend` → reconfigure → `audioRestart`. - `AudioSessionManager.handleRouteChange` also calls `audioRestart` on device - plug/unplug so the playback device picks up the new route. - 3. **Deprecated `.allowBluetooth` instead of `.allowBluetoothHFP`** (modernized). - TeamTalk5 uses `.allowBluetoothHFP` (iOS 17+). VoiceCat was using the deprecated - `.allowBluetooth` alias. Replaced; also added `.bluetoothHighQualityRecording` on - iOS 26+ (mirrors TeamTalk5 `UtilSound.swift:229-231`). - 4. **Capture channels not reset when switching stereo→mono** (BUG — fixed). Switching from - stereo to mono only changed the `AVAudioSession` polar pattern, not the `LocalStream`'s - `capture_channels` field — so the next engine start still opened 2 channels. **Fix:** - `AudioSessionManager` now tracks `activeMicStreamId` (set by `SessionState` on - join/leave voice); `selectCaptureChannels` and `applyPreset` call - `setCaptureChannels(streamId, channels.channelCount)` inside the suspend window so the - field is updated before the engine restarts. - 5. **Docs corrected.** `docs/voice.md` §8, `docs/tech-stack.md` §2, `docs/architecture.md` - §4 — removed stale `setPreferredInputNumberOfChannels(2)` references; documented the - actual recipe (`.stereo` polar pattern + `setPreferredInput` + `setInputDataSource` + - `vc_set_capture_channels`) and the new `vc_audio_restart` ABI + close→reconfigure→reopen - ordering rule. `voicecat.h` `vc_set_capture_channels` doc comment also corrected. - - **New C ABI:** `vc_result vc_audio_restart(vc_client* c)` — full audio engine restart - (stop + uninit + re-init), append-only addition. Skeleton stub added. Swift wrapper: - `VoiceCatClient.audioRestart()`. - - **Verified:** `cmake --build --preset dev` + `ctest --preset dev` — **21/21 green** - (including `test_stereo_mic_capture`). `xcodebuild -arch arm64 ONLY_ACTIVE_ARCH=YES` — - **BUILD SUCCEEDED** (iOS client compiles + links). **On-device verification still - pending** user test: (a) Voice Chat → Stereo Mic, (b) BT Headphones + Mono Mic → Stereo - Mic, (c) Stereo Mic → Voice Chat, (d) unplug/replug Bluetooth mid-stereo-session. +All items below are `[x]` done; `ctest --preset dev` 21/21 on Windows after each. - a. **Stereo mic killed A2DP output (BUG — fixed correctly this time).** My first follow-up - wrongly concluded stereo ⊥ Bluetooth was a hardware limit and forced stereo → speaker. - WRONG — TeamTalk5 (`Client/iTeamTalk/iTeamTalk/UtilSound.swift`) and Ferrite both do - built-in stereo mic + A2DP output. The actual cause was *how* stereo was requested: - `setPreferredInputNumberOfChannels(2)` + `setPreferredInput(builtInMic)` + - `mode=.videoRecording` together collapse the A2DP output route. **Fix (TT5 recipe):** - stereo is now enabled purely by setting the built-in mic data source's `.stereo` polar - pattern (`configureStereoCapture`); NO `setPreferredInputNumberOfChannels`, NO - `setPreferredInput` (with HFP disabled the system already routes input to the built-in - mic); `mode=.default` for stereo (`.voiceChat`/VPIO forces mono). The channel count is - requested by miniaudio at the AU level via `vc_set_capture_channels(2)`. Reverted the - force-to-speaker workaround: `.stereoMic`/`.studio` presets use A2DP again (speaker - fallback when no BT); re-removed `showsStereoForcesSpeakerWarning`; `.allowAirPlay` - added to BT presets. `clearStereoPolarPattern()` resets the capsule when returning to - mono. So stereo mic + A2DP output now coexist. - - **Verified:** rebuilt — **BUILD SUCCEEDED**. On-device confirmation (hearing remote - clients; stereo mic + A2DP output with both channels) still pending user test. +- **iOS A2DP + stereo root cause fix** (2026-06-20): miniaudio's NULL-context `ma_device_init` + was calling `AVAudioSession setCategory(Record)` on every device open, wiping the session + config `IOSAudioRouter` had set. Fixed by sharing a `ma_context` with + `sessionCategory=none` + `noAudioSessionActivate/Deactivate=MA_TRUE` — miniaudio never + touches `AVAudioSession`; `IOSAudioRouter` is the sole owner. Files: `audio_engine.{h,cpp}`. -- **Done:** **iOS audio overhaul + Join/Leave Voice + channel-id sync fix** (2026-06-19). - Three problems found while reviewing the iOS client, all fixed: - 1. **Mic button permanently dimmed (BUG — fixed).** `VoiceControlsView.swift:26` gated the - mic button on `session.currentChannelId == 0`, but `SessionState` never synced - `currentChannelId` from the self user's `channelId` on connect. The server auto-places - every newly-authed user into the Lobby (channel 1, `server/src/session_registry.cpp:111`), - but the iOS client ignored it. **Fix:** added `syncSelfChannel()` (mirrors macOS - `MainWindowController.swift:461,491,522`); called from `init`, `.channelList`, - `.userJoined`/`.userLeft`/`.userUpdated`, `.joinResult`. Added - `applyServerMuteState(muted:deafened:)` (mirrors macOS lines 693-700); called from - `.userUpdated`. Added `serverMuted`/`serverDeafened` to `VoiceState`. - 2. **No Join/Leave Voice button (fixed — parity with macOS).** Replaced the icon-only mic - toggle with an explicit "Join Voice"/"Leave Voice" text button (mirrors macOS - `micToggleButton`). Mute/deafen buttons now disable when not in voice. PTT path kept. - 3. **Limited audio input/output options (fixed — full `IOSAudioRouter.swift`).** New - `IOSAudioRouter` singleton drives all iOS audio routing via `AVAudioSession` before the - core (miniaudio) opens its device: input port selection (`availableInputs`), built-in mic - orientation (`setPreferredDataSource`: front/back/top/bottom), polar patterns - (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), Bluetooth mode - (`.allowBluetooth` HFP voice / `.allowBluetoothA2DP` stereo output / neither), mic - processing mode (`.voiceChat` Standard with AEC/AGC/HPF / `.measurement` Raw with all - processing off + speaker echo warning), stereo capture - (`setPreferredInputNumberOfChannels(2)` → `vc_set_capture_channels`), AirPlay via - `AVRoutePickerView`. All choices persisted in `UserDefaults`; re-applied on route changes. - `AudioSessionManager` refactored to delegate routing to `IOSAudioRouter`. - 4. **Core stereo-mic capture (new C ABI: `vc_set_capture_channels`).** Append-only ABI - addition: `vc_result vc_set_capture_channels(vc_client*, uint32_t stream_id, uint32_t - channels)` (1=mono, 2=stereo). `LocalStream` gained a `capture_channels` field; - `ensure_audio_running()` reads it into `AudioParams.capture_channels` before the device - opens. `audio_engine.cpp` `capture_accum_` sized to `frame_samples_ * capture_channels`; - `on_capture` updated to the same channel-aware accumulation pattern as `on_loopback`. - Skeleton stub added. Test `test_stereo_mic_capture` (headless, feeds L≠R stereo through - the mic accumulator path, asserts L≠R end-to-end). `ctest --preset dev` — **21/21 green**. - Swift wrapper: `VoiceCatClient.setCaptureChannels(streamId:channels:)`. - 5. **Settings UI rework.** `SettingsView` replaced the miniaudio-based device picker with - the AVAudioSession-derived tree: Audio Input (port picker → built-in mic - orientation/polar pattern sub-pickers + mic mode Standard/Raw + channels Mono/Stereo), - Audio Output (bluetooth mode + current route read-only + AirPlay), Voice (input mode, - VAD threshold). - 6. **Deployment target raised to iOS 18.0.** `Package.swift` + `project.pbxproj` (4 - occurrences). Unlocks newest AVAudioSession APIs. - 7. **Docs updated:** `docs/tech-stack.md` §2 (iOS audio routing via IOSAudioRouter, stereo - mic, deployment 18.0, fixed stale "deferred" claim about `vc_audio_suspend`/`resume`), - `docs/architecture.md` §4 (Swift binding notes — IOSAudioRouter, fixed stale "deferred" - claim), `docs/voice.md` (new iOS mic capture subsection), `docs/roadmap.md` (iOS pending - list updated), `docs/building.md` §9 (deployment target 18.0), `PROGRESS.md` (this entry). - - **Verified:** `cmake --build --preset dev` + `ctest --preset dev` — **21/21 green** - (including new `test_stereo_mic_capture`: `total_diff=8433549, seen_channels=2`). - - **Original plan (kept for reference):** - Three problems found while reviewing - the iOS client: - 1. **Mic button permanently dimmed (BUG — root cause).** `VoiceControlsView.swift:26` gates the - mic button on `session.currentChannelId == 0`, but `SessionState` never syncs - `currentChannelId` from the self user's `channelId` on connect. The server auto-places every - newly-authed user into the Lobby (channel 1, `server/src/session_registry.cpp:111`), but the - iOS client ignores it — `currentChannelId` is set in only two places: `.joinResult` - (`SessionState.swift:97`) and reset to 0 by `leaveChannel()` (`SessionState.swift:135`). The - `init` (`SessionState.swift:55-57`) calls `refreshUsers()` but throws away self's channelId; - the `.channelList`/`.userUpdated` handlers (`SessionState.swift:76-79`) likewise only re-fetch - the list. macOS does this sync correctly (`MainWindowController.swift:461,491,522`). Because - the button stays dimmed, `startMicStream()` (and thus the mic permission prompt) can never be - triggered — a chicken-and-egg that *looks* like a permission issue. **Not** a permission bug: - `Info.plist:23` declares `NSMicrophoneUsageDescription` and `SessionState.swift:139` calls - `AVAudioApplication.requestRecordPermission`. Fix = mirror macOS: add `syncSelfChannel()`, - call it from `init`/`.channelList`/`.userJoined`/`.userLeft`/`.userUpdated`/`.joinResult`; - also add `applyServerMuteState(muted:deafened:)` (iOS currently ignores server mute/deafen). - 2. **No Join/Leave Voice button (parity gap).** macOS (`MainWindowController.swift:303,767-797`) - and Windows have an explicit "Join Voice"/"Leave Voice" button; iOS has only an icon mic - toggle. Add an explicit button. **Screen-audio button deferred** to the ReplayKit Broadcast - Upload Extension milestone (PROGRESS.md line below) — needs the extension to actually function. - 3. **Limited audio input/output options (feature gap).** Current `SettingsView.swift:41-60` uses - miniaudio's `vc_list_devices` which on iOS returns ~2 entries ("iPhone mic"/"Default"). - miniaudio does NOT touch `AVAudioSession` on iOS — it opens the current default route via - CoreAudio (AudioUnit/AudioQueue) and that's it. All iOS audio routing (input port selection, - mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) must be - driven manually from Swift via `AVAudioSession` *before* the core opens its device. The - current `AudioSessionManager.swift:13-15` uses `.voiceChat` + `.allowBluetooth`, which forces - HFP (mono 8/16 kHz + heavy output processing) whenever a Bluetooth headset is connected — - this is why stereo output is degraded with BT headsets. +- **iOS audio routing overhaul** (2026-06-19): Full `IOSAudioRouter` singleton drives all + `AVAudioSession` config before miniaudio opens devices. Fixed stereo mic polar-pattern setup + (WWDC20 recipe: `setPreferredInput` + `setInputDataSource` + `.stereo` polar pattern + no + `setPreferredInputNumberOfChannels`). Added `vc_audio_restart` ABI (full stop+reinit for + close→reconfigure→reopen ordering). Added `vc_set_capture_channels` ABI (core stereo-mic + support). AVAudioSession activated proactively on `.authResult`, not lazily on + `.streamStarted`. Join/Leave Voice button added (parity with macOS). Channel-id sync fixed + (mic button was permanently dimmed). iOS deployment target raised to 18.0. - **Plan (decisions locked with user 2026-06-19):** - - **Part A — Channel-id sync (bug fix, smallest, unblocks mic button):** `SessionState` gains - `syncSelfChannel()` (finds self in `users`, sets `currentChannelId = user.channelId`); called - from `init`, `.channelList`, `.userJoined`, `.userLeft`, `.userUpdated`, `.joinResult`. Add - `applyServerMuteState(muted:deafened:)` (mirror macOS lines 693-700); call from `.userUpdated`. - - **Part B — Join/Leave Voice button:** Replace the icon-only mic toggle in `VoiceControlsView` - with an explicit "Join Voice"/"Leave Voice" button (mirror macOS `micToggleButton`). Keep PTT - path, mute/deafen, disconnect, level meter. No screen-audio button (deferred). - - **Part C — iOS audio routing layer (Swift, new `IOSAudioRouter.swift`):** Drive selection via - `AVAudioSession` before the core opens its device. Input enumeration via - `availableInputs` → ports (builtInMic/bluetoothHFP/headsetMic/usbAudio/airPlay); for builtInMic - walk `port.dataSources` → expose **orientation** (front/back/top/bottom) + **polar patterns** - (`supportedPolarPatterns`: omni/cardioid/subcardioid/bidirectional). Apply via - `setPreferredInput`/`setPreferredDataSource`/`setPreferredPolarPattern`/ - `setPreferredInputNumberOfChannels(2)` for stereo. Bluetooth mode as category options: "BT HFP - voice" (`.allowBluetooth`, mono voice, BT mic) / "Built-in Mic + BT A2DP stereo" - (`.allowBluetoothA2DP` only — stereo output, built-in mic, **no HFP processing**) / "Built-in - Mic + Speaker" (neither). Mic processing mode: "Standard" (`.voiceChat`, AEC/AGC/HPF on) / - "Raw / Studio" (`.measurement`, all processing off — **allowed always with a warning** when - output route is the speaker, echo risk since no AEC). Output: read-only `currentRoute.outputs` - display + AirPlay via `AVRoutePickerView`. Persist choices in `UserDefaults`; re-apply on route - changes. Note: Voice Isolation / Wide Spectrum (iOS 17+/18+) are user-toggleable in Control - Center for `.voiceChat` apps — surface as a hint, not a programmatic toggle. - - **Part D — Core stereo-mic capture (C ABI change, append-only):** `AudioParams.capture_channels` - is hardcoded to 1 (`audio_engine.h:85`); encode path already stereo-capable (proven by WASAPI - loopback stereo work, 2026-06-17 entry below). New C ABI: `vc_result - vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels);` (1 or 2) — new - setter, not a struct change, keeps `vc_stream_desc` stable. `client.{h,cpp}`: `set_capture_channels` - → per-stream, flows into `AudioParams.capture_channels` before `AudioEngine::start` for MIC kind. - `client.cpp` `on_capture_frame`: mic path with `channels==2` reuses the existing stereo encode - branch. `audio_engine.cpp`: capture device already opens with `p.capture_channels` — just needs - the value to propagate. Swift `VoiceCatCore`: add `setCaptureChannels(streamId:channels:)`; - `IOSAudioRouter` calls it when user picks stereo built-in mic. - - **Part E — Settings UI rework:** Audio Input section (input port picker → if builtInMic, show - orientation + polar pattern sub-pickers + mic mode Standard/Raw + channels Mono/Stereo); Audio - Output section (bluetooth mode HFP/A2DP/Off + current route read-only + AirPlay button); Voice - section (existing: input mode, VAD threshold, PTT). Replace the current miniaudio-based device - picker (`SettingsView.swift:41-60`) with the AVAudioSession-derived tree. - - **Part F — Docs + deployment target:** Raise iOS deployment target to **18.0** (user decision — - unlocks newest audio APIs; `project.pbxproj` + `Package.swift` `.iOS(.v18)`). Update - `docs/tech-stack.md` §2 (iOS audio routing via AVAudioSession, stereo mic, deployment 18.0), - `docs/voice.md` (new iOS mic capture subsection), `docs/roadmap.md` (iOS pending list), - `docs/architecture.md` §4 (Swift binding notes — iOS audio routing in Swift layer, core - stereo-mic via new ABI setter), `PROGRESS.md` (this entry + next-action pointer). - - **Implementation order:** A (channel-id sync, testable immediately) → B (Join/Leave Voice) → - D (core stereo-mic + ABI, verify with `ctest --preset dev`) → C (Swift routing layer, depends - on D) → E (Settings UI, depends on C) → F (docs, same commit as work). - - **Verification:** `xcodebuild -target VoiceCatiOS -sdk iphonesimulator -configuration Debug - build` → BUILD SUCCEEDED; `swift test` green + stereo-mic capture round-trip test; `ctest - --preset dev` green (core changed in D); manual (simulator/device): join channel → Join Voice - works → input picker shows built-in mic sub-options (orientation/polar pattern) → toggle Raw - mode (warning appears on speaker) → toggle Bluetooth A2DP stereo output → toggle stereo capture. - - **Files to touch:** - - iOS Swift: `SessionState.swift`, `AudioSessionManager.swift` (+ new `IOSAudioRouter.swift`), - `Views/VoiceControlsView.swift`, `Views/SettingsView.swift`, `VoiceCatiOSApp.swift`, - `VoiceCatiOS.xcodeproj/project.pbxproj` (iOS 18.0). - - Swift core: `Sources/VoiceCatCore/VoiceCatClient.swift` (`setCaptureChannels`), - `Sources/VoiceCatCore/Models.swift` (input-port model), `Package.swift` (`.iOS(.v18)`). - - Core C++: `core/include/voicecat.h` (`vc_set_capture_channels`), `core/src/voicecat.cpp`, - `core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`. - - Docs: `docs/tech-stack.md`, `docs/voice.md`, `docs/roadmap.md`, `docs/architecture.md`, - `PROGRESS.md`. - - Tests: `clients/apple/Tests/VoiceCatCoreTests/`, `tests/` (stereo-mic encode test). - - **Why miniaudio can't do this alone:** On iOS, miniaudio uses CoreAudio (AudioUnit/AudioQueue) - to open the *current default route* for PCM I/O — it never touches `AVAudioSession`. - `ma_context_get_devices` on iOS returns a near-empty list (iOS doesn't expose a real CoreAudio - device list like macOS does). All iOS audio routing (input ports, data sources, polar patterns, - HFP/A2DP, measurement mode, `preferredInputNumberOfChannels`) must be driven from Swift via - `AVAudioSession` *before* miniaudio opens its device. miniaudio just reads whatever route - AVAudioSession has established. +- **iOS SwiftUI client** (2026-06-19): `VoiceCatiOS.xcodeproj` at `clients/apple/iOS/`. + Full feature parity with macOS/Windows: saved server list (JSON + Keychain, App Group + `group.cat.voice.VoiceCat`), TOFU, connect flow, channel tree, user list with context menus, + chat, admin sheets, voice controls, settings. `xcodebuild` → BUILD SUCCEEDED. -- **Done:** **iOS SwiftUI client — `VoiceCatiOS`** (2026-06-19). Full SwiftUI app at - `clients/apple/iOS/VoiceCatiOS.xcodeproj`. Mirrors the macOS AppKit and Windows WinForms - clients feature-for-feature: saved server list (JSON + Keychain passwords via App Group - `group.cat.voice.VoiceCat`), TOFU server-identity sheet (first-connect and mismatch), connect - flow (guest + account auth, connection state labels, password prompt), main window - (`NavigationSplitView` on iPad, `TabView` on iPhone via `horizontalSizeClass`), channel tree - (`OutlineGroup` recursive — `children: nil` for leaf nodes), user list with context menu - (kick/ban/move/set-permissions/server-mute), chat (`ScrollViewReader` + `LazyVStack`), activity - log, voice controls (mic toggle, PTT via `DragGesture(minimumDistance: 0)` + `@GestureState`, - level meter `Canvas`), per-user tuning (gain slider, mute, NR toggles), admin sheets (channels - CRUD, accounts CRUD, ban duration, move-to-channel), settings (input mode, VAD threshold, - device picker, disconnect). All 24 Swift source files in place (10 model/state + 14 views). - `@Observable` + `@MainActor` throughout — both `AppState` and `SessionState` are `@MainActor`. - `onEvent` closures dispatch to main actor via `Task { @MainActor in ... }`. - **C ABI additions:** `vc_audio_suspend` / `vc_audio_resume` (no-op when engine stopped) — - called by `AudioSessionManager` on AVAudioSession interruption (phone calls, Siri). - **XCFramework:** iOS arm64 + arm64-simulator slices added alongside existing macOS-arm64 slice; - `Package.swift` updated with `.iOS(.v17)`. - **Verified:** `xcodebuild -target VoiceCatiOS -sdk iphonesimulator26.5 -configuration Debug ... - SYMROOT=... OBJROOT=... build` → **BUILD SUCCEEDED** (clean build). - - **Next:** ReplayKit Broadcast Upload Extension (`VoiceCatBroadcast`) for screen/system audio - sharing on iOS — separate Xcode target, App Group credential sharing, `SampleHandler.swift`. - Or: DRED/audio-quality polish (M5). +- **macOS AppKit client** (2026-06-18): `VoiceCatMac.xcodeproj` at `clients/apple/macOS/`. + Fixed compile errors (`NSAccessibility` call-site arg order, `StreamSummary.id` vs + `.streamId`) and linker issues (`OTHER_LDFLAGS = -lc++`, `ONLY_ACTIVE_ARCH = YES` for + Release). Debug + Release both BUILD SUCCEEDED. -- **Done:** **macOS AppKit client builds clean (Debug + Release)** (2026-06-18). The previous - "shipped" claim in the entry below was incorrect — `xcodebuild` actually failed on a fresh - clone. Two real defect classes fixed, no source architecture changed: - 1. **Swift compile errors in `MainWindowController.swift`** (the previous agent wrote AppKit - API calls from memory that didn't match the SDK): - - `NSAccessibility.post(notification:element:userInfo:)` — wrong argument order. The Swift - import (verified against `AppKit.apinotes` in the macOS 26.5 SDK) is - `NSAccessibility.post(element:notification:userInfo:)`. 3 call sites fixed. - - `NSAccessibilityPriorityMedium` — not a Swift symbol. The C constants - `NSAccessibilityPriorityHigh/Medium/Low` are imported by Swift as cases of the - `NSAccessibilityPriorityLevel` enum (it's an `NS_ENUM(NSInteger, ...)` in - `NSAccessibilityConstants.h`, no apinotes rename). Replaced with - `NSAccessibilityPriorityLevel.medium` at 3 call sites. - - `streams.first(where: { $0.streamId == event.streamId })` — `StreamSummary` has no - `streamId` property; the init parameter is named `streamId` but the stored property is - `id` (per `VoiceCatCore/Models.swift:102-110`, `Identifiable` conformance). The bad - property access made the closure fail to type-check, which made the compiler treat - `streams.first` as a property (returning `StreamSummary?`) and then try to call it — - hence "cannot call value of non-function type 'StreamSummary?'". Fixed to `$0.id`. - - Removed a redundant `fileprivate var description` extension on `VoiceCatResult` in - `ConnectWindowController.swift` — `VoiceCatResult` already has a `public var description` - in `VoiceCatCore/Enums.swift`, so the redeclaration would have errored ("invalid - redeclaration") once the compiler got past `MainWindowController.swift`. - 2. **Linker error — `libvoicecat-fat.a` is C++ but the app target didn't link libc++** - (the previous agent's `project.pbxproj` had `OTHER_LDFLAGS` empty). The pure-Swift app - pulls in `libvoicecat-fat.a` (a static C++20 archive that references `std::__1::*`, - `__cxa_*`, `operator new/delete`, etc.), but the linker has no reason to pull in libc++ - on its own — there are no `.cpp` sources in the app target. The VoiceCatCore package's - *test* target sidesteps this with `linkerSettings: [.linkedLibrary("c++")]` in - `Package.swift:54-56`, which is why `swift test` was green but `xcodebuild` wasn't. - Fixed by adding `OTHER_LDFLAGS = ("$(inherited)", "-lc++")` to BOTH the Debug and Release - target configurations in `VoiceCatMac.xcodeproj/project.pbxproj`. `otool -L` on the - produced dylib confirms `/usr/lib/libc++.1.dylib` is now linked. - 3. **Release config tried to build x86_64 (XCFramework only has arm64)** — the project-level - Release config (`AAAA…000C`) lacked `ONLY_ACTIVE_ARCH = YES`, so Release built the - standard `ARCHS_STANDARD` (arm64 + x86_64) and the x86_64 slice failed with - `Undefined symbols for architecture x86_64: _vc_authenticate_guest …` because - `VoiceCatCore.xcframework/macos-arm64` only contains an arm64 slice. Added - `ONLY_ACTIVE_ARCH = YES` to the project-level Release config to match the XCFramework. - For distribution (App Store / universal binary), the right fix is to make - `build-xcframework.sh --all` also produce an x86_64 macOS slice — left as a future - enhancement; the current setup builds a working arm64 Release on Apple Silicon. - - **Verified:** `xcodebuild -project … -scheme VoiceCatMac -configuration Debug build` → - **BUILD SUCCEEDED** (clean build, not incremental). Release config → **BUILD SUCCEEDED**. - `otool -L` on `VoiceCatMac.debug.dylib` shows `libc++.1.dylib`, `Security.framework`, - `AppKit`, `Foundation`, `CoreFoundation`, swift runtime libs. `nm` shows `_vc_client_create` - + `_vc_version_string` are present (the static `libvoicecat-fat.a` linked in). App - launches and stays running (verified via background launch + `kill -0`). - - **Lesson:** the previous "shipped" entry was written without ever running `xcodebuild` — - a clean compile is the floor, not the goal (AGENTS.md). The `swift test` green status was - real but tested only the VoiceCatCore package, not the macOS app target. The app target - had never been built. +- **Swift `VoiceCatCore` package + XCFramework** (2026-06-18): Shared Swift wrapper at + `clients/apple/`. `build-xcframework.sh` merges `libvoicecat.a` + 107 vcpkg static deps into + a fat `.a` via `libtool -static`. 6/6 Swift tests green (real server, mirrors C# Interop + tests). Supports macOS-arm64 + iOS-arm64 + iOS-sim slices. -- **Done:** **macOS AppKit UI — `VoiceCatMac`** (2026-06-18). Full AppKit application at - `clients/apple/macOS/VoiceCatMac.xcodeproj`. Mirrors the Windows WinForms client feature-for-feature: - saved server list (JSON + Keychain passwords), TOFU server-identity sheet (first-connect and mismatch - paths), connect flow (guest + account auth, connection state labels), main window (NSSplitView - layout with NSOutlineView channel tree, NSTableView user list, NSTextView chat, activity log), - voice controls panel (Join/Leave mic, screen audio, mute/deafen checkboxes, VAD/PTT/Always-On - segmented control, VAD sensitivity slider, PTT key capture, input device picker, level meter), - compose bar (scope picker for channel/private text, NSTextField + Send), right-click context - menus on channels and users (join, create/edit/delete channels, kick/ban/move/server-mute/deafen/ - set permissions), admin menu (server accounts sheet with CRUD), 10 sheet view controllers. - Full VoiceOver accessibility: every control has `setAccessibilityLabel`; `NSAccessibility.post - (.announcementRequested)` on join, talk-state, stream events. All 17 Swift source files in - place; `PttKeyCaptureSheet` uses a `KeyCaptureView: NSView` subclass that becomes first - responder and captures `keyDown`. **Build prerequisite:** run - `clients/apple/scripts/build-xcframework.sh` first; then `xcodebuild -project - clients/apple/macOS/VoiceCatMac.xcodeproj -scheme VoiceCatMac build`. - - **Next:** iOS SwiftUI client (`clients/apple/iOS/`) — same VoiceCatCore Swift package, - SwiftUI instead of AppKit. Or: DRED/audio-quality polish (M5). +- **macOS port validated** (2026-06-18): 21/21 on macOS. Three cross-platform bugs fixed: + missing `` in POSIX test branch; SIGPIPE kills (added `SIG_IGN`); use-after-free of + Asio kqueue reactor on server shutdown (fixed `TcpAcceptor` shutdown/connection-drain + sequence). -- **Done:** **macOS/iOS Swift core — `VoiceCatCore` package + tests** (2026-06-18). The - shared Swift core for the macOS (AppKit) and iOS (SwiftUI) clients is built and verified. - This is the foundation that both platform UIs will build on — mirrors the Windows client's - proven `VoiceCat.Interop` layer using Swift-native C interop. - - **Architecture decision:** macOS UI = **AppKit** (not SwiftUI), for the most mature - VoiceOver accessibility story — same rationale as the Windows client's WinForms-over- - WinUI-3 decision. iOS stays SwiftUI. Recorded in `docs/roadmap.md` §2 and - `docs/tech-stack.md` §2. - - **`VoiceCatCore` Swift Package** (`clients/apple/Package.swift`): binary target - (`VoiceCatCoreXCF` → `VoiceCatCore.xcframework`) + library target (`VoiceCatCore` — the - Swift wrapper) + test target (`VoiceCatCoreTests`). 7 source files: - `Enums.swift` (Swift mirrors of the 9 C enums), `Config.swift`, `Event.swift` - (copies `ev.text` to `String` inside the callback — the #1 lifetime rule), - `Models.swift` (Channel/User/Stream/Device/Permissions/Account/AudioConfig/…), - `Marshaling.swift` (C arrays → Swift arrays + immediate `vc_free_*`), - `Callbacks.swift` (`@convention(c)` + `Unmanaged.passUnretained` — the Swift analog of - C#'s `[UnmanagedCallersOnly]` + `GCHandle`), `VoiceCatClient.swift` (owns `vc_client*`, - all 38 C functions, `deinit` → `vc_client_destroy` then frees config CStrings, event - delivery on `@MainActor` via coalesced `DispatchQueue.main` drain). - - **`build-xcframework.sh`** (`clients/apple/scripts/`): runs `cmake --preset apple-dev`, - merges `libvoicecat.a` + 107 vcpkg static deps into a single ~30 MB fat static library - (`libvoicecat-fat.a`) via `libtool -static`, stages `voicecat.h` + a generated - `module.modulemap` (`module VoiceCatC { header "voicecat.h" }`) into the XCFramework - headers, runs `xcodebuild -create-xcframework` → `clients/apple/VoiceCatCore.xcframework`. - The fat library is needed because SPM binary targets link ONE `.a` per slice — without - it, the final executable gets undefined-symbol errors for protobuf/mbedtls/sodium/opus/… - (the Apple equivalent of how the Windows client ships a single `voicecat.dll` with all - deps statically linked). - - **Tests — 6/6 green:** `swift test` against a real `voicecat-server` (built by - `cmake --preset dev`). Tests mirror the C# `VoiceCat.Interop.Tests`: - `testVersionStringIsNonEmpty`, `testResultStringRoundTrips`, - `testConnectTofuAuthListChannelsRoundTrips` (connect → TOFU → confirm → guest auth → - channels → permissions → guest ListAccounts rejected), - `testAdminChannelCrudAccountCrudRoundTrips` (admin auth → channel CRUD → account CRUD), - `testScreenAudioStreamStartsAndStops` (screen-audio stream lifecycle through Swift), - `testPerStreamRecvControlsRoundTrip` (two clients, per-stream gain/mute/NR round-trip). - Catches Swift-specific interop bugs (`@convention(c)` callback lifetime, `Unmanaged` - pointer resolution, CString memory management, enum raw-value bridging, struct layout) - that C++ ctest can't. - - **Key C interop discoveries:** (1) Swift imports `vc_client*` (incomplete C struct) as - `OpaquePointer?`, not a named type — `private var handle: OpaquePointer?`. (2) C enums - are imported as `UInt32`-backed (not `Int32`) — all Swift enum mirrors use `UInt32`; - `vc_event.result` is `int32_t` (signed), bridged via `UInt32(bitPattern:)`. (3) Swift - auto-marshals `String` to `const char*` for function params, but NOT for C struct fields - — `vc_stream_desc`/`vc_channel_info` string fields need `strdup` + `defer { free }`. - (4) `selfPointer` must be a computed property (not stored) to break the circular init - dependency (`Unmanaged.passUnretained(self)` needs `self` fully initialized, but stored - properties must be set first). - - **Docs updated:** `docs/tech-stack.md` §2 (AppKit macOS / SwiftUI iOS / shared - `VoiceCatCore` package / fat static lib), `docs/architecture.md` §4 (Swift binding - notes), `docs/roadmap.md` M4 + §2 (AppKit decision recorded), `clients/apple/README.md` - (full rewrite), `PROGRESS.md` (this entry), `.gitignore` (XCFramework + SPM artifacts). - - **Next:** macOS AppKit app (`clients/apple/macOS/`) — connect dialog, saved-server list - (Keychain), TOFU identity dialog, main window (NSOutlineView + NSTableView + NSTextView - + activity log), voice controls, per-user tuning, full VoiceOver accessibility. Mirrors - the Windows `VoiceCat.App` feature set. +- **CMake preset cleanup** (2026-06-18): `m1-dev`→`dev`, `dev`→`skeleton`, `m2-dev` dropped. + New `release`, `server-release` (stripped), `apple-dev`/`apple-ios`/`apple-ios-sim`. Cross- + platform triplet auto-resolved by `cmake/voicecat-toolchain.cmake`. -- **Done:** **macOS port — `dev` + `apple-dev` presets validated** (2026-06-18). The core, - server, tools, and tests now build and run on macOS (Apple Silicon, macOS 26.5, Apple clang - 21). This lays the groundwork for the macOS/iOS Swift client. Three real bugs found and - fixed (all were cross-platform issues that manifested on macOS but were latent on - Windows/Linux): - 1. **Missing `` in `test_m2_voice.cpp` POSIX branch** — the raw-socket test's - `#else` branch included ``/``/``/`` - but not `` (needed for `addrinfo`/`getaddrinfo`/`freeaddrinfo`). On Linux glibc - these headers transitively include ``; on macOS they don't. Fixed by adding - `# include ` to the POSIX branch (mirrors `core/src/core/client.cpp:16` which - already had it). Real latent bug — would fail on any strict POSIX system. - 2. **SIGPIPE killing processes on macOS** — on macOS, writing to a closed TCP socket - raises `SIGPIPE` by default (unlike Windows where it doesn't exist, or Linux where it's - often benign). This killed `test_tofu_flow` (intermittent SIGPIPE/SEGFAULT) and would - also kill `voicecat-server` and `vccli` in production when a peer dropped mid-write. - Fixed by ignoring SIGPIPE (`std::signal(SIGPIPE, SIG_IGN)`) in both the core client - init (`core/src/core/client.cpp` POSIX branch of the `#ifdef _WIN32` WSAStartup block) - and the server startup (`server/src/server.cpp` before the `asio::signal_set`). Both are - POSIX-only (`#ifndef _WIN32`), process-global, and idempotent. The server's - `asio::signal_set(SIGINT, SIGTERM)` is unaffected (independent signals). - 3. **Use-after-free of Asio's kqueue reactor on server shutdown** — the root cause of the - deterministic `test_tofu_flow` segfault (EXC_BAD_ACCESS in - `kqueue_reactor::deregister_descriptor(this=0x0000000000000000)`). `TcpServerConn`'s - `tls_read_loop` runs on a dedicated blocking-I/O thread (not async on `io_context`). - When `Server::stop()` → `io.stop()` → `Server::run()` returned, the local - `asio::io_context` was destroyed while `tls_read_loop` threads were still running. When - a thread detected the disconnect and called `TcpServerConn::close()` → `socket.close()` - → Asio tried to deregister from the kqueue reactor — but the reactor (owned by - `io_context`) was already destroyed, and on macOS kqueue the reactor pointer is null'd - immediately. Latent on Windows (IOCP) and Linux (epoll) where the timing is more - forgiving. **Fix:** `TcpAcceptor` now tracks its connections (new `conns_` vector + - `conns_mu_`); `TcpAcceptor::stop()` closes all tracked connections while `io_context` is - still alive; new `TcpAcceptor::shutdown()` method calls `stop()` then - `wait_closed()` on each connection (new `TcpServerConn::wait_closed()` joins - `tls_thread_`); `Server::run()` calls `acceptor.shutdown()` after `io.run()` returns and - before `io` is destroyed; `Server::stop()`'s `stop_fn_` now calls `acceptor.stop()` + - `media_relay->stop()` + `io.stop()` (was just `io.stop()`). After `acceptor.shutdown()`, - when `tls_read_loop` threads exit and call `on_disconnected` → `ConnSession::close()` → - `TcpServerConn::close()`, the `close()` is a no-op (`closing_.exchange(true)` returns - true) — no reactor access occurs after `io` is destroyed. - - **Environment setup:** vcpkg cloned to `~/code/vcpkg` + bootstrapped. `VCPKG_ROOT` must - be set. Homebrew `autoconf-archive` is required (vcpkg's libsodium port needs it for - autoreconf — `brew install autoconf-archive`). `autoconf`/`automake`/`libtool` were - already installed; `glibtoolize` (Homebrew's macOS name for `libtoolize`) is handled by - vcpkg automatically. - - **Verified:** `cmake --preset dev` + `cmake --build --preset dev` green (21 binaries). - `ctest --preset dev --parallel 1` — **21/21 green** (2 consecutive runs, 64s each). - `cmake --preset apple-dev` + `cmake --build --preset apple-dev` green → valid 1.9 MB - arm64 `libvoicecat.a` (167 exported C ABI symbols, correct visibility). - `xcodebuild -create-xcframework` → valid `VoiceCatCore.xcframework` (macOS-arm64 slice - with `voicecat.h` headers). Server runtime: `voicecat-server` starts, generates identity, - SQLite, Lobby, binds TCP+UDP, clean SIGINT shutdown. Two `vccli` text chat over TLS - (M1 exit criterion on Mac). `vccli --voice --input-mode vad` starts a MIC stream via - CoreAudio (M2 protocol-level on Mac — ear test pending). `vccli --list-devices` - enumerates 4 CoreAudio input + 3 output devices with correct defaults. - - **Apple framework linking:** NOT needed — modern macOS ld (Xcode 26.5) auto-discovers - CoreAudio/CoreFoundation frameworks in `/System/Library/Frameworks` without explicit - `-framework` flags. miniaudio's `MINIAUDIO_IMPLEMENTATION` compiles the CoreAudio calls - inline, and the linker resolves them automatically. No `if(APPLE)` CMake block was added. - - **Docs updated:** `docs/building.md` (`apple-dev` row + §6 updated from "scaffolding" to - "validated"), `clients/apple/README.md` (macOS slice build confirmed, iOS slices still - scaffolding), `PROGRESS.md` (this entry). - - **Still deferred (per scope):** macOS `SCREEN_AUDIO` loopback via ScreenCaptureKit (stub - returns `false` — per `docs/voice.md §9`); iOS cross-compile presets (`apple-ios`/ - `apple-ios-sim` — scaffolding); `vc_audio_suspend`/`vc_audio_resume` ABI hooks (defer to - iOS client milestone, keep ABI stable). +- **Disconnect, keepalive & reaper** (2026-06-18): Client sends `Ping` every 15 s; server + reaper drops sessions after 45 s; UDP `KEEPALIVE` every 5 s keeps NAT alive. `vc_disconnect` + sends graceful `Disconnect` proto. Stale-user LEFT broadcast on drop. PLC capped at ~2 s. + Three new tests: `test_disconnect_left`, `test_plc_cap`, `test_reaper_timeout`. -- **Done:** **CMake preset cleanup + cross-platform build config** (2026-06-18). The preset - set was a mess — `dev` (never used), `m1-dev` (the one everyone used), `m2-dev` - (cache-identical to `m1-dev`, never used), no optimized+tests preset, no stripping. - Cleaned up to a sensible set + added cross-platform triplet auto-resolution + Apple - platform scaffolding: - - **Renames:** `dev`→`skeleton` (no-deps stub smoke — accurately named now); `m1-dev`→`dev` - (the default development preset — milestone-named presets were misleading since the - project is past M5); `m2-dev` **dropped** (cache-identical to `m1-dev`). - - **New presets:** `release` (optimized Release + tests on, symbols kept — run the suite - against optimized code or profile); `apple-dev` / `apple-ios` / `apple-ios-sim` - (scaffolding — static `libvoicecat.a` slices for the Swift Package / XCFramework; marked - "not yet CI-validated, build on macOS to verify"). - - **`server-release`** now strips binaries (`CMAKE_EXE_LINKER_FLAGS=-s` + - `CMAKE_SHARED_LINKER_FLAGS=-s`) — smaller executables for deployment. - - **Cross-platform triplet auto-resolution:** new `cmake/voicecat-toolchain.cmake` wraps - vcpkg's toolchain and resolves `VCPKG_TARGET_TRIPLET` / `VCPKG_HOST_TRIPLET` from - `CMAKE_HOST_SYSTEM_NAME` + `CMAKE_HOST_SYSTEM_PROCESSOR` — `x64-mingw-static` on Windows, - `x64-linux` on Linux, `arm64-osx` on Apple Silicon. The main presets (`dev`, `release`, - `server-release`) now work on all three platforms without per-OS variants. Cross-compile - presets (`apple-ios`, `apple-ios-sim`) override `VCPKG_TARGET_TRIPLET` explicitly. Hidden - base preset renamed `vcpkg-base`→`vcpkg-common` (now points at the wrapper toolchain - instead of vcpkg's toolchain directly). - - **No C++ source changes** — the core was already portable (every `#ifdef _WIN32` in - `client.cpp` already had a POSIX `#else`; `voicecat.h`'s export macro already handled - GCC visibility; `transport.cpp` is pure Asio; miniaudio abstracts WASAPI/CoreAudio/ALSA). - - **Docs updated:** `docs/building.md` (full rewrite — 8-preset table, platform matrix, - preset history mapping old names to new, Apple scaffolding section), `CLAUDE.md` (build - section reframed around `dev` as default, status line updated to M5-done), `README.md` - (stale "M0 skeleton" framing replaced with current M5 reality), `AGENTS.md` (build - section updated, stale "placeholder builtin-baseline" sentence deleted), - `docs/deployment.md` (server-release now stripped + cross-platform note), - `docs/tech-stack.md` §4 (triplet auto-resolution + Apple scaffolding note), - `clients/apple/README.md` (new "Building the core for Apple platforms" section with - XCFramework workflow), `clients/windows/README.md` (m1-dev→dev). - - **Code comments updated:** `core/include/voicecat.h`, `core/src/net/transport.h`, - `core/src/voicecat.cpp`, `tests/CMakeLists.txt` — preset name references updated. - - **Historical `PROGRESS.md` entries left intact** — `m1-dev`/`m2-dev` mentions in older - entries are a true record of what was run; rewriting them would falsify history. The - preset history table in `docs/building.md` §1 maps old names to new. - - **Verified:** `cmake --list-presets` shows all 8 presets. `skeleton` + `dev` configure - and build green on Windows. Apple presets are scaffolding (won't build on Windows — - expected; they require macOS). +- **Stereo screen-audio loopback** (2026-06-17): WASAPI loopback opens in channel's + stereo/mono mode (was hardcoded mono). Real stereo flows end-to-end through loopback → encode + → decode → mixer. New `test_loopback_stereo_capture`. -- **Done:** **Disconnect, timeout & keepalive system** (2026-06-18). Three reported bugs - traced to one root cause + two missing designed features, all fixed: - 1. **Stale users after disconnect + eternal PLC hiss** (root cause): `ConnSession::close()` - silently erased dropped users from the registry without broadcasting `UserEvent::LEFT`. - Peers never learned the user left → their user lists stayed stale AND their audio - engines never called `remove_stream` → Opus PLC synthesized comfort noise forever - (the "soft hissing that never goes away"). **Fix:** `SessionRegistry::broadcast_left()` - helper (mirrors `kick_user`'s first half); `ConnSession::close()` now broadcasts LEFT - before erasing. Regression test `test_disconnect_left` (`tests/`). - 2. **PLC cap** (defense-in-depth): `AudioEngine::on_playback` now caps pure-PLC at ~2 s - (`kPlcCapSamples`); after that it emits digital silence instead of more comfort noise, - so a stale stream can never hiss forever even if `remove_stream` is never called. Resets - automatically when fresh packets arrive. Test `test_plc_cap` (`tests/`). - 3. **No timeout / no ping** (missing feature): the client never sent `Ping`, the server had - no `last_seen` / reaper, and half-open connections (NAT timeout, wifi loss, sleep) left - ghost users forever. **Fix:** client sends `Ping` every 15 s from the io thread (RTT - measured from `Pong` nonce); `ConnSession::last_seen` bumped on every inbound TCP/UDP - frame; `asio::steady_timer` reaper sweeps every 15 s and drops sessions older than 45 s - (configurable via `server::Config::reaper_timeout_ms`/`reaper_sweep_ms`). Each drop - broadcasts LEFT via fix #1. Test `test_reaper_timeout` (`tests/`, 2 s timeout for fast - turnaround). - 4. **UDP KEEPALIVE** (missing feature): client sends a plaintext `KEEPALIVE` frame every - 5 s (`voice_frame.h::kFrameKeepalive`); server bumps `last_seen` + echoes back. Keeps - NAT bindings alive and lets media activity defer the reaper independently of TCP. - 5. **Graceful client disconnect:** `vc_disconnect()` now sends `Disconnect{code=0}` when - fully authenticated (`VC_STATE_CONNECTED`): it queues the envelope, sets a - `graceful_disconnect_pending_` flag, and joins the io thread — the io thread's - `drain_sends()` sends the Disconnect, sees the flag, sets `io_stop_`, and exits - naturally (its cleanup handles `teardown_voice()` + socket close). No main-thread - socket close → no double-close race. Server handles client-sent `Disconnect` - (`handle_client_disconnect` → `close()` → immediate LEFT broadcast, no reaper/EOF - wait). In non-authenticated states, the original force-close path runs (with TOFU - unblock). - - Docs updated: `docs/protocol.md §6/§7` (graceful disconnect, pinned N=3, reaper, last_seen), - `docs/voice.md §6` (KEEPALIVE plaintext + echo + last_seen), `docs/architecture.md §5` - (reaper timer), `server::Config` (reaper fields). `ctest --preset m2-dev` — **21/21 green** - (3 new tests: disconnect_left, plc_cap, reaper_timeout). +- **Windows screen-audio UI wired** (2026-06-17): `btnScreenShareToggle` in `MainForm.cs`. + No core/proto/ABI changes — all the plumbing was already there. `dotnet test` 4/4 green. -- **Done:** **Stereo screen-audio loopback capture on Windows** (2026-06-17). The WASAPI - loopback path (`start_loopback_capture`) used to hardcode `cfg.capture.channels = 1`, - downmixing the system's stereo mix to mono before the encoder ever saw it — so even on a - stereo channel, `SCREEN_AUDIO` was effectively mono (the encoder then upmixed L=R to - produce a *fake* stereo bitstream). Now the loopback device opens in the channel's mode: - stereo (interleaved L/R) when the channel is configured stereo, mono when mono. Real stereo - flows end-to-end through loopback → Opus encode → decode → stereo playback mixer. - - `audio_engine.h` — `CaptureCallback` gained an `int channels` parameter (the encoder - needs to know whether the PCM is real stereo or mono to avoid upmixing real stereo). - `start_loopback_capture(int kind)` → `start_loopback_capture(int kind, int channels)`. - New `loopback_channels_` member; new `feed_loopback_for_test` test hook (self-contained, - works on headless CI where the real WASAPI device can't init). - - `audio_engine.cpp` — `on_loopback` accumulator is now channel-aware (sized to - `frame_samples_*loopback_channels_`); `start_loopback_capture` sizes the accumulator off - the RT thread before `ma_device_start`, opens the device with `channels`, and falls back - to mono if the render endpoint rejects stereo (mirrors the playback path's fallback). - `on_capture`/`inject_capture`/`feed_capture_for_test` forward `channels` through the - callback (mic path always 1; loopback path 1 or 2). - - `client.cpp` — `on_capture_frame` takes `channels`; for `channels==2` (real stereo - loopback PCM) it encodes directly with no upmix; for `channels==1` on a stereo channel it - keeps the existing L=R upmix (mic stays mono in v1). `handle_stream_announce_result` reads - `effective_params.stereo` under the lock and passes `2` or `1` to `start_loopback_capture`. - - `test_vad_ptt_devices.cpp` — new `test_loopback_stereo_capture` behavior test: feeds a - loud-L / silent-R stereo signal through `feed_loopback_for_test`, encodes (as - `on_capture_frame` now does for `channels==2`), decodes, mixes, and asserts L≠R across - the frame (total_diff ~8.2M, well above the 960k threshold). A mono-downmixed-then- - upmixed bitstream would have L==R. Existing test lambda updated for the 4-arg callback. - - `docs/voice.md §8/§9` — diagram + notes updated: mic stays mono; `SCREEN_AUDIO` loopback - captures stereo when the channel is stereo. - - `cmake --build --preset m2-dev` + `ctest --preset m2-dev --parallel 1` — **18/18 green**. - The `vad_ptt_devices` VAD-gate sub-test has a pre-existing parallel-run timing flake - (passes serially and in isolation); unrelated to this change (VAD gate logic is unchanged - for `channels==1`, the only path the MIC uses). -- **Done:** **Screen-audio sharing wired into the Windows WinForms client** (2026-06-17). - The core already fully supported `SCREEN_AUDIO` capture on Windows (post-M3 WASAPI - loopback via `VOICECAT_HAS_LOOPBACK`, always on for the `windows-client` preset — - `core/CMakeLists.txt:85`; loopback start/stop at `client.cpp:1093`/`audio_engine.cpp:529`; - VAD/PTT/self-mute/server-mute correctly bypassed for non-MIC kinds at `client.cpp:862-879`) - and the C# Interop layer was already complete (`VcStreamKind.ScreenAudio`, - `StartStream`/`StopStream`/`SetRemoteStream`/`ListUserStreams` all generic). The gap was - purely UI wiring. **No core, proto, or C ABI changes were needed** — confirming the - "the core should support it already" assessment. - - `MainForm.Designer.cs` — new `btnScreenShareToggle` button in the voice panel top row - (`flpVoiceTop`), right after `btnMicToggle`, with full `AccessibleName`/ - `AccessibleDescription` per the existing accessibility convention. - - `MainForm.cs` — new `_screenStreamId` field; `BtnScreenShareToggle_Click` handler - mirroring `BtnMicToggle_Click` but with no device picker / VAD / PTT / mode / mute - (screen audio bypasses all of those in the core). Independent of mic — can share - without joining voice and vice versa. `HandleDisconnected` now resets - `_screenStreamId` and disables the screen toggle. `OnFormClosed` now explicitly stops - both mic and screen streams before `Disconnect()` (clean `StreamStop` messages go out - before the control channel closes). `HandleStreamStarted` already labeled - `ScreenAudio => "screen audio"`; `PerUserTuningDialog.ApplySettings` already iterates - all of a peer's streams — both unchanged, peers can independently volume-tune a - screen-audio stream vs that user's mic. - - `VoiceCatClientSmokeTests.cs` — new `ScreenAudioStream_Starts_And_Stops` test: - connects + TOFU + guest auth, `StartStream(ScreenAudio)`, asserts - `VC_EVENT_STREAM_STARTED` arrives with matching `StreamId`, `StopStream`, asserts - `VC_EVENT_STREAM_STOPPED`. Passes headless (the `StreamAnnounce` succeeds regardless - of whether the loopback device initializes on a CI box). - - `dotnet build` — 0 warnings/errors. `dotnet test` — **4/4 tests green** - (3 existing + 1 new). Event trace confirms the full - `StreamStarted → UserUpdated → StreamStopped` path through P/Invoke against a live - `voicecat-server.exe`. - - **Not yet confirmed audible by ear** — pending manual two-instance live test (one - shares screen audio while something plays on the default render endpoint, the other - hears it). This is the observable-behavior exit criterion per `AGENTS.md`. - - **Documented caveat** (`docs/voice.md §9`, unchanged): whole-device WASAPI loopback - inherently re-captures this app's own incoming voice mix (self-echo loop) — accepted - characteristic, not a bug. Process-specific loopback (Windows 10 2004+ - `AUDIOCLIENT_ACTIVATION_PARAMS`) is a future enhancement; miniaudio doesn't expose it. -- **In progress:** **M5 — moderation & admin** (2026-06-17). Server-side and C ABI are - implemented and tested: permissions, kick/ban/move/server-mute, channel CRUD, in-app account - management. Four new tests pass: `test_m5_permissions`, `test_m5_kick_ban_move_mute`, - `test_m5_admin_accounts`, `test_m5_channel_crud`. `vccli` now exposes all M5 operations via - CLI flags (`--kick`, `--ban`, `--move`, `--server-mute`/`-unmute`/`-deafen`/`-undeafen`, - `--set-permission`, `--create-channel`, `--edit-channel`, `--delete-channel`, - `--create-account`, `--reset-password`, `--delete-account`, `--list-accounts`) plus - `--username`/`--password` for account auth and `--self-mute`/`--self-deafen`. Docs updated: - `docs/protocol.md` (envelope tags for `ServerMuteRequest`/`ListAccountsResult`, `User.server_deafened`, - `GenericResult` usage), `docs/security.md` (BLAKE2b channel passwords, `bans` schema). - `ctest --preset m1-dev` — **18/18 green**. Windows WinForms UI now exposes all M5 operations: - channel CRUD with full per-channel Opus audio config, user moderation (kick/ban/move/server - mute/server deafen/set permissions), and server account management. `dotnet test` of the - Windows solution passes. Still to do: DRED/audio-quality polish. -- **Done:** **Fixed multi-user voice — relayed frames failed AEAD decryption (nonce desync)** - (2026-06-17, reported live: with 2+ people in a channel, audio was one-directional — "I can - hear them but they can't hear me" — and a 3rd joiner heard nobody). Root cause was in the SFU - relay (`server/src/media_relay.cpp`). The media AEAD nonce is an *implicit per-direction - monotonic counter*; `open()` reconstructs it from the 14-byte header's `seq` field (the AAD), - so the wire contract is `header.seq == the counter seal() used` (the client honors this at - `client.cpp:907`). The relay decrypted each inbound frame with the sender's key, then re-sealed - with the **recipient's** `send_crypto` (its own counter) but **forwarded the sender's header - verbatim** — so `header.seq` carried the sender's counter, not the recipient's. The recipient's - `open()` rebuilt the wrong nonce → every relayed frame failed auth and was silently dropped. It - only "worked" while the sender's counter coincidentally equalled the server→recipient counter - (a single first-ever sender into a fresh recipient), which is exactly why the first/sole talker - was heard but reverse/3rd-party audio was not. **Fix:** before re-sealing, the relay rewrites the - outgoing header's `seq` (bytes [8..9]) to the recipient's `peek_send_counter()`, so each - server→client direction is one contiguous monotonic counter and the nonce always matches (the - anti-replay window also stops seeing false replays from interleaved senders). Safe because the - jitter buffer orders by `timestamp`, not `seq` (`audio_engine.h`); `seq` exists only to carry the - AEAD counter. No wire-format/proto/ABI change. Regression test added in `tests/test_media_aead.cpp` - (`test_relay_interleaved_reseal`): two senders interleaved into one recipient all decrypt with the - fix, and the verbatim-seq path is asserted to fail. `ctest --preset m1-dev` — **18/18 green** (run - via PowerShell; Git Bash can't resolve the runtime DLLs. `vad_ptt_devices` is timing-flaky over - loopback — passes on re-run — unrelated to this fix). **Latent, separate:** the secondary - "3rd joiner sometimes can't see other users" report is a control-plane (TCP snapshot/UserEvent) - issue, not this AEAD bug — re-verify after live testing before investigating. Also still latent: - the 16-bit `seq` wraps after 65536 frames per direction (faster on a busy relay) with no ROC, so - the implicit counter desyncs on long continuous sessions (`crypto.cpp` open() TODO). -- **Done:** **Fixed "randomly bumped to Lobby" in the Windows client — actors were excluded - from their own state-change broadcasts** (2026-06-17, reported live: a connected client would - intermittently snap from its joined channel back to Lobby in the UI). Root cause was a design - inconsistency, not a disconnect: the server delivered self-initiated state changes (channel - join/leave, stream announce/stop) only as a private `*Result` to the actor and broadcast the - authoritative `UserEvent::UPDATED` to *everyone else*. The core never applied the join result - to its `SessionModel`, so `vc_list_users()` kept self in the old channel; the Windows - `HandleUserUpdated` rebuilds `_currentChannelId` from `vc_list_users()` on **any** user's - UPDATED event, so the next unrelated event (someone joining, announcing/stopping a stream, - being muted) surfaced the stale self-channel → "bumped to Lobby." Flaky because it depended on - other users' activity. **Fix (broad, per the actor-sees-own-changes principle):** the server - now broadcasts these `UserEvent::UPDATED`s to **all** clients including the actor - (`server/src/conn_session.cpp`, `broadcast(…, /*exclude*/ 0)`), and text fan-out now includes - the sender (`resolve_text_targets`), so every client converges via one authoritative path. - The `*Result` is now purely ack/correlation/actor-private payload; response handlers no longer - mutate the local model. Windows client drops its optimistic text echo (the relay comes back) - and renders the sender's own message via `HandleTextMessage`. Documented the - response-vs-broadcast contract in `docs/protocol.md` §6. Registry-level admin broadcasts - (move/mute/kick/channel CRUD) already used `exclude=0` and were correct. `ctest --test-dir - build/m1-dev` — **18/18 green** (PowerShell); Windows `VoiceCat.App` builds 0 warnings. - ~~**Latent, not fixed:** the client never sends the `Ping` keepalive that `docs/protocol.md` §7 - describes (only the server answers pings) — unrelated to this bug, noted for later.~~ - **Resolved** (2026-06-18): full keepalive/timeout/disconnect system implemented — see - "disconnect, timeout & keepalive" entry below. -- **Done:** **Fixed a *second* silent-playback bug — the playout clock free-ran and drifted off - the stream** (2026-06-17, reported live: both `vccli` and the Windows client showed `talking=1/0` - correctly on VAD/PTT, mic + screen-share were recognized by peers, but nothing was audible). - Root cause: `RemoteStream::playout_ts` was only ever seeded to `0` and then advanced one Opus - frame per playback callback **via the PLC path too** (`core/src/audio/audio_engine.cpp` - `on_playback`), so it free-ran at ~1× wall-clock regardless of whether the sender was - transmitting. The sender's frame timestamps only advance while it actually sends (the VAD/PTT - gate in `core/src/core/client.cpp` returns before `ls.timestamp += samples`). Across a late join - or any VAD/PTT silence gap the two clocks diverged without bound; once past the jitter buffer's - 500 ms late-drop window, every real frame was dropped-as-late (clock ahead) or never-due (clock - behind) → permanent silence, while the talk indicator (driven by `push_recv_frame`, independent - of the jitter buffer) stayed lit. The M3 E2E test missed it because clients there talked - continuously right after joining, keeping the clocks aligned. **Fix:** `JitterBuffer` gained - `peek_front_ts()` (try-lock, RT-safe); `on_playback` now seeds/re-syncs `playout_ts` to the - earliest buffered frame on the first frame and whenever it has drifted past ±200/500 ms - (`kResyncAheadSamples`/`kResyncBehindSamples`), which both seeds startup and recovers after every - silence gap. New regression test `test_playout_resync` (`tests/test_vad_ptt_devices.cpp`): - free-runs the clock ~2 s past the drop window, pushes a `ts=0` frame, asserts audible output — - verified to fail (energy=0) with the fix disabled, pass (energy≈15M) with it. `ctest --test-dir - build/m1-dev` — **14/14 green** (run via PowerShell; Git Bash exec gotcha for these binaries, see - `docs/building.md`). **Not yet confirmed audible by ear** — pending the user re-running their - live test. -- **Done:** **Fixed silent-playback bug in `AudioEngine::on_playback`** (2026-06-16, found via - live manual test: two `vccli --voice` clients, control-plane events and VAD all correct, but - zero audible output). Root cause: `opus_decode()`'s `max_samples` was being passed the - *hardware playback callback's* frame count (miniaudio's own choice, frequently smaller than - one Opus frame — e.g. ~480 samples on default low-latency WASAPI periods), instead of the - decoder's fixed frame size (960 @ 20ms/48kHz). Since the real packet almost always decodes to - more samples than that, `opus_decode` returned `OPUS_BUFFER_TOO_SMALL` on nearly every - callback — frames were correctly received/decrypted/jitter-buffered, just never decoded into - audible PCM. `mix_for_test()`'s white-box test masked this because it always called - `on_playback` with `frames == frame_samples`, the one case where the bug is invisible. - Fix: `RemoteStream` (`core/src/audio/audio_engine.h`) gained a small ring buffer - (`init_ring`/`push_ring`/`pop_ring`) that decouples decode cadence from playback-callback - cadence — `on_playback` (`core/src/audio/audio_engine.cpp`) now tops the ring up by decoding - whole Opus frames (`decoder.frame_samples()`, never the hardware `frames`) and drains exactly - `frames` samples-per-channel from it each callback, silence-padding (PLC) on underrun. Also - fixes a latent `playout_ts` bug: it now advances by the actual decoded sample count per Opus - frame, not by the hardware callback's (unrelated) frame count, which was the wrong unit for - jitter-buffer timestamp comparisons. `ctest --test-dir build/m1-dev` — 12/12 green (run via - PowerShell; Git Bash exec gotcha for these binaries, see `docs/building.md`). **Not yet - confirmed audible by ear** — pending the user re-running their live two-`vccli` test. -- **Done:** **Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI - loopback** ✓ complete (2026-06-16). Closes all three items M3 explicitly carried forward as - out of scope (see the dated section below for the full file-by-file change list). - `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive full-suite runs), - including the new `test_vad_ptt_devices` (real `vc_client`s against a real server, plus a - white-box `AudioEngine` stereo-mix check — same ABI-level-coverage lesson as M2/M3). - Manually verified live: `vccli --list-devices` against real hardware, and - `vccli --voice --input-mode vad` connecting/streaming without incident. - **Still explicitly out of scope** (carried forward, not silently dropped): - - Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream; v1 ships a - lightweight energy/RMS VAD instead (see docs/roadmap.md §2, docs/voice.md §8/§11). There is - **no AEC, NS, or AGC implementation at all**, not just a deferred VAD. - `vc_set_remote_stream(..., noise_reduction)`'s per-stream NS toggle is unaffected by this - pass and stays exactly as inert as it was after M3 (`ApmPassthrough`, no PCM modification). - - macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — this pass is - Windows-only for real loopback capture; other platforms keep `vc_test_inject_capture` as - the only way to feed `SCREEN_AUDIO`. - - Process-specific WASAPI loopback — miniaudio's loopback mode captures the whole render - endpoint (including this app's own incoming voice mix), not a single process. - - The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture` - (mutex lock, heap allocation, blocking `sendto` on the miniaudio real-time callback - thread) — predates this work, documented but not fixed; fixing it needs the lock-free - ring-buffer hand-off `docs/architecture.md §3` specifies, a separate, larger refactor. -- **Done:** **M4 — Windows WinForms C# client** ✓ complete (2026-06-17). Full details in the - M4 section below. `ctest --preset m1-dev` — **14/14 tests** green. `dotnet build` — 0 - warnings/errors across all three C# projects. Manually verified: saved servers, TOFU - first-connect dialog, channel tree, join, voice (VAD/PTT/always-on + sensitivity slider + - per-user gain/mute/NR), text chat (channel + private), device pickers, level meter. -- **Next:** macOS/iOS Swift client (M4 continued) and/or **M5** (admin UI, moderation, kick/ - ban). The C ABI is complete and stable through M4 — both directions are unblocked. See - `docs/roadmap.md §M4–M5`. +- **Bug fixes** (2026-06-16 – 2026-06-17): + - *AEAD nonce desync in SFU relay* — relay forwarded sender's `seq` verbatim; recipient + nonce reconstruction used the wrong counter. Fixed by rewriting the outgoing `seq` field + to the recipient's `peek_send_counter()`. + - *Playout clock free-ran* — `playout_ts` advanced even during VAD/PTT silence gaps, + eventually dropping all frames as too-late. Fixed with resync in `on_playback` via + `JitterBuffer::peek_front_ts()`. + - *Stale users after disconnect* — `ConnSession::close()` didn't broadcast `UserEvent::LEFT` + before erasing. Fixed; PLC cap added as defense-in-depth. + - *"Randomly bumped to Lobby"* — server excluded the actor from its own state-change + broadcasts. Fixed: `UserEvent::UPDATED` now goes to all clients including the actor. + - *Silent playback after join* — `opus_decode` received hardware callback frame count as + `max_samples` instead of the Opus frame size. Fixed with a decode ring buffer. --- @@ -983,357 +287,91 @@ up instantly. Newest status at the top. --- -## M0 — Scaffolding ✓ (completed) +## M0 — Scaffolding ✓ -- [x] Repo layout (`core/ server/ tools/ clients/ tests/`), CMake + presets, vcpkg manifest. -- [x] C ABI header `core/include/voicecat.h` (full surface, stubbed). -- [x] Protocol source-of-truth `core/proto/voicecat.proto` (matches docs/protocol.md). -- [x] Core stubs for all six subsystems (net/crypto/codec/protocol/session/audio) + `vc_client`. -- [x] `voicecat-server` (arg parsing, config, stub run) and `vccli` (drives the C ABI). -- [x] CTest **smoke test** asserting the C ABI contract (not just "it compiles"). -- [x] `.gitattributes` (LF), `.gitignore`, `.clang-format`, onboarding docs. -- **Verified:** `cmake --preset dev && cmake --build --preset dev && ctest --preset dev` → green. +Repo layout (`core/ server/ tools/ clients/ tests/`), CMake + vcpkg manifest, C ABI header +(`voicecat.h`), proto source of truth, core stubs for all six subsystems, `voicecat-server` + +`vccli` skeletons, smoke CTest, `.clang-format`/`.gitattributes`/`.gitignore`. --- ## M1 — Control plane ✓ (completed 2026-06-15) -**Exit criterion:** ✓ `test_m1_integration` — two clients authenticate over TLS 1.3 (guest -+ Argon2id password), exchange channel and private text messages. Passes in ~1 s. +**Exit criterion:** `test_m1_integration` — two clients authenticate over TLS 1.3 (guest + +Argon2id), exchange channel + private text. ~1 s. -- [x] vcpkg baseline + `m1-dev` preset; `find_package` for protobuf/mbedTLS/libsodium/asio/sqlite3. -- [x] `FrameCodec` feed + emit; `encode_envelope` / `decode_envelope`. -- [x] Asio TCP acceptor + `TcpServerConn` (TLS path: blocking handshake thread + `tls_read_loop`). -- [x] `TlsContext` (mbedTLS 1.3, server cert/identity, ECDSA-P256 self-signed, TOFU on client). -- [x] `WorkerPool` (3 threads, used for Argon2id). -- [x] `Database` — SQLite, Argon2id via libsodium, `create_account` / `authenticate` / bootstrap admin. -- [x] `voicecat-admin` — account add/reset/del/list against live DB file. -- [x] `ServerIdentityManager` — generate/persist Ed25519 key + cert; fingerprint display. -- [x] `ConnSession` — WaitingHello → WaitingAuth → Authenticated state machine; full protocol relay. -- [x] `SessionRegistry` — channel tree, user map, broadcast, text routing. -- [x] `vc_client` (`client.cpp`) — full M1 C ABI: connect/TLS/ClientHello/AuthRequest/text/disconnect. -- [x] `Server::run()` — io_context, acceptor, worker pool, signal handling, `on_ready` callback. -- [x] `test_m1_integration` — M1 exit criterion. Verified green 2026-06-15. - -**Key bug fixed:** double-framing in `ConnSession::send_envelope` — `encode_envelope` was -pre-framing the protobuf, then `TcpServerConn::send_frame` re-framed it. Fixed by serializing -raw protobuf bytes directly and letting `send_frame` add the single `[4-byte len]` prefix. - ---- +`FrameCodec`, `TlsContext` (mbedTLS 1.3, ECDSA-P256 self-signed, TOFU pins TLS leaf-cert +SHA-256), `WorkerPool`, `Database` (SQLite + Argon2id), `ServerIdentityManager`, +`ConnSession` state machine, `SessionRegistry`, `vc_client` full M1 C ABI, `voicecat-admin` +CLI, dual-stack `TcpAcceptor`. **Key bug fixed:** `send_frame` double-framing — `encode_envelope` +was pre-framing the protobuf; fixed by passing raw protobuf bytes. --- ## M2 — Voice, single stream ✓ (completed 2026-06-16) -**Exit criterion:** ✓ `test_m2_voice` — two headless clients authenticate over TLS, bind UDP, -announce a MIC stream, send 50 encrypted Opus frames; server SFU relay re-encrypts + forwards -to the second client; B receives ≥ 25 frames and all decrypt correctly. Passes in ~4 s. +**Exit criterion:** `test_m2_voice` + `test_voice_client_abi` — two headless clients auth, bind +UDP, 50 Opus frames relayed + re-encrypted by SFU, B receives ≥25 and decrypts. ~4 s. -- [x] `m2-dev` preset (inherits `vcpkg-base`, binaryDir `build/m2-dev`); `m1-dev` also builds all M2 code. -- [x] `core/CMakeLists.txt` — `find_package(Opus)`, `find_path(MINIAUDIO_INCLUDE_DIR)`. -- [x] `core/src/net/voice_frame.h` — 14-byte UDP header (type/flags/codec/ssrc/seq/ts), serialize/parse, `make_udp_binding_packet`. -- [x] `SodiumMediaCrypto` — ChaCha20-Poly1305 AEAD; counter-nonce; 64-bit sliding-window anti-replay; `derive_send/recv` from TLS RFC 5705 exporter. -- [x] `OpusEncoder` / `OpusDecoder` — libopus 1.6, FEC, DTX, PLC (free; nullptr → decoder extrapolates). -- [x] `UdpMediaChannel` — async UDP socket (asio); thread-safe `send_to`; async recv loop. -- [x] `JitterBuffer` — per-ssrc, EWMA jitter estimation, adaptive depth 20–200 ms, late-drop at 500 ms. -- [x] `AudioEngine` — miniaudio capture+playback; `inject_capture()` bypass for headless tests; per-ssrc RemoteStream with OpusDecoder + JitterBuffer. -- [x] `ApmProcessor` — `ApmPassthrough` stub (VAD always open); WebRTC APM deferred until M3. -- [x] `on_tls_ready` callback in `TcpChannelCallbacks` — server derives and stores media AEAD keys immediately after TLS handshake. -- [x] `ConnSession` M2 — `udp_token` generated at construction; included in `AuthResult`; `handle_udp_binding` (verifies token, TCP ack); `handle_stream_announce` (assigns SSRC via registry); `udp_media_port` in `ServerHello`. -- [x] `SessionRegistry` M2 — `register_udp_token`, `find_by_udp_token`, `register_udp_endpoint`, `find_by_udp_endpoint`, `assign_ssrc`, `find_channel_sessions`, `user_channel`. -- [x] `MediaRelay` — SFU UDP relay; `kFrameUdpBinding` → endpoint binding; `kFrameVoice` → decrypt/re-encrypt/forward to channel members. -- [x] `Server::run()` — creates and binds `MediaRelay`; passes media port to `ConnSession`; wires `on_tls_ready` to derive per-connection media AEAD keys. -- [x] `test_voice_frame` — header round-trip, big-endian layout, binding packet format. -- [x] `test_media_aead` — seal/open round-trip, anti-replay, tamper detection, multi-packet sequence. -- [x] `test_opus_codec` — encode/decode round-trip energy check (within 3 dB), PLC, frame-samples helper. -- [x] `test_m2_voice` — M2 exit criterion (raw-socket harness). Verified green 2026-06-16. - -**Follow-up (same day):** the above made `test_m2_voice` pass, but `vc_client`'s public voice -methods were still stubs — the *actual* M2 exit criterion ("two vccli/early-GUI clients talk") -wasn't met. Closed the gap: - -- [x] `core/src/core/client.cpp` — real `stream_start`/`stream_stop`/`set_self_mute`/ - `set_remote_stream`; UDP-binding handshake (`start_udp_binding`/`handle_udp_binding_ack`/ - `finish_udp_binding`); media key derivation from `tls_` (RFC 5705 exporter); `run_udp_recv` - (AEAD-open → `JitterBuffer::Frame` → `audio_engine_.push_recv_frame`); `on_capture_frame` - (encode → seal → `sendto`); `sync_remote_streams` (diffs a `User` proto's `streams` against - `remote_streams_`, wiring up `OpusDecoder`s and emitting `STREAM_STARTED`/`STOPPED`). - `set_input_device`/`set_input_mode`/`set_push_to_talk`/`list_devices` remain - `VC_ERR_NOT_IMPLEMENTED` — no device-enumeration backend yet; scoped to M3 (VAD/PTT). -- [x] `core/src/session/session.cpp/h` — `SessionModel::find_user`, `find_user_by_ssrc`, - `Stream{stream_id, ssrc, kind, label, sample_rate, frame_ms}`. -- [x] `server/src/conn_session.cpp/h` — `handle_stream_announce`/`handle_stream_stop` now - broadcast via `SessionRegistry::set_user_stream`/`clear_user_stream` → `UserEvent::UPDATED`. -- [x] `server/src/session_registry.cpp/h` — `set_user_stream`/`clear_user_stream` (mutate a - user's `StreamInfo` list, return the updated `User` proto for broadcast). -- [x] `tests/test_voice_client_abi.cpp` — drives two real `vc_client` instances through - `vc_connect`/`vc_authenticate_guest`/`vc_stream_start`/`vc_stream_stop`; asserts client B - observes client A's `STREAM_STARTED`/`STOPPED` events. Verified green 2026-06-16. -- [x] `tools/vccli/src/main.cpp` — argv parsing (`--host/--port/--nick/--channel/--voice/ - --mute/--text`); `--voice` starts a MIC stream and blocks on SIGINT, printing `on_event` - callbacks live (unbuffered stdout — MinGW/MSVCRT treat `_IOLBF` as full buffering for - non-console streams). Dropped the originally-planned `--voice-loopback` and the - `tx=N rx=M lost=K jitter=J` stats line: `voicecat.h` exposes no PCM-injection hook or - jitter/loss stats getter publicly, only `on_event` + `on_level` (RMS). Manually verified: - two `vccli --voice` instances see each other's stream start in real time. +14-byte UDP voice header, `SodiumMediaCrypto` (ChaCha20-Poly1305 + 64-bit anti-replay), +`OpusEncoder`/`OpusDecoder` (FEC, PLC), `UdpMediaChannel`, `JitterBuffer`, `AudioEngine` +(miniaudio), `MediaRelay` SFU. **Key bug fixed:** `on_playback` passed hardware callback frame +count as `opus_decode` max_samples; fixed with a per-stream decode ring buffer. --- ## M3 — Multi-stream & per-channel tuning ✓ (completed 2026-06-16) -**Exit criterion:** ✓ `test_m3_multistream` — a real `vc_client` (A) runs two concurrent local -streams (MIC + SCREEN_AUDIO) with distinct stream ids; a second client (B) sees both as -separate `STREAM_STARTED` events and a `VC_EVENT_TALK_STATE` talking edge for A's MIC stream; -B independently sets gain/mute/noise-reduction on each of A's streams without one call -affecting the other; A then joins "Music Room" (channel 2: stereo/128kbps/`OPUS_AUDIO`/no -DTX) and announces a fresh MIC stream there, while B stays in "Lobby" (channel 1: mono/24kbps/ -`OPUS_VOIP`/DTX on) — `vc_get_stream_audio_config` shows their effective Opus config differs -exactly as the server enforces per channel. Passes in ~2.4s; verified across 8 consecutive -standalone runs + 3 consecutive full-suite runs with no flakes. +**Exit criterion:** `test_m3_multistream` — client A runs two concurrent streams (MIC + +SCREEN_AUDIO); B sees both; per-stream gain/mute/NR independent; effective Opus config matches +channel's server-enforced settings. ~2.4 s. -Exploration before implementing turned up several bugs/gaps where the wire format already -supported this milestone but the client/server logic didn't — these were fixed as part of M3, -not treated as pre-existing-and-out-of-scope: - -- [x] **Server `stream_id` bug** — `handle_stream_announce` always wrote `stream_id=1`, so a - second stream from the same user silently overwrote the first in - `SessionRegistry::set_user_stream`'s replace-by-id logic. Fixed with a per-session counter - (`ConnSession::next_stream_id_`) + `announced_stream_ids_` (also now validated in - `handle_stream_stop`, rejecting stops for ids the session never announced). -- [x] **Per-channel `AudioConfig` was modeled but never populated/enforced.** - `SessionRegistry::init_default_channels()` now seeds Lobby (id=1: mono, 24kbps, `OPUS_VOIP`, - FEC+DTX on) and a new "Music Room" (id=2: stereo, 128kbps, `OPUS_AUDIO`, FEC+DTX off) with - real `AudioConfig`s; new `SessionRegistry::channel_audio_config(channel_id)` accessor (there - was no per-id channel getter before, only `channel_snapshot()`). `handle_stream_announce` - now treats the channel's config as authoritative (mode/frame_ms/application/fec/dtx/ - complexity), clamping (not overriding) `bitrate_bps` to the channel's ceiling. -- [x] **Client silently dropped `mode`/`dtx`/`complexity`/`application` from `effective_audio`** - even for the single M2 stream — `handle_stream_announce_result` and `sync_remote_streams` - only copied `sample_rate`/`bitrate_bps`/`frame_ms`/`fec` into `OpusParams`. New shared - `opus_params_from_audio_config()` helper (`client.cpp`) fixes both the send and receive - paths. -- [x] `core/src/codec/opus_codec.h/.cpp` — new `OpusApplication` enum + `OpusParams::application` - field; `OpusEncoder::init` now honors it instead of hardcoding `OPUS_APPLICATION_VOIP`. -- [x] `core/src/session/session.h/.cpp` — `Stream` struct extended with the full `AudioConfig` - (mode/bitrate_bps/application/fec/expected_packet_loss/dtx/complexity), not just - sample_rate/frame_ms; `copy_streams()` now copies all of it. -- [x] `core/src/core/client.h/.cpp` — local-stream state is now a `std::unordered_map` keyed by `vc_stream_kind` (one active stream per kind — MIC/SCREEN_AUDIO/ - AUX_DEVICE are each singletons for a client), replacing the M2 single-stream fields. - `StreamAnnounce`/`StreamAnnounceResult` round-trips are now correlated by `request_id` - (already round-tripped on the wire; just wasn't read) via `pending_announce_kind_`, so - multiple concurrent announces from one client resolve to the right `LocalStream`. - `on_capture_frame` takes a `kind` and `channels` parameter; for mono capture (`channels==1`) - on a stereo channel it upmixes L=R, and for real stereo capture (`channels==2`, the - `SCREEN_AUDIO` loopback path on a stereo channel) it encodes directly with no upmix. `vc_set_self_mute`'s `mic_muted` only - gates the `MIC` kind — a concurrent `SCREEN_AUDIO` share keeps playing while muted. - `set_remote_stream` now actually wires `noise_reduction` through (previously parsed and - discarded). New `run_talk_timer()` (a small dedicated thread, started alongside the UDP - media path, never the miniaudio callback thread) polls both remote talk-state edges - (`AudioEngine::poll_talk_transitions()`) and local capture-activity edges, emitting - `VC_EVENT_TALK_STATE`. -- [x] **Fixed a thread-join race in `teardown_voice()`** — it's called both from `run_io()`'s - own cleanup and from `disconnect()`, on different threads; without serialization both could - see `udp_thread_`/`talk_timer_thread_` as `joinable()` simultaneously and race to `join()` - the same `std::thread` (UB; surfaced as an intermittent `std::system_error: No such process` - under `ctest`). Added a `teardown_mu_` guard around the whole function. This pre-existed for - `udp_thread_` alone (likely the same root cause as the `test_m1_integration`/`test_m2_voice` - cleanup-path flake noted in the M2 section above) — adding `talk_timer_thread_`'s join just - made it surface more often, so it was fixed properly here rather than carried forward again. -- [x] `core/src/audio/audio_engine.h/.cpp` — `CaptureCallback` gained a `kind` parameter - (the real miniaudio capture device is always tagged `kind=0`/MIC; a second concurrent local - stream is fed via its own `inject_capture(kind, ...)` ring buffer — `inject_taps_`, keyed by - kind — since there is only one real hardware capture device in M3). Fixed a buffer-sizing - bug in `on_playback`'s per-stream decode (`opus_decode`'s `frame_size` parameter is - samples-*per-channel*, not total samples — the old code passed `frames * params_.channels`, - which would have overflowed the decode buffer for any stereo stream). Stereo decoder output - is downmixed (avg L/R) into the engine's mono mix accumulator immediately after decode. - `RemoteStream` gained `recv_ns`/`noise_reduction_enabled` (lazy `ApmProcessor` instantiation - — freed on disable, so no separate instance cap is needed per the roadmap's guidance) and - `last_voice_ms`/`talking` (talk-indicator edge state, updated in `push_recv_frame`); new - `set_stream_noise_reduction()` and `poll_talk_transitions()`. Note: until `VOICECAT_HAS_APM` - is wired to a real WebRTC APM build, the NS toggle is plumbed end-to-end but behaviorally a - passthrough no-op (`ApmPassthrough` doesn't touch PCM) — same situation send-side APM has - been in since M2; M3's job was the plumbing, not the DSP backend. -- [x] **New C ABI surface** (`core/include/voicecat.h`, additive only): - `vc_audio_config` struct + `vc_get_stream_audio_config(c, user_id, stream_id, out)` — the - effective Opus config for a stream you own or a peer's, reading from the (now richer) - `LocalStream`/`session::Stream`. `vc_test_inject_capture(c, stream_id, pcm, samples)` — - clearly-marked **test-only**, forwards to `AudioEngine::inject_capture`, so - `test_m3_multistream` can drive two concurrent synthetic-audio streams through the real ABI - without a microphone. -- [x] `tests/test_m3_multistream.cpp` — the M3 exit criterion (ABI-level, mirrors - `test_voice_client_abi.cpp`'s approach per the M2 lesson). Registered in `tests/CMakeLists.txt`. - -**Explicitly out of scope for this pass** (confirmed with the user before implementing): -- `vc_set_input_device`/`vc_set_input_mode`/`vc_set_push_to_talk`/`vc_list_devices` (device - enumeration + VAD/PTT input gate) — still `VC_ERR_NOT_IMPLEMENTED`. These were mentioned as - "scoped to M3" in the M2 follow-up notes above, but docs/roadmap.md's M3 bullets never - actually listed them — deferred again, now tracked explicitly rather than implicitly. -- Real WASAPI desktop-audio loopback capture for `SCREEN_AUDIO` — the engine now supports - feeding a second concurrent local stream via `inject_capture`, but only synthetic PCM is - wired up; a real loopback capture device is a follow-up. -- True stereo *playback output* — `AudioEngine`'s mixer/output device stays mono; stereo - streams are downmixed after decode (see above). The Opus wire format itself is fully - stereo-correct. +Fixed server `stream_id` counter bug (always wrote `1`). Per-channel `AudioConfig` populated +(Lobby: mono/24kbps/VOIP + DTX; Music Room: stereo/128kbps/AUDIO). `LocalStream` map, +`pending_announce_kind_`, `run_talk_timer()`, thread-join race in `teardown_voice()` fixed. +New C ABI: `vc_get_stream_audio_config`, `vc_test_inject_capture`. --- -## Post-M3 follow-up — device enumeration, VAD/PTT gate, stereo playback, WASAPI loopback ✓ (completed 2026-06-16) +## Post-M3 follow-up ✓ (completed 2026-06-16) -Closes all three items the M3 section above explicitly carried forward as out of scope. +- **Device enumeration** — `vc_list_devices`/`vc_set_input_device`; opaque hex device ids; + `vc_free_device_list` now frees. Works pre-connect. +- **VAD/PTT gate** — `EnergyVadProcessor` (RMS threshold ~0.025, 300 ms hang-time); + `vc_set_input_mode`/`vc_set_push_to_talk`; MIC-only (SCREEN_AUDIO/AUX_DEVICE bypass). +- **True stereo playback** — `playback_channels=2`; stereo decoded L→L R→R in mixer; mono + upmixed L=R; hardware fallback to mono on failure. +- **WASAPI loopback** — `loopback_device_` with `ma_device_type_loopback`; + `VOICECAT_HAS_LOOPBACK` macro (Windows-only). `vccli --share-screen-audio`. -**Exit verification:** `ctest --test-dir build/m1-dev` — **12/12 tests** green (3 consecutive -full-suite runs), including the new `test_vad_ptt_devices` (device enumeration + VAD/PTT gate -through real `vc_client`s against a real server, plus a white-box `AudioEngine` stereo-mix -check — no audio hardware needed for that last part). Also verified 5 consecutive standalone -runs of the new test alone, no flakes. Manually verified live on Windows: `vccli ---list-devices` against real hardware (3 input / 4 output devices, correct `is_default` -flags), and `vccli --voice --input-mode vad` connecting + streaming without incident. - -- [x] **Device enumeration** (`vc_list_devices`) — `AudioEngine::enumerate_devices(bool - capture)` (static, works without a running engine — inits a throwaway `ma_context` via - `ma_context_get_devices`). `device_id`/`vc_device.id` is an opaque hex-encoded raw - `ma_device_id` (not the device name — names aren't guaranteed unique); documented as an - internal contract callers must round-trip, never construct by hand. `vc_client::list_devices` - works in any connection state (no `VC_STATE_CONNECTED` gate) since device pickers need to - populate pre-connect. `vc_free_device_list` is now a real free (was a no-op stub). -- [x] **Input device selection** (`vc_set_input_device`) — stores the device id on the - targeted `LocalStream` (new field); for the MIC stream, if the engine is already running, - restarts it (`stop()` + `ensure_audio_running()`) to pick up the new device. Simplified: it - restarts unconditionally rather than trying to detect whether the device id actually - changed (`AudioEngine` has no getter for "current device"). -- [x] **VAD/PTT input gate** (`vc_set_input_mode`, `vc_set_push_to_talk`) — new - `EnergyVadProcessor` (`core/src/audio/apm_processor.cpp`) implementing the existing - `ApmProcessor` interface: energy/RMS threshold (default ~0.025 normalized) + hang-time - (default 300 ms, matching `kTalkHangoverMs`). New factory `ApmProcessor::create_vad()`, - kept separate from `create()` (which recv-side per-stream NS still uses, unaffected by this - pass). `vc_client` gained `current_input_mode_`/`ptt_active_`/`mic_vad_`; the gate is - inserted in `on_capture_frame`, **MIC-only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it - (gating a desktop-audio share on the user's own voice activity would silently drop shared - music/video audio). `last_capture_ms` (drives the talk indicator) is now updated *after* the - gate check, not before, so a VAD/PTT-closed frame never shows as "talking". `mic_vad_` is - constructed once the MIC stream's `StreamAnnounceResult` lands (on `io_thread_`), not lazily - inside the capture path. -- [x] **True stereo playback** — `AudioParams::channels` split into `capture_channels` (stays - 1) and `playback_channels` (now 2, unconditionally). `AudioEngine::on_playback` no longer - downmixes decoded stereo streams to mono before mixing — stereo decode output is mixed - directly (L→L, R→R); mono decode output is upmixed (duplicated into both channels). Falls - back to a 1-channel playback device once if the 2-channel `ma_device_init` fails (unusual - hardware). New test-only `AudioEngine::mix_for_test()` exposes the mixer for white-box - testing without a real `ma_device`. -- [x] **WASAPI loopback capture for `SCREEN_AUDIO`** — new `VOICECAT_HAS_LOOPBACK` macro - (`core/CMakeLists.txt`, Windows-only). `AudioEngine` gained a separate `loopback_device_` - (own lifecycle, decoupled from the mic capture/playback devices) with - `start_loopback_capture()`/`stop_loopback_capture()`, using miniaudio's - `ma_device_type_loopback` against the default render endpoint. Its callback feeds - `capture_cb_` directly (same pattern as the real mic capture device), **not** through - `inject_capture()`'s test-only ring. Wired into `vc_client::handle_stream_announce_result` - (start, alongside `ensure_audio_running()`) and `stream_stop` (stop) for - `VC_STREAM_SCREEN_AUDIO`. Non-Windows builds keep `vc_test_inject_capture` as the only way to - feed `SCREEN_AUDIO`. -- [x] `tools/vccli/src/main.cpp` — new flags `--list-devices`, `--input-device`, - `--input-mode vad|ptt`, `--share-screen-audio`; while `--voice` is running, a background - stdin-reader thread accepts `ptt on`/`ptt off`/`mode vad`/`mode ptt` (the most portable way - to drive PTT interactively from a headless CLI — no SIGUSR1 equivalent on Windows). Also - prints `VC_EVENT_TALK_STATE`. Known minor caveat: on Windows the stdin-reader thread is - detached (not joined) on exit, since `std::getline` can't be interrupted from another thread - — a `vc_client*` use-after-free is theoretically possible if a command line arrives in the - brief window between teardown and process exit; acceptable for a headless test/dev tool. -- [x] `tests/test_vad_ptt_devices.cpp` — new test covering all four items above; registered in - `tests/CMakeLists.txt`. `tests/test_smoke.cpp`'s device-list assertion is now conditional on - `VOICECAT_HAS_AUDIO` (was a hard `VC_ERR_NOT_IMPLEMENTED` assertion) — `VC_OK` only, never - `count > 0` (a headless CI build agent may legitimately report zero audio devices). - -**Still explicitly out of scope** (carried forward, not silently dropped): -- Real `webrtc-audio-processing`/AEC — no working Windows/MSVC build upstream (see - docs/roadmap.md §2's superseding decision-log entry). There is **no AEC, NS, or AGC - implementation at all**, not just a deferred VAD. The per-stream NS toggle - (`vc_set_remote_stream(..., noise_reduction)`) is unaffected by this pass and stays exactly - as inert as it was after M3 (`ApmPassthrough`, no PCM modification) — don't mistake this - pass for having fixed it. -- macOS/iOS `SCREEN_AUDIO` capture (ScreenCaptureKit / ReplayKit) — Windows-only loopback in - this pass. -- Process-specific WASAPI loopback — whole-device capture only; inherently captures this - app's own incoming voice mix along with everything else playing. -- The pre-existing RT-thread rule violation in `on_capture_frame`/`AudioEngine::on_capture` - (mutex lock, heap allocation for the stereo-upmix path, blocking `sendto`, all on the - miniaudio real-time callback thread) — predates this work (was already present in M2/M3); - documented here explicitly rather than silently carried forward again. Fixing it properly - needs the lock-free ring-buffer hand-off `docs/architecture.md §3` specifies — a separate, - larger refactor, out of scope for this pass. +**Known deferred (still open):** AEC/NS/AGC (no working Windows/MSVC WebRTC APM build); +process-specific WASAPI loopback; RT-thread rule violation in `on_capture_frame` (mutex lock +on audio callback thread — pre-existing, needs lock-free ring-buffer refactor). --- -## M4 — Windows WinForms C# client ✓ (completed 2026-06-17) +## M4 — Native clients ✓ (completed 2026-06-17 – 2026-06-19) -**Exit criterion:** ✓ `ctest --preset m1-dev` — **14/14 tests** green (existing 12 + 2 new -C++ tests: `test_channel_user_list_abi`, `test_tofu_flow`). `dotnet build` — 0 warnings/errors. -Manually verified: connect, TOFU first-connect dialog, channel tree, join, voice, text, device -pickers, level meter. Accessibility: explicit `AccessibleName`/`AccessibleDescription` on every -control, `&` mnemonics on every button, activity-log `ListBox` as screen-reader record. +**Exit criterion:** `ctest --preset dev` 21/21 green; `dotnet build` 0 warnings; `xcodebuild` +BUILD SUCCEEDED (macOS + iOS); manually verified: connect, TOFU, channel tree, join, voice, +text, device pickers, level meter on each platform. -**New C++ ABI surface** (all additive, backward-compatible): -- `vc_list_channels` / `vc_list_users` / `vc_list_user_streams` — pull-based snapshot getters - for the channel tree + user list; `session_model_mu_` added for cross-thread safety. - `SessionModel::apply_snapshot`/`apply_channel_event` fixed to populate `parent_id`, - `password_protected`, `max_users` (were permanently zeroed despite the struct declaring them). -- `vc_join_channel(channel_id, password)` — join with optional password; server replies via - new `VC_EVENT_JOIN_RESULT`. -- `VC_EVENT_SERVER_IDENTITY` + `vc_confirm_server_identity(accept)` — TOFU gate that blocks - `io_thread_` until the UI approves or rejects. Pins the TLS leaf-cert SHA-256 fingerprint - (verifiable directly at handshake), **not** the declared Ed25519 value (see docs/security.md - §1.1 for why — the TLS cert and Ed25519 key are generated independently, no binding). - `vc_get_server_identity_display` exposes the Ed25519 fingerprint for human-readable display. -- `vc_config::tofu_store_path` — optional per-user pin file path; defaults to a relative - `"./voicecat_tofu_pins.txt"` so existing tests need no change. -- `TcpAcceptor` now dual-stacks (IPv6 + IPv4 fallback) so `localhost` → `::1` on Windows - connects correctly without forcing users to type `127.0.0.1`. -- `VC_INPUT_ALWAYS_ON = 2` in `vc_input_mode` — transmit unconditionally (no VAD gate). -- `vc_set_vad_threshold(float)` — live VAD threshold update; `EnergyVadProcessor` stores it - atomically so the RT capture path reads without a lock or allocation. +**New C ABI (additive):** `vc_list_channels`/`vc_list_users`/`vc_list_user_streams`, +`vc_join_channel`, `VC_EVENT_SERVER_IDENTITY` + `vc_confirm_server_identity`, +`vc_config::tofu_store_path`, `VC_INPUT_ALWAYS_ON`, `vc_set_vad_threshold`, +`vc_audio_suspend`/`vc_audio_resume`, `vc_audio_restart`, `vc_set_capture_channels`. -**New C++ tests:** -- `test_channel_user_list_abi` — snapshot getters, `parent_id`/`password_protected`/`max_users` - regression, per-user stream list, invalid-user-id error, double-free idempotency. -- `test_tofu_flow` — first-connect blocks until confirmed; reject doesn't persist; reconnect to - same identity reports `MATCHED`; rotated identity reports `MISMATCH`; `vc_confirm_*` with - nothing pending returns an error. +**Windows** (`clients/windows/`): `VoiceCat.Interop` (P/Invoke, `[UnmanagedCallersOnly]`), +`VoiceCat.App` (ConnectDialog, ServerIdentityDialog, MainForm with full M5 moderation UI, +PerUserTuningDialog, PttKeyCaptureDialog), `VoiceCat.Interop.Tests`. PTT is focus-scoped. -**`windows-client` CMake preset** — Release, `VOICECAT_BUILD_SHARED=ON`, static MinGW runtime -(`-static-libgcc -static-libstdc++ -static -lwinpthread`), no tools/tests. Outputs -`build/windows-client/bin/voicecat.dll` with zero MinGW DLL dependencies (only Windows system -DLLs remain — verified via `objdump -p`). +**macOS** (`clients/apple/macOS/VoiceCatMac.xcodeproj`): NSOutlineView channel tree, +NSTableView user list, NSTextView chat, voice controls, full VoiceOver accessibility, admin +menu, 17 Swift source files. `build-xcframework.sh` produces `VoiceCatCore.xcframework`. -**C# solution** (`clients/windows/`, .NET 10 LTS `net10.0-windows`): -- `VoiceCat.Interop` — `[LibraryImport]` P/Invoke surface, `[UnmanagedCallersOnly]` callbacks, - `System.Threading.Channels.Channel` event delivery drained by 30ms WinForms - Timer; `VoiceCatClientHandle : SafeHandle` guarantees `vc_client_destroy`. -- `VoiceCat.App` — WinForms UI: - - `ConnectDialog` — saved-server `ListBox`, Add/Remove/Edit; servers persisted to - `%AppData%\VoiceCat\servers.json`; passwords DPAPI-encrypted (`ProtectedData`, opt-in). - - `ServerIdentityDialog` — shown only on `FIRST_CONNECT`/`MISMATCH` (never `MATCHED`); - mismatch text and button ordering are starkly different ("WARNING" framing, Cancel default). - - `MainForm` — `TreeView` channel tree, `ListBox` user list, `RichTextBox` chat, scope - `ComboBox` (Channel/Private), activity-log `ListBox`, voice panel with mic toggle, - mute/deafen checkboxes, VAD/PTT/Always-On radio group, VAD sensitivity `TrackBar` - (1–100, hidden for non-VAD modes), device `ComboBox` + refresh, level `ProgressBar`. - - `PerUserTuningDialog` — real-time gain `TrackBar` + mute/NR checkboxes; applied to all - of a user's streams immediately (no OK/Cancel round-trip). - - `PttKeyCaptureDialog` — focus-scoped PTT key capture. -- `VoiceCat.Interop.Tests` — xunit smoke test: connect → TOFU → guest auth → list channels - purely via P/Invoke against a live `voicecat-server.exe`. - -**Explicitly out of scope for this pass:** -- macOS/iOS Swift client — pending. -- Admin/moderation UI (kick/ban/permissions/account provisioning) — server-side dispatch for - these messages is M5's job; building the UI now would require building the server side too. -- PTT hotkey is **focus-scoped only** (works while VoiceCat window has focus). A system-wide - `WH_KEYBOARD_LL` hook would require escalated permissions and risk AV flagging — documented - limitation, not silently omitted. -- Receive-side noise reduction (`vc_set_remote_stream(..., noise_reduction)`) is end-to-end - plumbed but behaviorally a passthrough no-op (`ApmPassthrough`, no PCM modification) — - same as before M4. The per-user NR checkbox in `PerUserTuningDialog` is labeled accordingly. +**iOS** (`clients/apple/iOS/VoiceCatiOS.xcodeproj`): SwiftUI, `NavigationSplitView`/`TabView`, +`OutlineGroup` channel tree, `IOSAudioRouter` AVAudioSession driver, 24 Swift source files, +iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sharing. --- @@ -1343,40 +381,26 @@ DLLs remain — verified via `objdump -p`). `test_m5_kick_ban_move_mute`, `test_m5_admin_accounts`, `test_m5_channel_crud`); `vccli` can drive all moderation/admin/channel operations against a live server. -- [x] **Server-side moderation & permissions:** - - `server/src/session_registry.h/.cpp` — per-session `Permissions`, permission helpers - (`can_kick`, `can_ban`, etc.), kick/ban/move/server-mute, channel CRUD, DB-backed channel - tree load/save, in-memory channel state. - - `server/src/conn_session.cpp` — M5 dispatch handlers, permission checks, channel-password - + `max_users` enforcement, `UserEvent::UPDATED` broadcast on join/leave. - - `core/proto/voicecat.proto` — `ServerMuteRequest`, `UserEvent.reason`, `User.server_deafened`, - `ListAccountsResult`, `AccountEntry`. -- [x] **C ABI / client-side:** - - `core/include/voicecat.h` — `vc_permissions`, `vc_channel_info`, `vc_kick_user`, - `vc_ban_user`, `vc_set_permission`, `vc_set_server_mute`, `vc_move_user`, - `vc_create_channel`, `vc_edit_channel`, `vc_delete_channel`, `vc_create_account`, - `vc_reset_password`, `vc_delete_account`, `vc_list_accounts`, `vc_get_permissions`; - new events `VC_EVENT_GENERIC_RESULT` and `VC_EVENT_ACCOUNT_LIST`. - - `core/src/voicecat.cpp`, `core/src/core/client.h/.cpp` — implementations + server-mute/deafen - gating on the client. -- [x] **Database:** `server/src/db.h/.cpp` schema v2 (`channels`, `bans`), Argon2id accounts, - BLAKE2b channel passwords, migrations. -- [x] **Tests:** four new M5 tests registered in `tests/CMakeLists.txt`: - - `test_m5_permissions` — grant/revoke permissions, verify enforcement. - - `test_m5_kick_ban_move_mute` — kick, ban, move, server-mute/deafen. - - `test_m5_admin_accounts` — create/reset/delete/list accounts. - - `test_m5_channel_crud` — create/edit/delete channels, password + max_users enforcement. -- [x] **vccli** (`tools/vccli/src/main.cpp`) — all M5 operations exposed via flags; account auth - via `--username`/`--password`; async `VC_EVENT_GENERIC_RESULT`/`VC_EVENT_ACCOUNT_LIST` handling. -- [x] **Docs** kept in sync: `docs/protocol.md`, `docs/security.md`, `PROGRESS.md`. - -**Key bug fixed:** `test_m5_channel_crud` failed because `SessionRegistry::create_channel` -broadcast `ChannelEvent::CREATED` from a moved-from `entry.proto` after -`channels_[id] = std::move(entry)`. Fixed by building the event before moving into the map. - - **Still to do:** - - DRED/audio-quality polish. - - macOS/iOS Swift client (carried from M4). +- [x] **Server-side moderation & permissions** — per-session `Permissions`, kick/ban/move/ + server-mute, channel CRUD, DB schema v2 (`channels`, `bans`), BLAKE2b channel passwords. +- [x] **C ABI** — `vc_kick_user`, `vc_ban_user`, `vc_set_permission`, `vc_set_server_mute`, + `vc_move_user`, `vc_create_channel`, `vc_edit_channel`, `vc_delete_channel`, + `vc_create_account`, `vc_reset_password`, `vc_delete_account`, `vc_list_accounts`, + `vc_get_permissions`; events `VC_EVENT_GENERIC_RESULT`, `VC_EVENT_ACCOUNT_LIST`. +- [x] **Four M5 tests** passing — `ctest --preset dev` 21/21. +- [x] **vccli** M5 flags: `--kick`, `--ban`, `--move`, `--server-mute`/`-unmute`/`-deafen`/ + `-undeafen`, `--set-permission`, channel CRUD, account CRUD, `--username`/`--password`. +- [x] **All three client UIs** (Windows WinForms, macOS AppKit, iOS SwiftUI) expose the full + M5 moderation and admin surface. +- [x] **Docs** — `docs/protocol.md`, `docs/security.md` kept in sync. +- [ ] **DRED/audio-quality polish** — not started. +- [ ] **macOS ScreenCaptureKit screen-audio** — `startStream(.screenAudio)` in macOS client + announces the stream but `start_loopback_capture()` returns false (no `VOICECAT_HAS_LOOPBACK` + on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships. +- [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target, + App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`. +- [ ] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — see full + plan in "Where we left off" above. ---