Files
voice-cat/PROGRESS.md

683 lines
49 KiB
Markdown
Raw Normal View History

# PROGRESS — VoiceCat
Living status. **Update this file in the same commit as your work** so the next agent picks
up instantly. Newest status at the top.
- **Date convention:** ISO (YYYY-MM-DD).
- Statuses: `[ ]` not started · `[~]` in progress · `[x]` done.
---
## ▶ Where we left off / next action
- **Awaiting on-device verification (2026-06-22):** **iOS real echo cancellation / noise suppression
via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS
AEC/NS/AGC come only from Apple's Voice-Processing I/O unit (VPIO), but the core uses miniaudio's
plain RemoteIO units — so `.voiceChat` mode alone never engaged AEC. Fix moves both mic capture and
playback to a native Swift `AVAudioEngine` (`setVoiceProcessingEnabled`) on the AEC presets, with the
core in external mode.
- **Core (done, builds + tests green):** new ABI `vc_set_mixed_output_sink` + `vc_set_external_playback`
(voicecat.h PATCH→2). `AudioEngine` gains a mixer-timer thread that drives `on_playback` (decode+mix)
on a ~20 ms cadence with NO hardware playback device and ships the final mix to the mixed-output
sink; `start()` also skips the hardware capture device when the MIC stream is `external_feed`
(`AudioParams.external_capture`). New white-box test `test_external_playback` (23/24;
pre-existing `external_pcm` teardown crash on Darwin 25.5 is UNRELATED — original tree crashes too).
- **Swift (done, builds):** `VoiceCatCore` wrappers (`externalFeed` on `StreamDescriptor`,
`setMixedOutputSink`, `setExternalPlayback`); new `IOSVoiceProcessingEngine.swift` (VPIO
`AVAudioEngine`: mic tap→`feedPcm`, mixed-sink lock-free ring→`AVAudioSourceNode`);
`IOSAudioRouter.currentConfigUsesVoiceProcessing` gates the path per preset; `SessionState`
join/leave + `reconcileVoicePath()` switch between VPIO and the miniaudio path; Voice Chat defaults
to speaker; SettingsView shows AEC/NS state. **Rebuild the xcframework** before building the app:
`clients/apple/scripts/build-xcframework.sh --all` (new ABI symbols). `xcodebuild` iOS sim Debug
BUILD SUCCEEDED.
- **Post-verification fixes (2026-06-22, Swift-only — no core/ABI change):** two on-device bugs fixed.
- **Voice Chat (VPIO) silent playback:** `SessionState.doStartMicStream()` called `audioRestart()`
BEFORE `startStream`, so when the engine was already running (a remote stream had started it) it
reopened with `external_capture=false` and opened a hardware miniaudio capture device; the
announce-result restart then early-returned (engine already running) so that device was never
dropped and fought the `AVAudioEngine` VPIO unit, silencing playback. Fix: set
`setExternalPlayback` first, then `startStream` (which stores `external_feed` synchronously), THEN
`audioRestart()` — the core reopens in full external mode (no hardware devices). Added VPIO
diagnostics (graph/route formats at start; ring written/read totals at teardown).
- **Stereo Mic / Studio quiet earpiece:** the `.builtInMicBtA2dp` presets omit `.defaultToSpeaker`
(it breaks A2DP) and skip `forceSpeaker`, so with no Bluetooth connected output pinned to the quiet
receiver. New `IOSAudioRouter.applyA2dpSpeakerFallback()` overrides to the built-in speaker when no
external (A2DP/wired/AirPlay) output is present, clears the override when one is — called after
activation and on device-change route changes (`AudioSessionManager`).
- **Next (user, on device):** two iPhones on speaker, Voice Chat preset → confirm (a) no echo, (b)
background noise suppressed, (c) speaker output by default AND remote audio is now audible; then
Stereo Mic / Studio with no BT → confirm loud speaker (not earpiece), and A2DP takes over when a BT
headset connects. Tune the mixer-timer/ring sizing if there's under/overrun.
- **Done (2026-06-21):** **Docker + Linux deployment + GitHub Actions cross-build.** Added the complete Linux server
deployment story (the only missing platform — Windows and macOS already have native
binaries):
- `Dockerfile` — multi-stage (builder: `ubuntu:24.04` + vcpkg + `cmake --preset
server-release`; runtime: `ubuntu:24.04`, non-root `voicecat` user, `/data` volume,
TCP+UDP 8384). vcpkg is fetched via the GitHub archive tarball at the exact
`builtin-baseline` commit (`d46283cf…`), avoiding a full git-history clone. BuildKit
cache mounts on `/vcpkg/downloads`, `/vcpkg/buildtrees`, `/vcpkg/packages` (scoped by
`TARGETARCH`) keep rebuilds fast. Both `voicecat-server` and `voicecat-admin` are
copied into the runtime image.
- `docker-compose.yml` — single-service compose file with `restart: unless-stopped`,
named volume `voicecat-data`, and port mappings for TCP+UDP 8384. `command:` shows
how to set `--name`.
- `.dockerignore` — excludes `.git/`, `build/`, `clients/` (Swift/C# code), `docs/`,
markdown, editor config; build context is just `core/`, `server/`, `tools/`, `cmake/`,
and the three root CMake/vcpkg files.
- `deploy/linux/voicecat.service` — hardened systemd unit (non-root, `ProtectSystem`,
`NoNewPrivileges`, `AmbientCapabilities=CAP_NET_BIND_SERVICE`) for bare-metal deploys.
- Multi-arch: `docker buildx build --platform linux/amd64,linux/arm64 .` works without
any triplet override — `cmake/voicecat-toolchain.cmake` auto-detects from the host
arch cmake sees inside the buildx container.
- Quick start: `docker compose up -d` (or `docker run -d -p 8384:8384/tcp -p
8384:8384/udp -v voicecat-data:/data voicecat`). First run auto-generates identity
+ cert + DB; check logs for fingerprint + admin password.
- **GitHub Actions** (`.github/workflows/build-linux.yml`): primary cross-platform
binary build path — amd64 uses `ubuntu-24.04`, arm64 uses `ubuntu-24.04-arm`
(native, not QEMU). Triggers on push to main (when C++/cmake files change) and
manually via `workflow_dispatch`. Downloads land as 90-day artifacts.
`scripts/build-linux-binaries.sh` is the local Docker fallback (needs ~1015 GB
free disk; suits Linux dev machines, not Windows Docker Desktop).
- **Done (2026-06-21):** **Fix permanent voice-loss bug + harden the UDP media path (protocol v2).**
Field report: two iOS users lost all audio mid-call after a bad-network blip and could not
recover even by restarting the apps. Root causes found in the UDP media path:
1. **Anti-replay window poisoned by unauthenticated packets (the trigger).**
`SodiumMediaCrypto::open()` advanced `recv_highest_` from the plaintext header `seq`
*before* verifying the AEAD tag and never rolled it back on failure. One corrupted/forged
frame (a bit-flip on flaky wifi) shoved the high-water mark far ahead, after which every
legitimate frame was rejected as "too old" — permanently. Fixed by reordering to
replay-check → authenticate → update (RFC 3711 §3.3): the window is now touched only after
a successful tag check. Regression test in `test_media_aead.cpp`
(`test_corrupted_seq_does_not_poison_window`) — fails on the old code, passes now.
2. **16-bit seq wrap with no rollover counter.** The wire header carried only the low 16 bits
of the nonce counter (zero-extended on receive); after 65,536 frames the reconstructed
nonce diverged and all frames failed auth. **Wire format widened to a full u64 seq**
(`voice_frame.h`: header 14 → 20 bytes, `seq` u16 → u64; `crypto.cpp`, `client.cpp`,
`media_relay.cpp` updated; `JitterBuffer::Frame::seq` widened). This is a **versioned wire
change → `VOICECAT_PROTOCOL_VERSION` 1 → 2**; the `Hello` handshake rejects on mismatch
(`conn_session.cpp`). The voice frame is parsed only in `core/`+`server/`+`tests/`, so the
Swift/C# clients need only a rebuild — no parser changes.
3. **Server leaked UDP state on disconnect.** `SessionRegistry::unregister_session()` now also
frees `udp_endpoints_`/`udp_tokens_`/`ssrc_to_session_` (scan-and-erase by session id).
4. **Diagnostics.** `MediaRelay` now emits rate-limited dropped-frame counters
(unmapped-endpoint / no-recv-crypto / open-failed) so a wedged media path is observable.
- **Verified:** `cmake --build --preset dev` clean; `ctest --preset dev -E external_pcm`
**22/22 pass** (incl. `m2_voice` e2e relay + the two new AEAD regressions). `external_pcm`
still aborts on the **pre-existing** CoreAudio shutdown mutex race (confirmed identical on a
clean baseline checkout under the same harness — unrelated to these changes). Docs updated:
`voice.md` §2 (header), `protocol.md` (v2 + negotiation), `security.md` (authenticate-then-advance).
- **Done (2026-06-21):** **Expose all channel codec params + guest nickname in every client.**
- **DRED everywhere + ABI fix.** `dred` (Opus 1.6 Deep REDundancy) existed in the C ABI
(`vc_audio_config.dred`) and proto but was absent from *both* client marshaling layers — a
latent ABI mismatch: Swift `AudioConfig` and the C# `VcAudioConfigNative` blittable struct
were each one `int` short of the native struct passed to `vc_create_channel`/`vc_edit_channel`.
Added `dred` through Swift (`Models.swift`, `Marshaling.swift`, `VoiceCatClient.toNative`) and
C# (`Structs.cs`, `Models.cs`, `Marshaling.cs`, `VoiceCatClient.cs`).
- **Windows:** added the one missing DRED checkbox to `ChannelEditDialog` (all other params
were already present).
- **macOS:** `ChannelEditSheet` now exposes the previously-hidden params — application profile,
sample rate, expected packet loss, complexity, and DRED (was only stereo/bitrate/frame/FEC/DTX).
- **iOS:** `ChannelEditView` was name+topic only; rebuilt into a full create **and edit** form
(General: name/topic/parent/password/max-users/sort-order; Audio: stereo/bitrate/sample-rate/
frame/application/packet-loss/complexity/FEC/DTX/DRED). Added `SessionState.editChannel` and an
"Edit" swipe action (admins) in `ChannelTreeView` + `ChannelBrowserView` (iOS previously had no
edit-channel UI at all). Note: the channel list doesn't carry the current audio config, so on
edit the audio fields start from codec defaults — same limitation as macOS/Windows.
- **Guest nickname.** Guests could not set a display name on iOS *or* macOS (the field was
absent/disabled; only Windows had it). Added a dedicated `nickname` to `SavedServer` on both
(backward-compatible Codable), a Nickname field shown in Guest mode (`AddServerView` /
`AddServerSheet`), and wired the guest auth path to use it (`AppState`, `ConnectWindowController`).
- **Verified:** `xcodebuild` Debug — macOS BUILD SUCCEEDED; iOS (sim, `ARCHS=arm64`) BUILD
SUCCEEDED. Core `ctest --preset dev` 22/23 (only `external_pcm` aborts on a pre-existing
shutdown mutex race; no C++ was changed). Windows C# not buildable on macOS — changes reviewed.
- **Done (2026-06-21):** **iOS iPhone-layout UX fixes.** (1) Channels are now a **drill-down**
on iPhone — new `ChannelBrowserView` (root list of top-level channels) → `ChannelDetailView`
(people in the channel + sub-channels + an explicit "Join Channel" button with password
prompt). The iPad 3-column `NavigationSplitView` is unchanged. (2) Extracted a self-contained
`UserRow` (context menu + sheets) from `UserListView` so admin actions are reused in the
drill-down. (3) Fixed the **off-screen chat compose box**: `MainView` now places
`VoiceControlsView` via `.safeAreaInset(edge: .bottom)` instead of a floating `.overlay`, so
it reserves layout space above the tab bar and cooperates with keyboard avoidance. (4)
**Collapsed Activity into Chat** like macOS/Windows: `ChatView` renders a merged, time-sorted
timeline of `messages` + `activityLog` (activity rows in gray); the separate Activity tab and
`ActivityLogView.swift` are removed. `xcodebuild` Debug for `generic/platform=iOS` BUILD
SUCCEEDED (sim slice still arm64-only → simulator run N/A). Next: on-device check of the
drill-down + compose box + unified timeline.
- **Done (2026-06-21):** **Screen-audio sharing on macOS + iOS.** macOS uses ScreenCaptureKit
(`ScreenAudioCapture.swift`) → `vc_stream_feed_pcm`; iOS uses a ReplayKit Broadcast Upload
Extension (`VoiceCatBroadcast`) that forwards captured `.audioApp` PCM through a shared App
Group SPSC ring (`BroadcastAudioRing.swift`) to the host's `BroadcastAudioPump`, which owns
the `SCREEN_AUDIO` stream and feeds it — single session, no creds on disk. No C++ changes
(the core was already ready). macOS `xcodebuild` Debug BUILD SUCCEEDED; iOS app + extension
build for device (the xcframework sim slice is arm64-only, so x86_64-simulator link is N/A).
Next: on-device end-to-end verification (two clients hear the shared audio; iOS broadcast
start/stop). NOTE: `ctest --preset dev` is 22/23 — `external_pcm` passes its assertions but
aborts at shutdown (`mutex lock failed`), a **pre-existing** teardown crash unrelated to this
change (no C++ was modified).
- **Done (2026-06-21):** **macOS per-app screen-audio selection.** Before sharing, a new
`ScreenSharePickerSheet` lets the user choose scope — share Everything / Only selected apps /
All except selected apps — plus a first-class **"Exclude screen reader (VoiceOver) audio"**
toggle. `ScreenAudioCapture` now takes a `ScreenAudioSelection` and builds the matching
`SCContentFilter` (`including:` / `excludingApplications:`); app list comes from
`SCShareableContent`. macOS `xcodebuild` Debug BUILD SUCCEEDED. iOS deliberately untouched —
ReplayKit only delivers the mixed system stream, so per-app/VoiceOver filtering is impossible
there (documented in voice.md §9). **Still to verify on-device:** which process actually
carries VoiceOver speech (VoiceOver app vs. `com.apple.speech.speechsynthesisd`) — the exclude
set covers both candidates in `ScreenAudioCapture.screenReaderBundleIDs`; confirm exclusion
actually silences it in a real share.
feat(macos): UI overhaul -- toolbar, unified log, PM windows, settings window, hotkeys Mirrors the Windows client's UI overhaul (commit 97fa659 + 540ec13) adapted to Mac-native conventions. The main window is now just toolbar + channels + users + chat; audio device settings moved to a modeless Settings window. - NSToolbar: Join Voice, Share Screen Audio, Mute, Deafen (SF Symbol toggle buttons) + Output Volume slider (NSSlider 0-100, default 80). Voice actions, mute/deafen, and output volume moved out of the bottom panel into the toolbar - Audio device settings (input mode, VAD sensitivity, PTT key, device picker, level meter) moved to a new SettingsWindowController -- a modeless window opened via the app menu's "Settings..." (Cmd+,) item. Source-of-truth for audio state lives in MainWindowController so voice start applies settings even before the window has been opened; SettingsWindowController reads from / writes back to those properties and applies changes live when voice is active. Level meter forwarded from handleLevel -> updateLevel(rms:) - Unified log: chat NSTextView + activity NSTableView collapsed into a single NSTextView -- activity events in secondaryLabelColor (gray), chat in default - Private messaging: scope dropdown removed; compose always sends to the current channel. Each PM conversation opens in its own modeless PrivateMessageWindowController. Incoming .textMessage with .private scope routed to the right window; outgoing PMs echoed by server arrive through the same path. "Send Private Message..." added to user context menu. - Messages menu: "New Private Message..." (Cmd+Shift+N) opens a UserPickerSheet listing all server users so you can PM anyone on the server - Channel tree now shows live user counts, e.g. "General (3)"; refreshChannelTree called on .userJoined/.userLeft (was missing) - Voice menu: Join Voice (Cmd+Shift+V), Share Screen Audio (Cmd+Shift+S), Mute (Cmd+Shift+M), Deafen (Cmd+Shift+D) -- NSMenuItem key equivalents with [.command, .shift] mask, dispatched by the responder chain - setOutputVolume(_:) wrapper added to VoiceCatClient.swift (was missing -- the C ABI + C# wrapper shipped in commit 97fa659 but the Swift wrapper was never added); wired end-to-end: toolbar slider -> client.setOutputVolume(gain) Part A -- fixed and verified the previously-uncompiled Swift from the external PCM feed/tap commit (615d2a8): - Rebuilt the macOS xcframework slice (regenerated the module map from current voicecat.h, exposing vc_pcm_sink_cb / vc_stream_feed_pcm / vc_set_pcm_sink) - Fixed feedPcm type bug: size_t imports as Int in Swift not UInt; the original UInt(samplesPerChannel) was wrong - Added VoiceCatPcmSinkCallback typealias -- a Swift-idiomatic public alias for the C vc_pcm_sink_cb so consumers (tests, the macOS app) can declare a sink callback without directly importing the VoiceCatC C module. Mirrors the C# VcPcmSinkCallback delegate - keyCodeName helper deduplicated (was in PttKeyCaptureSheet.swift + MainWindowController.swift -- now shared) Platform-specific adaptations (vs. Windows): NSToolbar instead of ToolStrip; global menu bar + NSMenuItem key equivalents (Cmd not Ctrl, responder-chain dispatched, no custom key monitor needed); PM windows as modeless NSWindows; picker as Mac sheet; gray = secondaryLabelColor; SF Symbols for toolbar icons. swift test 10/10 (4 ExternalPcmTests + 6 VoiceCatClientSmokeTests against a live server); xcodebuild Debug + Release BUILD SUCCEEDED with 0 Swift warnings.
2026-06-20 23:30:52 +02:00
- **Done (2026-06-20):** **macOS client UI overhaul** — mirrors the Windows client's UI
overhaul (commit 97fa659 + 540ec13), adapted to Mac-native conventions. Also fixed and
verified the previously-uncompiled Swift changes from the external PCM feed/tap commit
(615d2a8). The main window is now just toolbar + channels + users + chat; audio device
settings (input mode, VAD, PTT key, device picker, level meter) moved to a modeless
Settings window (⌘,). Details in M5 section below. `swift test` 10/10; `xcodebuild` Debug
+ Release BUILD SUCCEEDED with 0 Swift warnings.
Next: live manual verification (toolbar toggles, unified log colors, PM windows, channel
counts, volume slider, settings window); then iOS ReplayKit and macOS ScreenCaptureKit
consumers of `vc_stream_feed_pcm`.
fix(ios): stop miniaudio from clobbering AVAudioSession (stereo->A2DP output death) The real root cause of "selecting Stereo Mic kills headphone/A2DP output on Join Voice." Every prior fix worked on the Swift IOSAudioRouter under the false premise that "miniaudio does NOT touch AVAudioSession on iOS." It does: the core opened devices via ma_device_init(nullptr, ...), and with a NULL context miniaudio 0.11.25 runs an iOS "hack" that sets the session category by device type, then ma_context_init__coreaudio calls setCategory()+setActive() on every device open -- capture -> AVAudioSessionCategoryRecord with zero options. That wipes the .playAndRecord category, the mode, and .allowBluetoothA2DP / .mixWithOthers / .allowAirPlay that IOSAudioRouter had just configured, killing headphone/A2DP (and even wired) output. Stereo presets break worst because they rely on the A2DP output route the wipe removes. TeamTalk avoids this by opening RemoteIO/VPIO AudioUnits directly and leaving the session entirely to the app. Fix (core, cross-platform safe): AudioEngine now owns a ma_context built by make_context_config() with coreaudio.sessionCategory = ma_ios_session_category_none and noAudioSessionActivate/Deactivate = MA_TRUE, and routes all ma_device_init calls (playback, capture, loopback) plus enumerate_devices through it. miniaudio no longer touches AVAudioSession; IOSAudioRouter is the sole owner (the session is already activated on connect in AppState before any device opens). Context is lazily inited in start(), reused across restarts, uninited in ~AudioEngine. Adds TEMP AudioSessionManager.logSessionState() diagnostics (after activate, on route change, on .streamStarted) to verify on-device that the category stays PlayAndRecord+allowBluetoothA2DP instead of flipping to Record. Remove once confirmed. Windows: cmake --build --preset dev clean; ctest --preset dev 21/21. iOS build + on-device verification pending on Mac. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 02:34:07 +02:00
- **Awaiting on-device verification:** **iOS stereo mic kills headphone/A2DP output — REAL
root cause found & fixed** (2026-06-20, on Windows; verify on Mac). All prior "fixes" (the
2026-06-19 entries below) targeted the Swift `IOSAudioRouter` on the false premise that
"miniaudio does NOT touch AVAudioSession on iOS." **It does.** The core opened its miniaudio
devices with `ma_device_init(nullptr, ...)`; with a NULL context, miniaudio (0.11.25) runs an
iOS "hack" (`miniaudio.h` ~44057) that picks a session category by device type, then
`ma_context_init__coreaudio` (~36552) calls `setCategory()` + `setActive()` on **every device
open** — capture → `AVAudioSessionCategoryRecord` with **zero options**. That wiped the
`.playAndRecord` category, the mode, and `.allowBluetoothA2DP`/`.mixWithOthers`/`.allowAirPlay`
that `IOSAudioRouter` had just configured → headphone/A2DP (and even wired) output died. The
stereo presets broke worst because they depend on the A2DP output route the wipe removed.
TeamTalk never hits this: its SDK opens RemoteIO/VPIO AudioUnits directly and leaves the
session entirely to the app (`UtilSound.swift`); miniaudio insists on managing it.
- **Fix (core, cross-platform safe):** `AudioEngine` now owns a `ma_context` built by
`make_context_config()` with `coreaudio.sessionCategory = ma_ios_session_category_none` +
`noAudioSessionActivate`/`noAudioSessionDeactivate = MA_TRUE`, and passes it to **all**
`ma_device_init` calls (playback, capture, loopback) and to `enumerate_devices`'s context.
miniaudio now never touches AVAudioSession; the Swift `IOSAudioRouter` is the sole owner
(session is already activated on connect in `AppState.swift:authResult`, before any device
opens, so removing miniaudio's self-activation is safe). Context is lazily inited in
`start()`, reused across restarts, uninited in `~AudioEngine`.
Files: `core/src/audio/audio_engine.{h,cpp}`.
- **TEMP diagnostics (remove after verification):** `AudioSessionManager.logSessionState(_:)`
logs category/mode/options/route; called after `ensureSessionActive`, on every route change,
and on `.streamStarted` (right after the core opens its devices). On Mac, watch the log when
joining voice with the Stereo Mic preset: category must stay `…PlayAndRecord` with
`allowBluetoothA2DP` and the output route must remain the headphones/A2DP device — NOT flip
to `…Record`. If confirmed, delete the `logSessionState` calls + method and the prior
band-aid comments in `IOSAudioRouter`/`audio_engine.cpp` can be trimmed.
feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink) Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public, stereo-capable production API and adds a symmetric PCM tap on the receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, soundboards, and custom clients — all without a hardware audio device. Core C++: - voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef, vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias - audio_engine: stereo-aware inject_capture (channels param + ring reset on channel-count change); atomic pcm_sink_ fired per decoded frame in on_playback; RemoteStream carries user_id/stream_id for RT-safe sink metadata; init_recv_stream takes user_id+stream_id - client.cpp: stream_feed_pcm / set_pcm_sink implementations; sync_remote_streams passes user_id/stream_id to init_recv_stream - voicecat.cpp: trampolines + channels=1/2 validation Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip, stereo feed L≠R, sink metadata+disable). ctest 23/23. Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest smoke tests (ExternalPcmTests.swift). C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs (vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate, vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs). Docs: architecture.md §4 new subsection, voice.md §9 updated (macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-20 17:52:09 +02:00
- **Verified on Windows:** `cmake --build --preset dev` clean, `ctest --preset dev` 23/23
(22/22 prior + `test_external_pcm` new binary). iOS build & on-device run still to be done by the user on the Mac.
fix(ios): stop miniaudio from clobbering AVAudioSession (stereo->A2DP output death) The real root cause of "selecting Stereo Mic kills headphone/A2DP output on Join Voice." Every prior fix worked on the Swift IOSAudioRouter under the false premise that "miniaudio does NOT touch AVAudioSession on iOS." It does: the core opened devices via ma_device_init(nullptr, ...), and with a NULL context miniaudio 0.11.25 runs an iOS "hack" that sets the session category by device type, then ma_context_init__coreaudio calls setCategory()+setActive() on every device open -- capture -> AVAudioSessionCategoryRecord with zero options. That wipes the .playAndRecord category, the mode, and .allowBluetoothA2DP / .mixWithOthers / .allowAirPlay that IOSAudioRouter had just configured, killing headphone/A2DP (and even wired) output. Stereo presets break worst because they rely on the A2DP output route the wipe removes. TeamTalk avoids this by opening RemoteIO/VPIO AudioUnits directly and leaving the session entirely to the app. Fix (core, cross-platform safe): AudioEngine now owns a ma_context built by make_context_config() with coreaudio.sessionCategory = ma_ios_session_category_none and noAudioSessionActivate/Deactivate = MA_TRUE, and routes all ma_device_init calls (playback, capture, loopback) plus enumerate_devices through it. miniaudio no longer touches AVAudioSession; IOSAudioRouter is the sole owner (the session is already activated on connect in AppState before any device opens). Context is lazily inited in start(), reused across restarts, uninited in ~AudioEngine. Adds TEMP AudioSessionManager.logSessionState() diagnostics (after activate, on route change, on .streamStarted) to verify on-device that the category stays PlayAndRecord+allowBluetoothA2DP instead of flipping to Record. Remove once confirmed. Windows: cmake --build --preset dev clean; ctest --preset dev 21/21. iOS build + on-device verification pending on Mac. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 02:34:07 +02:00
feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink) Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public, stereo-capable production API and adds a symmetric PCM tap on the receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, soundboards, and custom clients — all without a hardware audio device. Core C++: - voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef, vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias - audio_engine: stereo-aware inject_capture (channels param + ring reset on channel-count change); atomic pcm_sink_ fired per decoded frame in on_playback; RemoteStream carries user_id/stream_id for RT-safe sink metadata; init_recv_stream takes user_id+stream_id - client.cpp: stream_feed_pcm / set_pcm_sink implementations; sync_remote_streams passes user_id/stream_id to init_recv_stream - voicecat.cpp: trampolines + channels=1/2 validation Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip, stereo feed L≠R, sink metadata+disable). ctest 23/23. Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest smoke tests (ExternalPcmTests.swift). C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs (vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate, vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs). Docs: architecture.md §4 new subsection, voice.md §9 updated (macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-20 17:52:09 +02:00
- **Done (2026-06-20):** **External PCM feed/tap API (`vc_stream_feed_pcm` +
`vc_set_pcm_sink`)** — see detail in M5 section below. `ctest --preset dev` 23/23 (was 22/22 + 1 new test binary with 3 sub-tests).
Next: iOS ReplayKit and macOS ScreenCaptureKit consumers of this API. A public,
documented API for driving audio streams with externally-provided PCM instead of (or in
addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our
roadmap — that the current "miniaudio owns the device" model can't serve:
1. **ReplayKit Broadcast Upload Extension (iOS `SCREEN_AUDIO`)** — the extension is a
*separate process* with a ~50 MB memory cap and can't link the full `AudioEngine`
(`ma_device`, capture/playback threads). It needs to feed `CMSampleBuffer` audio (system
app audio) into the encode path without any audio hardware. The current plan in
`docs/voice.md §9` says the extension links "a minimal slice of the core (Opus encode +
media send only)" — a public feed-PCM API *is* that minimal slice. The extension links
Opus + the feed entry point, no `ma_device` needed.
2. **ScreenCaptureKit (macOS `SCREEN_AUDIO`)**`SCStream` delivers `CMSampleBuffer` in a
callback; convert to int16 and feed. No need to route through miniaudio's device layer.
**This is how macOS screen-audio actually gets implemented** — today it does NOT work:
`VOICECAT_HAS_LOOPBACK` is Windows-only (`core/CMakeLists.txt:88-95`), so on macOS
`AudioEngine::start_loopback_capture()` hits the `#else` stub (`audio_engine.cpp:647-649`)
and returns `false`. The macOS client's "Share Screen Audio" button
(`MainWindowController.swift:800-816`) calls `startStream(.screenAudio)` which announces
the stream to peers but captures **zero audio** — peers hear silence. The button is left
in place (not touched per user request); it'll work once this API + a ScreenCaptureKit
tap ship on Mac.
3. **Bots** — music bot, TTS bot, radio relay, transcription bot. They create a
`SCREEN_AUDIO`/`AUX_DEVICE` stream and feed synthesized or decoded PCM via the feed API.
No audio hardware required — runs headless on a server. Today the only way to feed
external PCM is `vc_test_inject_capture` (TEST-ONLY, name signals "don't ship this") or
re-implementing Opus encode + AEAD + UDP framing yourself (~500 lines of duplicated
crypto/codec code per consumer).
4. **Custom clients / accessibility** — soundboard, DAW integration, TTS of incoming chat,
recording/transcription of remote audio. Need either feed (send) or tap (receive) or
both.
**What we already have (input half, gated as test-only):** `vc_test_inject_capture
(stream_id, pcm, samples)` (`voicecat.h`, `client.cpp:1452`) feeds raw int16 PCM into the
encode pipeline via `AudioEngine::inject_capture(kind, pcm, n)`. It works for any stream
kind, supports multiple concurrent injection taps (one ring buffer per local kind), and
goes through the full encode → AEAD → UDP path. The encode path already handles
`channels == 1 || 2` (proven by the WASAPI stereo loopback work, 2026-06-17 entry below).
The only problems: it's marked TEST-ONLY in the header, the name signals "don't use this in
production," and it hardcodes mono (no `channels` parameter).
**What's missing (output half):** today decoded remote audio is mixed and pushed to the
miniaudio playback device (`on_playback`). There's no way for an external consumer to
intercept the decoded PCM of a specific remote stream — it all goes to the hardware device.
A bot that wants to record, transcribe, or re-broadcast remote audio has no hook.
**Plan (API design — clean, append-only, no struct changes, ABI-stable):**
- **`vc_stream_feed_pcm`** — promote `vc_test_inject_capture` to a public, documented API
and add a `channels` parameter:
```c
/* External PCM feed — replaces the hardware capture device for this stream. Caller
provides interleaved int16 PCM at the stream's sample rate. The core frames it,
encodes (Opus), seals (AEAD), and sends (UDP). Works for any stream kind
(MIC/SCREEN_AUDIO/AUX_DEVICE). The stream must be started first (vc_stream_start);
this just replaces the capture source. channels = 1 (mono) or 2 (stereo interleaved).
Thread-safe; may be called from any thread including audio callbacks. */
vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id,
const int16_t* pcm, size_t samples_per_channel,
uint32_t channels);
```
- **`vc_set_pcm_sink`** — symmetric output side: receive decoded remote audio as int16 PCM
instead of (or in addition to) the hardware playback device:
```c
/* External PCM tap — receive decoded, mixed remote audio as int16 PCM. The callback
fires on the audio thread with the mixed output for a specific remote stream. Pass
cb=NULL to disable (default: disabled, hardware playback only). When enabled, PCM is
delivered to the sink AND the hardware device (dual output) so a bot can record
without disabling local monitoring. user_id+stream_id identify the source stream.
The callback MUST NOT block — copy what you need and return (same contract as
vc_callbacks.on_event). */
typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
const int16_t* pcm, size_t samples_per_channel,
uint32_t channels, uint32_t sample_rate);
vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
```
- **Core changes:**
- `core/include/voicecat.h` — add `vc_pcm_sink_cb` typedef + the two function
declarations (append-only, after `vc_test_inject_capture`). Full doc comments on both
(contract, thread-safety, lifetime, use cases).
- `core/src/voicecat.cpp` — thin C trampolines → `vc_client::stream_feed_pcm` /
`set_pcm_sink`.
- `core/src/core/client.{h,cpp}``stream_feed_pcm`: validates `stream_id`, looks up
the `LocalStream`'s kind, calls `audio_engine_.inject_capture(kind, pcm, n)` (existing
path) with the channel count forwarded. `set_pcm_sink`: stores the callback + user
pointer; `on_playback` (or a new fan-out in the mixer) invokes it per remote stream
alongside the existing hardware write. Keep `vc_test_inject_capture` as a deprecated
alias calling `stream_feed_pcm(..., channels=1)` for source compatibility.
- `core/src/audio/audio_engine.{h,cpp}``inject_capture` already exists per-kind; add
a `channels` parameter to the ring-buffer write path (or a parallel stereo-aware
variant). The encode path in `client.cpp::on_capture_frame` already handles
`channels==2` via the stereo encode branch — just plumb the value through. For the
sink: add a `pcm_sink_` member (callback + user); in `on_playback` after mixing, if the
sink is set, copy the mixed PCM for the current stream and invoke the callback. The
copy must stay off the RT-critical path — document the non-blocking contract.
- **Skeleton stub path:** update `client.cpp`'s `#else` (no-deps) stub section to add
`vc_stream_feed_pcm`/`vc_set_pcm_sink` returning `VC_ERR_NOT_IMPLEMENTED` — keeps the
skeleton preset green.
- **Swift `VoiceCatCore`:** add `feedPcm(streamId:pcm:samplesPerChannel:channels:)` and
`setPcmSink(_:user:)` (the Swift wrapper around `vc_pcm_sink_cb` — a
`@convention(c)` closure + `Unmanaged` context, mirroring `Callbacks.swift`). Wraps both
new ABI functions.
- **C# `VoiceCat.Interop`:** add `StreamFeedPcm(streamId, pcm, samples, channels)` (with
`int16[]` marshaling) and `SetPcmSink` (delegates via `[UnmanagedCallersOnly]` thunk,
mirroring the event-callback pattern). Wraps both new ABI functions.
- **Tests:**
- `tests/test_external_pcm.cpp` (new) — `test_feed_pcm_round_trip`: two clients, A feeds
a known mono sine wave via `vc_stream_feed_pcm` on a MIC stream, B receives via the
normal decode path and asserts energy matches. `test_feed_pcm_stereo`: same with
`channels=2`, assert L≠R end-to-end (mirrors the WASAPI loopback stereo test).
`test_pcm_sink`: B sets a `vc_pcm_sink_cb`, A feeds PCM, assert the sink callback
receives the decoded PCM with matching energy. All headless, no audio hardware.
- `clients/apple/Tests/VoiceCatCoreTests/` — Swift wrapper round-trip for `feedPcm`.
- `clients/windows/VoiceCat.Interop.Tests/` — C# wrapper round-trip.
- **Docs:**
- `docs/architecture.md §4` — new subsection on external PCM feed/tap: the contract
(caller provides interleaved int16 at the stream's sample rate; core frames/encodes/
seals/sends for feed; core decodes/mixes/delivers for sink; sink callback must not
block), the use cases (ReplayKit, ScreenCaptureKit, bots, custom clients), and the
relationship to `vc_test_inject_capture` (deprecated alias).
- `docs/voice.md §9` — update the iOS ReplayKit and macOS ScreenCaptureKit rows: both
now consume `vc_stream_feed_pcm` instead of a "minimal slice of the core." Update the
iOS detail bullets: the extension links Opus + `vc_stream_feed_pcm` (not a parallel
media stack). Add a macOS ScreenCaptureKit note: convert `CMSampleBuffer` → int16,
feed via `vc_stream_feed_pcm` — this is how macOS screen-audio actually ships.
- `docs/protocol.md` — no protocol changes (the feed/sink are client-local; the wire
format is identical whether PCM came from miniaudio or an external source). Note this
explicitly.
- `docs/roadmap.md` — add a milestone entry; update the iOS ReplayKit and macOS
ScreenCaptureKit pending items to reference `vc_stream_feed_pcm`.
- **Implementation order:**
1. C ABI + core (`voicecat.h`, `voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`) +
skeleton stub. Verify `ctest --preset dev` green.
2. `tests/test_external_pcm.cpp` — the three behavior tests. Verify green.
3. Swift `VoiceCatCore` wrapper + `VoiceCatCoreTests` round-trip.
4. C# `VoiceCat.Interop` wrapper + `VoiceCatClientSmokeTests` round-trip.
5. Docs (`architecture.md`, `voice.md`, `protocol.md`, `roadmap.md`, header comments).
6. **Then** ReplayKit (iOS) and ScreenCaptureKit (macOS) become ~100-line consumers of
this API instead of parallel media stacks.
- **Verification:** `ctest --preset dev` green (3 new tests); `swift test` green; `dotnet
test` green; `xcodebuild` (skeleton) green. The feed/sink tests are fully headless — no
audio hardware, no simulator, no device — so they run in CI on every platform.
- **Files to touch:**
- Core C++: `core/include/voicecat.h`, `core/src/voicecat.cpp`,
`core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`.
- Tests: `tests/test_external_pcm.cpp` (new), `tests/CMakeLists.txt`.
- Swift: `clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift`,
`clients/apple/Sources/VoiceCatCore/Callbacks.swift`,
`clients/apple/Tests/VoiceCatCoreTests/ExternalPcmTests.swift` (new).
- C#: `clients/windows/VoiceCat.Interop/VoiceCatClient.cs`,
`clients/windows/VoiceCat.Interop/NativeMethods.cs`,
`clients/windows/VoiceCat.Interop.Tests/ExternalPcmTests.cs` (new).
- Docs: `docs/architecture.md`, `docs/voice.md`, `docs/protocol.md`, `docs/roadmap.md`.
- **ABI stability:** append-only — two new functions + one new typedef, no existing
structs/enums changed. `vc_test_inject_capture` stays as a deprecated alias for source
compatibility. Treat as a deliberate, versioned ABI event per `docs/protocol.md §8`.
- **Relationship to the iOS audio routing plan:** orthogonal. The iOS routing layer controls
*which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan is about
*bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship; they don't
conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer controls the
*mic* path which still uses miniaudio's device.
---
## Recent completed work
feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink) Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public, stereo-capable production API and adds a symmetric PCM tap on the receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, soundboards, and custom clients — all without a hardware audio device. Core C++: - voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef, vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias - audio_engine: stereo-aware inject_capture (channels param + ring reset on channel-count change); atomic pcm_sink_ fired per decoded frame in on_playback; RemoteStream carries user_id/stream_id for RT-safe sink metadata; init_recv_stream takes user_id+stream_id - client.cpp: stream_feed_pcm / set_pcm_sink implementations; sync_remote_streams passes user_id/stream_id to init_recv_stream - voicecat.cpp: trampolines + channels=1/2 validation Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip, stereo feed L≠R, sink metadata+disable). ctest 23/23. Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest smoke tests (ExternalPcmTests.swift). C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs (vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate, vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs). Docs: architecture.md §4 new subsection, voice.md §9 updated (macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-20 17:52:09 +02:00
All items below are `[x]` done; `ctest --preset dev` 23/23 on Windows after all.
- **External PCM feed/tap API** (2026-06-20): `vc_stream_feed_pcm` + `vc_set_pcm_sink` shipped.
Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public, stereo-capable API.
Adds symmetric PCM sink on the playback thread. Swift wrapper (`feedPcm`/`setPcmSink` in
`VoiceCatClient.swift`, 4 XCTest smoke tests). C# wrapper (`StreamFeedPcm`/`SetPcmSink` in
`VoiceCatClient.cs` + `NativeMethods.cs`, 4 xUnit smoke tests in `ExternalPcmTests.cs`).
Three new headless C++ ctests. Docs: architecture.md §4 new subsection, voice.md §9 updated,
protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Files: `voicecat.h`,
`voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`, `tests/test_external_pcm.cpp`,
`tests/CMakeLists.txt`, Swift + C# wrappers.
- **iOS A2DP + stereo root cause fix** (2026-06-20): miniaudio's NULL-context `ma_device_init`
was calling `AVAudioSession setCategory(Record)` on every device open, wiping the session
config `IOSAudioRouter` had set. Fixed by sharing a `ma_context` with
`sessionCategory=none` + `noAudioSessionActivate/Deactivate=MA_TRUE` — miniaudio never
touches `AVAudioSession`; `IOSAudioRouter` is the sole owner. Files: `audio_engine.{h,cpp}`.
- **iOS audio routing overhaul** (2026-06-19): Full `IOSAudioRouter` singleton drives all
`AVAudioSession` config before miniaudio opens devices. Fixed stereo mic polar-pattern setup
(WWDC20 recipe: `setPreferredInput` + `setInputDataSource` + `.stereo` polar pattern + no
`setPreferredInputNumberOfChannels`). Added `vc_audio_restart` ABI (full stop+reinit for
close→reconfigure→reopen ordering). Added `vc_set_capture_channels` ABI (core stereo-mic
support). AVAudioSession activated proactively on `.authResult`, not lazily on
`.streamStarted`. Join/Leave Voice button added (parity with macOS). Channel-id sync fixed
(mic button was permanently dimmed). iOS deployment target raised to 18.0.
- **iOS SwiftUI client** (2026-06-19): `VoiceCatiOS.xcodeproj` at `clients/apple/iOS/`.
Full feature parity with macOS/Windows: saved server list (JSON + Keychain, App Group
`group.cat.voice.VoiceCat`), TOFU, connect flow, channel tree, user list with context menus,
chat, admin sheets, voice controls, settings. `xcodebuild` → BUILD SUCCEEDED.
- **macOS AppKit client** (2026-06-18): `VoiceCatMac.xcodeproj` at `clients/apple/macOS/`.
Fixed compile errors (`NSAccessibility` call-site arg order, `StreamSummary.id` vs
`.streamId`) and linker issues (`OTHER_LDFLAGS = -lc++`, `ONLY_ACTIVE_ARCH = YES` for
Release). Debug + Release both BUILD SUCCEEDED.
- **Swift `VoiceCatCore` package + XCFramework** (2026-06-18): Shared Swift wrapper at
`clients/apple/`. `build-xcframework.sh` merges `libvoicecat.a` + 107 vcpkg static deps into
a fat `.a` via `libtool -static`. 6/6 Swift tests green (real server, mirrors C# Interop
tests). Supports macOS-arm64 + iOS-arm64 + iOS-sim slices.
- **macOS port validated** (2026-06-18): 21/21 on macOS. Three cross-platform bugs fixed:
missing `<netdb.h>` in POSIX test branch; SIGPIPE kills (added `SIG_IGN`); use-after-free of
Asio kqueue reactor on server shutdown (fixed `TcpAcceptor` shutdown/connection-drain
sequence).
- **CMake preset cleanup** (2026-06-18): `m1-dev``dev`, `dev``skeleton`, `m2-dev` dropped.
New `release`, `server-release` (stripped), `apple-dev`/`apple-ios`/`apple-ios-sim`. Cross-
platform triplet auto-resolved by `cmake/voicecat-toolchain.cmake`.
- **Disconnect, keepalive & reaper** (2026-06-18): Client sends `Ping` every 15 s; server
reaper drops sessions after 45 s; UDP `KEEPALIVE` every 5 s keeps NAT alive. `vc_disconnect`
sends graceful `Disconnect` proto. Stale-user LEFT broadcast on drop. PLC capped at ~2 s.
Three new tests: `test_disconnect_left`, `test_plc_cap`, `test_reaper_timeout`.
- **Stereo screen-audio loopback** (2026-06-17): WASAPI loopback opens in channel's
stereo/mono mode (was hardcoded mono). Real stereo flows end-to-end through loopback → encode
→ decode → mixer. New `test_loopback_stereo_capture`.
- **Windows screen-audio UI wired** (2026-06-17): `btnScreenShareToggle` in `MainForm.cs`.
No core/proto/ABI changes — all the plumbing was already there. `dotnet test` 4/4 green.
- **Bug fixes** (2026-06-16 2026-06-17):
- *AEAD nonce desync in SFU relay* — relay forwarded sender's `seq` verbatim; recipient
nonce reconstruction used the wrong counter. Fixed by rewriting the outgoing `seq` field
to the recipient's `peek_send_counter()`.
- *Playout clock free-ran* — `playout_ts` advanced even during VAD/PTT silence gaps,
eventually dropping all frames as too-late. Fixed with resync in `on_playback` via
`JitterBuffer::peek_front_ts()`.
- *Stale users after disconnect* — `ConnSession::close()` didn't broadcast `UserEvent::LEFT`
before erasing. Fixed; PLC cap added as defense-in-depth.
- *"Randomly bumped to Lobby"* — server excluded the actor from its own state-change
broadcasts. Fixed: `UserEvent::UPDATED` now goes to all clients including the actor.
- *Silent playback after join* — `opus_decode` received hardware callback frame count as
`max_samples` instead of the Opus frame size. Fixed with a decode ring buffer.
---
## Milestones (see [docs/roadmap.md](docs/roadmap.md) for full detail)
- [x] **M0 — Scaffolding** ✓ complete
2026-06-15 23:48:44 +02:00
- [x] **M1 — Control plane** ✓ complete (2026-06-15)
- [x] **M2 — Voice, single stream** ✓ complete (2026-06-16)
feat(M3): multi-stream & per-channel tuning Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC + SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise- reduction, talk indicators, and enforced per-channel Opus configurability (mono/stereo, bitrate, frame size, FEC/DTX, application). Bugs fixed along the way (found while implementing, not pre-existing scope): - Server hard-coded stream_id=1 for every announce, so a second stream from the same user silently overwrote the first in SessionRegistry::set_user_stream. Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop validates against announced_stream_ids_ before clearing. - Client dropped mode/dtx/complexity/application from effective_audio even for the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever applied to OpusParams. Fixed on both the send (handle_stream_announce_result) and receive (sync_remote_streams) paths via a shared opus_params_from_audio_config() helper. - OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application and wired it through. - on_playback's per-stream decode passed the wrong frame_size to opus_decode (total samples instead of samples-per-channel), which would have overflowed the decode buffer for any stereo stream. - teardown_voice() raced when called concurrently from run_io()'s own cleanup and from disconnect() on a different thread -- both could see udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the same std::thread (intermittent std::system_error under ctest). Fixed with a teardown_mu_ guard instead of carrying the flake forward. New: - Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/ FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX); handle_stream_announce enforces the channel's config, clamping (not overriding) bitrate_bps to its ceiling. - core/src/core/client.h/.cpp: local-stream state is now a std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with request_id-correlated announce/result handling (request_id already round-tripped on the wire; just wasn't read before). on_capture_frame is kind-aware and upmixes mono capture to stereo when a stream's config calls for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired through set_remote_stream. New run_talk_timer() thread emits VC_EVENT_TALK_STATE from both remote and local edge detection. - core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps (inject_capture), stereo-to-mono downmix at the decode/mix boundary, RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled and last_voice_ms/talking; new set_stream_noise_reduction() and poll_talk_transitions(). - core/src/session/session.h/.cpp: Stream now carries the full AudioConfig, not just sample_rate/frame_ms. - New additive C ABI (core/include/voicecat.h): vc_audio_config + vc_get_stream_audio_config (effective Opus config for any stream you own or a peer's); vc_test_inject_capture (test-only synthetic PCM injection, clearly marked, mirrors AudioEngine::inject_capture). - tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two concurrent local streams, independent gain/mute/NS control, per-channel config divergence via vc_get_stream_audio_config, talk indicators. Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback capture for SCREEN_AUDIO (synthetic injection only); true stereo playback output (AudioEngine's mixer/output device stays mono -- Opus itself is fully stereo-correct on the wire). ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive full-suite runs plus 8 standalone runs of the new test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
- [x] **M3 — Multi-stream & per-channel tuning** ✓ complete (2026-06-16)
- [x] **M4 — Native clients** — Windows WinForms ✓ (2026-06-17); macOS AppKit ✓ (2026-06-18); iOS SwiftUI ✓ (2026-06-19)
- [~] **M5 — Moderation, polish, beyond** (perms, bans, DRED; then file transfer, E2EE, …)
---
## M0 — Scaffolding ✓
Repo layout (`core/ server/ tools/ clients/ tests/`), CMake + vcpkg manifest, C ABI header
(`voicecat.h`), proto source of truth, core stubs for all six subsystems, `voicecat-server` +
`vccli` skeletons, smoke CTest, `.clang-format`/`.gitattributes`/`.gitignore`.
---
2026-06-15 23:48:44 +02:00
## M1 — Control plane ✓ (completed 2026-06-15)
**Exit criterion:** `test_m1_integration` — two clients authenticate over TLS 1.3 (guest +
Argon2id), exchange channel + private text. ~1 s.
2026-06-15 23:48:44 +02:00
`FrameCodec`, `TlsContext` (mbedTLS 1.3, ECDSA-P256 self-signed, TOFU pins TLS leaf-cert
SHA-256), `WorkerPool`, `Database` (SQLite + Argon2id), `ServerIdentityManager`,
`ConnSession` state machine, `SessionRegistry`, `vc_client` full M1 C ABI, `voicecat-admin`
CLI, dual-stack `TcpAcceptor`. **Key bug fixed:** `send_frame` double-framing — `encode_envelope`
was pre-framing the protobuf; fixed by passing raw protobuf bytes.
---
## M2 — Voice, single stream ✓ (completed 2026-06-16)
**Exit criterion:** `test_m2_voice` + `test_voice_client_abi` — two headless clients auth, bind
UDP, 50 Opus frames relayed + re-encrypted by SFU, B receives ≥25 and decrypts. ~4 s.
14-byte UDP voice header, `SodiumMediaCrypto` (ChaCha20-Poly1305 + 64-bit anti-replay),
`OpusEncoder`/`OpusDecoder` (FEC, PLC), `UdpMediaChannel`, `JitterBuffer`, `AudioEngine`
(miniaudio), `MediaRelay` SFU. **Key bug fixed:** `on_playback` passed hardware callback frame
count as `opus_decode` max_samples; fixed with a per-stream decode ring buffer.
---
feat(M3): multi-stream & per-channel tuning Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC + SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise- reduction, talk indicators, and enforced per-channel Opus configurability (mono/stereo, bitrate, frame size, FEC/DTX, application). Bugs fixed along the way (found while implementing, not pre-existing scope): - Server hard-coded stream_id=1 for every announce, so a second stream from the same user silently overwrote the first in SessionRegistry::set_user_stream. Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop validates against announced_stream_ids_ before clearing. - Client dropped mode/dtx/complexity/application from effective_audio even for the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever applied to OpusParams. Fixed on both the send (handle_stream_announce_result) and receive (sync_remote_streams) paths via a shared opus_params_from_audio_config() helper. - OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application and wired it through. - on_playback's per-stream decode passed the wrong frame_size to opus_decode (total samples instead of samples-per-channel), which would have overflowed the decode buffer for any stereo stream. - teardown_voice() raced when called concurrently from run_io()'s own cleanup and from disconnect() on a different thread -- both could see udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the same std::thread (intermittent std::system_error under ctest). Fixed with a teardown_mu_ guard instead of carrying the flake forward. New: - Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/ FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX); handle_stream_announce enforces the channel's config, clamping (not overriding) bitrate_bps to its ceiling. - core/src/core/client.h/.cpp: local-stream state is now a std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with request_id-correlated announce/result handling (request_id already round-tripped on the wire; just wasn't read before). on_capture_frame is kind-aware and upmixes mono capture to stereo when a stream's config calls for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired through set_remote_stream. New run_talk_timer() thread emits VC_EVENT_TALK_STATE from both remote and local edge detection. - core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps (inject_capture), stereo-to-mono downmix at the decode/mix boundary, RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled and last_voice_ms/talking; new set_stream_noise_reduction() and poll_talk_transitions(). - core/src/session/session.h/.cpp: Stream now carries the full AudioConfig, not just sample_rate/frame_ms. - New additive C ABI (core/include/voicecat.h): vc_audio_config + vc_get_stream_audio_config (effective Opus config for any stream you own or a peer's); vc_test_inject_capture (test-only synthetic PCM injection, clearly marked, mirrors AudioEngine::inject_capture). - tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two concurrent local streams, independent gain/mute/NS control, per-channel config divergence via vc_get_stream_audio_config, talk indicators. Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback capture for SCREEN_AUDIO (synthetic injection only); true stereo playback output (AudioEngine's mixer/output device stays mono -- Opus itself is fully stereo-correct on the wire). ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive full-suite runs plus 8 standalone runs of the new test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
## M3 — Multi-stream & per-channel tuning ✓ (completed 2026-06-16)
**Exit criterion:** `test_m3_multistream` — client A runs two concurrent streams (MIC +
SCREEN_AUDIO); B sees both; per-stream gain/mute/NR independent; effective Opus config matches
channel's server-enforced settings. ~2.4 s.
feat(M3): multi-stream & per-channel tuning Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC + SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise- reduction, talk indicators, and enforced per-channel Opus configurability (mono/stereo, bitrate, frame size, FEC/DTX, application). Bugs fixed along the way (found while implementing, not pre-existing scope): - Server hard-coded stream_id=1 for every announce, so a second stream from the same user silently overwrote the first in SessionRegistry::set_user_stream. Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop validates against announced_stream_ids_ before clearing. - Client dropped mode/dtx/complexity/application from effective_audio even for the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever applied to OpusParams. Fixed on both the send (handle_stream_announce_result) and receive (sync_remote_streams) paths via a shared opus_params_from_audio_config() helper. - OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application and wired it through. - on_playback's per-stream decode passed the wrong frame_size to opus_decode (total samples instead of samples-per-channel), which would have overflowed the decode buffer for any stereo stream. - teardown_voice() raced when called concurrently from run_io()'s own cleanup and from disconnect() on a different thread -- both could see udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the same std::thread (intermittent std::system_error under ctest). Fixed with a teardown_mu_ guard instead of carrying the flake forward. New: - Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/ FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX); handle_stream_announce enforces the channel's config, clamping (not overriding) bitrate_bps to its ceiling. - core/src/core/client.h/.cpp: local-stream state is now a std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with request_id-correlated announce/result handling (request_id already round-tripped on the wire; just wasn't read before). on_capture_frame is kind-aware and upmixes mono capture to stereo when a stream's config calls for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired through set_remote_stream. New run_talk_timer() thread emits VC_EVENT_TALK_STATE from both remote and local edge detection. - core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps (inject_capture), stereo-to-mono downmix at the decode/mix boundary, RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled and last_voice_ms/talking; new set_stream_noise_reduction() and poll_talk_transitions(). - core/src/session/session.h/.cpp: Stream now carries the full AudioConfig, not just sample_rate/frame_ms. - New additive C ABI (core/include/voicecat.h): vc_audio_config + vc_get_stream_audio_config (effective Opus config for any stream you own or a peer's); vc_test_inject_capture (test-only synthetic PCM injection, clearly marked, mirrors AudioEngine::inject_capture). - tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two concurrent local streams, independent gain/mute/NS control, per-channel config divergence via vc_get_stream_audio_config, talk indicators. Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback capture for SCREEN_AUDIO (synthetic injection only); true stereo playback output (AudioEngine's mixer/output device stays mono -- Opus itself is fully stereo-correct on the wire). ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive full-suite runs plus 8 standalone runs of the new test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
Fixed server `stream_id` counter bug (always wrote `1`). Per-channel `AudioConfig` populated
(Lobby: mono/24kbps/VOIP + DTX; Music Room: stereo/128kbps/AUDIO). `LocalStream` map,
`pending_announce_kind_`, `run_talk_timer()`, thread-join race in `teardown_voice()` fixed.
New C ABI: `vc_get_stream_audio_config`, `vc_test_inject_capture`.
feat(M3): multi-stream & per-channel tuning Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC + SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise- reduction, talk indicators, and enforced per-channel Opus configurability (mono/stereo, bitrate, frame size, FEC/DTX, application). Bugs fixed along the way (found while implementing, not pre-existing scope): - Server hard-coded stream_id=1 for every announce, so a second stream from the same user silently overwrote the first in SessionRegistry::set_user_stream. Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop validates against announced_stream_ids_ before clearing. - Client dropped mode/dtx/complexity/application from effective_audio even for the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever applied to OpusParams. Fixed on both the send (handle_stream_announce_result) and receive (sync_remote_streams) paths via a shared opus_params_from_audio_config() helper. - OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application and wired it through. - on_playback's per-stream decode passed the wrong frame_size to opus_decode (total samples instead of samples-per-channel), which would have overflowed the decode buffer for any stereo stream. - teardown_voice() raced when called concurrently from run_io()'s own cleanup and from disconnect() on a different thread -- both could see udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the same std::thread (intermittent std::system_error under ctest). Fixed with a teardown_mu_ guard instead of carrying the flake forward. New: - Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/ FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX); handle_stream_announce enforces the channel's config, clamping (not overriding) bitrate_bps to its ceiling. - core/src/core/client.h/.cpp: local-stream state is now a std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with request_id-correlated announce/result handling (request_id already round-tripped on the wire; just wasn't read before). on_capture_frame is kind-aware and upmixes mono capture to stereo when a stream's config calls for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired through set_remote_stream. New run_talk_timer() thread emits VC_EVENT_TALK_STATE from both remote and local edge detection. - core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps (inject_capture), stereo-to-mono downmix at the decode/mix boundary, RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled and last_voice_ms/talking; new set_stream_noise_reduction() and poll_talk_transitions(). - core/src/session/session.h/.cpp: Stream now carries the full AudioConfig, not just sample_rate/frame_ms. - New additive C ABI (core/include/voicecat.h): vc_audio_config + vc_get_stream_audio_config (effective Opus config for any stream you own or a peer's); vc_test_inject_capture (test-only synthetic PCM injection, clearly marked, mirrors AudioEngine::inject_capture). - tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two concurrent local streams, independent gain/mute/NS control, per-channel config divergence via vc_get_stream_audio_config, talk indicators. Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback capture for SCREEN_AUDIO (synthetic injection only); true stereo playback output (AudioEngine's mixer/output device stays mono -- Opus itself is fully stereo-correct on the wire). ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive full-suite runs plus 8 standalone runs of the new test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
---
## Post-M3 follow-up ✓ (completed 2026-06-16)
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
- **Device enumeration** — `vc_list_devices`/`vc_set_input_device`; opaque hex device ids;
`vc_free_device_list` now frees. Works pre-connect.
- **VAD/PTT gate** — `EnergyVadProcessor` (RMS threshold ~0.025, 300 ms hang-time);
`vc_set_input_mode`/`vc_set_push_to_talk`; MIC-only (SCREEN_AUDIO/AUX_DEVICE bypass).
- **True stereo playback** — `playback_channels=2`; stereo decoded L→L R→R in mixer; mono
upmixed L=R; hardware fallback to mono on failure.
- **WASAPI loopback** — `loopback_device_` with `ma_device_type_loopback`;
`VOICECAT_HAS_LOOPBACK` macro (Windows-only). `vccli --share-screen-audio`.
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
**Known deferred (still open):** AEC/NS/AGC (no working Windows/MSVC WebRTC APM build);
process-specific WASAPI loopback; RT-thread rule violation in `on_capture_frame` (mutex lock
on audio callback thread — pre-existing, needs lock-free ring-buffer refactor).
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback Closes the three items PROGRESS.md's M3 section explicitly carried forward as out of scope: - Device enumeration (vc_list_devices) + input device selection (vc_set_input_device), backed by AudioEngine::enumerate_devices() via miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded ma_device_id strings. - VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk). webrtc-audio-processing (the originally-planned APM) has no working Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard abseil-cpp dependency), so VAD is a new lightweight, dependency-free energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it. - True stereo playback: AudioEngine's mixer and output device now carry stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded stereo streams to mono before mixing. - Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via miniaudio's loopback device type), replacing test-only injection as the production capture path. Also: vccli gains --list-devices, --input-device, --input-mode, and --share-screen-audio flags, plus a stdin command loop (ptt on/off, mode vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all four items (ABI-level + a white-box AudioEngine stereo-mix check). Docs updated to match: voice.md, roadmap.md (decision-log entry superseding the original webrtc-audio-processing choice), tech-stack.md, README.md, architecture.md, CLAUDE.md, PROGRESS.md. Still explicitly out of scope, documented not silently dropped: real webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS SCREEN_AUDIO capture, process-specific loopback, and a pre-existing RT-thread rule violation in the capture path that predates this work. Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive standalone runs; manually verified live (vccli --list-devices against real hardware, vccli --voice --input-mode vad streaming without incident). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
---
## M4 — Native clients ✓ (completed 2026-06-17 2026-06-19)
2026-06-17 00:52:02 +02:00
**Exit criterion:** `ctest --preset dev` 21/21 green; `dotnet build` 0 warnings; `xcodebuild`
BUILD SUCCEEDED (macOS + iOS); manually verified: connect, TOFU, channel tree, join, voice,
text, device pickers, level meter on each platform.
2026-06-17 00:52:02 +02:00
**New C ABI (additive):** `vc_list_channels`/`vc_list_users`/`vc_list_user_streams`,
`vc_join_channel`, `VC_EVENT_SERVER_IDENTITY` + `vc_confirm_server_identity`,
`vc_config::tofu_store_path`, `VC_INPUT_ALWAYS_ON`, `vc_set_vad_threshold`,
`vc_audio_suspend`/`vc_audio_resume`, `vc_audio_restart`, `vc_set_capture_channels`.
2026-06-17 00:52:02 +02:00
**Windows** (`clients/windows/`): `VoiceCat.Interop` (P/Invoke, `[UnmanagedCallersOnly]`),
`VoiceCat.App` (ConnectDialog, ServerIdentityDialog, MainForm with full M5 moderation UI,
PerUserTuningDialog, PttKeyCaptureDialog), `VoiceCat.Interop.Tests`. PTT is focus-scoped.
2026-06-17 00:52:02 +02:00
**macOS** (`clients/apple/macOS/VoiceCatMac.xcodeproj`): NSOutlineView channel tree,
NSTableView user list, NSTextView chat, voice controls, full VoiceOver accessibility, admin
menu, 17 Swift source files. `build-xcframework.sh` produces `VoiceCatCore.xcframework`.
2026-06-17 00:52:02 +02:00
**iOS** (`clients/apple/iOS/VoiceCatiOS.xcodeproj`): SwiftUI, `NavigationSplitView`/`TabView`,
`OutlineGroup` channel tree, `IOSAudioRouter` AVAudioSession driver, 24 Swift source files,
iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sharing.
2026-06-17 00:52:02 +02:00
---
## M5 — Moderation, polish, and beyond [~] (in progress 2026-06-17)
**Exit criterion:** four ABI-level tests green (`test_m5_permissions`,
`test_m5_kick_ban_move_mute`, `test_m5_admin_accounts`, `test_m5_channel_crud`);
`vccli` can drive all moderation/admin/channel operations against a live server.
- [x] **Server-side moderation & permissions** — per-session `Permissions`, kick/ban/move/
server-mute, channel CRUD, DB schema v2 (`channels`, `bans`), BLAKE2b channel passwords.
- [x] **C ABI**`vc_kick_user`, `vc_ban_user`, `vc_set_permission`, `vc_set_server_mute`,
`vc_move_user`, `vc_create_channel`, `vc_edit_channel`, `vc_delete_channel`,
`vc_create_account`, `vc_reset_password`, `vc_delete_account`, `vc_list_accounts`,
`vc_get_permissions`; events `VC_EVENT_GENERIC_RESULT`, `VC_EVENT_ACCOUNT_LIST`.
- [x] **Four M5 tests** passing — `ctest --preset dev` 21/21.
- [x] **vccli** M5 flags: `--kick`, `--ban`, `--move`, `--server-mute`/`-unmute`/`-deafen`/
`-undeafen`, `--set-permission`, channel CRUD, account CRUD, `--username`/`--password`.
- [x] **All three client UIs** (Windows WinForms, macOS AppKit, iOS SwiftUI) expose the full
M5 moderation and admin surface.
- [x] **Docs**`docs/protocol.md`, `docs/security.md` kept in sync.
- [x] **DRED/audio-quality polish** — done (2026-06-20). `bool dred` added to `AudioConfig`
proto (field 11) and `vc_audio_config` C ABI. Encoder: `OPUS_SET_DRED_DURATION(2)` when
enabled (20 ms of ML redundancy per packet). Decoder: `OpusDREDDecoder` + per-stream
`OpusDRED` scratch pre-allocated; `JitterBuffer::try_copy_front_payload` peeks at the next
buffered packet on every PLC step; if DRED data is present, `opus_decoder_dred_decode`
reconstructs the lost frame — otherwise falls back to standard PLC. New test:
`test_dred_toggle` (ctest 22/22). Files: `voicecat.proto`, `voicecat.h`,
`opus_codec.{h,cpp}`, `audio_engine.{h,cpp}`, `client.cpp`, `session.{h,cpp}`.
- [ ] **DRED toggle in client UIs** — expose the `dred` flag in all three channel-config UIs
so admins can enable it per channel. Windows: `ChannelEditForm` / `vc_channel_info.audio.dred`
checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs
already have full channel CRUD wired; this is an additive checkbox on the existing audio-config
section. (Core/protocol/ABI all done — this is UI-only work.)
- [x] **macOS ScreenCaptureKit screen-audio** — done 2026-06-21. `ScreenAudioCapture.swift`
drives an `SCStream` (audio-only, `excludesCurrentProcessAudio`), converts Float32 →
int16 in the channel's mono/stereo mode, and calls `vc_stream_feed_pcm`. Capture starts on
the self `.streamStarted` event (when the effective config is known); wired into
`MainWindowController.screenAudioClicked()`.
- [x] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — done 2026-06-21.
Forward-to-host design: the extension (`SampleHandler.swift`) captures `.audioApp`,
converts to 48 kHz int16 stereo, and writes a shared App Group SPSC ring
(`BroadcastAudioRing.swift`); the host's `BroadcastAudioPump` owns the `SCREEN_AUDIO`
stream and feeds via `vc_stream_feed_pcm` (single session, no creds on disk). UI is an
`RPSystemBroadcastPickerView` in `VoiceControlsView`. (Replaced the speculative
`BroadcastCredentials.swift` self-connecting design, now removed.)
feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink) Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public, stereo-capable production API and adds a symmetric PCM tap on the receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, soundboards, and custom clients — all without a hardware audio device. Core C++: - voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef, vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias - audio_engine: stereo-aware inject_capture (channels param + ring reset on channel-count change); atomic pcm_sink_ fired per decoded frame in on_playback; RemoteStream carries user_id/stream_id for RT-safe sink metadata; init_recv_stream takes user_id+stream_id - client.cpp: stream_feed_pcm / set_pcm_sink implementations; sync_remote_streams passes user_id/stream_id to init_recv_stream - voicecat.cpp: trampolines + channels=1/2 validation Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip, stereo feed L≠R, sink metadata+disable). ctest 23/23. Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest smoke tests (ExternalPcmTests.swift). C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs (vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate, vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs). Docs: architecture.md §4 new subsection, voice.md §9 updated (macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-20 17:52:09 +02:00
- [x] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — done
2026-06-20. Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public API with
stereo support. Adds a symmetric PCM sink fired on the playback thread per decoded remote
stream. Full wrappers for Swift (`feedPcm`/`setPcmSink`) and C# (`StreamFeedPcm`/
`SetPcmSink`). Three new C++ ctests (`test_feed_pcm_round_trip`, `test_feed_pcm_stereo`,
`test_pcm_sink`), 4 Swift XCTest smoke tests, 4 C# xUnit smoke tests. Docs updated
(architecture.md §4 new subsection, voice.md §9 updated, protocol.md §8 explicit
no-protocol-change note, roadmap.md M5 entry). `ctest --preset dev` 23/23.
feat(macos): UI overhaul -- toolbar, unified log, PM windows, settings window, hotkeys Mirrors the Windows client's UI overhaul (commit 97fa659 + 540ec13) adapted to Mac-native conventions. The main window is now just toolbar + channels + users + chat; audio device settings moved to a modeless Settings window. - NSToolbar: Join Voice, Share Screen Audio, Mute, Deafen (SF Symbol toggle buttons) + Output Volume slider (NSSlider 0-100, default 80). Voice actions, mute/deafen, and output volume moved out of the bottom panel into the toolbar - Audio device settings (input mode, VAD sensitivity, PTT key, device picker, level meter) moved to a new SettingsWindowController -- a modeless window opened via the app menu's "Settings..." (Cmd+,) item. Source-of-truth for audio state lives in MainWindowController so voice start applies settings even before the window has been opened; SettingsWindowController reads from / writes back to those properties and applies changes live when voice is active. Level meter forwarded from handleLevel -> updateLevel(rms:) - Unified log: chat NSTextView + activity NSTableView collapsed into a single NSTextView -- activity events in secondaryLabelColor (gray), chat in default - Private messaging: scope dropdown removed; compose always sends to the current channel. Each PM conversation opens in its own modeless PrivateMessageWindowController. Incoming .textMessage with .private scope routed to the right window; outgoing PMs echoed by server arrive through the same path. "Send Private Message..." added to user context menu. - Messages menu: "New Private Message..." (Cmd+Shift+N) opens a UserPickerSheet listing all server users so you can PM anyone on the server - Channel tree now shows live user counts, e.g. "General (3)"; refreshChannelTree called on .userJoined/.userLeft (was missing) - Voice menu: Join Voice (Cmd+Shift+V), Share Screen Audio (Cmd+Shift+S), Mute (Cmd+Shift+M), Deafen (Cmd+Shift+D) -- NSMenuItem key equivalents with [.command, .shift] mask, dispatched by the responder chain - setOutputVolume(_:) wrapper added to VoiceCatClient.swift (was missing -- the C ABI + C# wrapper shipped in commit 97fa659 but the Swift wrapper was never added); wired end-to-end: toolbar slider -> client.setOutputVolume(gain) Part A -- fixed and verified the previously-uncompiled Swift from the external PCM feed/tap commit (615d2a8): - Rebuilt the macOS xcframework slice (regenerated the module map from current voicecat.h, exposing vc_pcm_sink_cb / vc_stream_feed_pcm / vc_set_pcm_sink) - Fixed feedPcm type bug: size_t imports as Int in Swift not UInt; the original UInt(samplesPerChannel) was wrong - Added VoiceCatPcmSinkCallback typealias -- a Swift-idiomatic public alias for the C vc_pcm_sink_cb so consumers (tests, the macOS app) can declare a sink callback without directly importing the VoiceCatC C module. Mirrors the C# VcPcmSinkCallback delegate - keyCodeName helper deduplicated (was in PttKeyCaptureSheet.swift + MainWindowController.swift -- now shared) Platform-specific adaptations (vs. Windows): NSToolbar instead of ToolStrip; global menu bar + NSMenuItem key equivalents (Cmd not Ctrl, responder-chain dispatched, no custom key monitor needed); PM windows as modeless NSWindows; picker as Mac sheet; gray = secondaryLabelColor; SF Symbols for toolbar icons. swift test 10/10 (4 ExternalPcmTests + 6 VoiceCatClientSmokeTests against a live server); xcodebuild Debug + Release BUILD SUCCEEDED with 0 Swift warnings.
2026-06-20 23:30:52 +02:00
- [x] **macOS client UI overhaul** — done 2026-06-20. Mirrors the Windows client's UI
overhaul (toolbar, unified log, PM windows, channel counts, output volume, keyboard
shortcuts), adapted to Mac-native conventions:
- **NSToolbar**: Join Voice, Share Screen Audio, Mute, Deafen (SF Symbol toggle buttons),
and Output Volume slider (NSSlider 0100, default 80). Voice actions + mute/deafen +
output volume moved out of the bottom voice panel into the toolbar. Bottom panel keeps
input-mode segmented control / VAD slider / PTT key / device picker / level meter.
- **Unified log**: chat `NSTextView` + activity `NSTableView` collapsed into a single
`NSTextView` — activity events in `secondaryLabelColor` (gray), chat in default color.
Removed `activityTableView` and `activityLog` array.
- **Private messaging**: scope dropdown removed; compose bar always sends to the current
channel. Each PM conversation opens in its own modeless `PrivateMessageWindowController`
(NSWindow). Incoming `.textMessage` with `.private` scope routed to the right window;
outgoing PMs echoed by server arrive through the same path. "Send Private Message…"
added to user context menu. "New Private Message…" (⌘⇧N) opens `UserPickerSheet`
listing all server users.
- **Channel counts**: outline view renders `"Name (n)"` with live user counts;
`refreshChannelTree()` called on `.userJoined`/`.userLeft` (was missing).
- **Voice menu** (⌘⇧V join/leave, ⌘⇧S share screen, ⌘⇧M mute, ⌘⇧D deafen) and **Messages
menu** (⌘⇧N new PM) added to `NSApp.mainMenu` via `NSMenuItem` key equivalents with
`[.command, .shift]` mask. Removed on `windowWillClose`. Mac-native: ⌘ not Ctrl, dispatched
by the responder chain (no custom key monitor needed).
- **Output volume**: `setOutputVolume(_:)` wrapper added to `VoiceCatClient.swift` (was
missing — the C ABI + C# wrapper shipped in commit 97fa659 but the Swift wrapper was
never added). Wired end-to-end: toolbar slider → `client.setOutputVolume(gain)`.
- **Part A (uncompiled Swift fix)**: the external PCM feed/tap Swift wrapper (commit
615d2a8) was never compiled — the local xcframework predating the `voicecat.h` PCM
additions. Fixed: rebuilt xcframework (regenerated module map), fixed `UInt``Int` type
mismatch in `feedPcm` (Swift imports `size_t` as `Int` not `UInt`), added
`VoiceCatPcmSinkCallback` typealias (Swift-idiomatic alias for the C `vc_pcm_sink_cb`
so consumers don't need to directly import `VoiceCatC`). `swift test` 10/10 green.
- **Audio settings moved to Settings window**: the bottom voice panel (input mode, VAD
slider, PTT key, device picker, level meter) was removed from the main window and moved
into a new `SettingsWindowController` — a modeless window opened via the app menu's
"Settings…" (⌘,) item. The main window is now just toolbar + channels + users + chat.
Source-of-truth for audio settings (`selectedInputMode`, `vadThresholdValue`,
`selectedInputDeviceId`, `pttKeyCode`) lives in `MainWindowController` so voice start can
apply them even before the settings window has been opened; `SettingsWindowController`
reads from and writes back to those properties and applies changes to the client
immediately when voice is active. The level meter is forwarded from
`MainWindowController.handleLevel``settingsWindowController.updateLevel(rms:)`.
`keyCodeName` helper deduplicated (was duplicated in `PttKeyCaptureSheet.swift` +
`MainWindowController.swift` — now shared from `MainWindowController.swift`).
- Files: `MainWindowController.swift` (overhauled), `PrivateMessageWindowController.swift`
(new), `UserPickerSheet.swift` (new), `SettingsWindowController.swift` (new),
`VoiceCatClient.swift` (setOutputVolume + VoiceCatPcmSinkCallback typealias + feedPcm
type fix), `ExternalPcmTests.swift` (use typealias), `PttKeyCaptureSheet.swift` (removed
duplicate `keyCodeName`), `VoiceCatMac.xcodeproj/project.pbxproj` (register 3 new files).
- **Platform-specific adaptations** (vs. Windows): `NSToolbar` instead of `ToolStrip`;
global menu bar + `NSMenuItem` key equivalents (⌘ not Ctrl, responder-chain dispatched);
PM windows as modeless `NSWindow`s; picker as Mac sheet; gray = `secondaryLabelColor`;
SF Symbols for toolbar icons.
---
## Decisions log
All architecture/scope decisions are settled and recorded in
[docs/roadmap.md §2 "Resolved decisions"](docs/roadmap.md) and reflected across `docs/`.
If you make a *new* decision, record it there and link it here.
---
## How to update this file
1. Check off tasks as you complete them; flip a milestone to `[x]` only when its **exit
criterion test** passes.
2. Keep the **"Where we left off / next action"** block at the top accurate — it's the first
thing the next agent reads.
3. When you start a milestone, copy its task list from `docs/roadmap.md` into a section here.