Talon
fdcd8d1427
Adds Opus 1.6 DRED support end-to-end: encoder embeds 20 ms of ML redundancy in every packet when enabled; decoder recovers lost frames from the next buffered packet's DRED extension rather than falling back to PLC comfort noise. Protocol: bool dred = 11 added to AudioConfig (backward-compatible, defaults false). C ABI: int dred added to vc_audio_config. Encoder: OPUS_SET_DRED_DURATION(2) when dred=true. Decoder: OpusDREDDecoder + per-stream OpusDRED scratch pre-allocated off the RT thread; JitterBuffer::try_copy_front_payload peeks at the next packet without popping on every PLC step; opus_decoder_dred_decode reconstructs the lost frame if DRED data is present, otherwise falls back to PLC. New test: test_dred_toggle (22/22 ctest green). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
435 lines
28 KiB
Markdown
435 lines
28 KiB
Markdown
# PROGRESS — VoiceCat
|
||
|
||
Living status. **Update this file in the same commit as your work** so the next agent picks
|
||
up instantly. Newest status at the top.
|
||
|
||
- **Date convention:** ISO (YYYY-MM-DD).
|
||
- Statuses: `[ ]` not started · `[~]` in progress · `[x]` done.
|
||
|
||
---
|
||
|
||
## ▶ Where we left off / next action
|
||
|
||
- **Awaiting on-device verification:** **iOS stereo mic kills headphone/A2DP output — REAL
|
||
root cause found & fixed** (2026-06-20, on Windows; verify on Mac). All prior "fixes" (the
|
||
2026-06-19 entries below) targeted the Swift `IOSAudioRouter` on the false premise that
|
||
"miniaudio does NOT touch AVAudioSession on iOS." **It does.** The core opened its miniaudio
|
||
devices with `ma_device_init(nullptr, ...)`; with a NULL context, miniaudio (0.11.25) runs an
|
||
iOS "hack" (`miniaudio.h` ~44057) that picks a session category by device type, then
|
||
`ma_context_init__coreaudio` (~36552) calls `setCategory()` + `setActive()` on **every device
|
||
open** — capture → `AVAudioSessionCategoryRecord` with **zero options**. That wiped the
|
||
`.playAndRecord` category, the mode, and `.allowBluetoothA2DP`/`.mixWithOthers`/`.allowAirPlay`
|
||
that `IOSAudioRouter` had just configured → headphone/A2DP (and even wired) output died. The
|
||
stereo presets broke worst because they depend on the A2DP output route the wipe removed.
|
||
TeamTalk never hits this: its SDK opens RemoteIO/VPIO AudioUnits directly and leaves the
|
||
session entirely to the app (`UtilSound.swift`); miniaudio insists on managing it.
|
||
- **Fix (core, cross-platform safe):** `AudioEngine` now owns a `ma_context` built by
|
||
`make_context_config()` with `coreaudio.sessionCategory = ma_ios_session_category_none` +
|
||
`noAudioSessionActivate`/`noAudioSessionDeactivate = MA_TRUE`, and passes it to **all**
|
||
`ma_device_init` calls (playback, capture, loopback) and to `enumerate_devices`'s context.
|
||
miniaudio now never touches AVAudioSession; the Swift `IOSAudioRouter` is the sole owner
|
||
(session is already activated on connect in `AppState.swift:authResult`, before any device
|
||
opens, so removing miniaudio's self-activation is safe). Context is lazily inited in
|
||
`start()`, reused across restarts, uninited in `~AudioEngine`.
|
||
Files: `core/src/audio/audio_engine.{h,cpp}`.
|
||
- **TEMP diagnostics (remove after verification):** `AudioSessionManager.logSessionState(_:)`
|
||
logs category/mode/options/route; called after `ensureSessionActive`, on every route change,
|
||
and on `.streamStarted` (right after the core opens its devices). On Mac, watch the log when
|
||
joining voice with the Stereo Mic preset: category must stay `…PlayAndRecord` with
|
||
`allowBluetoothA2DP` and the output route must remain the headphones/A2DP device — NOT flip
|
||
to `…Record`. If confirmed, delete the `logSessionState` calls + method and the prior
|
||
band-aid comments in `IOSAudioRouter`/`audio_engine.cpp` can be trimmed.
|
||
- **Verified on Windows:** `cmake --build --preset dev` clean, `ctest --preset dev` 22/22
|
||
(21/21 prior + new `test_dred_toggle`).
|
||
iOS build & on-device run still to be done by the user on the Mac.
|
||
|
||
- **Planned (not started):** **External PCM feed/tap API (`vc_stream_feed_pcm` +
|
||
`vc_set_pcm_sink`)** (2026-06-19, plan written on Windows; implement on Mac). A public,
|
||
documented API for driving audio streams with externally-provided PCM instead of (or in
|
||
addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our
|
||
roadmap — that the current "miniaudio owns the device" model can't serve:
|
||
1. **ReplayKit Broadcast Upload Extension (iOS `SCREEN_AUDIO`)** — the extension is a
|
||
*separate process* with a ~50 MB memory cap and can't link the full `AudioEngine`
|
||
(`ma_device`, capture/playback threads). It needs to feed `CMSampleBuffer` audio (system
|
||
app audio) into the encode path without any audio hardware. The current plan in
|
||
`docs/voice.md §9` says the extension links "a minimal slice of the core (Opus encode +
|
||
media send only)" — a public feed-PCM API *is* that minimal slice. The extension links
|
||
Opus + the feed entry point, no `ma_device` needed.
|
||
2. **ScreenCaptureKit (macOS `SCREEN_AUDIO`)** — `SCStream` delivers `CMSampleBuffer` in a
|
||
callback; convert to int16 and feed. No need to route through miniaudio's device layer.
|
||
**This is how macOS screen-audio actually gets implemented** — today it does NOT work:
|
||
`VOICECAT_HAS_LOOPBACK` is Windows-only (`core/CMakeLists.txt:88-95`), so on macOS
|
||
`AudioEngine::start_loopback_capture()` hits the `#else` stub (`audio_engine.cpp:647-649`)
|
||
and returns `false`. The macOS client's "Share Screen Audio" button
|
||
(`MainWindowController.swift:800-816`) calls `startStream(.screenAudio)` which announces
|
||
the stream to peers but captures **zero audio** — peers hear silence. The button is left
|
||
in place (not touched per user request); it'll work once this API + a ScreenCaptureKit
|
||
tap ship on Mac.
|
||
3. **Bots** — music bot, TTS bot, radio relay, transcription bot. They create a
|
||
`SCREEN_AUDIO`/`AUX_DEVICE` stream and feed synthesized or decoded PCM via the feed API.
|
||
No audio hardware required — runs headless on a server. Today the only way to feed
|
||
external PCM is `vc_test_inject_capture` (TEST-ONLY, name signals "don't ship this") or
|
||
re-implementing Opus encode + AEAD + UDP framing yourself (~500 lines of duplicated
|
||
crypto/codec code per consumer).
|
||
4. **Custom clients / accessibility** — soundboard, DAW integration, TTS of incoming chat,
|
||
recording/transcription of remote audio. Need either feed (send) or tap (receive) or
|
||
both.
|
||
|
||
**What we already have (input half, gated as test-only):** `vc_test_inject_capture
|
||
(stream_id, pcm, samples)` (`voicecat.h`, `client.cpp:1452`) feeds raw int16 PCM into the
|
||
encode pipeline via `AudioEngine::inject_capture(kind, pcm, n)`. It works for any stream
|
||
kind, supports multiple concurrent injection taps (one ring buffer per local kind), and
|
||
goes through the full encode → AEAD → UDP path. The encode path already handles
|
||
`channels == 1 || 2` (proven by the WASAPI stereo loopback work, 2026-06-17 entry below).
|
||
The only problems: it's marked TEST-ONLY in the header, the name signals "don't use this in
|
||
production," and it hardcodes mono (no `channels` parameter).
|
||
|
||
**What's missing (output half):** today decoded remote audio is mixed and pushed to the
|
||
miniaudio playback device (`on_playback`). There's no way for an external consumer to
|
||
intercept the decoded PCM of a specific remote stream — it all goes to the hardware device.
|
||
A bot that wants to record, transcribe, or re-broadcast remote audio has no hook.
|
||
|
||
**Plan (API design — clean, append-only, no struct changes, ABI-stable):**
|
||
- **`vc_stream_feed_pcm`** — promote `vc_test_inject_capture` to a public, documented API
|
||
and add a `channels` parameter:
|
||
```c
|
||
/* External PCM feed — replaces the hardware capture device for this stream. Caller
|
||
provides interleaved int16 PCM at the stream's sample rate. The core frames it,
|
||
encodes (Opus), seals (AEAD), and sends (UDP). Works for any stream kind
|
||
(MIC/SCREEN_AUDIO/AUX_DEVICE). The stream must be started first (vc_stream_start);
|
||
this just replaces the capture source. channels = 1 (mono) or 2 (stereo interleaved).
|
||
Thread-safe; may be called from any thread including audio callbacks. */
|
||
vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id,
|
||
const int16_t* pcm, size_t samples_per_channel,
|
||
uint32_t channels);
|
||
```
|
||
- **`vc_set_pcm_sink`** — symmetric output side: receive decoded remote audio as int16 PCM
|
||
instead of (or in addition to) the hardware playback device:
|
||
```c
|
||
/* External PCM tap — receive decoded, mixed remote audio as int16 PCM. The callback
|
||
fires on the audio thread with the mixed output for a specific remote stream. Pass
|
||
cb=NULL to disable (default: disabled, hardware playback only). When enabled, PCM is
|
||
delivered to the sink AND the hardware device (dual output) so a bot can record
|
||
without disabling local monitoring. user_id+stream_id identify the source stream.
|
||
The callback MUST NOT block — copy what you need and return (same contract as
|
||
vc_callbacks.on_event). */
|
||
typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
|
||
const int16_t* pcm, size_t samples_per_channel,
|
||
uint32_t channels, uint32_t sample_rate);
|
||
vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
|
||
```
|
||
- **Core changes:**
|
||
- `core/include/voicecat.h` — add `vc_pcm_sink_cb` typedef + the two function
|
||
declarations (append-only, after `vc_test_inject_capture`). Full doc comments on both
|
||
(contract, thread-safety, lifetime, use cases).
|
||
- `core/src/voicecat.cpp` — thin C trampolines → `vc_client::stream_feed_pcm` /
|
||
`set_pcm_sink`.
|
||
- `core/src/core/client.{h,cpp}` — `stream_feed_pcm`: validates `stream_id`, looks up
|
||
the `LocalStream`'s kind, calls `audio_engine_.inject_capture(kind, pcm, n)` (existing
|
||
path) with the channel count forwarded. `set_pcm_sink`: stores the callback + user
|
||
pointer; `on_playback` (or a new fan-out in the mixer) invokes it per remote stream
|
||
alongside the existing hardware write. Keep `vc_test_inject_capture` as a deprecated
|
||
alias calling `stream_feed_pcm(..., channels=1)` for source compatibility.
|
||
- `core/src/audio/audio_engine.{h,cpp}` — `inject_capture` already exists per-kind; add
|
||
a `channels` parameter to the ring-buffer write path (or a parallel stereo-aware
|
||
variant). The encode path in `client.cpp::on_capture_frame` already handles
|
||
`channels==2` via the stereo encode branch — just plumb the value through. For the
|
||
sink: add a `pcm_sink_` member (callback + user); in `on_playback` after mixing, if the
|
||
sink is set, copy the mixed PCM for the current stream and invoke the callback. The
|
||
copy must stay off the RT-critical path — document the non-blocking contract.
|
||
- **Skeleton stub path:** update `client.cpp`'s `#else` (no-deps) stub section to add
|
||
`vc_stream_feed_pcm`/`vc_set_pcm_sink` returning `VC_ERR_NOT_IMPLEMENTED` — keeps the
|
||
skeleton preset green.
|
||
- **Swift `VoiceCatCore`:** add `feedPcm(streamId:pcm:samplesPerChannel:channels:)` and
|
||
`setPcmSink(_:user:)` (the Swift wrapper around `vc_pcm_sink_cb` — a
|
||
`@convention(c)` closure + `Unmanaged` context, mirroring `Callbacks.swift`). Wraps both
|
||
new ABI functions.
|
||
- **C# `VoiceCat.Interop`:** add `StreamFeedPcm(streamId, pcm, samples, channels)` (with
|
||
`int16[]` marshaling) and `SetPcmSink` (delegates via `[UnmanagedCallersOnly]` thunk,
|
||
mirroring the event-callback pattern). Wraps both new ABI functions.
|
||
- **Tests:**
|
||
- `tests/test_external_pcm.cpp` (new) — `test_feed_pcm_round_trip`: two clients, A feeds
|
||
a known mono sine wave via `vc_stream_feed_pcm` on a MIC stream, B receives via the
|
||
normal decode path and asserts energy matches. `test_feed_pcm_stereo`: same with
|
||
`channels=2`, assert L≠R end-to-end (mirrors the WASAPI loopback stereo test).
|
||
`test_pcm_sink`: B sets a `vc_pcm_sink_cb`, A feeds PCM, assert the sink callback
|
||
receives the decoded PCM with matching energy. All headless, no audio hardware.
|
||
- `clients/apple/Tests/VoiceCatCoreTests/` — Swift wrapper round-trip for `feedPcm`.
|
||
- `clients/windows/VoiceCat.Interop.Tests/` — C# wrapper round-trip.
|
||
- **Docs:**
|
||
- `docs/architecture.md §4` — new subsection on external PCM feed/tap: the contract
|
||
(caller provides interleaved int16 at the stream's sample rate; core frames/encodes/
|
||
seals/sends for feed; core decodes/mixes/delivers for sink; sink callback must not
|
||
block), the use cases (ReplayKit, ScreenCaptureKit, bots, custom clients), and the
|
||
relationship to `vc_test_inject_capture` (deprecated alias).
|
||
- `docs/voice.md §9` — update the iOS ReplayKit and macOS ScreenCaptureKit rows: both
|
||
now consume `vc_stream_feed_pcm` instead of a "minimal slice of the core." Update the
|
||
iOS detail bullets: the extension links Opus + `vc_stream_feed_pcm` (not a parallel
|
||
media stack). Add a macOS ScreenCaptureKit note: convert `CMSampleBuffer` → int16,
|
||
feed via `vc_stream_feed_pcm` — this is how macOS screen-audio actually ships.
|
||
- `docs/protocol.md` — no protocol changes (the feed/sink are client-local; the wire
|
||
format is identical whether PCM came from miniaudio or an external source). Note this
|
||
explicitly.
|
||
- `docs/roadmap.md` — add a milestone entry; update the iOS ReplayKit and macOS
|
||
ScreenCaptureKit pending items to reference `vc_stream_feed_pcm`.
|
||
- **Implementation order:**
|
||
1. C ABI + core (`voicecat.h`, `voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`) +
|
||
skeleton stub. Verify `ctest --preset dev` green.
|
||
2. `tests/test_external_pcm.cpp` — the three behavior tests. Verify green.
|
||
3. Swift `VoiceCatCore` wrapper + `VoiceCatCoreTests` round-trip.
|
||
4. C# `VoiceCat.Interop` wrapper + `VoiceCatClientSmokeTests` round-trip.
|
||
5. Docs (`architecture.md`, `voice.md`, `protocol.md`, `roadmap.md`, header comments).
|
||
6. **Then** ReplayKit (iOS) and ScreenCaptureKit (macOS) become ~100-line consumers of
|
||
this API instead of parallel media stacks.
|
||
- **Verification:** `ctest --preset dev` green (3 new tests); `swift test` green; `dotnet
|
||
test` green; `xcodebuild` (skeleton) green. The feed/sink tests are fully headless — no
|
||
audio hardware, no simulator, no device — so they run in CI on every platform.
|
||
- **Files to touch:**
|
||
- Core C++: `core/include/voicecat.h`, `core/src/voicecat.cpp`,
|
||
`core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`.
|
||
- Tests: `tests/test_external_pcm.cpp` (new), `tests/CMakeLists.txt`.
|
||
- Swift: `clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift`,
|
||
`clients/apple/Sources/VoiceCatCore/Callbacks.swift`,
|
||
`clients/apple/Tests/VoiceCatCoreTests/ExternalPcmTests.swift` (new).
|
||
- C#: `clients/windows/VoiceCat.Interop/VoiceCatClient.cs`,
|
||
`clients/windows/VoiceCat.Interop/NativeMethods.cs`,
|
||
`clients/windows/VoiceCat.Interop.Tests/ExternalPcmTests.cs` (new).
|
||
- Docs: `docs/architecture.md`, `docs/voice.md`, `docs/protocol.md`, `docs/roadmap.md`.
|
||
- **ABI stability:** append-only — two new functions + one new typedef, no existing
|
||
structs/enums changed. `vc_test_inject_capture` stays as a deprecated alias for source
|
||
compatibility. Treat as a deliberate, versioned ABI event per `docs/protocol.md §8`.
|
||
- **Relationship to the iOS audio routing plan:** orthogonal. The iOS routing layer controls
|
||
*which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan is about
|
||
*bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship; they don't
|
||
conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer controls the
|
||
*mic* path which still uses miniaudio's device.
|
||
|
||
---
|
||
|
||
## Recent completed work
|
||
|
||
All items below are `[x]` done; `ctest --preset dev` 21/21 on Windows after each.
|
||
|
||
- **iOS A2DP + stereo root cause fix** (2026-06-20): miniaudio's NULL-context `ma_device_init`
|
||
was calling `AVAudioSession setCategory(Record)` on every device open, wiping the session
|
||
config `IOSAudioRouter` had set. Fixed by sharing a `ma_context` with
|
||
`sessionCategory=none` + `noAudioSessionActivate/Deactivate=MA_TRUE` — miniaudio never
|
||
touches `AVAudioSession`; `IOSAudioRouter` is the sole owner. Files: `audio_engine.{h,cpp}`.
|
||
|
||
- **iOS audio routing overhaul** (2026-06-19): Full `IOSAudioRouter` singleton drives all
|
||
`AVAudioSession` config before miniaudio opens devices. Fixed stereo mic polar-pattern setup
|
||
(WWDC20 recipe: `setPreferredInput` + `setInputDataSource` + `.stereo` polar pattern + no
|
||
`setPreferredInputNumberOfChannels`). Added `vc_audio_restart` ABI (full stop+reinit for
|
||
close→reconfigure→reopen ordering). Added `vc_set_capture_channels` ABI (core stereo-mic
|
||
support). AVAudioSession activated proactively on `.authResult`, not lazily on
|
||
`.streamStarted`. Join/Leave Voice button added (parity with macOS). Channel-id sync fixed
|
||
(mic button was permanently dimmed). iOS deployment target raised to 18.0.
|
||
|
||
- **iOS SwiftUI client** (2026-06-19): `VoiceCatiOS.xcodeproj` at `clients/apple/iOS/`.
|
||
Full feature parity with macOS/Windows: saved server list (JSON + Keychain, App Group
|
||
`group.cat.voice.VoiceCat`), TOFU, connect flow, channel tree, user list with context menus,
|
||
chat, admin sheets, voice controls, settings. `xcodebuild` → BUILD SUCCEEDED.
|
||
|
||
- **macOS AppKit client** (2026-06-18): `VoiceCatMac.xcodeproj` at `clients/apple/macOS/`.
|
||
Fixed compile errors (`NSAccessibility` call-site arg order, `StreamSummary.id` vs
|
||
`.streamId`) and linker issues (`OTHER_LDFLAGS = -lc++`, `ONLY_ACTIVE_ARCH = YES` for
|
||
Release). Debug + Release both BUILD SUCCEEDED.
|
||
|
||
- **Swift `VoiceCatCore` package + XCFramework** (2026-06-18): Shared Swift wrapper at
|
||
`clients/apple/`. `build-xcframework.sh` merges `libvoicecat.a` + 107 vcpkg static deps into
|
||
a fat `.a` via `libtool -static`. 6/6 Swift tests green (real server, mirrors C# Interop
|
||
tests). Supports macOS-arm64 + iOS-arm64 + iOS-sim slices.
|
||
|
||
- **macOS port validated** (2026-06-18): 21/21 on macOS. Three cross-platform bugs fixed:
|
||
missing `<netdb.h>` in POSIX test branch; SIGPIPE kills (added `SIG_IGN`); use-after-free of
|
||
Asio kqueue reactor on server shutdown (fixed `TcpAcceptor` shutdown/connection-drain
|
||
sequence).
|
||
|
||
- **CMake preset cleanup** (2026-06-18): `m1-dev`→`dev`, `dev`→`skeleton`, `m2-dev` dropped.
|
||
New `release`, `server-release` (stripped), `apple-dev`/`apple-ios`/`apple-ios-sim`. Cross-
|
||
platform triplet auto-resolved by `cmake/voicecat-toolchain.cmake`.
|
||
|
||
- **Disconnect, keepalive & reaper** (2026-06-18): Client sends `Ping` every 15 s; server
|
||
reaper drops sessions after 45 s; UDP `KEEPALIVE` every 5 s keeps NAT alive. `vc_disconnect`
|
||
sends graceful `Disconnect` proto. Stale-user LEFT broadcast on drop. PLC capped at ~2 s.
|
||
Three new tests: `test_disconnect_left`, `test_plc_cap`, `test_reaper_timeout`.
|
||
|
||
- **Stereo screen-audio loopback** (2026-06-17): WASAPI loopback opens in channel's
|
||
stereo/mono mode (was hardcoded mono). Real stereo flows end-to-end through loopback → encode
|
||
→ decode → mixer. New `test_loopback_stereo_capture`.
|
||
|
||
- **Windows screen-audio UI wired** (2026-06-17): `btnScreenShareToggle` in `MainForm.cs`.
|
||
No core/proto/ABI changes — all the plumbing was already there. `dotnet test` 4/4 green.
|
||
|
||
- **Bug fixes** (2026-06-16 – 2026-06-17):
|
||
- *AEAD nonce desync in SFU relay* — relay forwarded sender's `seq` verbatim; recipient
|
||
nonce reconstruction used the wrong counter. Fixed by rewriting the outgoing `seq` field
|
||
to the recipient's `peek_send_counter()`.
|
||
- *Playout clock free-ran* — `playout_ts` advanced even during VAD/PTT silence gaps,
|
||
eventually dropping all frames as too-late. Fixed with resync in `on_playback` via
|
||
`JitterBuffer::peek_front_ts()`.
|
||
- *Stale users after disconnect* — `ConnSession::close()` didn't broadcast `UserEvent::LEFT`
|
||
before erasing. Fixed; PLC cap added as defense-in-depth.
|
||
- *"Randomly bumped to Lobby"* — server excluded the actor from its own state-change
|
||
broadcasts. Fixed: `UserEvent::UPDATED` now goes to all clients including the actor.
|
||
- *Silent playback after join* — `opus_decode` received hardware callback frame count as
|
||
`max_samples` instead of the Opus frame size. Fixed with a decode ring buffer.
|
||
|
||
---
|
||
|
||
## Milestones (see [docs/roadmap.md](docs/roadmap.md) for full detail)
|
||
|
||
- [x] **M0 — Scaffolding** ✓ complete
|
||
- [x] **M1 — Control plane** ✓ complete (2026-06-15)
|
||
- [x] **M2 — Voice, single stream** ✓ complete (2026-06-16)
|
||
- [x] **M3 — Multi-stream & per-channel tuning** ✓ complete (2026-06-16)
|
||
- [x] **M4 — Native clients** — Windows WinForms ✓ (2026-06-17); macOS AppKit ✓ (2026-06-18); iOS SwiftUI ✓ (2026-06-19)
|
||
- [~] **M5 — Moderation, polish, beyond** (perms, bans, DRED; then file transfer, E2EE, …)
|
||
|
||
---
|
||
|
||
## M0 — Scaffolding ✓
|
||
|
||
Repo layout (`core/ server/ tools/ clients/ tests/`), CMake + vcpkg manifest, C ABI header
|
||
(`voicecat.h`), proto source of truth, core stubs for all six subsystems, `voicecat-server` +
|
||
`vccli` skeletons, smoke CTest, `.clang-format`/`.gitattributes`/`.gitignore`.
|
||
|
||
---
|
||
|
||
## M1 — Control plane ✓ (completed 2026-06-15)
|
||
|
||
**Exit criterion:** `test_m1_integration` — two clients authenticate over TLS 1.3 (guest +
|
||
Argon2id), exchange channel + private text. ~1 s.
|
||
|
||
`FrameCodec`, `TlsContext` (mbedTLS 1.3, ECDSA-P256 self-signed, TOFU pins TLS leaf-cert
|
||
SHA-256), `WorkerPool`, `Database` (SQLite + Argon2id), `ServerIdentityManager`,
|
||
`ConnSession` state machine, `SessionRegistry`, `vc_client` full M1 C ABI, `voicecat-admin`
|
||
CLI, dual-stack `TcpAcceptor`. **Key bug fixed:** `send_frame` double-framing — `encode_envelope`
|
||
was pre-framing the protobuf; fixed by passing raw protobuf bytes.
|
||
|
||
---
|
||
|
||
## M2 — Voice, single stream ✓ (completed 2026-06-16)
|
||
|
||
**Exit criterion:** `test_m2_voice` + `test_voice_client_abi` — two headless clients auth, bind
|
||
UDP, 50 Opus frames relayed + re-encrypted by SFU, B receives ≥25 and decrypts. ~4 s.
|
||
|
||
14-byte UDP voice header, `SodiumMediaCrypto` (ChaCha20-Poly1305 + 64-bit anti-replay),
|
||
`OpusEncoder`/`OpusDecoder` (FEC, PLC), `UdpMediaChannel`, `JitterBuffer`, `AudioEngine`
|
||
(miniaudio), `MediaRelay` SFU. **Key bug fixed:** `on_playback` passed hardware callback frame
|
||
count as `opus_decode` max_samples; fixed with a per-stream decode ring buffer.
|
||
|
||
---
|
||
|
||
## M3 — Multi-stream & per-channel tuning ✓ (completed 2026-06-16)
|
||
|
||
**Exit criterion:** `test_m3_multistream` — client A runs two concurrent streams (MIC +
|
||
SCREEN_AUDIO); B sees both; per-stream gain/mute/NR independent; effective Opus config matches
|
||
channel's server-enforced settings. ~2.4 s.
|
||
|
||
Fixed server `stream_id` counter bug (always wrote `1`). Per-channel `AudioConfig` populated
|
||
(Lobby: mono/24kbps/VOIP + DTX; Music Room: stereo/128kbps/AUDIO). `LocalStream` map,
|
||
`pending_announce_kind_`, `run_talk_timer()`, thread-join race in `teardown_voice()` fixed.
|
||
New C ABI: `vc_get_stream_audio_config`, `vc_test_inject_capture`.
|
||
|
||
---
|
||
|
||
## Post-M3 follow-up ✓ (completed 2026-06-16)
|
||
|
||
- **Device enumeration** — `vc_list_devices`/`vc_set_input_device`; opaque hex device ids;
|
||
`vc_free_device_list` now frees. Works pre-connect.
|
||
- **VAD/PTT gate** — `EnergyVadProcessor` (RMS threshold ~0.025, 300 ms hang-time);
|
||
`vc_set_input_mode`/`vc_set_push_to_talk`; MIC-only (SCREEN_AUDIO/AUX_DEVICE bypass).
|
||
- **True stereo playback** — `playback_channels=2`; stereo decoded L→L R→R in mixer; mono
|
||
upmixed L=R; hardware fallback to mono on failure.
|
||
- **WASAPI loopback** — `loopback_device_` with `ma_device_type_loopback`;
|
||
`VOICECAT_HAS_LOOPBACK` macro (Windows-only). `vccli --share-screen-audio`.
|
||
|
||
**Known deferred (still open):** AEC/NS/AGC (no working Windows/MSVC WebRTC APM build);
|
||
process-specific WASAPI loopback; RT-thread rule violation in `on_capture_frame` (mutex lock
|
||
on audio callback thread — pre-existing, needs lock-free ring-buffer refactor).
|
||
|
||
---
|
||
|
||
## M4 — Native clients ✓ (completed 2026-06-17 – 2026-06-19)
|
||
|
||
**Exit criterion:** `ctest --preset dev` 21/21 green; `dotnet build` 0 warnings; `xcodebuild`
|
||
BUILD SUCCEEDED (macOS + iOS); manually verified: connect, TOFU, channel tree, join, voice,
|
||
text, device pickers, level meter on each platform.
|
||
|
||
**New C ABI (additive):** `vc_list_channels`/`vc_list_users`/`vc_list_user_streams`,
|
||
`vc_join_channel`, `VC_EVENT_SERVER_IDENTITY` + `vc_confirm_server_identity`,
|
||
`vc_config::tofu_store_path`, `VC_INPUT_ALWAYS_ON`, `vc_set_vad_threshold`,
|
||
`vc_audio_suspend`/`vc_audio_resume`, `vc_audio_restart`, `vc_set_capture_channels`.
|
||
|
||
**Windows** (`clients/windows/`): `VoiceCat.Interop` (P/Invoke, `[UnmanagedCallersOnly]`),
|
||
`VoiceCat.App` (ConnectDialog, ServerIdentityDialog, MainForm with full M5 moderation UI,
|
||
PerUserTuningDialog, PttKeyCaptureDialog), `VoiceCat.Interop.Tests`. PTT is focus-scoped.
|
||
|
||
**macOS** (`clients/apple/macOS/VoiceCatMac.xcodeproj`): NSOutlineView channel tree,
|
||
NSTableView user list, NSTextView chat, voice controls, full VoiceOver accessibility, admin
|
||
menu, 17 Swift source files. `build-xcframework.sh` produces `VoiceCatCore.xcframework`.
|
||
|
||
**iOS** (`clients/apple/iOS/VoiceCatiOS.xcodeproj`): SwiftUI, `NavigationSplitView`/`TabView`,
|
||
`OutlineGroup` channel tree, `IOSAudioRouter` AVAudioSession driver, 24 Swift source files,
|
||
iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sharing.
|
||
|
||
---
|
||
|
||
## M5 — Moderation, polish, and beyond [~] (in progress 2026-06-17)
|
||
|
||
**Exit criterion:** four ABI-level tests green (`test_m5_permissions`,
|
||
`test_m5_kick_ban_move_mute`, `test_m5_admin_accounts`, `test_m5_channel_crud`);
|
||
`vccli` can drive all moderation/admin/channel operations against a live server.
|
||
|
||
- [x] **Server-side moderation & permissions** — per-session `Permissions`, kick/ban/move/
|
||
server-mute, channel CRUD, DB schema v2 (`channels`, `bans`), BLAKE2b channel passwords.
|
||
- [x] **C ABI** — `vc_kick_user`, `vc_ban_user`, `vc_set_permission`, `vc_set_server_mute`,
|
||
`vc_move_user`, `vc_create_channel`, `vc_edit_channel`, `vc_delete_channel`,
|
||
`vc_create_account`, `vc_reset_password`, `vc_delete_account`, `vc_list_accounts`,
|
||
`vc_get_permissions`; events `VC_EVENT_GENERIC_RESULT`, `VC_EVENT_ACCOUNT_LIST`.
|
||
- [x] **Four M5 tests** passing — `ctest --preset dev` 21/21.
|
||
- [x] **vccli** M5 flags: `--kick`, `--ban`, `--move`, `--server-mute`/`-unmute`/`-deafen`/
|
||
`-undeafen`, `--set-permission`, channel CRUD, account CRUD, `--username`/`--password`.
|
||
- [x] **All three client UIs** (Windows WinForms, macOS AppKit, iOS SwiftUI) expose the full
|
||
M5 moderation and admin surface.
|
||
- [x] **Docs** — `docs/protocol.md`, `docs/security.md` kept in sync.
|
||
- [x] **DRED/audio-quality polish** — done (2026-06-20). `bool dred` added to `AudioConfig`
|
||
proto (field 11) and `vc_audio_config` C ABI. Encoder: `OPUS_SET_DRED_DURATION(2)` when
|
||
enabled (20 ms of ML redundancy per packet). Decoder: `OpusDREDDecoder` + per-stream
|
||
`OpusDRED` scratch pre-allocated; `JitterBuffer::try_copy_front_payload` peeks at the next
|
||
buffered packet on every PLC step; if DRED data is present, `opus_decoder_dred_decode`
|
||
reconstructs the lost frame — otherwise falls back to standard PLC. New test:
|
||
`test_dred_toggle` (ctest 22/22). Files: `voicecat.proto`, `voicecat.h`,
|
||
`opus_codec.{h,cpp}`, `audio_engine.{h,cpp}`, `client.cpp`, `session.{h,cpp}`.
|
||
- [ ] **DRED toggle in client UIs** — expose the `dred` flag in all three channel-config UIs
|
||
so admins can enable it per channel. Windows: `ChannelEditForm` / `vc_channel_info.audio.dred`
|
||
checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs
|
||
already have full channel CRUD wired; this is an additive checkbox on the existing audio-config
|
||
section. (Core/protocol/ABI all done — this is UI-only work.)
|
||
- [ ] **macOS ScreenCaptureKit screen-audio** — `startStream(.screenAudio)` in macOS client
|
||
announces the stream but `start_loopback_capture()` returns false (no `VOICECAT_HAS_LOOPBACK`
|
||
on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships.
|
||
- [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target,
|
||
App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`.
|
||
- [ ] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — see full
|
||
plan in "Where we left off" above.
|
||
|
||
---
|
||
|
||
## Decisions log
|
||
|
||
All architecture/scope decisions are settled and recorded in
|
||
[docs/roadmap.md §2 "Resolved decisions"](docs/roadmap.md) and reflected across `docs/`.
|
||
If you make a *new* decision, record it there and link it here.
|
||
|
||
---
|
||
|
||
## How to update this file
|
||
|
||
1. Check off tasks as you complete them; flip a milestone to `[x]` only when its **exit
|
||
criterion test** passes.
|
||
2. Keep the **"Where we left off / next action"** block at the top accurate — it's the first
|
||
thing the next agent reads.
|
||
3. When you start a milestone, copy its task list from `docs/roadmap.md` into a section here.
|