scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
# PROGRESS — VoiceCat
|
|
|
|
|
|
|
|
|
|
|
|
Living status. **Update this file in the same commit as your work** so the next agent picks
|
|
|
|
|
|
up instantly. Newest status at the top.
|
|
|
|
|
|
|
|
|
|
|
|
- **Date convention:** ISO (YYYY-MM-DD).
|
|
|
|
|
|
- Statuses: `[ ]` not started · `[~]` in progress · `[x]` done.
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## ▶ Where we left off / next action
|
|
|
|
|
|
|
fix(ios): stop miniaudio from clobbering AVAudioSession (stereo->A2DP output death)
The real root cause of "selecting Stereo Mic kills headphone/A2DP output on Join
Voice." Every prior fix worked on the Swift IOSAudioRouter under the false premise
that "miniaudio does NOT touch AVAudioSession on iOS." It does: the core opened
devices via ma_device_init(nullptr, ...), and with a NULL context miniaudio 0.11.25
runs an iOS "hack" that sets the session category by device type, then
ma_context_init__coreaudio calls setCategory()+setActive() on every device open --
capture -> AVAudioSessionCategoryRecord with zero options. That wipes the
.playAndRecord category, the mode, and .allowBluetoothA2DP / .mixWithOthers /
.allowAirPlay that IOSAudioRouter had just configured, killing headphone/A2DP (and
even wired) output. Stereo presets break worst because they rely on the A2DP output
route the wipe removes. TeamTalk avoids this by opening RemoteIO/VPIO AudioUnits
directly and leaving the session entirely to the app.
Fix (core, cross-platform safe): AudioEngine now owns a ma_context built by
make_context_config() with coreaudio.sessionCategory = ma_ios_session_category_none
and noAudioSessionActivate/Deactivate = MA_TRUE, and routes all ma_device_init calls
(playback, capture, loopback) plus enumerate_devices through it. miniaudio no longer
touches AVAudioSession; IOSAudioRouter is the sole owner (the session is already
activated on connect in AppState before any device opens). Context is lazily inited
in start(), reused across restarts, uninited in ~AudioEngine.
Adds TEMP AudioSessionManager.logSessionState() diagnostics (after activate, on route
change, on .streamStarted) to verify on-device that the category stays
PlayAndRecord+allowBluetoothA2DP instead of flipping to Record. Remove once confirmed.
Windows: cmake --build --preset dev clean; ctest --preset dev 21/21.
iOS build + on-device verification pending on Mac.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 02:34:07 +02:00
|
|
|
|
- **Awaiting on-device verification:** **iOS stereo mic kills headphone/A2DP output — REAL
|
|
|
|
|
|
root cause found & fixed** (2026-06-20, on Windows; verify on Mac). All prior "fixes" (the
|
|
|
|
|
|
2026-06-19 entries below) targeted the Swift `IOSAudioRouter` on the false premise that
|
|
|
|
|
|
"miniaudio does NOT touch AVAudioSession on iOS." **It does.** The core opened its miniaudio
|
|
|
|
|
|
devices with `ma_device_init(nullptr, ...)`; with a NULL context, miniaudio (0.11.25) runs an
|
|
|
|
|
|
iOS "hack" (`miniaudio.h` ~44057) that picks a session category by device type, then
|
|
|
|
|
|
`ma_context_init__coreaudio` (~36552) calls `setCategory()` + `setActive()` on **every device
|
|
|
|
|
|
open** — capture → `AVAudioSessionCategoryRecord` with **zero options**. That wiped the
|
|
|
|
|
|
`.playAndRecord` category, the mode, and `.allowBluetoothA2DP`/`.mixWithOthers`/`.allowAirPlay`
|
|
|
|
|
|
that `IOSAudioRouter` had just configured → headphone/A2DP (and even wired) output died. The
|
|
|
|
|
|
stereo presets broke worst because they depend on the A2DP output route the wipe removed.
|
|
|
|
|
|
TeamTalk never hits this: its SDK opens RemoteIO/VPIO AudioUnits directly and leaves the
|
|
|
|
|
|
session entirely to the app (`UtilSound.swift`); miniaudio insists on managing it.
|
|
|
|
|
|
- **Fix (core, cross-platform safe):** `AudioEngine` now owns a `ma_context` built by
|
|
|
|
|
|
`make_context_config()` with `coreaudio.sessionCategory = ma_ios_session_category_none` +
|
|
|
|
|
|
`noAudioSessionActivate`/`noAudioSessionDeactivate = MA_TRUE`, and passes it to **all**
|
|
|
|
|
|
`ma_device_init` calls (playback, capture, loopback) and to `enumerate_devices`'s context.
|
|
|
|
|
|
miniaudio now never touches AVAudioSession; the Swift `IOSAudioRouter` is the sole owner
|
|
|
|
|
|
(session is already activated on connect in `AppState.swift:authResult`, before any device
|
|
|
|
|
|
opens, so removing miniaudio's self-activation is safe). Context is lazily inited in
|
|
|
|
|
|
`start()`, reused across restarts, uninited in `~AudioEngine`.
|
|
|
|
|
|
Files: `core/src/audio/audio_engine.{h,cpp}`.
|
|
|
|
|
|
- **TEMP diagnostics (remove after verification):** `AudioSessionManager.logSessionState(_:)`
|
|
|
|
|
|
logs category/mode/options/route; called after `ensureSessionActive`, on every route change,
|
|
|
|
|
|
and on `.streamStarted` (right after the core opens its devices). On Mac, watch the log when
|
|
|
|
|
|
joining voice with the Stereo Mic preset: category must stay `…PlayAndRecord` with
|
|
|
|
|
|
`allowBluetoothA2DP` and the output route must remain the headphones/A2DP device — NOT flip
|
|
|
|
|
|
to `…Record`. If confirmed, delete the `logSessionState` calls + method and the prior
|
|
|
|
|
|
band-aid comments in `IOSAudioRouter`/`audio_engine.cpp` can be trimmed.
|
|
|
|
|
|
- **Verified on Windows:** `cmake --build --preset dev` clean, `ctest --preset dev` 21/21.
|
|
|
|
|
|
iOS build & on-device run still to be done by the user on the Mac.
|
|
|
|
|
|
|
2026-06-19 03:51:14 +02:00
|
|
|
|
- **Planned (not started):** **External PCM feed/tap API (`vc_stream_feed_pcm` +
|
|
|
|
|
|
`vc_set_pcm_sink`)** (2026-06-19, plan written on Windows; implement on Mac). A public,
|
|
|
|
|
|
documented API for driving audio streams with externally-provided PCM instead of (or in
|
|
|
|
|
|
addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our
|
|
|
|
|
|
roadmap — that the current "miniaudio owns the device" model can't serve:
|
|
|
|
|
|
1. **ReplayKit Broadcast Upload Extension (iOS `SCREEN_AUDIO`)** — the extension is a
|
|
|
|
|
|
*separate process* with a ~50 MB memory cap and can't link the full `AudioEngine`
|
|
|
|
|
|
(`ma_device`, capture/playback threads). It needs to feed `CMSampleBuffer` audio (system
|
|
|
|
|
|
app audio) into the encode path without any audio hardware. The current plan in
|
|
|
|
|
|
`docs/voice.md §9` says the extension links "a minimal slice of the core (Opus encode +
|
|
|
|
|
|
media send only)" — a public feed-PCM API *is* that minimal slice. The extension links
|
|
|
|
|
|
Opus + the feed entry point, no `ma_device` needed.
|
|
|
|
|
|
2. **ScreenCaptureKit (macOS `SCREEN_AUDIO`)** — `SCStream` delivers `CMSampleBuffer` in a
|
|
|
|
|
|
callback; convert to int16 and feed. No need to route through miniaudio's device layer.
|
|
|
|
|
|
**This is how macOS screen-audio actually gets implemented** — today it does NOT work:
|
|
|
|
|
|
`VOICECAT_HAS_LOOPBACK` is Windows-only (`core/CMakeLists.txt:88-95`), so on macOS
|
|
|
|
|
|
`AudioEngine::start_loopback_capture()` hits the `#else` stub (`audio_engine.cpp:647-649`)
|
|
|
|
|
|
and returns `false`. The macOS client's "Share Screen Audio" button
|
|
|
|
|
|
(`MainWindowController.swift:800-816`) calls `startStream(.screenAudio)` which announces
|
|
|
|
|
|
the stream to peers but captures **zero audio** — peers hear silence. The button is left
|
|
|
|
|
|
in place (not touched per user request); it'll work once this API + a ScreenCaptureKit
|
|
|
|
|
|
tap ship on Mac.
|
|
|
|
|
|
3. **Bots** — music bot, TTS bot, radio relay, transcription bot. They create a
|
|
|
|
|
|
`SCREEN_AUDIO`/`AUX_DEVICE` stream and feed synthesized or decoded PCM via the feed API.
|
|
|
|
|
|
No audio hardware required — runs headless on a server. Today the only way to feed
|
|
|
|
|
|
external PCM is `vc_test_inject_capture` (TEST-ONLY, name signals "don't ship this") or
|
|
|
|
|
|
re-implementing Opus encode + AEAD + UDP framing yourself (~500 lines of duplicated
|
|
|
|
|
|
crypto/codec code per consumer).
|
|
|
|
|
|
4. **Custom clients / accessibility** — soundboard, DAW integration, TTS of incoming chat,
|
|
|
|
|
|
recording/transcription of remote audio. Need either feed (send) or tap (receive) or
|
|
|
|
|
|
both.
|
|
|
|
|
|
|
|
|
|
|
|
**What we already have (input half, gated as test-only):** `vc_test_inject_capture
|
|
|
|
|
|
(stream_id, pcm, samples)` (`voicecat.h`, `client.cpp:1452`) feeds raw int16 PCM into the
|
|
|
|
|
|
encode pipeline via `AudioEngine::inject_capture(kind, pcm, n)`. It works for any stream
|
|
|
|
|
|
kind, supports multiple concurrent injection taps (one ring buffer per local kind), and
|
|
|
|
|
|
goes through the full encode → AEAD → UDP path. The encode path already handles
|
|
|
|
|
|
`channels == 1 || 2` (proven by the WASAPI stereo loopback work, 2026-06-17 entry below).
|
|
|
|
|
|
The only problems: it's marked TEST-ONLY in the header, the name signals "don't use this in
|
|
|
|
|
|
production," and it hardcodes mono (no `channels` parameter).
|
|
|
|
|
|
|
|
|
|
|
|
**What's missing (output half):** today decoded remote audio is mixed and pushed to the
|
|
|
|
|
|
miniaudio playback device (`on_playback`). There's no way for an external consumer to
|
|
|
|
|
|
intercept the decoded PCM of a specific remote stream — it all goes to the hardware device.
|
|
|
|
|
|
A bot that wants to record, transcribe, or re-broadcast remote audio has no hook.
|
|
|
|
|
|
|
|
|
|
|
|
**Plan (API design — clean, append-only, no struct changes, ABI-stable):**
|
|
|
|
|
|
- **`vc_stream_feed_pcm`** — promote `vc_test_inject_capture` to a public, documented API
|
|
|
|
|
|
and add a `channels` parameter:
|
|
|
|
|
|
```c
|
|
|
|
|
|
/* External PCM feed — replaces the hardware capture device for this stream. Caller
|
|
|
|
|
|
provides interleaved int16 PCM at the stream's sample rate. The core frames it,
|
|
|
|
|
|
encodes (Opus), seals (AEAD), and sends (UDP). Works for any stream kind
|
|
|
|
|
|
(MIC/SCREEN_AUDIO/AUX_DEVICE). The stream must be started first (vc_stream_start);
|
|
|
|
|
|
this just replaces the capture source. channels = 1 (mono) or 2 (stereo interleaved).
|
|
|
|
|
|
Thread-safe; may be called from any thread including audio callbacks. */
|
|
|
|
|
|
vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id,
|
|
|
|
|
|
const int16_t* pcm, size_t samples_per_channel,
|
|
|
|
|
|
uint32_t channels);
|
|
|
|
|
|
```
|
|
|
|
|
|
- **`vc_set_pcm_sink`** — symmetric output side: receive decoded remote audio as int16 PCM
|
|
|
|
|
|
instead of (or in addition to) the hardware playback device:
|
|
|
|
|
|
```c
|
|
|
|
|
|
/* External PCM tap — receive decoded, mixed remote audio as int16 PCM. The callback
|
|
|
|
|
|
fires on the audio thread with the mixed output for a specific remote stream. Pass
|
|
|
|
|
|
cb=NULL to disable (default: disabled, hardware playback only). When enabled, PCM is
|
|
|
|
|
|
delivered to the sink AND the hardware device (dual output) so a bot can record
|
|
|
|
|
|
without disabling local monitoring. user_id+stream_id identify the source stream.
|
|
|
|
|
|
The callback MUST NOT block — copy what you need and return (same contract as
|
|
|
|
|
|
vc_callbacks.on_event). */
|
|
|
|
|
|
typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
|
|
|
|
|
|
const int16_t* pcm, size_t samples_per_channel,
|
|
|
|
|
|
uint32_t channels, uint32_t sample_rate);
|
|
|
|
|
|
vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
|
|
|
|
|
|
```
|
|
|
|
|
|
- **Core changes:**
|
|
|
|
|
|
- `core/include/voicecat.h` — add `vc_pcm_sink_cb` typedef + the two function
|
|
|
|
|
|
declarations (append-only, after `vc_test_inject_capture`). Full doc comments on both
|
|
|
|
|
|
(contract, thread-safety, lifetime, use cases).
|
|
|
|
|
|
- `core/src/voicecat.cpp` — thin C trampolines → `vc_client::stream_feed_pcm` /
|
|
|
|
|
|
`set_pcm_sink`.
|
|
|
|
|
|
- `core/src/core/client.{h,cpp}` — `stream_feed_pcm`: validates `stream_id`, looks up
|
|
|
|
|
|
the `LocalStream`'s kind, calls `audio_engine_.inject_capture(kind, pcm, n)` (existing
|
|
|
|
|
|
path) with the channel count forwarded. `set_pcm_sink`: stores the callback + user
|
|
|
|
|
|
pointer; `on_playback` (or a new fan-out in the mixer) invokes it per remote stream
|
|
|
|
|
|
alongside the existing hardware write. Keep `vc_test_inject_capture` as a deprecated
|
|
|
|
|
|
alias calling `stream_feed_pcm(..., channels=1)` for source compatibility.
|
|
|
|
|
|
- `core/src/audio/audio_engine.{h,cpp}` — `inject_capture` already exists per-kind; add
|
|
|
|
|
|
a `channels` parameter to the ring-buffer write path (or a parallel stereo-aware
|
|
|
|
|
|
variant). The encode path in `client.cpp::on_capture_frame` already handles
|
|
|
|
|
|
`channels==2` via the stereo encode branch — just plumb the value through. For the
|
|
|
|
|
|
sink: add a `pcm_sink_` member (callback + user); in `on_playback` after mixing, if the
|
|
|
|
|
|
sink is set, copy the mixed PCM for the current stream and invoke the callback. The
|
|
|
|
|
|
copy must stay off the RT-critical path — document the non-blocking contract.
|
|
|
|
|
|
- **Skeleton stub path:** update `client.cpp`'s `#else` (no-deps) stub section to add
|
|
|
|
|
|
`vc_stream_feed_pcm`/`vc_set_pcm_sink` returning `VC_ERR_NOT_IMPLEMENTED` — keeps the
|
|
|
|
|
|
skeleton preset green.
|
|
|
|
|
|
- **Swift `VoiceCatCore`:** add `feedPcm(streamId:pcm:samplesPerChannel:channels:)` and
|
|
|
|
|
|
`setPcmSink(_:user:)` (the Swift wrapper around `vc_pcm_sink_cb` — a
|
|
|
|
|
|
`@convention(c)` closure + `Unmanaged` context, mirroring `Callbacks.swift`). Wraps both
|
|
|
|
|
|
new ABI functions.
|
|
|
|
|
|
- **C# `VoiceCat.Interop`:** add `StreamFeedPcm(streamId, pcm, samples, channels)` (with
|
|
|
|
|
|
`int16[]` marshaling) and `SetPcmSink` (delegates via `[UnmanagedCallersOnly]` thunk,
|
|
|
|
|
|
mirroring the event-callback pattern). Wraps both new ABI functions.
|
|
|
|
|
|
- **Tests:**
|
|
|
|
|
|
- `tests/test_external_pcm.cpp` (new) — `test_feed_pcm_round_trip`: two clients, A feeds
|
|
|
|
|
|
a known mono sine wave via `vc_stream_feed_pcm` on a MIC stream, B receives via the
|
|
|
|
|
|
normal decode path and asserts energy matches. `test_feed_pcm_stereo`: same with
|
|
|
|
|
|
`channels=2`, assert L≠R end-to-end (mirrors the WASAPI loopback stereo test).
|
|
|
|
|
|
`test_pcm_sink`: B sets a `vc_pcm_sink_cb`, A feeds PCM, assert the sink callback
|
|
|
|
|
|
receives the decoded PCM with matching energy. All headless, no audio hardware.
|
|
|
|
|
|
- `clients/apple/Tests/VoiceCatCoreTests/` — Swift wrapper round-trip for `feedPcm`.
|
|
|
|
|
|
- `clients/windows/VoiceCat.Interop.Tests/` — C# wrapper round-trip.
|
|
|
|
|
|
- **Docs:**
|
|
|
|
|
|
- `docs/architecture.md §4` — new subsection on external PCM feed/tap: the contract
|
|
|
|
|
|
(caller provides interleaved int16 at the stream's sample rate; core frames/encodes/
|
|
|
|
|
|
seals/sends for feed; core decodes/mixes/delivers for sink; sink callback must not
|
|
|
|
|
|
block), the use cases (ReplayKit, ScreenCaptureKit, bots, custom clients), and the
|
|
|
|
|
|
relationship to `vc_test_inject_capture` (deprecated alias).
|
|
|
|
|
|
- `docs/voice.md §9` — update the iOS ReplayKit and macOS ScreenCaptureKit rows: both
|
|
|
|
|
|
now consume `vc_stream_feed_pcm` instead of a "minimal slice of the core." Update the
|
|
|
|
|
|
iOS detail bullets: the extension links Opus + `vc_stream_feed_pcm` (not a parallel
|
|
|
|
|
|
media stack). Add a macOS ScreenCaptureKit note: convert `CMSampleBuffer` → int16,
|
|
|
|
|
|
feed via `vc_stream_feed_pcm` — this is how macOS screen-audio actually ships.
|
|
|
|
|
|
- `docs/protocol.md` — no protocol changes (the feed/sink are client-local; the wire
|
|
|
|
|
|
format is identical whether PCM came from miniaudio or an external source). Note this
|
|
|
|
|
|
explicitly.
|
|
|
|
|
|
- `docs/roadmap.md` — add a milestone entry; update the iOS ReplayKit and macOS
|
|
|
|
|
|
ScreenCaptureKit pending items to reference `vc_stream_feed_pcm`.
|
|
|
|
|
|
- **Implementation order:**
|
|
|
|
|
|
1. C ABI + core (`voicecat.h`, `voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`) +
|
|
|
|
|
|
skeleton stub. Verify `ctest --preset dev` green.
|
|
|
|
|
|
2. `tests/test_external_pcm.cpp` — the three behavior tests. Verify green.
|
|
|
|
|
|
3. Swift `VoiceCatCore` wrapper + `VoiceCatCoreTests` round-trip.
|
|
|
|
|
|
4. C# `VoiceCat.Interop` wrapper + `VoiceCatClientSmokeTests` round-trip.
|
|
|
|
|
|
5. Docs (`architecture.md`, `voice.md`, `protocol.md`, `roadmap.md`, header comments).
|
|
|
|
|
|
6. **Then** ReplayKit (iOS) and ScreenCaptureKit (macOS) become ~100-line consumers of
|
|
|
|
|
|
this API instead of parallel media stacks.
|
|
|
|
|
|
- **Verification:** `ctest --preset dev` green (3 new tests); `swift test` green; `dotnet
|
|
|
|
|
|
test` green; `xcodebuild` (skeleton) green. The feed/sink tests are fully headless — no
|
|
|
|
|
|
audio hardware, no simulator, no device — so they run in CI on every platform.
|
|
|
|
|
|
- **Files to touch:**
|
|
|
|
|
|
- Core C++: `core/include/voicecat.h`, `core/src/voicecat.cpp`,
|
|
|
|
|
|
`core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`.
|
|
|
|
|
|
- Tests: `tests/test_external_pcm.cpp` (new), `tests/CMakeLists.txt`.
|
|
|
|
|
|
- Swift: `clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift`,
|
|
|
|
|
|
`clients/apple/Sources/VoiceCatCore/Callbacks.swift`,
|
|
|
|
|
|
`clients/apple/Tests/VoiceCatCoreTests/ExternalPcmTests.swift` (new).
|
|
|
|
|
|
- C#: `clients/windows/VoiceCat.Interop/VoiceCatClient.cs`,
|
|
|
|
|
|
`clients/windows/VoiceCat.Interop/NativeMethods.cs`,
|
|
|
|
|
|
`clients/windows/VoiceCat.Interop.Tests/ExternalPcmTests.cs` (new).
|
|
|
|
|
|
- Docs: `docs/architecture.md`, `docs/voice.md`, `docs/protocol.md`, `docs/roadmap.md`.
|
|
|
|
|
|
- **ABI stability:** append-only — two new functions + one new typedef, no existing
|
|
|
|
|
|
structs/enums changed. `vc_test_inject_capture` stays as a deprecated alias for source
|
|
|
|
|
|
compatibility. Treat as a deliberate, versioned ABI event per `docs/protocol.md §8`.
|
2026-06-20 13:14:39 +02:00
|
|
|
|
- **Relationship to the iOS audio routing plan:** orthogonal. The iOS routing layer controls
|
|
|
|
|
|
*which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan is about
|
|
|
|
|
|
*bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship; they don't
|
|
|
|
|
|
conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer controls the
|
|
|
|
|
|
*mic* path which still uses miniaudio's device.
|
2026-06-19 03:51:14 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
---
|
2026-06-19 03:51:14 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
## Recent completed work
|
|
|
|
|
|
|
|
|
|
|
|
All items below are `[x]` done; `ctest --preset dev` 21/21 on Windows after each.
|
|
|
|
|
|
|
|
|
|
|
|
- **iOS A2DP + stereo root cause fix** (2026-06-20): miniaudio's NULL-context `ma_device_init`
|
|
|
|
|
|
was calling `AVAudioSession setCategory(Record)` on every device open, wiping the session
|
|
|
|
|
|
config `IOSAudioRouter` had set. Fixed by sharing a `ma_context` with
|
|
|
|
|
|
`sessionCategory=none` + `noAudioSessionActivate/Deactivate=MA_TRUE` — miniaudio never
|
|
|
|
|
|
touches `AVAudioSession`; `IOSAudioRouter` is the sole owner. Files: `audio_engine.{h,cpp}`.
|
|
|
|
|
|
|
|
|
|
|
|
- **iOS audio routing overhaul** (2026-06-19): Full `IOSAudioRouter` singleton drives all
|
|
|
|
|
|
`AVAudioSession` config before miniaudio opens devices. Fixed stereo mic polar-pattern setup
|
|
|
|
|
|
(WWDC20 recipe: `setPreferredInput` + `setInputDataSource` + `.stereo` polar pattern + no
|
|
|
|
|
|
`setPreferredInputNumberOfChannels`). Added `vc_audio_restart` ABI (full stop+reinit for
|
|
|
|
|
|
close→reconfigure→reopen ordering). Added `vc_set_capture_channels` ABI (core stereo-mic
|
|
|
|
|
|
support). AVAudioSession activated proactively on `.authResult`, not lazily on
|
|
|
|
|
|
`.streamStarted`. Join/Leave Voice button added (parity with macOS). Channel-id sync fixed
|
|
|
|
|
|
(mic button was permanently dimmed). iOS deployment target raised to 18.0.
|
|
|
|
|
|
|
|
|
|
|
|
- **iOS SwiftUI client** (2026-06-19): `VoiceCatiOS.xcodeproj` at `clients/apple/iOS/`.
|
|
|
|
|
|
Full feature parity with macOS/Windows: saved server list (JSON + Keychain, App Group
|
|
|
|
|
|
`group.cat.voice.VoiceCat`), TOFU, connect flow, channel tree, user list with context menus,
|
|
|
|
|
|
chat, admin sheets, voice controls, settings. `xcodebuild` → BUILD SUCCEEDED.
|
|
|
|
|
|
|
|
|
|
|
|
- **macOS AppKit client** (2026-06-18): `VoiceCatMac.xcodeproj` at `clients/apple/macOS/`.
|
|
|
|
|
|
Fixed compile errors (`NSAccessibility` call-site arg order, `StreamSummary.id` vs
|
|
|
|
|
|
`.streamId`) and linker issues (`OTHER_LDFLAGS = -lc++`, `ONLY_ACTIVE_ARCH = YES` for
|
|
|
|
|
|
Release). Debug + Release both BUILD SUCCEEDED.
|
|
|
|
|
|
|
|
|
|
|
|
- **Swift `VoiceCatCore` package + XCFramework** (2026-06-18): Shared Swift wrapper at
|
|
|
|
|
|
`clients/apple/`. `build-xcframework.sh` merges `libvoicecat.a` + 107 vcpkg static deps into
|
|
|
|
|
|
a fat `.a` via `libtool -static`. 6/6 Swift tests green (real server, mirrors C# Interop
|
|
|
|
|
|
tests). Supports macOS-arm64 + iOS-arm64 + iOS-sim slices.
|
|
|
|
|
|
|
|
|
|
|
|
- **macOS port validated** (2026-06-18): 21/21 on macOS. Three cross-platform bugs fixed:
|
|
|
|
|
|
missing `<netdb.h>` in POSIX test branch; SIGPIPE kills (added `SIG_IGN`); use-after-free of
|
|
|
|
|
|
Asio kqueue reactor on server shutdown (fixed `TcpAcceptor` shutdown/connection-drain
|
|
|
|
|
|
sequence).
|
|
|
|
|
|
|
|
|
|
|
|
- **CMake preset cleanup** (2026-06-18): `m1-dev`→`dev`, `dev`→`skeleton`, `m2-dev` dropped.
|
|
|
|
|
|
New `release`, `server-release` (stripped), `apple-dev`/`apple-ios`/`apple-ios-sim`. Cross-
|
|
|
|
|
|
platform triplet auto-resolved by `cmake/voicecat-toolchain.cmake`.
|
|
|
|
|
|
|
|
|
|
|
|
- **Disconnect, keepalive & reaper** (2026-06-18): Client sends `Ping` every 15 s; server
|
|
|
|
|
|
reaper drops sessions after 45 s; UDP `KEEPALIVE` every 5 s keeps NAT alive. `vc_disconnect`
|
|
|
|
|
|
sends graceful `Disconnect` proto. Stale-user LEFT broadcast on drop. PLC capped at ~2 s.
|
|
|
|
|
|
Three new tests: `test_disconnect_left`, `test_plc_cap`, `test_reaper_timeout`.
|
|
|
|
|
|
|
|
|
|
|
|
- **Stereo screen-audio loopback** (2026-06-17): WASAPI loopback opens in channel's
|
|
|
|
|
|
stereo/mono mode (was hardcoded mono). Real stereo flows end-to-end through loopback → encode
|
|
|
|
|
|
→ decode → mixer. New `test_loopback_stereo_capture`.
|
|
|
|
|
|
|
|
|
|
|
|
- **Windows screen-audio UI wired** (2026-06-17): `btnScreenShareToggle` in `MainForm.cs`.
|
|
|
|
|
|
No core/proto/ABI changes — all the plumbing was already there. `dotnet test` 4/4 green.
|
|
|
|
|
|
|
|
|
|
|
|
- **Bug fixes** (2026-06-16 – 2026-06-17):
|
|
|
|
|
|
- *AEAD nonce desync in SFU relay* — relay forwarded sender's `seq` verbatim; recipient
|
|
|
|
|
|
nonce reconstruction used the wrong counter. Fixed by rewriting the outgoing `seq` field
|
|
|
|
|
|
to the recipient's `peek_send_counter()`.
|
|
|
|
|
|
- *Playout clock free-ran* — `playout_ts` advanced even during VAD/PTT silence gaps,
|
|
|
|
|
|
eventually dropping all frames as too-late. Fixed with resync in `on_playback` via
|
|
|
|
|
|
`JitterBuffer::peek_front_ts()`.
|
|
|
|
|
|
- *Stale users after disconnect* — `ConnSession::close()` didn't broadcast `UserEvent::LEFT`
|
|
|
|
|
|
before erasing. Fixed; PLC cap added as defense-in-depth.
|
|
|
|
|
|
- *"Randomly bumped to Lobby"* — server excluded the actor from its own state-change
|
|
|
|
|
|
broadcasts. Fixed: `UserEvent::UPDATED` now goes to all clients including the actor.
|
|
|
|
|
|
- *Silent playback after join* — `opus_decode` received hardware callback frame count as
|
|
|
|
|
|
`max_samples` instead of the Opus frame size. Fixed with a decode ring buffer.
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## Milestones (see [docs/roadmap.md](docs/roadmap.md) for full detail)
|
|
|
|
|
|
|
|
|
|
|
|
- [x] **M0 — Scaffolding** ✓ complete
|
feat(M1): TCP/TLS control plane -- auth, channels, ephemeral text
Implements the full M1 milestone. Two clients authenticate over TLS 1.3
(guest + Argon2id password) and exchange channel + private text messages
through a real server. All five ctest --preset m1-dev tests pass in ~1 s.
Key components added:
- vcpkg baseline + m1-dev preset (protobuf/mbedTLS/libsodium/asio/sqlite3)
- FrameCodec feed+emit, encode/decode_envelope, protobuf codegen
- TcpServerConn with blocking TLS handshake thread + tls_read_loop
- TlsContext (mbedTLS 1.3, ECDSA-P256 self-signed cert, TOFU on client)
- WorkerPool (3 threads, used for Argon2id)
- Database: SQLite + libsodium Argon2id, account lifecycle, bootstrap admin
- ServerIdentityManager: Ed25519 key + cert generate/persist/fingerprint
- ConnSession state machine: WaitingHello -> WaitingAuth -> Authenticated
- SessionRegistry: channel tree, user map, text routing, broadcast
- vc_client full M1 C ABI: connect/TLS/handshake/auth/text/disconnect
- voicecat-admin CLI: account add/reset/del/list
- test_m1_integration: M1 exit criterion, verified green
Bug fixed: double-framing in ConnSession::send_envelope -- encode_envelope
was adding the [4-byte len] prefix, then TcpServerConn::send_frame added
a second one, causing the client to parse [len][proto] as protobuf (silent
failure). Fixed by serializing raw protobuf bytes in send_envelope and
letting send_frame apply the single length prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 23:48:44 +02:00
|
|
|
|
- [x] **M1 — Control plane** ✓ complete (2026-06-15)
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
- [x] **M2 — Voice, single stream** ✓ complete (2026-06-16)
|
feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).
Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
the same user silently overwrote the first in SessionRegistry::set_user_stream.
Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
and receive (sync_remote_streams) paths via a shared
opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
(total samples instead of samples-per-channel), which would have overflowed
the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
and from disconnect() on a different thread -- both could see
udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
same std::thread (intermittent std::system_error under ctest). Fixed with a
teardown_mu_ guard instead of carrying the flake forward.
New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
handle_stream_announce enforces the channel's config, clamping (not
overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
request_id-correlated announce/result handling (request_id already
round-tripped on the wire; just wasn't read before). on_capture_frame is
kind-aware and upmixes mono capture to stereo when a stream's config calls
for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
through set_remote_stream. New run_talk_timer() thread emits
VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
(inject_capture), stereo-to-mono downmix at the decode/mix boundary,
RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
and last_voice_ms/talking; new set_stream_noise_reduction() and
poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
vc_get_stream_audio_config (effective Opus config for any stream you own or
a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
(mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
concurrent local streams, independent gain/mute/NS control, per-channel
config divergence via vc_get_stream_audio_config, talk indicators.
Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).
ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
|
|
|
|
- [x] **M3 — Multi-stream & per-channel tuning** ✓ complete (2026-06-16)
|
2026-06-19 02:10:25 +02:00
|
|
|
|
- [x] **M4 — Native clients** — Windows WinForms ✓ (2026-06-17); macOS AppKit ✓ (2026-06-18); iOS SwiftUI ✓ (2026-06-19)
|
M5: moderation, permissions, channel CRUD, in-app account management
- Server-side moderation & permissions (kick/ban/move/server-mute, channel CRUD).
- Database schema v2: channels, bans; BLAKE2b channel passwords, Argon2id accounts.
- C ABI additions and client-side handling (vc_kick_user, vc_ban_user, vc_set_permission, vc_set_server_mute, vc_move_user, vc_create/edit/delete_channel, vc_create/reset/delete/list_account).
- vccli flags for all M5 operations plus --username/--password auth.
- Four new tests covering permissions, kick/ban/move/mute, admin accounts, channel CRUD.
- Docs: protocol.md envelope updates, security.md channel-password hashing, PROGRESS.md.
2026-06-17 15:08:05 +02:00
|
|
|
|
- [~] **M5 — Moderation, polish, beyond** (perms, bans, DRED; then file transfer, E2EE, …)
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
## M0 — Scaffolding ✓
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
Repo layout (`core/ server/ tools/ clients/ tests/`), CMake + vcpkg manifest, C ABI header
|
|
|
|
|
|
(`voicecat.h`), proto source of truth, core stubs for all six subsystems, `voicecat-server` +
|
|
|
|
|
|
`vccli` skeletons, smoke CTest, `.clang-format`/`.gitattributes`/`.gitignore`.
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
feat(M1): TCP/TLS control plane -- auth, channels, ephemeral text
Implements the full M1 milestone. Two clients authenticate over TLS 1.3
(guest + Argon2id password) and exchange channel + private text messages
through a real server. All five ctest --preset m1-dev tests pass in ~1 s.
Key components added:
- vcpkg baseline + m1-dev preset (protobuf/mbedTLS/libsodium/asio/sqlite3)
- FrameCodec feed+emit, encode/decode_envelope, protobuf codegen
- TcpServerConn with blocking TLS handshake thread + tls_read_loop
- TlsContext (mbedTLS 1.3, ECDSA-P256 self-signed cert, TOFU on client)
- WorkerPool (3 threads, used for Argon2id)
- Database: SQLite + libsodium Argon2id, account lifecycle, bootstrap admin
- ServerIdentityManager: Ed25519 key + cert generate/persist/fingerprint
- ConnSession state machine: WaitingHello -> WaitingAuth -> Authenticated
- SessionRegistry: channel tree, user map, text routing, broadcast
- vc_client full M1 C ABI: connect/TLS/handshake/auth/text/disconnect
- voicecat-admin CLI: account add/reset/del/list
- test_m1_integration: M1 exit criterion, verified green
Bug fixed: double-framing in ConnSession::send_envelope -- encode_envelope
was adding the [4-byte len] prefix, then TcpServerConn::send_frame added
a second one, causing the client to parse [len][proto] as protobuf (silent
failure). Fixed by serializing raw protobuf bytes in send_envelope and
letting send_frame apply the single length prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 23:48:44 +02:00
|
|
|
|
## M1 — Control plane ✓ (completed 2026-06-15)
|
|
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**Exit criterion:** `test_m1_integration` — two clients authenticate over TLS 1.3 (guest +
|
|
|
|
|
|
Argon2id), exchange channel + private text. ~1 s.
|
feat(M1): TCP/TLS control plane -- auth, channels, ephemeral text
Implements the full M1 milestone. Two clients authenticate over TLS 1.3
(guest + Argon2id password) and exchange channel + private text messages
through a real server. All five ctest --preset m1-dev tests pass in ~1 s.
Key components added:
- vcpkg baseline + m1-dev preset (protobuf/mbedTLS/libsodium/asio/sqlite3)
- FrameCodec feed+emit, encode/decode_envelope, protobuf codegen
- TcpServerConn with blocking TLS handshake thread + tls_read_loop
- TlsContext (mbedTLS 1.3, ECDSA-P256 self-signed cert, TOFU on client)
- WorkerPool (3 threads, used for Argon2id)
- Database: SQLite + libsodium Argon2id, account lifecycle, bootstrap admin
- ServerIdentityManager: Ed25519 key + cert generate/persist/fingerprint
- ConnSession state machine: WaitingHello -> WaitingAuth -> Authenticated
- SessionRegistry: channel tree, user map, text routing, broadcast
- vc_client full M1 C ABI: connect/TLS/handshake/auth/text/disconnect
- voicecat-admin CLI: account add/reset/del/list
- test_m1_integration: M1 exit criterion, verified green
Bug fixed: double-framing in ConnSession::send_envelope -- encode_envelope
was adding the [4-byte len] prefix, then TcpServerConn::send_frame added
a second one, causing the client to parse [len][proto] as protobuf (silent
failure). Fixed by serializing raw protobuf bytes in send_envelope and
letting send_frame apply the single length prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 23:48:44 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
`FrameCodec`, `TlsContext` (mbedTLS 1.3, ECDSA-P256 self-signed, TOFU pins TLS leaf-cert
|
|
|
|
|
|
SHA-256), `WorkerPool`, `Database` (SQLite + Argon2id), `ServerIdentityManager`,
|
|
|
|
|
|
`ConnSession` state machine, `SessionRegistry`, `vc_client` full M1 C ABI, `voicecat-admin`
|
|
|
|
|
|
CLI, dual-stack `TcpAcceptor`. **Key bug fixed:** `send_frame` double-framing — `encode_envelope`
|
|
|
|
|
|
was pre-framing the protobuf; fixed by passing raw protobuf bytes.
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## M2 — Voice, single stream ✓ (completed 2026-06-16)
|
|
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**Exit criterion:** `test_m2_voice` + `test_voice_client_abi` — two headless clients auth, bind
|
|
|
|
|
|
UDP, 50 Opus frames relayed + re-encrypted by SFU, B receives ≥25 and decrypts. ~4 s.
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
14-byte UDP voice header, `SodiumMediaCrypto` (ChaCha20-Poly1305 + 64-bit anti-replay),
|
|
|
|
|
|
`OpusEncoder`/`OpusDecoder` (FEC, PLC), `UdpMediaChannel`, `JitterBuffer`, `AudioEngine`
|
|
|
|
|
|
(miniaudio), `MediaRelay` SFU. **Key bug fixed:** `on_playback` passed hardware callback frame
|
|
|
|
|
|
count as `opus_decode` max_samples; fixed with a per-stream decode ring buffer.
|
feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).
Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
the same user silently overwrote the first in SessionRegistry::set_user_stream.
Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
and receive (sync_remote_streams) paths via a shared
opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
(total samples instead of samples-per-channel), which would have overflowed
the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
and from disconnect() on a different thread -- both could see
udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
same std::thread (intermittent std::system_error under ctest). Fixed with a
teardown_mu_ guard instead of carrying the flake forward.
New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
handle_stream_announce enforces the channel's config, clamping (not
overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
request_id-correlated announce/result handling (request_id already
round-tripped on the wire; just wasn't read before). on_capture_frame is
kind-aware and upmixes mono capture to stereo when a stream's config calls
for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
through set_remote_stream. New run_talk_timer() thread emits
VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
(inject_capture), stereo-to-mono downmix at the decode/mix boundary,
RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
and last_voice_ms/talking; new set_stream_noise_reduction() and
poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
vc_get_stream_audio_config (effective Opus config for any stream you own or
a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
(mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
concurrent local streams, independent gain/mute/NS control, per-channel
config divergence via vc_get_stream_audio_config, talk indicators.
Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).
ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
|
|
|
|
## M3 — Multi-stream & per-channel tuning ✓ (completed 2026-06-16)
|
|
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**Exit criterion:** `test_m3_multistream` — client A runs two concurrent streams (MIC +
|
|
|
|
|
|
SCREEN_AUDIO); B sees both; per-stream gain/mute/NR independent; effective Opus config matches
|
|
|
|
|
|
channel's server-enforced settings. ~2.4 s.
|
feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).
Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
the same user silently overwrote the first in SessionRegistry::set_user_stream.
Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
and receive (sync_remote_streams) paths via a shared
opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
(total samples instead of samples-per-channel), which would have overflowed
the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
and from disconnect() on a different thread -- both could see
udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
same std::thread (intermittent std::system_error under ctest). Fixed with a
teardown_mu_ guard instead of carrying the flake forward.
New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
handle_stream_announce enforces the channel's config, clamping (not
overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
request_id-correlated announce/result handling (request_id already
round-tripped on the wire; just wasn't read before). on_capture_frame is
kind-aware and upmixes mono capture to stereo when a stream's config calls
for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
through set_remote_stream. New run_talk_timer() thread emits
VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
(inject_capture), stereo-to-mono downmix at the decode/mix boundary,
RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
and last_voice_ms/talking; new set_stream_noise_reduction() and
poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
vc_get_stream_audio_config (effective Opus config for any stream you own or
a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
(mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
concurrent local streams, independent gain/mute/NS control, per-channel
config divergence via vc_get_stream_audio_config, talk indicators.
Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).
ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
Fixed server `stream_id` counter bug (always wrote `1`). Per-channel `AudioConfig` populated
|
|
|
|
|
|
(Lobby: mono/24kbps/VOIP + DTX; Music Room: stereo/128kbps/AUDIO). `LocalStream` map,
|
|
|
|
|
|
`pending_announce_kind_`, `run_talk_timer()`, thread-join race in `teardown_voice()` fixed.
|
|
|
|
|
|
New C ABI: `vc_get_stream_audio_config`, `vc_test_inject_capture`.
|
feat(M3): multi-stream & per-channel tuning
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).
Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
the same user silently overwrote the first in SessionRegistry::set_user_stream.
Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
and receive (sync_remote_streams) paths via a shared
opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
(total samples instead of samples-per-channel), which would have overflowed
the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
and from disconnect() on a different thread -- both could see
udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
same std::thread (intermittent std::system_error under ctest). Fixed with a
teardown_mu_ guard instead of carrying the flake forward.
New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
handle_stream_announce enforces the channel's config, clamping (not
overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
request_id-correlated announce/result handling (request_id already
round-tripped on the wire; just wasn't read before). on_capture_frame is
kind-aware and upmixes mono capture to stereo when a stream's config calls
for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
through set_remote_stream. New run_talk_timer() thread emits
VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
(inject_capture), stereo-to-mono downmix at the decode/mix boundary,
RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
and last_voice_ms/talking; new set_stream_noise_reduction() and
poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
vc_get_stream_audio_config (effective Opus config for any stream you own or
a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
(mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
concurrent local streams, independent gain/mute/NS control, per-channel
config divergence via vc_get_stream_audio_config, talk indicators.
Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).
ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 14:12:37 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
## Post-M3 follow-up ✓ (completed 2026-06-16)
|
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
- **Device enumeration** — `vc_list_devices`/`vc_set_input_device`; opaque hex device ids;
|
|
|
|
|
|
`vc_free_device_list` now frees. Works pre-connect.
|
|
|
|
|
|
- **VAD/PTT gate** — `EnergyVadProcessor` (RMS threshold ~0.025, 300 ms hang-time);
|
|
|
|
|
|
`vc_set_input_mode`/`vc_set_push_to_talk`; MIC-only (SCREEN_AUDIO/AUX_DEVICE bypass).
|
|
|
|
|
|
- **True stereo playback** — `playback_channels=2`; stereo decoded L→L R→R in mixer; mono
|
|
|
|
|
|
upmixed L=R; hardware fallback to mono on failure.
|
|
|
|
|
|
- **WASAPI loopback** — `loopback_device_` with `ma_device_type_loopback`;
|
|
|
|
|
|
`VOICECAT_HAS_LOOPBACK` macro (Windows-only). `vccli --share-screen-audio`.
|
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**Known deferred (still open):** AEC/NS/AGC (no working Windows/MSVC WebRTC APM build);
|
|
|
|
|
|
process-specific WASAPI loopback; RT-thread rule violation in `on_capture_frame` (mutex lock
|
|
|
|
|
|
on audio callback thread — pre-existing, needs lock-free ring-buffer refactor).
|
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
## M4 — Native clients ✓ (completed 2026-06-17 – 2026-06-19)
|
2026-06-17 00:52:02 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**Exit criterion:** `ctest --preset dev` 21/21 green; `dotnet build` 0 warnings; `xcodebuild`
|
|
|
|
|
|
BUILD SUCCEEDED (macOS + iOS); manually verified: connect, TOFU, channel tree, join, voice,
|
|
|
|
|
|
text, device pickers, level meter on each platform.
|
2026-06-17 00:52:02 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**New C ABI (additive):** `vc_list_channels`/`vc_list_users`/`vc_list_user_streams`,
|
|
|
|
|
|
`vc_join_channel`, `VC_EVENT_SERVER_IDENTITY` + `vc_confirm_server_identity`,
|
|
|
|
|
|
`vc_config::tofu_store_path`, `VC_INPUT_ALWAYS_ON`, `vc_set_vad_threshold`,
|
|
|
|
|
|
`vc_audio_suspend`/`vc_audio_resume`, `vc_audio_restart`, `vc_set_capture_channels`.
|
2026-06-17 00:52:02 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**Windows** (`clients/windows/`): `VoiceCat.Interop` (P/Invoke, `[UnmanagedCallersOnly]`),
|
|
|
|
|
|
`VoiceCat.App` (ConnectDialog, ServerIdentityDialog, MainForm with full M5 moderation UI,
|
|
|
|
|
|
PerUserTuningDialog, PttKeyCaptureDialog), `VoiceCat.Interop.Tests`. PTT is focus-scoped.
|
2026-06-17 00:52:02 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**macOS** (`clients/apple/macOS/VoiceCatMac.xcodeproj`): NSOutlineView channel tree,
|
|
|
|
|
|
NSTableView user list, NSTextView chat, voice controls, full VoiceOver accessibility, admin
|
|
|
|
|
|
menu, 17 Swift source files. `build-xcframework.sh` produces `VoiceCatCore.xcframework`.
|
2026-06-17 00:52:02 +02:00
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
**iOS** (`clients/apple/iOS/VoiceCatiOS.xcodeproj`): SwiftUI, `NavigationSplitView`/`TabView`,
|
|
|
|
|
|
`OutlineGroup` channel tree, `IOSAudioRouter` AVAudioSession driver, 24 Swift source files,
|
|
|
|
|
|
iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sharing.
|
2026-06-17 00:52:02 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
M5: moderation, permissions, channel CRUD, in-app account management
- Server-side moderation & permissions (kick/ban/move/server-mute, channel CRUD).
- Database schema v2: channels, bans; BLAKE2b channel passwords, Argon2id accounts.
- C ABI additions and client-side handling (vc_kick_user, vc_ban_user, vc_set_permission, vc_set_server_mute, vc_move_user, vc_create/edit/delete_channel, vc_create/reset/delete/list_account).
- vccli flags for all M5 operations plus --username/--password auth.
- Four new tests covering permissions, kick/ban/move/mute, admin accounts, channel CRUD.
- Docs: protocol.md envelope updates, security.md channel-password hashing, PROGRESS.md.
2026-06-17 15:08:05 +02:00
|
|
|
|
## M5 — Moderation, polish, and beyond [~] (in progress 2026-06-17)
|
|
|
|
|
|
|
|
|
|
|
|
**Exit criterion:** four ABI-level tests green (`test_m5_permissions`,
|
|
|
|
|
|
`test_m5_kick_ban_move_mute`, `test_m5_admin_accounts`, `test_m5_channel_crud`);
|
|
|
|
|
|
`vccli` can drive all moderation/admin/channel operations against a live server.
|
|
|
|
|
|
|
2026-06-20 13:14:39 +02:00
|
|
|
|
- [x] **Server-side moderation & permissions** — per-session `Permissions`, kick/ban/move/
|
|
|
|
|
|
server-mute, channel CRUD, DB schema v2 (`channels`, `bans`), BLAKE2b channel passwords.
|
|
|
|
|
|
- [x] **C ABI** — `vc_kick_user`, `vc_ban_user`, `vc_set_permission`, `vc_set_server_mute`,
|
|
|
|
|
|
`vc_move_user`, `vc_create_channel`, `vc_edit_channel`, `vc_delete_channel`,
|
|
|
|
|
|
`vc_create_account`, `vc_reset_password`, `vc_delete_account`, `vc_list_accounts`,
|
|
|
|
|
|
`vc_get_permissions`; events `VC_EVENT_GENERIC_RESULT`, `VC_EVENT_ACCOUNT_LIST`.
|
|
|
|
|
|
- [x] **Four M5 tests** passing — `ctest --preset dev` 21/21.
|
|
|
|
|
|
- [x] **vccli** M5 flags: `--kick`, `--ban`, `--move`, `--server-mute`/`-unmute`/`-deafen`/
|
|
|
|
|
|
`-undeafen`, `--set-permission`, channel CRUD, account CRUD, `--username`/`--password`.
|
|
|
|
|
|
- [x] **All three client UIs** (Windows WinForms, macOS AppKit, iOS SwiftUI) expose the full
|
|
|
|
|
|
M5 moderation and admin surface.
|
|
|
|
|
|
- [x] **Docs** — `docs/protocol.md`, `docs/security.md` kept in sync.
|
|
|
|
|
|
- [ ] **DRED/audio-quality polish** — not started.
|
|
|
|
|
|
- [ ] **macOS ScreenCaptureKit screen-audio** — `startStream(.screenAudio)` in macOS client
|
|
|
|
|
|
announces the stream but `start_loopback_capture()` returns false (no `VOICECAT_HAS_LOOPBACK`
|
|
|
|
|
|
on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships.
|
|
|
|
|
|
- [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target,
|
|
|
|
|
|
App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`.
|
|
|
|
|
|
- [ ] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — see full
|
|
|
|
|
|
plan in "Where we left off" above.
|
M5: moderation, permissions, channel CRUD, in-app account management
- Server-side moderation & permissions (kick/ban/move/server-mute, channel CRUD).
- Database schema v2: channels, bans; BLAKE2b channel passwords, Argon2id accounts.
- C ABI additions and client-side handling (vc_kick_user, vc_ban_user, vc_set_permission, vc_set_server_mute, vc_move_user, vc_create/edit/delete_channel, vc_create/reset/delete/list_account).
- vccli flags for all M5 operations plus --username/--password auth.
- Four new tests covering permissions, kick/ban/move/mute, admin accounts, channel CRUD.
- Docs: protocol.md envelope updates, security.md channel-password hashing, PROGRESS.md.
2026-06-17 15:08:05 +02:00
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
scaffold: M0 skeleton + agent onboarding (build, architecture, progress)
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.
Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format
Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders
Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"
Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:09:09 +02:00
|
|
|
|
## Decisions log
|
|
|
|
|
|
|
|
|
|
|
|
All architecture/scope decisions are settled and recorded in
|
|
|
|
|
|
[docs/roadmap.md §2 "Resolved decisions"](docs/roadmap.md) and reflected across `docs/`.
|
|
|
|
|
|
If you make a *new* decision, record it there and link it here.
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## How to update this file
|
|
|
|
|
|
|
|
|
|
|
|
1. Check off tasks as you complete them; flip a milestone to `[x]` only when its **exit
|
|
|
|
|
|
criterion test** passes.
|
|
|
|
|
|
2. Keep the **"Where we left off / next action"** block at the top accurate — it's the first
|
|
|
|
|
|
thing the next agent reads.
|
|
|
|
|
|
3. When you start a milestone, copy its task list from `docs/roadmap.md` into a section here.
|