From a10a18aebede004b19d5190a1524e6c3b4038df0 Mon Sep 17 00:00:00 2001 From: Talon Date: Fri, 19 Jun 2026 03:51:14 +0200 Subject: [PATCH] docs(progress): plan external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink) --- PROGRESS.md | 162 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index e432011..1a35a0e 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -10,6 +10,168 @@ up instantly. Newest status at the top. ## ▶ Where we left off / next action +- **Planned (not started):** **External PCM feed/tap API (`vc_stream_feed_pcm` + + `vc_set_pcm_sink`)** (2026-06-19, plan written on Windows; implement on Mac). A public, + documented API for driving audio streams with externally-provided PCM instead of (or in + addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our + roadmap — that the current "miniaudio owns the device" model can't serve: + 1. **ReplayKit Broadcast Upload Extension (iOS `SCREEN_AUDIO`)** — the extension is a + *separate process* with a ~50 MB memory cap and can't link the full `AudioEngine` + (`ma_device`, capture/playback threads). It needs to feed `CMSampleBuffer` audio (system + app audio) into the encode path without any audio hardware. The current plan in + `docs/voice.md §9` says the extension links "a minimal slice of the core (Opus encode + + media send only)" — a public feed-PCM API *is* that minimal slice. The extension links + Opus + the feed entry point, no `ma_device` needed. + 2. **ScreenCaptureKit (macOS `SCREEN_AUDIO`)** — `SCStream` delivers `CMSampleBuffer` in a + callback; convert to int16 and feed. No need to route through miniaudio's device layer. + **This is how macOS screen-audio actually gets implemented** — today it does NOT work: + `VOICECAT_HAS_LOOPBACK` is Windows-only (`core/CMakeLists.txt:88-95`), so on macOS + `AudioEngine::start_loopback_capture()` hits the `#else` stub (`audio_engine.cpp:647-649`) + and returns `false`. The macOS client's "Share Screen Audio" button + (`MainWindowController.swift:800-816`) calls `startStream(.screenAudio)` which announces + the stream to peers but captures **zero audio** — peers hear silence. The button is left + in place (not touched per user request); it'll work once this API + a ScreenCaptureKit + tap ship on Mac. + 3. **Bots** — music bot, TTS bot, radio relay, transcription bot. They create a + `SCREEN_AUDIO`/`AUX_DEVICE` stream and feed synthesized or decoded PCM via the feed API. + No audio hardware required — runs headless on a server. Today the only way to feed + external PCM is `vc_test_inject_capture` (TEST-ONLY, name signals "don't ship this") or + re-implementing Opus encode + AEAD + UDP framing yourself (~500 lines of duplicated + crypto/codec code per consumer). + 4. **Custom clients / accessibility** — soundboard, DAW integration, TTS of incoming chat, + recording/transcription of remote audio. Need either feed (send) or tap (receive) or + both. + + **What we already have (input half, gated as test-only):** `vc_test_inject_capture + (stream_id, pcm, samples)` (`voicecat.h`, `client.cpp:1452`) feeds raw int16 PCM into the + encode pipeline via `AudioEngine::inject_capture(kind, pcm, n)`. It works for any stream + kind, supports multiple concurrent injection taps (one ring buffer per local kind), and + goes through the full encode → AEAD → UDP path. The encode path already handles + `channels == 1 || 2` (proven by the WASAPI stereo loopback work, 2026-06-17 entry below). + The only problems: it's marked TEST-ONLY in the header, the name signals "don't use this in + production," and it hardcodes mono (no `channels` parameter). + + **What's missing (output half):** today decoded remote audio is mixed and pushed to the + miniaudio playback device (`on_playback`). There's no way for an external consumer to + intercept the decoded PCM of a specific remote stream — it all goes to the hardware device. + A bot that wants to record, transcribe, or re-broadcast remote audio has no hook. + + **Plan (API design — clean, append-only, no struct changes, ABI-stable):** + - **`vc_stream_feed_pcm`** — promote `vc_test_inject_capture` to a public, documented API + and add a `channels` parameter: + ```c + /* External PCM feed — replaces the hardware capture device for this stream. Caller + provides interleaved int16 PCM at the stream's sample rate. The core frames it, + encodes (Opus), seals (AEAD), and sends (UDP). Works for any stream kind + (MIC/SCREEN_AUDIO/AUX_DEVICE). The stream must be started first (vc_stream_start); + this just replaces the capture source. channels = 1 (mono) or 2 (stereo interleaved). + Thread-safe; may be called from any thread including audio callbacks. */ + vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id, + const int16_t* pcm, size_t samples_per_channel, + uint32_t channels); + ``` + - **`vc_set_pcm_sink`** — symmetric output side: receive decoded remote audio as int16 PCM + instead of (or in addition to) the hardware playback device: + ```c + /* External PCM tap — receive decoded, mixed remote audio as int16 PCM. The callback + fires on the audio thread with the mixed output for a specific remote stream. Pass + cb=NULL to disable (default: disabled, hardware playback only). When enabled, PCM is + delivered to the sink AND the hardware device (dual output) so a bot can record + without disabling local monitoring. user_id+stream_id identify the source stream. + The callback MUST NOT block — copy what you need and return (same contract as + vc_callbacks.on_event). */ + typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id, + const int16_t* pcm, size_t samples_per_channel, + uint32_t channels, uint32_t sample_rate); + vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user); + ``` + - **Core changes:** + - `core/include/voicecat.h` — add `vc_pcm_sink_cb` typedef + the two function + declarations (append-only, after `vc_test_inject_capture`). Full doc comments on both + (contract, thread-safety, lifetime, use cases). + - `core/src/voicecat.cpp` — thin C trampolines → `vc_client::stream_feed_pcm` / + `set_pcm_sink`. + - `core/src/core/client.{h,cpp}` — `stream_feed_pcm`: validates `stream_id`, looks up + the `LocalStream`'s kind, calls `audio_engine_.inject_capture(kind, pcm, n)` (existing + path) with the channel count forwarded. `set_pcm_sink`: stores the callback + user + pointer; `on_playback` (or a new fan-out in the mixer) invokes it per remote stream + alongside the existing hardware write. Keep `vc_test_inject_capture` as a deprecated + alias calling `stream_feed_pcm(..., channels=1)` for source compatibility. + - `core/src/audio/audio_engine.{h,cpp}` — `inject_capture` already exists per-kind; add + a `channels` parameter to the ring-buffer write path (or a parallel stereo-aware + variant). The encode path in `client.cpp::on_capture_frame` already handles + `channels==2` via the stereo encode branch — just plumb the value through. For the + sink: add a `pcm_sink_` member (callback + user); in `on_playback` after mixing, if the + sink is set, copy the mixed PCM for the current stream and invoke the callback. The + copy must stay off the RT-critical path — document the non-blocking contract. + - **Skeleton stub path:** update `client.cpp`'s `#else` (no-deps) stub section to add + `vc_stream_feed_pcm`/`vc_set_pcm_sink` returning `VC_ERR_NOT_IMPLEMENTED` — keeps the + skeleton preset green. + - **Swift `VoiceCatCore`:** add `feedPcm(streamId:pcm:samplesPerChannel:channels:)` and + `setPcmSink(_:user:)` (the Swift wrapper around `vc_pcm_sink_cb` — a + `@convention(c)` closure + `Unmanaged` context, mirroring `Callbacks.swift`). Wraps both + new ABI functions. + - **C# `VoiceCat.Interop`:** add `StreamFeedPcm(streamId, pcm, samples, channels)` (with + `int16[]` marshaling) and `SetPcmSink` (delegates via `[UnmanagedCallersOnly]` thunk, + mirroring the event-callback pattern). Wraps both new ABI functions. + - **Tests:** + - `tests/test_external_pcm.cpp` (new) — `test_feed_pcm_round_trip`: two clients, A feeds + a known mono sine wave via `vc_stream_feed_pcm` on a MIC stream, B receives via the + normal decode path and asserts energy matches. `test_feed_pcm_stereo`: same with + `channels=2`, assert L≠R end-to-end (mirrors the WASAPI loopback stereo test). + `test_pcm_sink`: B sets a `vc_pcm_sink_cb`, A feeds PCM, assert the sink callback + receives the decoded PCM with matching energy. All headless, no audio hardware. + - `clients/apple/Tests/VoiceCatCoreTests/` — Swift wrapper round-trip for `feedPcm`. + - `clients/windows/VoiceCat.Interop.Tests/` — C# wrapper round-trip. + - **Docs:** + - `docs/architecture.md §4` — new subsection on external PCM feed/tap: the contract + (caller provides interleaved int16 at the stream's sample rate; core frames/encodes/ + seals/sends for feed; core decodes/mixes/delivers for sink; sink callback must not + block), the use cases (ReplayKit, ScreenCaptureKit, bots, custom clients), and the + relationship to `vc_test_inject_capture` (deprecated alias). + - `docs/voice.md §9` — update the iOS ReplayKit and macOS ScreenCaptureKit rows: both + now consume `vc_stream_feed_pcm` instead of a "minimal slice of the core." Update the + iOS detail bullets: the extension links Opus + `vc_stream_feed_pcm` (not a parallel + media stack). Add a macOS ScreenCaptureKit note: convert `CMSampleBuffer` → int16, + feed via `vc_stream_feed_pcm` — this is how macOS screen-audio actually ships. + - `docs/protocol.md` — no protocol changes (the feed/sink are client-local; the wire + format is identical whether PCM came from miniaudio or an external source). Note this + explicitly. + - `docs/roadmap.md` — add a milestone entry; update the iOS ReplayKit and macOS + ScreenCaptureKit pending items to reference `vc_stream_feed_pcm`. + - **Implementation order:** + 1. C ABI + core (`voicecat.h`, `voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`) + + skeleton stub. Verify `ctest --preset dev` green. + 2. `tests/test_external_pcm.cpp` — the three behavior tests. Verify green. + 3. Swift `VoiceCatCore` wrapper + `VoiceCatCoreTests` round-trip. + 4. C# `VoiceCat.Interop` wrapper + `VoiceCatClientSmokeTests` round-trip. + 5. Docs (`architecture.md`, `voice.md`, `protocol.md`, `roadmap.md`, header comments). + 6. **Then** ReplayKit (iOS) and ScreenCaptureKit (macOS) become ~100-line consumers of + this API instead of parallel media stacks. + - **Verification:** `ctest --preset dev` green (3 new tests); `swift test` green; `dotnet + test` green; `xcodebuild` (skeleton) green. The feed/sink tests are fully headless — no + audio hardware, no simulator, no device — so they run in CI on every platform. + - **Files to touch:** + - Core C++: `core/include/voicecat.h`, `core/src/voicecat.cpp`, + `core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`. + - Tests: `tests/test_external_pcm.cpp` (new), `tests/CMakeLists.txt`. + - Swift: `clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift`, + `clients/apple/Sources/VoiceCatCore/Callbacks.swift`, + `clients/apple/Tests/VoiceCatCoreTests/ExternalPcmTests.swift` (new). + - C#: `clients/windows/VoiceCat.Interop/VoiceCatClient.cs`, + `clients/windows/VoiceCat.Interop/NativeMethods.cs`, + `clients/windows/VoiceCat.Interop.Tests/ExternalPcmTests.cs` (new). + - Docs: `docs/architecture.md`, `docs/voice.md`, `docs/protocol.md`, `docs/roadmap.md`. + - **ABI stability:** append-only — two new functions + one new typedef, no existing + structs/enums changed. `vc_test_inject_capture` stays as a deprecated alias for source + compatibility. Treat as a deliberate, versioned ABI event per `docs/protocol.md §8`. + - **Relationship to the iOS audio routing plan (entry below):** orthogonal. That plan is + about *which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan + is about *bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship; + they don't conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer + controls the *mic* path which still uses miniaudio's device. + + - **Planned (not started):** **iOS audio overhaul + Join/Leave Voice + channel-id sync fix** (2026-06-19, plan written on Windows; implement on Mac). Three problems found while reviewing the iOS client: