docs(progress): plan external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink)

PROGRESS.md

@@ -10,6 +10,168 @@ up instantly. Newest status at the top.
 
 ## ▶ Where we left off / next action
 
+- **Planned (not started):** **External PCM feed/tap API (`vc_stream_feed_pcm` +
+  `vc_set_pcm_sink`)** (2026-06-19, plan written on Windows; implement on Mac). A public,
+  documented API for driving audio streams with externally-provided PCM instead of (or in
+  addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our
+  roadmap — that the current "miniaudio owns the device" model can't serve:
+  1. **ReplayKit Broadcast Upload Extension (iOS `SCREEN_AUDIO`)** — the extension is a
+     *separate process* with a ~50 MB memory cap and can't link the full `AudioEngine`
+     (`ma_device`, capture/playback threads). It needs to feed `CMSampleBuffer` audio (system
+     app audio) into the encode path without any audio hardware. The current plan in
+     `docs/voice.md §9` says the extension links "a minimal slice of the core (Opus encode +
+     media send only)" — a public feed-PCM API *is* that minimal slice. The extension links
+     Opus + the feed entry point, no `ma_device` needed.
+  2. **ScreenCaptureKit (macOS `SCREEN_AUDIO`)** — `SCStream` delivers `CMSampleBuffer` in a
+     callback; convert to int16 and feed. No need to route through miniaudio's device layer.
+     **This is how macOS screen-audio actually gets implemented** — today it does NOT work:
+     `VOICECAT_HAS_LOOPBACK` is Windows-only (`core/CMakeLists.txt:88-95`), so on macOS
+     `AudioEngine::start_loopback_capture()` hits the `#else` stub (`audio_engine.cpp:647-649`)
+     and returns `false`. The macOS client's "Share Screen Audio" button
+     (`MainWindowController.swift:800-816`) calls `startStream(.screenAudio)` which announces
+     the stream to peers but captures **zero audio** — peers hear silence. The button is left
+     in place (not touched per user request); it'll work once this API + a ScreenCaptureKit
+     tap ship on Mac.
+  3. **Bots** — music bot, TTS bot, radio relay, transcription bot. They create a
+     `SCREEN_AUDIO`/`AUX_DEVICE` stream and feed synthesized or decoded PCM via the feed API.
+     No audio hardware required — runs headless on a server. Today the only way to feed
+     external PCM is `vc_test_inject_capture` (TEST-ONLY, name signals "don't ship this") or
+     re-implementing Opus encode + AEAD + UDP framing yourself (~500 lines of duplicated
+     crypto/codec code per consumer).
+  4. **Custom clients / accessibility** — soundboard, DAW integration, TTS of incoming chat,
+     recording/transcription of remote audio. Need either feed (send) or tap (receive) or
+     both.
+
+  **What we already have (input half, gated as test-only):** `vc_test_inject_capture
+  (stream_id, pcm, samples)` (`voicecat.h`, `client.cpp:1452`) feeds raw int16 PCM into the
+  encode pipeline via `AudioEngine::inject_capture(kind, pcm, n)`. It works for any stream
+  kind, supports multiple concurrent injection taps (one ring buffer per local kind), and
+  goes through the full encode → AEAD → UDP path. The encode path already handles
+  `channels == 1 || 2` (proven by the WASAPI stereo loopback work, 2026-06-17 entry below).
+  The only problems: it's marked TEST-ONLY in the header, the name signals "don't use this in
+  production," and it hardcodes mono (no `channels` parameter).
+
+  **What's missing (output half):** today decoded remote audio is mixed and pushed to the
+  miniaudio playback device (`on_playback`). There's no way for an external consumer to
+  intercept the decoded PCM of a specific remote stream — it all goes to the hardware device.
+  A bot that wants to record, transcribe, or re-broadcast remote audio has no hook.
+
+  **Plan (API design — clean, append-only, no struct changes, ABI-stable):**
+  - **`vc_stream_feed_pcm`** — promote `vc_test_inject_capture` to a public, documented API
+    and add a `channels` parameter:
+    ```c
+    /* External PCM feed — replaces the hardware capture device for this stream. Caller
+       provides interleaved int16 PCM at the stream's sample rate. The core frames it,
+       encodes (Opus), seals (AEAD), and sends (UDP). Works for any stream kind
+       (MIC/SCREEN_AUDIO/AUX_DEVICE). The stream must be started first (vc_stream_start);
+       this just replaces the capture source. channels = 1 (mono) or 2 (stereo interleaved).
+       Thread-safe; may be called from any thread including audio callbacks. */
+    vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id,
+                                 const int16_t* pcm, size_t samples_per_channel,
+                                 uint32_t channels);
+    ```
+  - **`vc_set_pcm_sink`** — symmetric output side: receive decoded remote audio as int16 PCM
+    instead of (or in addition to) the hardware playback device:
+    ```c
+    /* External PCM tap — receive decoded, mixed remote audio as int16 PCM. The callback
+       fires on the audio thread with the mixed output for a specific remote stream. Pass
+       cb=NULL to disable (default: disabled, hardware playback only). When enabled, PCM is
+       delivered to the sink AND the hardware device (dual output) so a bot can record
+       without disabling local monitoring. user_id+stream_id identify the source stream.
+       The callback MUST NOT block — copy what you need and return (same contract as
+       vc_callbacks.on_event). */
+    typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
+                                   const int16_t* pcm, size_t samples_per_channel,
+                                   uint32_t channels, uint32_t sample_rate);
+    vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
+    ```
+  - **Core changes:**
+    - `core/include/voicecat.h` — add `vc_pcm_sink_cb` typedef + the two function
+      declarations (append-only, after `vc_test_inject_capture`). Full doc comments on both
+      (contract, thread-safety, lifetime, use cases).
+    - `core/src/voicecat.cpp` — thin C trampolines → `vc_client::stream_feed_pcm` /
+      `set_pcm_sink`.
+    - `core/src/core/client.{h,cpp}` — `stream_feed_pcm`: validates `stream_id`, looks up
+      the `LocalStream`'s kind, calls `audio_engine_.inject_capture(kind, pcm, n)` (existing
+      path) with the channel count forwarded. `set_pcm_sink`: stores the callback + user
+      pointer; `on_playback` (or a new fan-out in the mixer) invokes it per remote stream
+      alongside the existing hardware write. Keep `vc_test_inject_capture` as a deprecated
+      alias calling `stream_feed_pcm(..., channels=1)` for source compatibility.
+    - `core/src/audio/audio_engine.{h,cpp}` — `inject_capture` already exists per-kind; add
+      a `channels` parameter to the ring-buffer write path (or a parallel stereo-aware
+      variant). The encode path in `client.cpp::on_capture_frame` already handles
+      `channels==2` via the stereo encode branch — just plumb the value through. For the
+      sink: add a `pcm_sink_` member (callback + user); in `on_playback` after mixing, if the
+      sink is set, copy the mixed PCM for the current stream and invoke the callback. The
+      copy must stay off the RT-critical path — document the non-blocking contract.
+  - **Skeleton stub path:** update `client.cpp`'s `#else` (no-deps) stub section to add
+    `vc_stream_feed_pcm`/`vc_set_pcm_sink` returning `VC_ERR_NOT_IMPLEMENTED` — keeps the
+    skeleton preset green.
+  - **Swift `VoiceCatCore`:** add `feedPcm(streamId:pcm:samplesPerChannel:channels:)` and
+    `setPcmSink(_:user:)` (the Swift wrapper around `vc_pcm_sink_cb` — a
+    `@convention(c)` closure + `Unmanaged` context, mirroring `Callbacks.swift`). Wraps both
+    new ABI functions.
+  - **C# `VoiceCat.Interop`:** add `StreamFeedPcm(streamId, pcm, samples, channels)` (with
+    `int16[]` marshaling) and `SetPcmSink` (delegates via `[UnmanagedCallersOnly]` thunk,
+    mirroring the event-callback pattern). Wraps both new ABI functions.
+  - **Tests:**
+    - `tests/test_external_pcm.cpp` (new) — `test_feed_pcm_round_trip`: two clients, A feeds
+      a known mono sine wave via `vc_stream_feed_pcm` on a MIC stream, B receives via the
+      normal decode path and asserts energy matches. `test_feed_pcm_stereo`: same with
+      `channels=2`, assert L≠R end-to-end (mirrors the WASAPI loopback stereo test).
+      `test_pcm_sink`: B sets a `vc_pcm_sink_cb`, A feeds PCM, assert the sink callback
+      receives the decoded PCM with matching energy. All headless, no audio hardware.
+    - `clients/apple/Tests/VoiceCatCoreTests/` — Swift wrapper round-trip for `feedPcm`.
+    - `clients/windows/VoiceCat.Interop.Tests/` — C# wrapper round-trip.
+  - **Docs:**
+    - `docs/architecture.md §4` — new subsection on external PCM feed/tap: the contract
+      (caller provides interleaved int16 at the stream's sample rate; core frames/encodes/
+      seals/sends for feed; core decodes/mixes/delivers for sink; sink callback must not
+      block), the use cases (ReplayKit, ScreenCaptureKit, bots, custom clients), and the
+      relationship to `vc_test_inject_capture` (deprecated alias).
+    - `docs/voice.md §9` — update the iOS ReplayKit and macOS ScreenCaptureKit rows: both
+      now consume `vc_stream_feed_pcm` instead of a "minimal slice of the core." Update the
+      iOS detail bullets: the extension links Opus + `vc_stream_feed_pcm` (not a parallel
+      media stack). Add a macOS ScreenCaptureKit note: convert `CMSampleBuffer` → int16,
+      feed via `vc_stream_feed_pcm` — this is how macOS screen-audio actually ships.
+    - `docs/protocol.md` — no protocol changes (the feed/sink are client-local; the wire
+      format is identical whether PCM came from miniaudio or an external source). Note this
+      explicitly.
+    - `docs/roadmap.md` — add a milestone entry; update the iOS ReplayKit and macOS
+      ScreenCaptureKit pending items to reference `vc_stream_feed_pcm`.
+  - **Implementation order:**
+    1. C ABI + core (`voicecat.h`, `voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`) +
+       skeleton stub. Verify `ctest --preset dev` green.
+    2. `tests/test_external_pcm.cpp` — the three behavior tests. Verify green.
+    3. Swift `VoiceCatCore` wrapper + `VoiceCatCoreTests` round-trip.
+    4. C# `VoiceCat.Interop` wrapper + `VoiceCatClientSmokeTests` round-trip.
+    5. Docs (`architecture.md`, `voice.md`, `protocol.md`, `roadmap.md`, header comments).
+    6. **Then** ReplayKit (iOS) and ScreenCaptureKit (macOS) become ~100-line consumers of
+       this API instead of parallel media stacks.
+  - **Verification:** `ctest --preset dev` green (3 new tests); `swift test` green; `dotnet
+    test` green; `xcodebuild` (skeleton) green. The feed/sink tests are fully headless — no
+    audio hardware, no simulator, no device — so they run in CI on every platform.
+  - **Files to touch:**
+    - Core C++: `core/include/voicecat.h`, `core/src/voicecat.cpp`,
+      `core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`.
+    - Tests: `tests/test_external_pcm.cpp` (new), `tests/CMakeLists.txt`.
+    - Swift: `clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift`,
+      `clients/apple/Sources/VoiceCatCore/Callbacks.swift`,
+      `clients/apple/Tests/VoiceCatCoreTests/ExternalPcmTests.swift` (new).
+    - C#: `clients/windows/VoiceCat.Interop/VoiceCatClient.cs`,
+      `clients/windows/VoiceCat.Interop/NativeMethods.cs`,
+      `clients/windows/VoiceCat.Interop.Tests/ExternalPcmTests.cs` (new).
+    - Docs: `docs/architecture.md`, `docs/voice.md`, `docs/protocol.md`, `docs/roadmap.md`.
+  - **ABI stability:** append-only — two new functions + one new typedef, no existing
+    structs/enums changed. `vc_test_inject_capture` stays as a deprecated alias for source
+    compatibility. Treat as a deliberate, versioned ABI event per `docs/protocol.md §8`.
+  - **Relationship to the iOS audio routing plan (entry below):** orthogonal. That plan is
+    about *which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan
+    is about *bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship;
+    they don't conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer
+    controls the *mic* path which still uses miniaudio's device.
+
+
 - **Planned (not started):** **iOS audio overhaul + Join/Leave Voice + channel-id sync fix**
   (2026-06-19, plan written on Windows; implement on Mac). Three problems found while reviewing
   the iOS client: