feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink)

Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public,
stereo-capable production API and adds a symmetric PCM tap on the
receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots,
soundboards, and custom clients — all without a hardware audio device.

Core C++:
- voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef,
  vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias
- audio_engine: stereo-aware inject_capture (channels param + ring
  reset on channel-count change); atomic pcm_sink_ fired per decoded
  frame in on_playback; RemoteStream carries user_id/stream_id for
  RT-safe sink metadata; init_recv_stream takes user_id+stream_id
- client.cpp: stream_feed_pcm / set_pcm_sink implementations;
  sync_remote_streams passes user_id/stream_id to init_recv_stream
- voicecat.cpp: trampolines + channels=1/2 validation

Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip,
stereo feed L≠R, sink metadata+disable). ctest 23/23.

Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest
smoke tests (ExternalPcmTests.swift).

C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs
(vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate,
vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs).

Docs: architecture.md §4 new subsection, voice.md §9 updated
(macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit
no-protocol-change note, roadmap.md M5 entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-20 17:52:09 +02:00
parent 540ec13a63
commit 615d2a8e5f
21 changed files with 891 additions and 39 deletions

View File

@@ -155,6 +155,26 @@ Design notes:
`PostMessage` with no meaningful latency cost. `VoiceCatClientHandle : SafeHandle` wraps
the `vc_client*` and guarantees `vc_client_destroy` runs on GC/Dispose.
### External PCM feed/tap
Two API functions let callers bypass miniaudio entirely for a stream:
| Function | Direction | Contract |
|----------|-----------|----------|
| `vc_stream_feed_pcm(c, stream_id, pcm, samples_per_channel, channels)` | **Send** — caller → network | Caller supplies interleaved int16 at the stream's sample rate (`channels` = 1 mono, 2 stereo). The core frames, Opus-encodes, AEAD-seals, and sends over UDP — identical wire path to hardware capture. The stream must already be started with `vc_stream_start`. Thread-safe; may be called from any thread (audio callback, ReplayKit delegate, SCStream callback). |
| `vc_set_pcm_sink(c, cb, user)` | **Receive** — network → caller | `cb` is called on the audio (playback) thread once per decoded Opus frame per remote stream, with `(user_id, stream_id, pcm, samples_per_channel, channels, sample_rate)`. PCM is delivered to the sink **and** the hardware device — dual output; the hardware mix is unaffected. Pass `cb=NULL` to disable (default). **Must not block** — copy what you need and return. |
`vc_test_inject_capture` (the old TEST-ONLY mono-only predecessor) is a deprecated alias
for `vc_stream_feed_pcm(..., channels=1)` — kept for source compatibility.
**Use cases:** ReplayKit Broadcast Extension (iOS `SCREEN_AUDIO`), ScreenCaptureKit (macOS
`SCREEN_AUDIO`), music/TTS/relay bots, soundboards, transcription clients. The extension or
bot links Opus + the feed entry point — no `ma_device`, no hardware, headless.
**Threading:** the feed path is thread-safe (ring buffer, no lock on the RT path). The sink
callback runs on the miniaudio playback thread — observe the same rules as the capture
callback: no allocations, no blocking calls.
## 5. Server architecture
`voicecat-server` is a headless process linking the core.

View File

@@ -309,7 +309,21 @@ message TextMessage {
disconnect; server-sent fatal `Disconnect` uses `code ≥ 1` (1 = protocol error,
2 = kicked).
## 8. Extensibility checklist
## 8. Client-local features (no protocol changes)
Some features are entirely client-side and involve no changes to the wire format:
- **External PCM feed (`vc_stream_feed_pcm`)** — the caller supplies interleaved int16 PCM
that the core frames, encodes, and sends over the existing UDP media path. From the server
and peers' perspective the stream is indistinguishable from a hardware-captured stream. No
new messages, fields, or tags are needed.
- **PCM tap (`vc_set_pcm_sink`)** — receives decoded per-stream audio before hardware mixing.
Entirely local to the listener; no protocol traffic of any kind.
These are noted here to prevent future contributors from looking for corresponding protocol
changes: there are none.
## 9. Extensibility checklist
When adding a feature later (e.g. **file transfer**), the rules are:

View File

@@ -69,7 +69,8 @@ exists from M1 so the protocol can be exercised long before any GUI.
- ~~AVAudioSession, mic permission, foreground voice.~~ ✓ Done — `IOSAudioRouter` drives
all iOS audio routing (input ports, orientation/polar patterns, HFP/A2DP, Standard/Raw
mic mode, stereo capture), `vc_audio_suspend`/`vc_audio_resume` for interruptions.
- ReplayKit broadcast extension for `SCREEN_AUDIO`.
- ReplayKit broadcast extension for `SCREEN_AUDIO` — feeds `CMSampleBuffer` audio via
`vc_stream_feed_pcm` (see architecture.md §4).
**Exit:** non-technical user installs a client, saves a server, and joins.
@@ -78,6 +79,13 @@ exists from M1 so the protocol can be exercised long before any GUI.
(channel CRUD with full Opus config, user moderation, server account management); macOS/iOS
Swift UI pending.
- DRED toggle, audio-quality polish. (AEC and VAD/PTT already shipped in M2.)
- **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) ✓ shipped
(2026-06-20) — promotes `vc_test_inject_capture` to a public, stereo-capable API and adds
a symmetric PCM sink. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, and custom
clients. See architecture.md §4 "External PCM feed/tap" and protocol.md §8 for the
no-protocol-change rationale. One new C++ ctest binary (`test_external_pcm`) covering 3
sub-tests (`test_feed_pcm_round_trip`, `test_feed_pcm_stereo`, `test_pcm_sink`) — ctest
23/23; Swift wrapper + 4 XCTest smoke tests; C# wrapper + 4 xUnit smoke tests.
- **Then (post-v1, protocol already reserves space):** file transfer, E2EE option,
CallKit/PushKit background voice, key-based identity, server-side text history,
multi-node server.

View File

@@ -274,8 +274,8 @@ normal stream; only the *source* is platform-specific.
| Platform | Mechanism | Notes |
|----------|-----------|-------|
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | OS requires screen-recording permission; capture happens in the main app. Swift converts each `CMSampleBuffer` (Float32) → int16 and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). The extension calls `vc_stream_feed_pcm` to drive the encode path without a hardware device. |
### iOS detail
@@ -285,11 +285,11 @@ normal stream; only the *source* is platform-specific.
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
audio-only consumer stays comfortably inside it.
- The extension is a *separate process*. It links a **minimal slice of the core** (Opus
encode + media send only — not the full client), reads the active session token and
server endpoint from a shared **App Group** container that the host app wrote at join
time, derives its own media keys, and publishes the `SCREEN_AUDIO` stream directly. The
host app announces the stream over its control channel (`StreamAnnounce`) so the server and
- The extension is a *separate process*. It reads the active session token and server
endpoint from a shared **App Group** container, opens a UDP media channel, and calls
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the
Opus encode + AEAD + send path directly, with no `ma_device` or audio hardware. The host
app announces the stream over its control channel (`StreamAnnounce`) so the server and
peers learn about it.
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final