feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink)
Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public, stereo-capable production API and adds a symmetric PCM tap on the receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, soundboards, and custom clients — all without a hardware audio device. Core C++: - voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef, vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias - audio_engine: stereo-aware inject_capture (channels param + ring reset on channel-count change); atomic pcm_sink_ fired per decoded frame in on_playback; RemoteStream carries user_id/stream_id for RT-safe sink metadata; init_recv_stream takes user_id+stream_id - client.cpp: stream_feed_pcm / set_pcm_sink implementations; sync_remote_streams passes user_id/stream_id to init_recv_stream - voicecat.cpp: trampolines + channels=1/2 validation Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip, stereo feed L≠R, sink metadata+disable). ctest 23/23. Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest smoke tests (ExternalPcmTests.swift). C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs (vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate, vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs). Docs: architecture.md §4 new subsection, voice.md §9 updated (macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -155,6 +155,26 @@ Design notes:
|
||||
`PostMessage` with no meaningful latency cost. `VoiceCatClientHandle : SafeHandle` wraps
|
||||
the `vc_client*` and guarantees `vc_client_destroy` runs on GC/Dispose.
|
||||
|
||||
### External PCM feed/tap
|
||||
|
||||
Two API functions let callers bypass miniaudio entirely for a stream:
|
||||
|
||||
| Function | Direction | Contract |
|
||||
|----------|-----------|----------|
|
||||
| `vc_stream_feed_pcm(c, stream_id, pcm, samples_per_channel, channels)` | **Send** — caller → network | Caller supplies interleaved int16 at the stream's sample rate (`channels` = 1 mono, 2 stereo). The core frames, Opus-encodes, AEAD-seals, and sends over UDP — identical wire path to hardware capture. The stream must already be started with `vc_stream_start`. Thread-safe; may be called from any thread (audio callback, ReplayKit delegate, SCStream callback). |
|
||||
| `vc_set_pcm_sink(c, cb, user)` | **Receive** — network → caller | `cb` is called on the audio (playback) thread once per decoded Opus frame per remote stream, with `(user_id, stream_id, pcm, samples_per_channel, channels, sample_rate)`. PCM is delivered to the sink **and** the hardware device — dual output; the hardware mix is unaffected. Pass `cb=NULL` to disable (default). **Must not block** — copy what you need and return. |
|
||||
|
||||
`vc_test_inject_capture` (the old TEST-ONLY mono-only predecessor) is a deprecated alias
|
||||
for `vc_stream_feed_pcm(..., channels=1)` — kept for source compatibility.
|
||||
|
||||
**Use cases:** ReplayKit Broadcast Extension (iOS `SCREEN_AUDIO`), ScreenCaptureKit (macOS
|
||||
`SCREEN_AUDIO`), music/TTS/relay bots, soundboards, transcription clients. The extension or
|
||||
bot links Opus + the feed entry point — no `ma_device`, no hardware, headless.
|
||||
|
||||
**Threading:** the feed path is thread-safe (ring buffer, no lock on the RT path). The sink
|
||||
callback runs on the miniaudio playback thread — observe the same rules as the capture
|
||||
callback: no allocations, no blocking calls.
|
||||
|
||||
## 5. Server architecture
|
||||
|
||||
`voicecat-server` is a headless process linking the core.
|
||||
|
||||
@@ -309,7 +309,21 @@ message TextMessage {
|
||||
disconnect; server-sent fatal `Disconnect` uses `code ≥ 1` (1 = protocol error,
|
||||
2 = kicked).
|
||||
|
||||
## 8. Extensibility checklist
|
||||
## 8. Client-local features (no protocol changes)
|
||||
|
||||
Some features are entirely client-side and involve no changes to the wire format:
|
||||
|
||||
- **External PCM feed (`vc_stream_feed_pcm`)** — the caller supplies interleaved int16 PCM
|
||||
that the core frames, encodes, and sends over the existing UDP media path. From the server
|
||||
and peers' perspective the stream is indistinguishable from a hardware-captured stream. No
|
||||
new messages, fields, or tags are needed.
|
||||
- **PCM tap (`vc_set_pcm_sink`)** — receives decoded per-stream audio before hardware mixing.
|
||||
Entirely local to the listener; no protocol traffic of any kind.
|
||||
|
||||
These are noted here to prevent future contributors from looking for corresponding protocol
|
||||
changes: there are none.
|
||||
|
||||
## 9. Extensibility checklist
|
||||
|
||||
When adding a feature later (e.g. **file transfer**), the rules are:
|
||||
|
||||
|
||||
@@ -69,7 +69,8 @@ exists from M1 so the protocol can be exercised long before any GUI.
|
||||
- ~~AVAudioSession, mic permission, foreground voice.~~ ✓ Done — `IOSAudioRouter` drives
|
||||
all iOS audio routing (input ports, orientation/polar patterns, HFP/A2DP, Standard/Raw
|
||||
mic mode, stereo capture), `vc_audio_suspend`/`vc_audio_resume` for interruptions.
|
||||
- ReplayKit broadcast extension for `SCREEN_AUDIO`.
|
||||
- ReplayKit broadcast extension for `SCREEN_AUDIO` — feeds `CMSampleBuffer` audio via
|
||||
`vc_stream_feed_pcm` (see architecture.md §4).
|
||||
|
||||
**Exit:** non-technical user installs a client, saves a server, and joins.
|
||||
|
||||
@@ -78,6 +79,13 @@ exists from M1 so the protocol can be exercised long before any GUI.
|
||||
(channel CRUD with full Opus config, user moderation, server account management); macOS/iOS
|
||||
Swift UI pending.
|
||||
- DRED toggle, audio-quality polish. (AEC and VAD/PTT already shipped in M2.)
|
||||
- **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) ✓ shipped
|
||||
(2026-06-20) — promotes `vc_test_inject_capture` to a public, stereo-capable API and adds
|
||||
a symmetric PCM sink. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, and custom
|
||||
clients. See architecture.md §4 "External PCM feed/tap" and protocol.md §8 for the
|
||||
no-protocol-change rationale. One new C++ ctest binary (`test_external_pcm`) covering 3
|
||||
sub-tests (`test_feed_pcm_round_trip`, `test_feed_pcm_stereo`, `test_pcm_sink`) — ctest
|
||||
23/23; Swift wrapper + 4 XCTest smoke tests; C# wrapper + 4 xUnit smoke tests.
|
||||
- **Then (post-v1, protocol already reserves space):** file transfer, E2EE option,
|
||||
CallKit/PushKit background voice, key-based identity, server-side text history,
|
||||
multi-node server.
|
||||
|
||||
@@ -274,8 +274,8 @@ normal stream; only the *source* is platform-specific.
|
||||
| Platform | Mechanism | Notes |
|
||||
|----------|-----------|-------|
|
||||
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | OS requires screen-recording permission; capture happens in the main app. Swift converts each `CMSampleBuffer` (Float32) → int16 and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). The extension calls `vc_stream_feed_pcm` to drive the encode path without a hardware device. |
|
||||
|
||||
### iOS detail
|
||||
|
||||
@@ -285,11 +285,11 @@ normal stream; only the *source* is platform-specific.
|
||||
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
|
||||
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
|
||||
audio-only consumer stays comfortably inside it.
|
||||
- The extension is a *separate process*. It links a **minimal slice of the core** (Opus
|
||||
encode + media send only — not the full client), reads the active session token and
|
||||
server endpoint from a shared **App Group** container that the host app wrote at join
|
||||
time, derives its own media keys, and publishes the `SCREEN_AUDIO` stream directly. The
|
||||
host app announces the stream over its control channel (`StreamAnnounce`) so the server and
|
||||
- The extension is a *separate process*. It reads the active session token and server
|
||||
endpoint from a shared **App Group** container, opens a UDP media channel, and calls
|
||||
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the
|
||||
Opus encode + AEAD + send path directly, with no `ma_device` or audio hardware. The host
|
||||
app announces the stream over its control channel (`StreamAnnounce`) so the server and
|
||||
peers learn about it.
|
||||
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the
|
||||
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final
|
||||
|
||||
Reference in New Issue
Block a user