feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink)
Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public, stereo-capable production API and adds a symmetric PCM tap on the receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, soundboards, and custom clients — all without a hardware audio device. Core C++: - voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef, vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias - audio_engine: stereo-aware inject_capture (channels param + ring reset on channel-count change); atomic pcm_sink_ fired per decoded frame in on_playback; RemoteStream carries user_id/stream_id for RT-safe sink metadata; init_recv_stream takes user_id+stream_id - client.cpp: stream_feed_pcm / set_pcm_sink implementations; sync_remote_streams passes user_id/stream_id to init_recv_stream - voicecat.cpp: trampolines + channels=1/2 validation Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip, stereo feed L≠R, sink metadata+disable). ctest 23/23. Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest smoke tests (ExternalPcmTests.swift). C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs (vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate, vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs). Docs: architecture.md §4 new subsection, voice.md §9 updated (macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -274,8 +274,8 @@ normal stream; only the *source* is platform-specific.
|
||||
| Platform | Mechanism | Notes |
|
||||
|----------|-----------|-------|
|
||||
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | OS requires screen-recording permission; capture happens in the main app. Swift converts each `CMSampleBuffer` (Float32) → int16 and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). The extension calls `vc_stream_feed_pcm` to drive the encode path without a hardware device. |
|
||||
|
||||
### iOS detail
|
||||
|
||||
@@ -285,11 +285,11 @@ normal stream; only the *source* is platform-specific.
|
||||
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
|
||||
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
|
||||
audio-only consumer stays comfortably inside it.
|
||||
- The extension is a *separate process*. It links a **minimal slice of the core** (Opus
|
||||
encode + media send only — not the full client), reads the active session token and
|
||||
server endpoint from a shared **App Group** container that the host app wrote at join
|
||||
time, derives its own media keys, and publishes the `SCREEN_AUDIO` stream directly. The
|
||||
host app announces the stream over its control channel (`StreamAnnounce`) so the server and
|
||||
- The extension is a *separate process*. It reads the active session token and server
|
||||
endpoint from a shared **App Group** container, opens a UDP media channel, and calls
|
||||
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the
|
||||
Opus encode + AEAD + send path directly, with no `ma_device` or audio hardware. The host
|
||||
app announces the stream over its control channel (`StreamAnnounce`) so the server and
|
||||
peers learn about it.
|
||||
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the
|
||||
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final
|
||||
|
||||
Reference in New Issue
Block a user