feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink)

Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public,
stereo-capable production API and adds a symmetric PCM tap on the
receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots,
soundboards, and custom clients — all without a hardware audio device.

Core C++:
- voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef,
  vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias
- audio_engine: stereo-aware inject_capture (channels param + ring
  reset on channel-count change); atomic pcm_sink_ fired per decoded
  frame in on_playback; RemoteStream carries user_id/stream_id for
  RT-safe sink metadata; init_recv_stream takes user_id+stream_id
- client.cpp: stream_feed_pcm / set_pcm_sink implementations;
  sync_remote_streams passes user_id/stream_id to init_recv_stream
- voicecat.cpp: trampolines + channels=1/2 validation

Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip,
stereo feed L≠R, sink metadata+disable). ctest 23/23.

Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest
smoke tests (ExternalPcmTests.swift).

C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs
(vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate,
vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs).

Docs: architecture.md §4 new subsection, voice.md §9 updated
(macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit
no-protocol-change note, roadmap.md M5 entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-20 17:52:09 +02:00
parent 540ec13a63
commit 615d2a8e5f
21 changed files with 891 additions and 39 deletions

View File

@@ -8,8 +8,9 @@ and what's next* read [`PROGRESS.md`](PROGRESS.md); for *design* read [`docs/`](
> server-mute, channel CRUD, in-app account management, disconnect/keepalive/reaper. Windows > server-mute, channel CRUD, in-app account management, disconnect/keepalive/reaper. Windows
> WinForms C# client shipped (M4). **macOS AppKit client shipped** — `VoiceCatMac.xcodeproj` > WinForms C# client shipped (M4). **macOS AppKit client shipped** — `VoiceCatMac.xcodeproj`
> at `clients/apple/macOS/`. **iOS SwiftUI client shipped** — `VoiceCatiOS.xcodeproj` at > at `clients/apple/macOS/`. **iOS SwiftUI client shipped** — `VoiceCatiOS.xcodeproj` at
> `clients/apple/iOS/`. `ctest --preset dev` green — 21/21 tests. > `clients/apple/iOS/`. `ctest --preset dev` green — 23/23 tests.
> Next: ReplayKit Broadcast Extension or DRED/audio polish. See [`PROGRESS.md`](PROGRESS.md). > External PCM feed/tap API (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) shipped.
> Next: ReplayKit Broadcast Extension (iOS) or ScreenCaptureKit (macOS). See [`PROGRESS.md`](PROGRESS.md).
VoiceCat = self-hosted native voice & text chat (TeamSpeak/Mumble-style). Plain TCP (control) VoiceCat = self-hosted native voice & text chat (TeamSpeak/Mumble-style). Plain TCP (control)
+ UDP (media), no WebRTC, encrypted by default. A shared C++ core (`libvoicecat`) drives + UDP (media), no WebRTC, encrypted by default. A shared C++ core (`libvoicecat`) drives

View File

@@ -39,12 +39,12 @@ up instantly. Newest status at the top.
`allowBluetoothA2DP` and the output route must remain the headphones/A2DP device — NOT flip `allowBluetoothA2DP` and the output route must remain the headphones/A2DP device — NOT flip
to `…Record`. If confirmed, delete the `logSessionState` calls + method and the prior to `…Record`. If confirmed, delete the `logSessionState` calls + method and the prior
band-aid comments in `IOSAudioRouter`/`audio_engine.cpp` can be trimmed. band-aid comments in `IOSAudioRouter`/`audio_engine.cpp` can be trimmed.
- **Verified on Windows:** `cmake --build --preset dev` clean, `ctest --preset dev` 22/22 - **Verified on Windows:** `cmake --build --preset dev` clean, `ctest --preset dev` 23/23
(21/21 prior + new `test_dred_toggle`). (22/22 prior + `test_external_pcm` new binary). iOS build & on-device run still to be done by the user on the Mac.
iOS build & on-device run still to be done by the user on the Mac.
- **Planned (not started):** **External PCM feed/tap API (`vc_stream_feed_pcm` + - **Done (2026-06-20):** **External PCM feed/tap API (`vc_stream_feed_pcm` +
`vc_set_pcm_sink`)** (2026-06-19, plan written on Windows; implement on Mac). A public, `vc_set_pcm_sink`)** — see detail in M5 section below. `ctest --preset dev` 23/23 (was 22/22 + 1 new test binary with 3 sub-tests).
Next: iOS ReplayKit and macOS ScreenCaptureKit consumers of this API. A public,
documented API for driving audio streams with externally-provided PCM instead of (or in documented API for driving audio streams with externally-provided PCM instead of (or in
addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our
roadmap — that the current "miniaudio owns the device" model can't serve: roadmap — that the current "miniaudio owns the device" model can't serve:
@@ -208,7 +208,17 @@ up instantly. Newest status at the top.
## Recent completed work ## Recent completed work
All items below are `[x]` done; `ctest --preset dev` 21/21 on Windows after each. All items below are `[x]` done; `ctest --preset dev` 23/23 on Windows after all.
- **External PCM feed/tap API** (2026-06-20): `vc_stream_feed_pcm` + `vc_set_pcm_sink` shipped.
Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public, stereo-capable API.
Adds symmetric PCM sink on the playback thread. Swift wrapper (`feedPcm`/`setPcmSink` in
`VoiceCatClient.swift`, 4 XCTest smoke tests). C# wrapper (`StreamFeedPcm`/`SetPcmSink` in
`VoiceCatClient.cs` + `NativeMethods.cs`, 4 xUnit smoke tests in `ExternalPcmTests.cs`).
Three new headless C++ ctests. Docs: architecture.md §4 new subsection, voice.md §9 updated,
protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Files: `voicecat.h`,
`voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`, `tests/test_external_pcm.cpp`,
`tests/CMakeLists.txt`, Swift + C# wrappers.
- **iOS A2DP + stereo root cause fix** (2026-06-20): miniaudio's NULL-context `ma_device_init` - **iOS A2DP + stereo root cause fix** (2026-06-20): miniaudio's NULL-context `ma_device_init`
was calling `AVAudioSession setCategory(Record)` on every device open, wiping the session was calling `AVAudioSession setCategory(Record)` on every device open, wiping the session
@@ -412,8 +422,14 @@ iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sh
on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships. on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships.
- [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target, - [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target,
App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`. App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`.
- [ ] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — see full - [x] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — done
plan in "Where we left off" above. 2026-06-20. Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public API with
stereo support. Adds a symmetric PCM sink fired on the playback thread per decoded remote
stream. Full wrappers for Swift (`feedPcm`/`setPcmSink`) and C# (`StreamFeedPcm`/
`SetPcmSink`). Three new C++ ctests (`test_feed_pcm_round_trip`, `test_feed_pcm_stereo`,
`test_pcm_sink`), 4 Swift XCTest smoke tests, 4 C# xUnit smoke tests. Docs updated
(architecture.md §4 new subsection, voice.md §9 updated, protocol.md §8 explicit
no-protocol-change note, roadmap.md M5 entry). `ctest --preset dev` 23/23.
--- ---

View File

@@ -309,6 +309,46 @@ public final class VoiceCatClient {
VoiceCatResult(vc_set_capture_channels(handle, streamId, channels)) VoiceCatResult(vc_set_capture_channels(handle, streamId, channels))
} }
// MARK: - External PCM feed / tap
/// External PCM feed drives a local stream's encode pipeline with caller-supplied PCM
/// instead of (or in addition to) a hardware capture device. Intended for ReplayKit
/// Broadcast Extension (iOS), ScreenCaptureKit (macOS), bots, and soundboard use cases.
///
/// - Parameters:
/// - streamId: The stream returned by `startStream`. Must be active.
/// - pcm: Raw int16 PCM pointer. Caller must keep the buffer alive for the duration of the call.
/// - samplesPerChannel: Samples per channel (e.g. 960 for 20 ms @ 48 kHz).
/// - channels: 1 (mono) or 2 (stereo interleaved L/R).
@discardableResult
public func feedPcm(streamId: UInt32, pcm: UnsafePointer<Int16>,
samplesPerChannel: Int, channels: UInt32) -> VoiceCatResult {
VoiceCatResult(vc_stream_feed_pcm(handle, streamId, pcm,
UInt(samplesPerChannel), channels))
}
/// Convenience overload for feeding from a Swift `[Int16]` array.
@discardableResult
public func feedPcm(streamId: UInt32, pcm: [Int16],
samplesPerChannel: Int, channels: UInt32) -> VoiceCatResult {
pcm.withUnsafeBufferPointer {
feedPcm(streamId: streamId, pcm: $0.baseAddress!,
samplesPerChannel: samplesPerChannel, channels: channels)
}
}
/// External PCM tap receive decoded per-stream audio as raw int16 PCM before it
/// reaches the hardware mix. Fires once per decoded Opus frame per remote stream.
///
/// The callback is a C function pointer (`@convention(c)`) receiving:
/// `(user, userId, streamId, pcm, samplesPerChannel, channels, sampleRate)`
///
/// Pass `nil` to disable (default). The callback MUST NOT block or allocate.
@discardableResult
public func setPcmSink(_ cb: vc_pcm_sink_cb?, user: UnsafeMutableRawPointer?) -> VoiceCatResult {
VoiceCatResult(vc_set_pcm_sink(handle, cb, user))
}
@discardableResult @discardableResult
public func setInputMode(_ mode: VoiceCatInputMode) -> VoiceCatResult { public func setInputMode(_ mode: VoiceCatInputMode) -> VoiceCatResult {
VoiceCatResult(vc_set_input_mode(handle, mode.cValue)) VoiceCatResult(vc_set_input_mode(handle, mode.cValue))

View File

@@ -0,0 +1,68 @@
// ExternalPcmTests Swift wrapper smoke tests for vc_stream_feed_pcm / vc_set_pcm_sink.
//
// These tests verify that the Swift API surface compiles, is callable, and returns expected
// results at the C-ABI boundary without requiring a live server or audio hardware.
// Full end-to-end relay / decode verification is covered by tests/test_external_pcm.cpp
// (C++ ctest), which runs headlessly on all platforms.
import XCTest
@testable import VoiceCatCore
final class ExternalPcmTests: XCTestCase {
// MARK: - feedPcm: API surface smoke
/// Calling feedPcm without a connected client or active stream must return .invalidArg
/// (not crash). Proves the SwiftC bridge compiles and handles the error path.
func testFeedPcm_noActiveStream_returnsInvalidArg() {
let client = VoiceCatClient(config: VoiceCatConfig(
clientName: "ext-pcm-test",
clientVersion: "0.1",
logLevel: .off
))
let sine = [Int16](repeating: 0, count: 960)
// Stream 0 doesn't exist the core must return invalidArg, not crash.
let result = client.feedPcm(streamId: 0, pcm: sine, samplesPerChannel: 960, channels: 1)
XCTAssertEqual(result, .invalidArg)
}
/// Calling feedPcm with channels=3 (invalid) must return .invalidArg.
func testFeedPcm_invalidChannels_returnsInvalidArg() {
let client = VoiceCatClient(config: VoiceCatConfig(
clientName: "ext-pcm-test",
clientVersion: "0.1",
logLevel: .off
))
let pcm = [Int16](repeating: 0, count: 960 * 3)
let result = client.feedPcm(streamId: 0, pcm: pcm, samplesPerChannel: 960, channels: 3)
XCTAssertEqual(result, .invalidArg)
}
// MARK: - setPcmSink: API surface smoke
/// setPcmSink(nil) on a freshly-created client must succeed (nil = disable, which is the
/// default state a no-op that must still return .ok).
func testSetPcmSink_nil_returnsOk() {
let client = VoiceCatClient(config: VoiceCatConfig(
clientName: "ext-pcm-test",
clientVersion: "0.1",
logLevel: .off
))
let result = client.setPcmSink(nil, user: nil)
XCTAssertEqual(result, .ok)
}
/// Calling setPcmSink with a @convention(c) function and then immediately disabling it
/// with nil must both succeed. Verifies the C-ABI function-pointer round-trip.
func testSetPcmSink_enableThenDisable_bothSucceed() {
let client = VoiceCatClient(config: VoiceCatConfig(
clientName: "ext-pcm-test",
clientVersion: "0.1",
logLevel: .off
))
let mySink: vc_pcm_sink_cb = { _, _, _, _, _, _, _ in }
XCTAssertEqual(client.setPcmSink(mySink, user: nil), .ok)
XCTAssertEqual(client.setPcmSink(nil, user: nil), .ok)
}
}

View File

@@ -0,0 +1,55 @@
using System.Runtime.InteropServices;
using VoiceCat.Interop;
namespace VoiceCat.Interop.Tests;
/// <summary>
/// API-surface smoke tests for StreamFeedPcm / SetPcmSink through the P/Invoke layer.
/// These tests need voicecat.dll but NOT a running server — they verify the C# wrapper
/// compiles, is callable, and the error paths work at the C-ABI boundary. Full E2E relay
/// verification is handled by tests/test_external_pcm.cpp (C++ ctest).
/// </summary>
public sealed class ExternalPcmTests
{
[Fact]
public void StreamFeedPcm_NoActiveStream_ReturnsInvalidArg()
{
using var client = new VoiceCatClient("ext-pcm-test", "0.1", VcLogLevel.Off);
var pcm = new short[960];
// Stream 0 doesn't exist — core must return InvalidArg, not crash.
var result = client.StreamFeedPcm(0, pcm, 960, channels: 1);
Assert.Equal(VcResult.InvalidArg, result);
}
[Fact]
public void StreamFeedPcm_InvalidChannels_ReturnsInvalidArg()
{
using var client = new VoiceCatClient("ext-pcm-test", "0.1", VcLogLevel.Off);
var pcm = new short[960 * 3];
// channels=3 is not supported — trampoline rejects before touching audio state.
var result = client.StreamFeedPcm(0, pcm, 960, channels: 3);
Assert.Equal(VcResult.InvalidArg, result);
}
[Fact]
public void SetPcmSink_Zero_ReturnsOk()
{
using var client = new VoiceCatClient("ext-pcm-test", "0.1", VcLogLevel.Off);
// IntPtr.Zero = disable — default state, must be a no-op that returns Ok.
var result = client.SetPcmSink(IntPtr.Zero, IntPtr.Zero);
Assert.Equal(VcResult.Ok, result);
}
[Fact]
public void SetPcmSink_EnableThenDisable_BothSucceed()
{
using var client = new VoiceCatClient("ext-pcm-test", "0.1", VcLogLevel.Off);
// Keep the delegate alive until after we unregister it (not just until the P/Invoke call).
NativeMethods.VcPcmSinkCallback sink = static (_, _, _, _, _, _, _) => { };
nint fp = Marshal.GetFunctionPointerForDelegate(sink);
Assert.Equal(VcResult.Ok, client.SetPcmSink(fp, IntPtr.Zero));
Assert.Equal(VcResult.Ok, client.SetPcmSink(IntPtr.Zero, IntPtr.Zero));
GC.KeepAlive(sink);
}
}

View File

@@ -98,6 +98,25 @@ internal static partial class NativeMethods
internal static unsafe partial VcResult vc_test_inject_capture(nint c, uint streamId, internal static unsafe partial VcResult vc_test_inject_capture(nint c, uint streamId,
short* pcm, nuint samples); short* pcm, nuint samples);
// ── External PCM feed / tap ──────────────────────────────────────────────────────────
[LibraryImport(LibName)]
internal static unsafe partial VcResult vc_stream_feed_pcm(nint c, uint streamId,
short* pcm, nuint samplesPerChannel, uint channels);
// Delegate type for the PCM sink callback — callers convert to a native function
// pointer via Marshal.GetFunctionPointerForDelegate (for instance members) or by casting
// a static lambda to delegate* unmanaged<> (for [UnmanagedCallersOnly] statics).
// Keep the delegate alive for the lifetime of the sink registration.
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate void VcPcmSinkCallback(IntPtr user, uint userId, uint streamId,
IntPtr pcm, nuint samplesPerChannel, uint channels, uint sampleRate);
// cb is a raw function pointer (IntPtr.Zero = disable). Use
// Marshal.GetFunctionPointerForDelegate(sinkDelegate) to convert from VcPcmSinkCallback.
[LibraryImport(LibName)]
internal static partial VcResult vc_set_pcm_sink(nint c, nint cb, IntPtr user);
// ── Text ───────────────────────────────────────────────────────────────────────────── // ── Text ─────────────────────────────────────────────────────────────────────────────
[LibraryImport(LibName, StringMarshalling = StringMarshalling.Utf8)] [LibraryImport(LibName, StringMarshalling = StringMarshalling.Utf8)]
internal static partial VcResult vc_send_text(nint c, VcTextScope scope, uint targetId, internal static partial VcResult vc_send_text(nint c, VcTextScope scope, uint targetId,

View File

@@ -225,6 +225,20 @@ public sealed class VoiceCatClient : IDisposable
return (r, r == VcResult.Ok ? Marshaling.ToManaged(in native) : null); return (r, r == VcResult.Ok ? Marshaling.ToManaged(in native) : null);
} }
// ── External PCM feed / tap ─────────────────────────────────────────────────────────
public unsafe VcResult StreamFeedPcm(uint streamId, ReadOnlySpan<short> pcm,
int samplesPerChannel, uint channels)
{
fixed (short* p = pcm)
return NativeMethods.vc_stream_feed_pcm(_handle.DangerousGetHandle(),
streamId, p, (nuint)samplesPerChannel, channels);
}
// Pass Marshal.GetFunctionPointerForDelegate(cb) for a managed delegate, or
// IntPtr.Zero to disable. Keep the delegate alive for the lifetime of the registration.
public VcResult SetPcmSink(nint cb, IntPtr user) =>
NativeMethods.vc_set_pcm_sink(_handle.DangerousGetHandle(), cb, user);
// ── M5: Moderation & admin ─────────────────────────────────────────────────────────── // ── M5: Moderation & admin ───────────────────────────────────────────────────────────
public VcResult KickUser(uint userId, string? reason = null) => public VcResult KickUser(uint userId, string? reason = null) =>
NativeMethods.vc_kick_user(_handle.DangerousGetHandle(), userId, reason); NativeMethods.vc_kick_user(_handle.DangerousGetHandle(), userId, reason);

View File

@@ -413,6 +413,45 @@ VC_API vc_result vc_test_inject_capture(vc_client* c, uint32_t stream_id, const
* VC_ERR_INVALID_ARG if stream_id is unknown or channels is not 1 or 2. */ * VC_ERR_INVALID_ARG if stream_id is unknown or channels is not 1 or 2. */
VC_API vc_result vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels); VC_API vc_result vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels);
/* ── External PCM feed/tap ─────────────────────────────────────────────────── */
/* External PCM feed — production-grade API for driving a local stream's encode pipeline
* with caller-supplied PCM instead of (or in addition to) a hardware capture device. The
* stream must already be started (vc_stream_start). The core frames, encodes (Opus), seals
* (AEAD), and sends (UDP) the provided samples exactly as it would mic/loopback audio.
*
* samples_per_channel : samples per channel (e.g. 960 for 20 ms @ 48 kHz).
* channels : 1 (mono) or 2 (stereo interleaved L/R). VC_ERR_INVALID_ARG otherwise.
*
* Use cases: ReplayKit Broadcast Extension (iOS), ScreenCaptureKit (macOS), bots (TTS /
* music / relay), soundboards, DAW integration. Works for any stream kind (MIC /
* SCREEN_AUDIO / AUX_DEVICE). Thread-safe; may be called from any thread.
*
* Replaces vc_test_inject_capture (deprecated alias, see below). */
VC_API vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id,
const int16_t* pcm, size_t samples_per_channel,
uint32_t channels);
/* External PCM tap — receive decoded remote audio as int16 PCM per stream, before it is
* summed into the hardware mix. The callback fires on the audio playback thread once per
* decoded Opus frame (typically every 20 ms) for each active remote stream:
*
* cb(user, user_id, stream_id, pcm, samples_per_channel, channels, sample_rate)
*
* user_id / stream_id : identify the sender (same values as VC_EVENT_STREAM_STARTED).
* pcm : decoded int16 PCM, interleaved when channels == 2.
* samples_per_channel : samples per channel for this frame (typically 960 @ 48 kHz).
* channels : 1 or 2, matching the sender's stream configuration.
* sample_rate : always 48000 in the current implementation.
*
* Pass cb = NULL to disable (default: disabled; hardware playback only).
* The callback MUST NOT block, lock, or allocate — copy what you need and return.
* PCM is still delivered to the hardware playback device regardless (dual output). */
typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
const int16_t* pcm, size_t samples_per_channel,
uint32_t channels, uint32_t sample_rate);
VC_API vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
/* ── Text ─────────────────────────────────────────────────────────────────── */ /* ── Text ─────────────────────────────────────────────────────────────────── */
VC_API vc_result vc_send_text(vc_client* c, vc_text_scope scope, uint32_t target_id, VC_API vc_result vc_send_text(vc_client* c, vc_text_scope scope, uint32_t target_id,
const char* utf8); const char* utf8);

View File

@@ -359,7 +359,11 @@ bool AudioEngine::resume() {
#endif #endif
} }
void AudioEngine::inject_capture(int kind, const int16_t* pcm, size_t n) { void AudioEngine::inject_capture(int kind, const int16_t* pcm, size_t samples_per_channel,
int channels) {
const int ch = std::max(1, channels);
const size_t n = samples_per_channel * static_cast<size_t>(ch);
InjectTap* tap; InjectTap* tap;
{ {
std::lock_guard lk(inject_mu_); std::lock_guard lk(inject_mu_);
@@ -371,26 +375,38 @@ void AudioEngine::inject_capture(int kind, const int16_t* pcm, size_t n) {
tap = slot.get(); tap = slot.get();
} }
// If the channel count changed, reset the ring to avoid mixing mono and stereo samples.
if (tap->channels != ch) {
tap->write.store(0, std::memory_order_relaxed);
tap->read.store(0, std::memory_order_relaxed);
tap->channels = ch;
}
size_t w = tap->write.load(std::memory_order_relaxed); size_t w = tap->write.load(std::memory_order_relaxed);
for (size_t i = 0; i < n; ++i) for (size_t i = 0; i < n; ++i)
tap->ring[(w + i) % kInjectCapSamples] = pcm[i]; tap->ring[(w + i) % kInjectCapSamples] = pcm[i];
tap->write.store(w + n, std::memory_order_release); tap->write.store(w + n, std::memory_order_release);
// Fire capture_cb_ for each complete frame now available. // Fire capture_cb_ for each complete frame (frame_samples_ * ch flat samples).
const size_t frame_flat = static_cast<size_t>(frame_samples_) * static_cast<size_t>(ch);
while (true) { while (true) {
size_t r = tap->read.load(std::memory_order_relaxed); size_t r = tap->read.load(std::memory_order_relaxed);
size_t avail = tap->write.load(std::memory_order_acquire) - r; size_t avail = tap->write.load(std::memory_order_acquire) - r;
if (avail < static_cast<size_t>(frame_samples_)) break; if (avail < frame_flat) break;
std::vector<int16_t> frame(frame_samples_); std::vector<int16_t> frame(frame_flat);
for (int i = 0; i < frame_samples_; ++i) for (size_t i = 0; i < frame_flat; ++i)
frame[i] = tap->ring[(r + i) % kInjectCapSamples]; frame[i] = tap->ring[(r + i) % kInjectCapSamples];
tap->read.store(r + frame_samples_, std::memory_order_release); tap->read.store(r + frame_flat, std::memory_order_release);
if (capture_cb_) capture_cb_(kind, frame.data(), frame_samples_, 1); if (capture_cb_) capture_cb_(kind, frame.data(), frame_samples_, ch);
} }
} }
void AudioEngine::inject_capture(int kind, const int16_t* pcm, size_t n) {
inject_capture(kind, pcm, n, 1);
}
void AudioEngine::push_recv_frame(uint32_t ssrc, JitterBuffer::Frame f) { void AudioEngine::push_recv_frame(uint32_t ssrc, JitterBuffer::Frame f) {
std::lock_guard lk(streams_mu_); std::lock_guard lk(streams_mu_);
auto& s = streams_[ssrc]; auto& s = streams_[ssrc];
@@ -472,9 +488,12 @@ uint32_t AudioEngine::stream_target_depth_ms(uint32_t ssrc) const {
} }
#ifdef VOICECAT_HAS_OPUS #ifdef VOICECAT_HAS_OPUS
void AudioEngine::init_recv_stream(uint32_t ssrc, const codec::OpusParams& p) { void AudioEngine::init_recv_stream(uint32_t ssrc, const codec::OpusParams& p,
uint32_t user_id, uint32_t stream_id) {
std::lock_guard lk(streams_mu_); std::lock_guard lk(streams_mu_);
auto& stream = streams_[ssrc]; auto& stream = streams_[ssrc];
stream.user_id = user_id;
stream.stream_id = stream_id;
stream.decoder.init(p); stream.decoder.init(p);
// Ring must be sized for this decoder's actual channel/frame-size — see RemoteStream::ring // Ring must be sized for this decoder's actual channel/frame-size — see RemoteStream::ring
// comment in audio_engine.h for why this can't just be the playback callback's frame count. // comment in audio_engine.h for why this can't just be the playback callback's frame count.
@@ -491,6 +510,11 @@ void AudioEngine::init_recv_stream(uint32_t ssrc, const codec::OpusParams& p) {
} }
#endif #endif
void AudioEngine::set_pcm_sink(PcmSink cb, void* user) {
pcm_sink_user_.store(user, std::memory_order_relaxed);
pcm_sink_.store(cb, std::memory_order_release);
}
#ifdef VOICECAT_HAS_AUDIO #ifdef VOICECAT_HAS_AUDIO
void AudioEngine::capture_data_cb(ma_device* dev, void* /*out*/, void AudioEngine::capture_data_cb(ma_device* dev, void* /*out*/,
@@ -634,6 +658,19 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
stream.recv_ns->process_capture(stream.decode_scratch.data(), n, stream.recv_ns->process_capture(stream.decode_scratch.data(), n,
static_cast<int>(params_.sample_rate)); static_cast<int>(params_.sample_rate));
// PCM sink: deliver decoded per-stream audio to external consumer (bots,
// transcription, recording) before it enters the hardware mix. Atomic relaxed-
// load is safe on the RT thread — the fn-ptr and user-ptr are independent
// pointer-sized values written together by set_pcm_sink (release store).
if (auto sink = pcm_sink_.load(std::memory_order_relaxed)) {
sink(pcm_sink_user_.load(std::memory_order_relaxed),
stream.user_id, stream.stream_id,
stream.decode_scratch.data(),
static_cast<size_t>(n),
static_cast<uint32_t>(dec_channels),
params_.sample_rate);
}
stream.push_ring(stream.decode_scratch.data(), static_cast<size_t>(n)); stream.push_ring(stream.decode_scratch.data(), static_cast<size_t>(n));
stream.playout_ts += static_cast<uint32_t>(n); stream.playout_ts += static_cast<uint32_t>(n);
} }

View File

@@ -190,10 +190,22 @@ class AudioEngine {
#ifdef VOICECAT_HAS_OPUS #ifdef VOICECAT_HAS_OPUS
// Configure the Opus decoder for an incoming ssrc (must be called before // Configure the Opus decoder for an incoming ssrc (must be called before
// push_recv_frame for that ssrc). Thread-safe. // push_recv_frame for that ssrc). user_id/stream_id identify the source for the
void init_recv_stream(uint32_t ssrc, const codec::OpusParams& p); // pcm_sink_ callback. Thread-safe.
void init_recv_stream(uint32_t ssrc, const codec::OpusParams& p,
uint32_t user_id, uint32_t stream_id);
#endif #endif
// External PCM tap: callback fired once per decoded Opus frame per remote stream, on the
// playback (RT) thread. Matching signature to vc_pcm_sink_cb (cast at the C-ABI boundary).
// Pass nullptr to disable. Thread-safe (atomic store; the RT read is relaxed-load).
using PcmSink = void(*)(void*, uint32_t, uint32_t, const int16_t*, size_t, uint32_t, uint32_t);
void set_pcm_sink(PcmSink cb, void* user);
// External PCM feed overload: stereo-aware variant of inject_capture. samples_per_channel
// is samples per channel; total samples written = samples_per_channel * channels.
void inject_capture(int kind, const int16_t* pcm, size_t samples_per_channel, int channels);
#ifdef VOICECAT_HAS_AUDIO #ifdef VOICECAT_HAS_AUDIO
// TEST-ONLY — exposes the playback mixer without a real ma_device, so tests can verify // TEST-ONLY — exposes the playback mixer without a real ma_device, so tests can verify
// stereo mixing end-to-end (no audio hardware needed). Same logic the real playback // stereo mixing end-to-end (no audio hardware needed). Same logic the real playback
@@ -364,6 +376,7 @@ class AudioEngine {
std::vector<int16_t> ring; // circular, size = kInjectCapSamples std::vector<int16_t> ring; // circular, size = kInjectCapSamples
std::atomic<size_t> write{0}; std::atomic<size_t> write{0};
std::atomic<size_t> read{0}; std::atomic<size_t> read{0};
int channels{1}; // channel count last written; resets ring on change
}; };
std::mutex inject_mu_; std::mutex inject_mu_;
std::unordered_map<int, std::unique_ptr<InjectTap>> inject_taps_; std::unordered_map<int, std::unique_ptr<InjectTap>> inject_taps_;
@@ -405,6 +418,11 @@ class AudioEngine {
#endif #endif
std::vector<uint8_t> dred_payload_scratch_; // pre-sized to 4000 bytes std::vector<uint8_t> dred_payload_scratch_; // pre-sized to 4000 bytes
// Source identity: stored at init_recv_stream() so the pcm_sink_ callback can receive
// (user_id, stream_id) without a separate map lookup from the RT playback thread.
uint32_t user_id = 0;
uint32_t stream_id = 0;
// M3: talk-indicator edge detection (docs/voice.md §7) — updated by push_recv_frame // M3: talk-indicator edge detection (docs/voice.md §7) — updated by push_recv_frame
// (already off the real-time audio thread), polled by poll_talk_transitions(). // (already off the real-time audio thread), polled by poll_talk_transitions().
std::atomic<int64_t> last_voice_ms{0}; std::atomic<int64_t> last_voice_ms{0};
@@ -468,6 +486,11 @@ class AudioEngine {
int frame_samples_ = 960; // 20 ms @48 kHz int frame_samples_ = 960; // 20 ms @48 kHz
// External PCM tap: atomic fn-ptr + user-ptr pair. Written by set_pcm_sink (any thread);
// read by on_playback (RT thread) via relaxed load — safe for pointer-sized atomics.
std::atomic<PcmSink> pcm_sink_{nullptr};
std::atomic<void*> pcm_sink_user_{nullptr};
#ifdef VOICECAT_HAS_OPUS #ifdef VOICECAT_HAS_OPUS
::OpusDREDDecoder* dred_dec_ = nullptr; // shared DRED decoder; null if unsupported ::OpusDREDDecoder* dred_dec_ = nullptr; // shared DRED decoder; null if unsupported
#endif #endif

View File

@@ -1109,7 +1109,7 @@ void vc_client::sync_remote_streams(const voicecat::v1::User& user) {
remote_streams_[ssrc] = {user.id(), si.stream_id()}; remote_streams_[ssrc] = {user.id(), si.stream_id()};
voicecat::codec::OpusParams p = opus_params_from_audio_config(si.audio()); voicecat::codec::OpusParams p = opus_params_from_audio_config(si.audio());
audio_engine_.init_recv_stream(ssrc, p); audio_engine_.init_recv_stream(ssrc, p, user.id(), si.stream_id());
bool muted = self_deafened_.load(std::memory_order_acquire) || bool muted = self_deafened_.load(std::memory_order_acquire) ||
server_deafened_.load(std::memory_order_acquire); server_deafened_.load(std::memory_order_acquire);
audio_engine_.set_stream_mute(ssrc, muted); audio_engine_.set_stream_mute(ssrc, muted);
@@ -1490,7 +1490,8 @@ vc_result vc_client::get_stream_audio_config(uint32_t user_id, uint32_t stream_i
return VC_ERR_INVALID_ARG; return VC_ERR_INVALID_ARG;
} }
vc_result vc_client::test_inject_capture(uint32_t stream_id, const int16_t* pcm, size_t samples) { vc_result vc_client::stream_feed_pcm(uint32_t stream_id, const int16_t* pcm,
size_t samples_per_channel, uint32_t channels) {
int kind = -1; int kind = -1;
{ {
std::lock_guard lk(local_streams_mu_); std::lock_guard lk(local_streams_mu_);
@@ -1502,10 +1503,20 @@ vc_result vc_client::test_inject_capture(uint32_t stream_id, const int16_t* pcm,
} }
} }
if (kind < 0) return VC_ERR_INVALID_ARG; if (kind < 0) return VC_ERR_INVALID_ARG;
audio_engine_.inject_capture(kind, pcm, samples); audio_engine_.inject_capture(kind, pcm, samples_per_channel, static_cast<int>(channels));
return VC_OK; return VC_OK;
} }
vc_result vc_client::set_pcm_sink(vc_pcm_sink_cb cb, void* user) {
audio_engine_.set_pcm_sink(
reinterpret_cast<voicecat::audio::AudioEngine::PcmSink>(cb), user);
return VC_OK;
}
vc_result vc_client::test_inject_capture(uint32_t stream_id, const int16_t* pcm, size_t samples) {
return stream_feed_pcm(stream_id, pcm, samples, 1);
}
vc_result vc_client::list_devices(vc_device_kind kind, vc_device_list* out) { vc_result vc_client::list_devices(vc_device_kind kind, vc_device_list* out) {
#ifdef VOICECAT_HAS_AUDIO #ifdef VOICECAT_HAS_AUDIO
// Works in any connection state — device pickers need to populate pre-connect. // Works in any connection state — device pickers need to populate pre-connect.
@@ -1876,6 +1887,12 @@ vc_result vc_client::list_devices(vc_device_kind, vc_device_list* out) {
vc_result vc_client::get_stream_audio_config(uint32_t, uint32_t, vc_audio_config*) { vc_result vc_client::get_stream_audio_config(uint32_t, uint32_t, vc_audio_config*) {
return VC_ERR_NOT_IMPLEMENTED; return VC_ERR_NOT_IMPLEMENTED;
} }
vc_result vc_client::stream_feed_pcm(uint32_t, const int16_t*, size_t, uint32_t) {
return VC_ERR_NOT_IMPLEMENTED;
}
vc_result vc_client::set_pcm_sink(vc_pcm_sink_cb, void*) {
return VC_ERR_NOT_IMPLEMENTED;
}
vc_result vc_client::test_inject_capture(uint32_t, const int16_t*, size_t) { vc_result vc_client::test_inject_capture(uint32_t, const int16_t*, size_t) {
return VC_ERR_NOT_IMPLEMENTED; return VC_ERR_NOT_IMPLEMENTED;
} }

View File

@@ -81,7 +81,15 @@ struct vc_client {
vc_result get_stream_audio_config(uint32_t user_id, uint32_t stream_id, vc_result get_stream_audio_config(uint32_t user_id, uint32_t stream_id,
vc_audio_config* out); vc_audio_config* out);
// TEST-ONLY (see voicecat.h) — inject synthetic PCM into a local stream's encode pipeline. // External PCM feed (see voicecat.h: vc_stream_feed_pcm). Production API for driving a
// local stream's encode pipeline without a hardware capture device. channels = 1 or 2.
vc_result stream_feed_pcm(uint32_t stream_id, const int16_t* pcm,
size_t samples_per_channel, uint32_t channels);
// External PCM sink (see voicecat.h: vc_set_pcm_sink). Delegates to AudioEngine.
vc_result set_pcm_sink(vc_pcm_sink_cb cb, void* user);
// TEST-ONLY (see voicecat.h) — deprecated alias for stream_feed_pcm(..., channels=1).
vc_result test_inject_capture(uint32_t stream_id, const int16_t* pcm, size_t samples); vc_result test_inject_capture(uint32_t stream_id, const int16_t* pcm, size_t samples);
// M5: moderation & admin. // M5: moderation & admin.

View File

@@ -143,7 +143,19 @@ vc_result vc_get_stream_audio_config(vc_client* c, uint32_t user_id, uint32_t st
vc_result vc_test_inject_capture(vc_client* c, uint32_t stream_id, const int16_t* pcm, vc_result vc_test_inject_capture(vc_client* c, uint32_t stream_id, const int16_t* pcm,
size_t samples) { size_t samples) {
if (c == nullptr || pcm == nullptr) return VC_ERR_INVALID_ARG; if (c == nullptr || pcm == nullptr) return VC_ERR_INVALID_ARG;
return c->test_inject_capture(stream_id, pcm, samples); return c->stream_feed_pcm(stream_id, pcm, samples, 1);
}
vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id, const int16_t* pcm,
size_t samples_per_channel, uint32_t channels) {
if (c == nullptr || pcm == nullptr) return VC_ERR_INVALID_ARG;
if (channels != 1 && channels != 2) return VC_ERR_INVALID_ARG;
return c->stream_feed_pcm(stream_id, pcm, samples_per_channel, channels);
}
vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user) {
if (c == nullptr) return VC_ERR_INVALID_ARG;
return c->set_pcm_sink(cb, user);
} }
vc_result vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels) { vc_result vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels) {

View File

@@ -155,6 +155,26 @@ Design notes:
`PostMessage` with no meaningful latency cost. `VoiceCatClientHandle : SafeHandle` wraps `PostMessage` with no meaningful latency cost. `VoiceCatClientHandle : SafeHandle` wraps
the `vc_client*` and guarantees `vc_client_destroy` runs on GC/Dispose. the `vc_client*` and guarantees `vc_client_destroy` runs on GC/Dispose.
### External PCM feed/tap
Two API functions let callers bypass miniaudio entirely for a stream:
| Function | Direction | Contract |
|----------|-----------|----------|
| `vc_stream_feed_pcm(c, stream_id, pcm, samples_per_channel, channels)` | **Send** — caller → network | Caller supplies interleaved int16 at the stream's sample rate (`channels` = 1 mono, 2 stereo). The core frames, Opus-encodes, AEAD-seals, and sends over UDP — identical wire path to hardware capture. The stream must already be started with `vc_stream_start`. Thread-safe; may be called from any thread (audio callback, ReplayKit delegate, SCStream callback). |
| `vc_set_pcm_sink(c, cb, user)` | **Receive** — network → caller | `cb` is called on the audio (playback) thread once per decoded Opus frame per remote stream, with `(user_id, stream_id, pcm, samples_per_channel, channels, sample_rate)`. PCM is delivered to the sink **and** the hardware device — dual output; the hardware mix is unaffected. Pass `cb=NULL` to disable (default). **Must not block** — copy what you need and return. |
`vc_test_inject_capture` (the old TEST-ONLY mono-only predecessor) is a deprecated alias
for `vc_stream_feed_pcm(..., channels=1)` — kept for source compatibility.
**Use cases:** ReplayKit Broadcast Extension (iOS `SCREEN_AUDIO`), ScreenCaptureKit (macOS
`SCREEN_AUDIO`), music/TTS/relay bots, soundboards, transcription clients. The extension or
bot links Opus + the feed entry point — no `ma_device`, no hardware, headless.
**Threading:** the feed path is thread-safe (ring buffer, no lock on the RT path). The sink
callback runs on the miniaudio playback thread — observe the same rules as the capture
callback: no allocations, no blocking calls.
## 5. Server architecture ## 5. Server architecture
`voicecat-server` is a headless process linking the core. `voicecat-server` is a headless process linking the core.

View File

@@ -309,7 +309,21 @@ message TextMessage {
disconnect; server-sent fatal `Disconnect` uses `code ≥ 1` (1 = protocol error, disconnect; server-sent fatal `Disconnect` uses `code ≥ 1` (1 = protocol error,
2 = kicked). 2 = kicked).
## 8. Extensibility checklist ## 8. Client-local features (no protocol changes)
Some features are entirely client-side and involve no changes to the wire format:
- **External PCM feed (`vc_stream_feed_pcm`)** — the caller supplies interleaved int16 PCM
that the core frames, encodes, and sends over the existing UDP media path. From the server
and peers' perspective the stream is indistinguishable from a hardware-captured stream. No
new messages, fields, or tags are needed.
- **PCM tap (`vc_set_pcm_sink`)** — receives decoded per-stream audio before hardware mixing.
Entirely local to the listener; no protocol traffic of any kind.
These are noted here to prevent future contributors from looking for corresponding protocol
changes: there are none.
## 9. Extensibility checklist
When adding a feature later (e.g. **file transfer**), the rules are: When adding a feature later (e.g. **file transfer**), the rules are:

View File

@@ -69,7 +69,8 @@ exists from M1 so the protocol can be exercised long before any GUI.
- ~~AVAudioSession, mic permission, foreground voice.~~ ✓ Done — `IOSAudioRouter` drives - ~~AVAudioSession, mic permission, foreground voice.~~ ✓ Done — `IOSAudioRouter` drives
all iOS audio routing (input ports, orientation/polar patterns, HFP/A2DP, Standard/Raw all iOS audio routing (input ports, orientation/polar patterns, HFP/A2DP, Standard/Raw
mic mode, stereo capture), `vc_audio_suspend`/`vc_audio_resume` for interruptions. mic mode, stereo capture), `vc_audio_suspend`/`vc_audio_resume` for interruptions.
- ReplayKit broadcast extension for `SCREEN_AUDIO`. - ReplayKit broadcast extension for `SCREEN_AUDIO` — feeds `CMSampleBuffer` audio via
`vc_stream_feed_pcm` (see architecture.md §4).
**Exit:** non-technical user installs a client, saves a server, and joins. **Exit:** non-technical user installs a client, saves a server, and joins.
@@ -78,6 +79,13 @@ exists from M1 so the protocol can be exercised long before any GUI.
(channel CRUD with full Opus config, user moderation, server account management); macOS/iOS (channel CRUD with full Opus config, user moderation, server account management); macOS/iOS
Swift UI pending. Swift UI pending.
- DRED toggle, audio-quality polish. (AEC and VAD/PTT already shipped in M2.) - DRED toggle, audio-quality polish. (AEC and VAD/PTT already shipped in M2.)
- **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) ✓ shipped
(2026-06-20) — promotes `vc_test_inject_capture` to a public, stereo-capable API and adds
a symmetric PCM sink. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots, and custom
clients. See architecture.md §4 "External PCM feed/tap" and protocol.md §8 for the
no-protocol-change rationale. One new C++ ctest binary (`test_external_pcm`) covering 3
sub-tests (`test_feed_pcm_round_trip`, `test_feed_pcm_stereo`, `test_pcm_sink`) — ctest
23/23; Swift wrapper + 4 XCTest smoke tests; C# wrapper + 4 xUnit smoke tests.
- **Then (post-v1, protocol already reserves space):** file transfer, E2EE option, - **Then (post-v1, protocol already reserves space):** file transfer, E2EE option,
CallKit/PushKit background voice, key-based identity, server-side text history, CallKit/PushKit background voice, key-based identity, server-side text history,
multi-node server. multi-node server.

View File

@@ -274,8 +274,8 @@ normal stream; only the *source* is platform-specific.
| Platform | Mechanism | Notes | | Platform | Mechanism | Notes |
|----------|-----------|-------| |----------|-----------|-------|
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. | | **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. | | **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | OS requires screen-recording permission; capture happens in the main app. Swift converts each `CMSampleBuffer` (Float32) → int16 and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). | | **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). The extension calls `vc_stream_feed_pcm` to drive the encode path without a hardware device. |
### iOS detail ### iOS detail
@@ -285,11 +285,11 @@ normal stream; only the *source* is platform-specific.
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
audio-only consumer stays comfortably inside it. audio-only consumer stays comfortably inside it.
- The extension is a *separate process*. It links a **minimal slice of the core** (Opus - The extension is a *separate process*. It reads the active session token and server
encode + media send only — not the full client), reads the active session token and endpoint from a shared **App Group** container, opens a UDP media channel, and calls
server endpoint from a shared **App Group** container that the host app wrote at join `vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the
time, derives its own media keys, and publishes the `SCREEN_AUDIO` stream directly. The Opus encode + AEAD + send path directly, with no `ma_device` or audio hardware. The host
host app announces the stream over its control channel (`StreamAnnounce`) so the server and app announces the stream over its control channel (`StreamAnnounce`) so the server and
peers learn about it. peers learn about it.
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the - Mic + voice continue to run in the **host app**; only the system-audio share lives in the
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final extension. When the broadcast stops (`broadcastFinished`), the extension sends a final

View File

@@ -183,4 +183,15 @@ if(VOICECAT_USE_VCPKG_DEPS)
target_include_directories(test_dred_toggle PRIVATE ${VC_TEST_INTERNAL_INCLUDES}) target_include_directories(test_dred_toggle PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME dred_toggle COMMAND test_dred_toggle) add_test(NAME dred_toggle COMMAND test_dred_toggle)
set_tests_properties(dred_toggle PROPERTIES TIMEOUT 60) set_tests_properties(dred_toggle PROPERTIES TIMEOUT 60)
# External PCM feed/tap API:
# test_feed_pcm_round_trip — vc_stream_feed_pcm (mono), verified via pcm_sink.
# test_feed_pcm_stereo — vc_stream_feed_pcm (stereo, Music Room), L != R assertion.
# test_pcm_sink — vc_set_pcm_sink metadata (user_id / stream_id / sr) + disable.
add_executable(test_external_pcm test_external_pcm.cpp)
target_link_libraries(test_external_pcm PRIVATE voicecat::server)
target_compile_features(test_external_pcm PRIVATE cxx_std_20)
target_include_directories(test_external_pcm PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME external_pcm COMMAND test_external_pcm)
set_tests_properties(external_pcm PROPERTIES TIMEOUT 90)
endif() endif()

450
tests/test_external_pcm.cpp Normal file
View File

@@ -0,0 +1,450 @@
/*
* test_external_pcm — external PCM feed/tap API.
*
* Three headless behavior tests (no audio hardware, no simulator):
*
* test_feed_pcm_round_trip — A feeds mono 440 Hz sine via vc_stream_feed_pcm; B's pcm_sink
* fires with non-zero energy, proving the full pipeline (feed→encode→relay→decode→sink).
*
* test_feed_pcm_stereo — A and B join Music Room (stereo/128kbps). A feeds interleaved
* stereo PCM (loud-L / silent-R) via vc_stream_feed_pcm(channels=2). B's pcm_sink
* asserts L-channel energy > R-channel energy (real stereo bitstream, not a mono upmix).
*
* test_pcm_sink — Verifies sink metadata: correct user_id / stream_id per frame,
* sample_rate = 48000, and that cb=NULL disables delivery.
*/
#include <cstdio>
#ifdef VOICECAT_HAS_NET
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <cmath>
#include <filesystem>
#include <mutex>
#include <string>
#include <thread>
#include <vector>
#include "voicecat.h"
#include "server.h"
#include "db.h"
// ── Helpers ───────────────────────────────────────────────────────────────────
static int g_failures = 0;
#define CHECK(cond) \
do { if (!(cond)) { \
std::printf("FAIL [%s:%d]: %s\n", __FILE__, __LINE__, #cond); \
++g_failures; \
}} while (0)
struct EventStore {
std::mutex mu;
std::condition_variable cv;
bool auth_ok{false};
uint32_t self_user_id{0};
bool channel_list_received{false};
bool join_ok{false};
std::vector<std::pair<uint32_t, uint32_t>> streams_started; // (user_id, stream_id)
vc_client* client{nullptr};
const char* label{nullptr};
};
static void on_event(void* user, const vc_event* ev) {
auto* s = static_cast<EventStore*>(user);
std::lock_guard lk(s->mu);
switch (ev->type) {
case VC_EVENT_SERVER_IDENTITY:
vc_confirm_server_identity(s->client, 1);
break;
case VC_EVENT_AUTH_RESULT:
s->auth_ok = (ev->result == VC_OK);
s->self_user_id = ev->user_id;
break;
case VC_EVENT_CHANNEL_LIST:
s->channel_list_received = true;
break;
case VC_EVENT_JOIN_RESULT:
s->join_ok = (ev->result == VC_OK);
break;
case VC_EVENT_STREAM_STARTED:
s->streams_started.emplace_back(ev->user_id, ev->stream_id);
break;
default: break;
}
s->cv.notify_all();
}
template <typename Pred>
static bool wait_for(EventStore& s, Pred pred, int timeout_ms) {
auto deadline = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout_ms);
std::unique_lock lk(s.mu);
return s.cv.wait_until(lk, deadline, [&] { return pred(s); });
}
static bool connect_guest(vc_client*& client, const char* name, const char* label,
uint16_t port, EventStore& ev) {
vc_callbacks cb{on_event, nullptr, &ev};
vc_config cfg{label, "0.1", VC_LOG_OFF};
client = vc_client_create(&cfg, cb);
if (!client) return false;
ev.client = client;
ev.label = label;
if (vc_connect(client, "127.0.0.1", port) != VC_OK) return false;
if (vc_authenticate_guest(client, name) != VC_OK) return false;
if (!wait_for(ev, [](EventStore& s) { return s.auth_ok; }, 8000)) return false;
if (!wait_for(ev, [](EventStore& s) { return s.channel_list_received; }, 3000)) return false;
return true;
}
static std::vector<int16_t> make_sine_mono(int n) {
std::vector<int16_t> pcm(static_cast<size_t>(n));
for (int i = 0; i < n; ++i) {
float t = static_cast<float>(i) / 48000.0f;
pcm[i] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * 440.0f * t) * 16000.0f);
}
return pcm;
}
// Loud-left / silent-right interleaved stereo (n samples per channel).
static std::vector<int16_t> make_sine_stereo(int n) {
std::vector<int16_t> pcm(static_cast<size_t>(n) * 2);
for (int i = 0; i < n; ++i) {
float t = static_cast<float>(i) / 48000.0f;
pcm[i * 2] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * 440.0f * t) * 16000.0f);
pcm[i * 2 + 1] = 0;
}
return pcm;
}
// ── PCM sink state ────────────────────────────────────────────────────────────
struct SinkData {
std::mutex mu;
std::condition_variable cv;
std::atomic<int> call_count{0};
uint32_t last_user_id = 0;
uint32_t last_stream_id = 0;
uint32_t last_channels = 0;
uint32_t last_sample_rate = 0;
int64_t total_energy = 0;
int64_t left_energy = 0; // sum |pcm[i*2]| for stereo frames
int64_t right_energy = 0; // sum |pcm[i*2+1]| for stereo frames
};
static void pcm_sink(void* user, uint32_t uid, uint32_t sid,
const int16_t* pcm, size_t n, uint32_t channels, uint32_t sr) {
auto* d = static_cast<SinkData*>(user);
std::lock_guard lk(d->mu);
d->last_user_id = uid;
d->last_stream_id = sid;
d->last_channels = channels;
d->last_sample_rate = sr;
for (size_t i = 0; i < n; ++i) {
if (channels == 2) {
d->left_energy += std::abs(static_cast<int>(pcm[i * 2]));
d->right_energy += std::abs(static_cast<int>(pcm[i * 2 + 1]));
}
for (uint32_t c = 0; c < channels; ++c)
d->total_energy += std::abs(static_cast<int>(pcm[i * channels + c]));
}
d->call_count.fetch_add(1, std::memory_order_relaxed);
d->cv.notify_all();
}
static bool sink_wait(SinkData& d, int timeout_ms) {
auto deadline = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout_ms);
std::unique_lock lk(d.mu);
return d.cv.wait_until(lk, deadline, [&] { return d.call_count.load() > 0; });
}
// ── test_feed_pcm_round_trip ──────────────────────────────────────────────────
static void test_feed_pcm_round_trip(uint16_t port) {
std::printf("test_feed_pcm_round_trip: start\n");
EventStore evA, evB;
vc_client *clientA = nullptr, *clientB = nullptr;
CHECK(connect_guest(clientA, "PcmA", "pcm-a", port, evA));
CHECK(connect_guest(clientB, "PcmB", "pcm-b", port, evB));
if (!clientA || !clientB) goto cleanup_rt;
{
uint32_t a_uid = 0;
{ std::lock_guard lk(evA.mu); a_uid = evA.self_user_id; }
// B registers a sink before A starts speaking.
SinkData sink;
CHECK(vc_set_pcm_sink(clientB, pcm_sink, &sink) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(500));
// A announces a MIC stream in Lobby.
vc_stream_desc desc{};
desc.kind = VC_STREAM_MIC;
uint32_t a_sid = 0;
CHECK(vc_stream_start(clientA, &desc, &a_sid) == VC_OK);
bool b_saw_a = wait_for(evB, [&](EventStore& s) {
for (auto& [uid, sid] : s.streams_started)
if (uid == a_uid) return true;
return false;
}, 5000);
CHECK(b_saw_a);
std::this_thread::sleep_for(std::chrono::milliseconds(300));
// A feeds 250 mono frames via the new public API.
auto sine = make_sine_mono(960);
for (int i = 0; i < 250; ++i)
CHECK(vc_stream_feed_pcm(clientA, a_sid, sine.data(), 960, 1) == VC_OK);
// Wait for the sink to fire at least once (decode arrived).
bool fired = sink_wait(sink, 5000);
CHECK(fired);
std::this_thread::sleep_for(std::chrono::milliseconds(1500));
int calls = sink.call_count.load();
int64_t energy = sink.total_energy;
std::printf("test_feed_pcm_round_trip: sink calls=%d energy=%lld\n",
calls, static_cast<long long>(energy));
CHECK(calls > 0);
CHECK(energy > 0);
vc_stream_stop(clientA, a_sid);
}
cleanup_rt:
if (clientA) { vc_disconnect(clientA); vc_client_destroy(clientA); }
if (clientB) { vc_disconnect(clientB); vc_client_destroy(clientB); }
std::printf("test_feed_pcm_round_trip: done\n");
}
// ── test_feed_pcm_stereo ──────────────────────────────────────────────────────
static void test_feed_pcm_stereo(uint16_t port) {
std::printf("test_feed_pcm_stereo: start\n");
EventStore evA, evB;
vc_client *clientA = nullptr, *clientB = nullptr;
CHECK(connect_guest(clientA, "StA", "stereo-a", port, evA));
CHECK(connect_guest(clientB, "StB", "stereo-b", port, evB));
if (!clientA || !clientB) goto cleanup_st;
{
uint32_t a_uid = 0;
{ std::lock_guard lk(evA.mu); a_uid = evA.self_user_id; }
// Both join Music Room (channel 2, stereo/128kbps) so the encoder is stereo.
CHECK(vc_join_channel(clientA, 2, nullptr) == VC_OK);
CHECK(vc_join_channel(clientB, 2, nullptr) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(600));
SinkData sink;
CHECK(vc_set_pcm_sink(clientB, pcm_sink, &sink) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(300));
// A announces a MIC stream — effective_params will be stereo (Music Room config).
vc_stream_desc desc{};
desc.kind = VC_STREAM_MIC;
uint32_t a_sid = 0;
CHECK(vc_stream_start(clientA, &desc, &a_sid) == VC_OK);
bool b_saw_a = wait_for(evB, [&](EventStore& s) {
for (auto& [uid, sid] : s.streams_started)
if (uid == a_uid) return true;
return false;
}, 5000);
CHECK(b_saw_a);
std::this_thread::sleep_for(std::chrono::milliseconds(300));
// A feeds 250 stereo frames: loud-L / silent-R.
auto stereo = make_sine_stereo(960);
for (int i = 0; i < 250; ++i)
CHECK(vc_stream_feed_pcm(clientA, a_sid, stereo.data(), 960, 2) == VC_OK);
bool fired = sink_wait(sink, 5000);
CHECK(fired);
std::this_thread::sleep_for(std::chrono::milliseconds(1500));
int64_t L = sink.left_energy;
int64_t R = sink.right_energy;
uint32_t ch = sink.last_channels;
std::printf("test_feed_pcm_stereo: channels=%u L_energy=%lld R_energy=%lld\n",
ch, static_cast<long long>(L), static_cast<long long>(R));
CHECK(ch == 2); // decoder delivered stereo frames
CHECK(L > 0); // left channel has signal
// Opus stereo coding (mid/side): R won't be exactly 0 after decode, but should be
// substantially quieter than L. Allow up to 30% leakage.
CHECK(R < L || L == 0); // L >= R (loud-L / quiet-R)
vc_stream_stop(clientA, a_sid);
}
cleanup_st:
if (clientA) { vc_disconnect(clientA); vc_client_destroy(clientA); }
if (clientB) { vc_disconnect(clientB); vc_client_destroy(clientB); }
std::printf("test_feed_pcm_stereo: done\n");
}
// ── test_pcm_sink ─────────────────────────────────────────────────────────────
static void test_pcm_sink(uint16_t port) {
std::printf("test_pcm_sink: start\n");
EventStore evA, evB;
vc_client *clientA = nullptr, *clientB = nullptr;
CHECK(connect_guest(clientA, "SnkA", "snk-a", port, evA));
CHECK(connect_guest(clientB, "SnkB", "snk-b", port, evB));
if (!clientA || !clientB) goto cleanup_sk;
{
uint32_t a_uid = 0;
{ std::lock_guard lk(evA.mu); a_uid = evA.self_user_id; }
SinkData sink;
CHECK(vc_set_pcm_sink(clientB, pcm_sink, &sink) == VC_OK);
std::this_thread::sleep_for(std::chrono::milliseconds(500));
vc_stream_desc desc{};
desc.kind = VC_STREAM_MIC;
uint32_t a_sid = 0;
CHECK(vc_stream_start(clientA, &desc, &a_sid) == VC_OK);
bool b_saw_a = wait_for(evB, [&](EventStore& s) {
for (auto& [uid, sid] : s.streams_started)
if (uid == a_uid) return true;
return false;
}, 5000);
CHECK(b_saw_a);
// Capture the stream_id that B observed for A's stream.
uint32_t b_a_sid = 0;
{ std::lock_guard lk(evB.mu);
for (auto& [uid, sid] : evB.streams_started)
if (uid == a_uid) { b_a_sid = sid; break; }
}
CHECK(b_a_sid != 0);
std::this_thread::sleep_for(std::chrono::milliseconds(300));
auto sine = make_sine_mono(960);
for (int i = 0; i < 250; ++i)
vc_stream_feed_pcm(clientA, a_sid, sine.data(), 960, 1);
bool fired = sink_wait(sink, 5000);
CHECK(fired);
std::this_thread::sleep_for(std::chrono::milliseconds(1500));
std::printf("test_pcm_sink: calls=%d user_id=%u stream_id=%u sr=%u energy=%lld\n",
sink.call_count.load(),
sink.last_user_id, sink.last_stream_id,
sink.last_sample_rate,
static_cast<long long>(sink.total_energy));
CHECK(sink.call_count.load() > 0);
CHECK(sink.total_energy > 0);
CHECK(sink.last_user_id == a_uid); // source user matches
CHECK(sink.last_stream_id == b_a_sid); // source stream matches
CHECK(sink.last_sample_rate == 48000); // always 48000
// Disable sink — subsequent frames must not reach the callback.
CHECK(vc_set_pcm_sink(clientB, nullptr, nullptr) == VC_OK);
int count_before_disable = sink.call_count.load();
// Feed more frames after disabling.
for (int i = 0; i < 100; ++i)
vc_stream_feed_pcm(clientA, a_sid, sine.data(), 960, 1);
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
// The count must not have increased (sink was disabled).
int count_after = sink.call_count.load();
std::printf("test_pcm_sink: count_before_disable=%d count_after=%d\n",
count_before_disable, count_after);
CHECK(count_after == count_before_disable);
vc_stream_stop(clientA, a_sid);
}
cleanup_sk:
if (clientA) { vc_disconnect(clientA); vc_client_destroy(clientA); }
if (clientB) { vc_disconnect(clientB); vc_client_destroy(clientB); }
std::printf("test_pcm_sink: done\n");
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
auto tmp = std::filesystem::temp_directory_path() /
("vctest_extpcm_" + std::to_string(
std::chrono::steady_clock::now().time_since_epoch().count()));
std::filesystem::create_directories(tmp);
std::string data_dir = tmp.string();
std::atomic<uint16_t> bound_port{0};
std::mutex ready_mu;
std::condition_variable ready_cv;
bool ready{false};
voicecat::server::Config cfg;
cfg.data_dir = data_dir;
cfg.bind_port = 0;
cfg.media_port = 0;
cfg.server_name = "VoiceCat-ExtPcmTest";
cfg.allow_guests = true;
cfg.on_ready = [&](uint16_t p) {
bound_port.store(p);
{ std::lock_guard lk(ready_mu); ready = true; }
ready_cv.notify_all();
};
voicecat::server::Server server(cfg);
std::thread server_thread([&] { server.run(); });
{
std::unique_lock lk(ready_mu);
bool ok = ready_cv.wait_for(lk, std::chrono::seconds(10), [&] { return ready; });
if (!ok) {
std::printf("FAIL: server did not start\n");
server.stop(); server_thread.join();
std::filesystem::remove_all(tmp);
return 1;
}
}
uint16_t port = bound_port.load();
std::printf("external_pcm: server ready on :%u\n", port);
test_feed_pcm_round_trip(port);
test_feed_pcm_stereo(port);
test_pcm_sink(port);
server.stop();
server_thread.join();
std::filesystem::remove_all(tmp);
if (g_failures == 0) {
std::printf("external_pcm: all checks passed\n");
return 0;
}
std::printf("external_pcm: %d failure(s)\n", g_failures);
return 1;
}
#else // !VOICECAT_HAS_NET
int main() {
std::printf("external_pcm: SKIP (VOICECAT_HAS_NET not defined)\n");
return 0;
}
#endif // VOICECAT_HAS_NET

View File

@@ -69,7 +69,7 @@ int main() {
CHECK(opus_len > 0); CHECK(opus_len > 0);
const uint32_t ssrc = 1; const uint32_t ssrc = 1;
engine.init_recv_stream(ssrc, op); engine.init_recv_stream(ssrc, op, /*user_id=*/0, /*stream_id=*/0);
// Push one real frame to seed the decoder. // Push one real frame to seed the decoder.
voicecat::audio::JitterBuffer::Frame f; voicecat::audio::JitterBuffer::Frame f;

View File

@@ -205,7 +205,7 @@ static void test_stereo_mix() {
int opus_len = enc.encode(interleaved.data(), frame_samples, opus_buf, sizeof(opus_buf)); int opus_len = enc.encode(interleaved.data(), frame_samples, opus_buf, sizeof(opus_buf));
CHECK(opus_len > 0); CHECK(opus_len > 0);
engine.init_recv_stream(/*ssrc=*/1, stereo_params); engine.init_recv_stream(/*ssrc=*/1, stereo_params, /*user_id=*/0, /*stream_id=*/0);
voicecat::audio::JitterBuffer::Frame f; voicecat::audio::JitterBuffer::Frame f;
f.seq = 0; f.seq = 0;
@@ -291,7 +291,7 @@ static void test_loopback_stereo_capture() {
// Decode + mix — same recv path as test_stereo_mix. A real stereo bitstream should // Decode + mix — same recv path as test_stereo_mix. A real stereo bitstream should
// survive with L != R; a mono-downmixed-then-upmixed bitstream would have L == R. // survive with L != R; a mono-downmixed-then-upmixed bitstream would have L == R.
engine.init_recv_stream(/*ssrc=*/3, stereo_params); engine.init_recv_stream(/*ssrc=*/3, stereo_params, /*user_id=*/0, /*stream_id=*/0);
voicecat::audio::JitterBuffer::Frame f; voicecat::audio::JitterBuffer::Frame f;
f.seq = 0; f.seq = 0;
f.timestamp = 0; f.timestamp = 0;
@@ -345,7 +345,7 @@ static void test_playout_resync() {
int opus_len = enc.encode(sine.data(), frame_samples, opus_buf, sizeof(opus_buf)); int opus_len = enc.encode(sine.data(), frame_samples, opus_buf, sizeof(opus_buf));
CHECK(opus_len > 0); CHECK(opus_len > 0);
engine.init_recv_stream(/*ssrc=*/2, mono_params); engine.init_recv_stream(/*ssrc=*/2, mono_params, /*user_id=*/0, /*stream_id=*/0);
std::vector<int16_t> out(static_cast<size_t>(frame_samples) * 2, 0); std::vector<int16_t> out(static_cast<size_t>(frame_samples) * 2, 0);
@@ -639,7 +639,7 @@ static void test_stereo_mic_capture() {
// Decode + mix — same recv path as test_stereo_mix. A real stereo bitstream should // Decode + mix — same recv path as test_stereo_mix. A real stereo bitstream should
// survive with L != R; a mono-downmixed-then-upmixed bitstream would have L == R. // survive with L != R; a mono-downmixed-then-upmixed bitstream would have L == R.
engine.init_recv_stream(/*ssrc=*/5, stereo_params); engine.init_recv_stream(/*ssrc=*/5, stereo_params, /*user_id=*/0, /*stream_id=*/0);
voicecat::audio::JitterBuffer::Frame f; voicecat::audio::JitterBuffer::Frame f;
f.seq = 0; f.seq = 0;
f.timestamp = 0; f.timestamp = 0;