feat(apple): screen-audio sharing -- macOS ScreenCaptureKit, iOS ReplayKit
Implement system/desktop audio sharing on the Apple clients, feeding the existing SCREEN_AUDIO Opus -> AEAD -> UDP path via vc_stream_feed_pcm. No C++/protocol/codec changes -- the core was already ready (the Windows-only loopback is #ifdef VOICECAT_HAS_LOOPBACK; off Windows the stream just waits for fed PCM). Audio only; video is dropped. macOS (in-process): - ScreenAudioCapture.swift drives an audio-only SCStream (excludesCurrentProcessAudio), converts Float32 -> int16 in the channel's mono/stereo mode, and calls feedPcm. Capture starts on the self .streamStarted event (effective config known then). Wired into MainWindowController.screenAudioClicked(). iOS (forward-to-host, single session): - VoiceCatBroadcast: a ReplayKit Broadcast Upload Extension consumes .audioApp only, resamples to 48kHz int16 stereo (AVAudioConverter), and writes a shared App Group SPSC ring (BroadcastAudioRing.swift). It does not link libvoicecat. - Host BroadcastAudioPump drains the ring (reacting to the extension's Darwin notifications) and feeds the SCREEN_AUDIO stream it owns, downmixing to mono when the channel is mono. Screen audio appears as a second stream of the same user; no credentials persisted. UI is RPSystemBroadcastPicker View in VoiceControlsView. Removes the speculative BroadcastCredentials. Docs: voice.md s9, CLAUDE.md status, PROGRESS.md.
This commit is contained in:
@@ -274,23 +274,31 @@ normal stream; only the *source* is platform-specific.
|
||||
| Platform | Mechanism | Notes |
|
||||
|----------|-----------|-------|
|
||||
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | OS requires screen-recording permission; capture happens in the main app. Swift converts each `CMSampleBuffer` (Float32) → int16 and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). The extension calls `vc_stream_feed_pcm` to drive the encode path without a hardware device. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
||||
|
||||
### iOS detail
|
||||
|
||||
The extension **captures**, the host app **sends**. Unlike a self-connecting extension, this
|
||||
keeps a **single session** — the screen-audio share appears as a second stream of the *same*
|
||||
user (exactly like macOS/Windows), and no credentials are ever persisted to disk.
|
||||
|
||||
- The user starts a broadcast from Control Center's screen-record button; we surface it via
|
||||
`RPSystemBroadcastPickerView` from inside the app for one-tap start.
|
||||
- The **Broadcast Upload Extension** receives `RPSampleBufferType.audioApp` (system/app
|
||||
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
|
||||
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
|
||||
audio-only consumer stays comfortably inside it.
|
||||
- The extension is a *separate process*. It reads the active session token and server
|
||||
endpoint from a shared **App Group** container, opens a UDP media channel, and calls
|
||||
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the
|
||||
Opus encode + AEAD + send path directly, with no `ma_device` or audio hardware. The host
|
||||
app announces the stream over its control channel (`StreamAnnounce`) so the server and
|
||||
peers learn about it.
|
||||
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the
|
||||
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final
|
||||
frame with the `last-frame-before-stop` flag and the host app emits `StreamStop`.
|
||||
`RPSystemBroadcastPickerView` from inside the app (`VoiceControlsView`) for one-tap start.
|
||||
- The **Broadcast Upload Extension** (`clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift`)
|
||||
receives `RPSampleBufferType.audioApp` (system/app audio), `.audioMic`, and `.video`. We
|
||||
consume **`.audioApp`** only and drop video + mic — video is what blows the **~50 MB**
|
||||
extension memory budget, so an audio-only consumer stays comfortably inside it. The extension
|
||||
does **not** link `libvoicecat`.
|
||||
- The extension converts each chunk to the core's canonical format (48 kHz int16 stereo, via
|
||||
`AVAudioConverter`) and writes it into a lock-free single-producer/single-consumer ring in a
|
||||
shared **App Group** mmap'd file (`clients/apple/iOS/Shared/BroadcastAudioRing.swift`). It
|
||||
posts Darwin notifications on start/stop so the host reacts promptly.
|
||||
- The **host app** owns the stream: its `BroadcastAudioPump` announces the `SCREEN_AUDIO`
|
||||
stream over the control channel (`StreamAnnounce`), drains the ring, and calls
|
||||
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the Opus
|
||||
encode + AEAD + send path. It downmixes to mono when the channel's effective config is mono.
|
||||
- Mic + voice also run in the host app. When the broadcast stops (`broadcastFinished`), the
|
||||
extension clears the ring's active flag (and posts a Darwin notification); the host stops
|
||||
feeding and emits `StreamStop`. The host must be alive to relay — always true while in a
|
||||
call (the app declares the `audio` background mode).
|
||||
|
||||
Reference in New Issue
Block a user