feat(apple): screen-audio sharing -- macOS ScreenCaptureKit, iOS ReplayKit

Implement system/desktop audio sharing on the Apple clients, feeding the
existing SCREEN_AUDIO Opus -> AEAD -> UDP path via vc_stream_feed_pcm. No
C++/protocol/codec changes -- the core was already ready (the Windows-only
loopback is #ifdef VOICECAT_HAS_LOOPBACK; off Windows the stream just waits
for fed PCM). Audio only; video is dropped.

macOS (in-process):
- ScreenAudioCapture.swift drives an audio-only SCStream
  (excludesCurrentProcessAudio), converts Float32 -> int16 in the channel's
  mono/stereo mode, and calls feedPcm. Capture starts on the self
  .streamStarted event (effective config known then). Wired into
  MainWindowController.screenAudioClicked().

iOS (forward-to-host, single session):
- VoiceCatBroadcast: a ReplayKit Broadcast Upload Extension consumes
  .audioApp only, resamples to 48kHz int16 stereo (AVAudioConverter), and
  writes a shared App Group SPSC ring (BroadcastAudioRing.swift). It does
  not link libvoicecat.
- Host BroadcastAudioPump drains the ring (reacting to the extension's
  Darwin notifications) and feeds the SCREEN_AUDIO stream it owns, downmixing
  to mono when the channel is mono. Screen audio appears as a second stream
  of the same user; no credentials persisted. UI is RPSystemBroadcastPicker
  View in VoiceControlsView. Removes the speculative BroadcastCredentials.

Docs: voice.md s9, CLAUDE.md status, PROGRESS.md.
This commit is contained in:
2026-06-21 00:14:31 +02:00
parent 6ab78fa792
commit 8c90e250f0
17 changed files with 985 additions and 69 deletions

View File

@@ -274,23 +274,31 @@ normal stream; only the *source* is platform-specific.
| Platform | Mechanism | Notes |
|----------|-----------|-------|
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | OS requires screen-recording permission; capture happens in the main app. Swift converts each `CMSampleBuffer` (Float32) → int16 and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). The extension calls `vc_stream_feed_pcm` to drive the encode path without a hardware device. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
### iOS detail
The extension **captures**, the host app **sends**. Unlike a self-connecting extension, this
keeps a **single session** — the screen-audio share appears as a second stream of the *same*
user (exactly like macOS/Windows), and no credentials are ever persisted to disk.
- The user starts a broadcast from Control Center's screen-record button; we surface it via
`RPSystemBroadcastPickerView` from inside the app for one-tap start.
- The **Broadcast Upload Extension** receives `RPSampleBufferType.audioApp` (system/app
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
audio-only consumer stays comfortably inside it.
- The extension is a *separate process*. It reads the active session token and server
endpoint from a shared **App Group** container, opens a UDP media channel, and calls
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the
Opus encode + AEAD + send path directly, with no `ma_device` or audio hardware. The host
app announces the stream over its control channel (`StreamAnnounce`) so the server and
peers learn about it.
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final
frame with the `last-frame-before-stop` flag and the host app emits `StreamStop`.
`RPSystemBroadcastPickerView` from inside the app (`VoiceControlsView`) for one-tap start.
- The **Broadcast Upload Extension** (`clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift`)
receives `RPSampleBufferType.audioApp` (system/app audio), `.audioMic`, and `.video`. We
consume **`.audioApp`** only and drop video + mic — video is what blows the **~50 MB**
extension memory budget, so an audio-only consumer stays comfortably inside it. The extension
does **not** link `libvoicecat`.
- The extension converts each chunk to the core's canonical format (48 kHz int16 stereo, via
`AVAudioConverter`) and writes it into a lock-free single-producer/single-consumer ring in a
shared **App Group** mmap'd file (`clients/apple/iOS/Shared/BroadcastAudioRing.swift`). It
posts Darwin notifications on start/stop so the host reacts promptly.
- The **host app** owns the stream: its `BroadcastAudioPump` announces the `SCREEN_AUDIO`
stream over the control channel (`StreamAnnounce`), drains the ring, and calls
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the Opus
encode + AEAD + send path. It downmixes to mono when the channel's effective config is mono.
- Mic + voice also run in the host app. When the broadcast stops (`broadcastFinished`), the
extension clears the ring's active flag (and posts a Darwin notification); the host stops
feeding and emits `StreamStop`. The host must be alive to relay — always true while in a
call (the app declares the `audio` background mode).