feat(apple): screen-audio sharing -- macOS ScreenCaptureKit, iOS ReplayKit

Implement system/desktop audio sharing on the Apple clients, feeding the
existing SCREEN_AUDIO Opus -> AEAD -> UDP path via vc_stream_feed_pcm. No
C++/protocol/codec changes -- the core was already ready (the Windows-only
loopback is #ifdef VOICECAT_HAS_LOOPBACK; off Windows the stream just waits
for fed PCM). Audio only; video is dropped.

macOS (in-process):
- ScreenAudioCapture.swift drives an audio-only SCStream
  (excludesCurrentProcessAudio), converts Float32 -> int16 in the channel's
  mono/stereo mode, and calls feedPcm. Capture starts on the self
  .streamStarted event (effective config known then). Wired into
  MainWindowController.screenAudioClicked().

iOS (forward-to-host, single session):
- VoiceCatBroadcast: a ReplayKit Broadcast Upload Extension consumes
  .audioApp only, resamples to 48kHz int16 stereo (AVAudioConverter), and
  writes a shared App Group SPSC ring (BroadcastAudioRing.swift). It does
  not link libvoicecat.
- Host BroadcastAudioPump drains the ring (reacting to the extension's
  Darwin notifications) and feeds the SCREEN_AUDIO stream it owns, downmixing
  to mono when the channel is mono. Screen audio appears as a second stream
  of the same user; no credentials persisted. UI is RPSystemBroadcastPicker
  View in VoiceControlsView. Removes the speculative BroadcastCredentials.

Docs: voice.md s9, CLAUDE.md status, PROGRESS.md.
This commit is contained in:
2026-06-21 00:14:31 +02:00
parent 6ab78fa792
commit 8c90e250f0
17 changed files with 985 additions and 69 deletions

View File

@@ -10,6 +10,18 @@ up instantly. Newest status at the top.
## ▶ Where we left off / next action
- **Done (2026-06-21):** **Screen-audio sharing on macOS + iOS.** macOS uses ScreenCaptureKit
(`ScreenAudioCapture.swift`) → `vc_stream_feed_pcm`; iOS uses a ReplayKit Broadcast Upload
Extension (`VoiceCatBroadcast`) that forwards captured `.audioApp` PCM through a shared App
Group SPSC ring (`BroadcastAudioRing.swift`) to the host's `BroadcastAudioPump`, which owns
the `SCREEN_AUDIO` stream and feeds it — single session, no creds on disk. No C++ changes
(the core was already ready). macOS `xcodebuild` Debug BUILD SUCCEEDED; iOS app + extension
build for device (the xcframework sim slice is arm64-only, so x86_64-simulator link is N/A).
Next: on-device end-to-end verification (two clients hear the shared audio; iOS broadcast
start/stop). NOTE: `ctest --preset dev` is 22/23 — `external_pcm` passes its assertions but
aborts at shutdown (`mutex lock failed`), a **pre-existing** teardown crash unrelated to this
change (no C++ was modified).
- **Done (2026-06-20):** **macOS client UI overhaul** — mirrors the Windows client's UI
overhaul (commit 97fa659 + 540ec13), adapted to Mac-native conventions. Also fixed and
verified the previously-uncompiled Swift changes from the external PCM feed/tap commit
@@ -428,11 +440,18 @@ iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sh
checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs
already have full channel CRUD wired; this is an additive checkbox on the existing audio-config
section. (Core/protocol/ABI all done — this is UI-only work.)
- [ ] **macOS ScreenCaptureKit screen-audio** — `startStream(.screenAudio)` in macOS client
announces the stream but `start_loopback_capture()` returns false (no `VOICECAT_HAS_LOOPBACK`
on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships.
- [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target,
App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`.
- [x] **macOS ScreenCaptureKit screen-audio** — done 2026-06-21. `ScreenAudioCapture.swift`
drives an `SCStream` (audio-only, `excludesCurrentProcessAudio`), converts Float32 →
int16 in the channel's mono/stereo mode, and calls `vc_stream_feed_pcm`. Capture starts on
the self `.streamStarted` event (when the effective config is known); wired into
`MainWindowController.screenAudioClicked()`.
- [x] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — done 2026-06-21.
Forward-to-host design: the extension (`SampleHandler.swift`) captures `.audioApp`,
converts to 48 kHz int16 stereo, and writes a shared App Group SPSC ring
(`BroadcastAudioRing.swift`); the host's `BroadcastAudioPump` owns the `SCREEN_AUDIO`
stream and feeds via `vc_stream_feed_pcm` (single session, no creds on disk). UI is an
`RPSystemBroadcastPickerView` in `VoiceControlsView`. (Replaced the speculative
`BroadcastCredentials.swift` self-connecting design, now removed.)
- [x] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — done
2026-06-20. Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public API with
stereo support. Adds a symmetric PCM sink fired on the playback thread per decoded remote