feat(apple): screen-audio sharing -- macOS ScreenCaptureKit, iOS ReplayKit
Implement system/desktop audio sharing on the Apple clients, feeding the existing SCREEN_AUDIO Opus -> AEAD -> UDP path via vc_stream_feed_pcm. No C++/protocol/codec changes -- the core was already ready (the Windows-only loopback is #ifdef VOICECAT_HAS_LOOPBACK; off Windows the stream just waits for fed PCM). Audio only; video is dropped. macOS (in-process): - ScreenAudioCapture.swift drives an audio-only SCStream (excludesCurrentProcessAudio), converts Float32 -> int16 in the channel's mono/stereo mode, and calls feedPcm. Capture starts on the self .streamStarted event (effective config known then). Wired into MainWindowController.screenAudioClicked(). iOS (forward-to-host, single session): - VoiceCatBroadcast: a ReplayKit Broadcast Upload Extension consumes .audioApp only, resamples to 48kHz int16 stereo (AVAudioConverter), and writes a shared App Group SPSC ring (BroadcastAudioRing.swift). It does not link libvoicecat. - Host BroadcastAudioPump drains the ring (reacting to the extension's Darwin notifications) and feeds the SCREEN_AUDIO stream it owns, downmixing to mono when the channel is mono. Screen audio appears as a second stream of the same user; no credentials persisted. UI is RPSystemBroadcastPicker View in VoiceControlsView. Removes the speculative BroadcastCredentials. Docs: voice.md s9, CLAUDE.md status, PROGRESS.md.
This commit is contained in:
29
PROGRESS.md
29
PROGRESS.md
@@ -10,6 +10,18 @@ up instantly. Newest status at the top.
|
||||
|
||||
## ▶ Where we left off / next action
|
||||
|
||||
- **Done (2026-06-21):** **Screen-audio sharing on macOS + iOS.** macOS uses ScreenCaptureKit
|
||||
(`ScreenAudioCapture.swift`) → `vc_stream_feed_pcm`; iOS uses a ReplayKit Broadcast Upload
|
||||
Extension (`VoiceCatBroadcast`) that forwards captured `.audioApp` PCM through a shared App
|
||||
Group SPSC ring (`BroadcastAudioRing.swift`) to the host's `BroadcastAudioPump`, which owns
|
||||
the `SCREEN_AUDIO` stream and feeds it — single session, no creds on disk. No C++ changes
|
||||
(the core was already ready). macOS `xcodebuild` Debug BUILD SUCCEEDED; iOS app + extension
|
||||
build for device (the xcframework sim slice is arm64-only, so x86_64-simulator link is N/A).
|
||||
Next: on-device end-to-end verification (two clients hear the shared audio; iOS broadcast
|
||||
start/stop). NOTE: `ctest --preset dev` is 22/23 — `external_pcm` passes its assertions but
|
||||
aborts at shutdown (`mutex lock failed`), a **pre-existing** teardown crash unrelated to this
|
||||
change (no C++ was modified).
|
||||
|
||||
- **Done (2026-06-20):** **macOS client UI overhaul** — mirrors the Windows client's UI
|
||||
overhaul (commit 97fa659 + 540ec13), adapted to Mac-native conventions. Also fixed and
|
||||
verified the previously-uncompiled Swift changes from the external PCM feed/tap commit
|
||||
@@ -428,11 +440,18 @@ iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sh
|
||||
checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs
|
||||
already have full channel CRUD wired; this is an additive checkbox on the existing audio-config
|
||||
section. (Core/protocol/ABI all done — this is UI-only work.)
|
||||
- [ ] **macOS ScreenCaptureKit screen-audio** — `startStream(.screenAudio)` in macOS client
|
||||
announces the stream but `start_loopback_capture()` returns false (no `VOICECAT_HAS_LOOPBACK`
|
||||
on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships.
|
||||
- [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target,
|
||||
App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`.
|
||||
- [x] **macOS ScreenCaptureKit screen-audio** — done 2026-06-21. `ScreenAudioCapture.swift`
|
||||
drives an `SCStream` (audio-only, `excludesCurrentProcessAudio`), converts Float32 →
|
||||
int16 in the channel's mono/stereo mode, and calls `vc_stream_feed_pcm`. Capture starts on
|
||||
the self `.streamStarted` event (when the effective config is known); wired into
|
||||
`MainWindowController.screenAudioClicked()`.
|
||||
- [x] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — done 2026-06-21.
|
||||
Forward-to-host design: the extension (`SampleHandler.swift`) captures `.audioApp`,
|
||||
converts to 48 kHz int16 stereo, and writes a shared App Group SPSC ring
|
||||
(`BroadcastAudioRing.swift`); the host's `BroadcastAudioPump` owns the `SCREEN_AUDIO`
|
||||
stream and feeds via `vc_stream_feed_pcm` (single session, no creds on disk). UI is an
|
||||
`RPSystemBroadcastPickerView` in `VoiceControlsView`. (Replaced the speculative
|
||||
`BroadcastCredentials.swift` self-connecting design, now removed.)
|
||||
- [x] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — done
|
||||
2026-06-20. Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public API with
|
||||
stereo support. Adds a symmetric PCM sink fired on the playback thread per decoded remote
|
||||
|
||||
Reference in New Issue
Block a user