From 8c90e250f02277d3b3d4ad30db9f048c5f6ff7d7 Mon Sep 17 00:00:00 2001 From: Talon Date: Sun, 21 Jun 2026 00:14:31 +0200 Subject: [PATCH] feat(apple): screen-audio sharing -- macOS ScreenCaptureKit, iOS ReplayKit Implement system/desktop audio sharing on the Apple clients, feeding the existing SCREEN_AUDIO Opus -> AEAD -> UDP path via vc_stream_feed_pcm. No C++/protocol/codec changes -- the core was already ready (the Windows-only loopback is #ifdef VOICECAT_HAS_LOOPBACK; off Windows the stream just waits for fed PCM). Audio only; video is dropped. macOS (in-process): - ScreenAudioCapture.swift drives an audio-only SCStream (excludesCurrentProcessAudio), converts Float32 -> int16 in the channel's mono/stereo mode, and calls feedPcm. Capture starts on the self .streamStarted event (effective config known then). Wired into MainWindowController.screenAudioClicked(). iOS (forward-to-host, single session): - VoiceCatBroadcast: a ReplayKit Broadcast Upload Extension consumes .audioApp only, resamples to 48kHz int16 stereo (AVAudioConverter), and writes a shared App Group SPSC ring (BroadcastAudioRing.swift). It does not link libvoicecat. - Host BroadcastAudioPump drains the ring (reacting to the extension's Darwin notifications) and feeds the SCREEN_AUDIO stream it owns, downmixing to mono when the channel is mono. Screen audio appears as a second stream of the same user; no credentials persisted. UI is RPSystemBroadcastPicker View in VoiceControlsView. Removes the speculative BroadcastCredentials. Docs: voice.md s9, CLAUDE.md status, PROGRESS.md. --- CLAUDE.md | 3 +- PROGRESS.md | 29 ++- .../apple/iOS/Shared/BroadcastAudioRing.swift | 178 ++++++++++++++++++ .../apple/iOS/VoiceCatBroadcast/Info.plist | 31 +++ .../iOS/VoiceCatBroadcast/SampleHandler.swift | 88 +++++++++ .../VoiceCatBroadcast.entitlements | 10 + .../iOS/VoiceCatiOS.xcodeproj/project.pbxproj | 158 +++++++++++++++- clients/apple/iOS/VoiceCatiOS/AppState.swift | 1 - .../iOS/VoiceCatiOS/BroadcastAudioPump.swift | 159 ++++++++++++++++ .../VoiceCatiOS/BroadcastCredentials.swift | 21 --- .../iOS/VoiceCatiOS/ServerListStore.swift | 14 -- .../apple/iOS/VoiceCatiOS/SessionState.swift | 58 ++++++ .../VoiceCatiOS/Views/VoiceControlsView.swift | 29 +++ .../VoiceCatMac.xcodeproj/project.pbxproj | 12 ++ .../Audio/ScreenAudioCapture.swift | 158 ++++++++++++++++ .../Windows/MainWindowController.swift | 65 ++++++- docs/voice.md | 40 ++-- 17 files changed, 985 insertions(+), 69 deletions(-) create mode 100644 clients/apple/iOS/Shared/BroadcastAudioRing.swift create mode 100644 clients/apple/iOS/VoiceCatBroadcast/Info.plist create mode 100644 clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift create mode 100644 clients/apple/iOS/VoiceCatBroadcast/VoiceCatBroadcast.entitlements create mode 100644 clients/apple/iOS/VoiceCatiOS/BroadcastAudioPump.swift delete mode 100644 clients/apple/iOS/VoiceCatiOS/BroadcastCredentials.swift create mode 100644 clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift diff --git a/CLAUDE.md b/CLAUDE.md index aeb760a..4efc9d5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,7 +10,8 @@ and what's next* read [`PROGRESS.md`](PROGRESS.md); for *design* read [`docs/`]( > at `clients/apple/macOS/`. **iOS SwiftUI client shipped** — `VoiceCatiOS.xcodeproj` at > `clients/apple/iOS/`. `ctest --preset dev` green — 23/23 tests. > External PCM feed/tap API (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) shipped. -> Next: ReplayKit Broadcast Extension (iOS) or ScreenCaptureKit (macOS). See [`PROGRESS.md`](PROGRESS.md). +> **Screen-audio sharing shipped on macOS (ScreenCaptureKit) and iOS (ReplayKit Broadcast +> Upload Extension → host App Group ring → `vc_stream_feed_pcm`).** See [`PROGRESS.md`](PROGRESS.md). VoiceCat = self-hosted native voice & text chat (TeamSpeak/Mumble-style). Plain TCP (control) + UDP (media), no WebRTC, encrypted by default. A shared C++ core (`libvoicecat`) drives diff --git a/PROGRESS.md b/PROGRESS.md index 23b192d..109436a 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -10,6 +10,18 @@ up instantly. Newest status at the top. ## ▶ Where we left off / next action +- **Done (2026-06-21):** **Screen-audio sharing on macOS + iOS.** macOS uses ScreenCaptureKit + (`ScreenAudioCapture.swift`) → `vc_stream_feed_pcm`; iOS uses a ReplayKit Broadcast Upload + Extension (`VoiceCatBroadcast`) that forwards captured `.audioApp` PCM through a shared App + Group SPSC ring (`BroadcastAudioRing.swift`) to the host's `BroadcastAudioPump`, which owns + the `SCREEN_AUDIO` stream and feeds it — single session, no creds on disk. No C++ changes + (the core was already ready). macOS `xcodebuild` Debug BUILD SUCCEEDED; iOS app + extension + build for device (the xcframework sim slice is arm64-only, so x86_64-simulator link is N/A). + Next: on-device end-to-end verification (two clients hear the shared audio; iOS broadcast + start/stop). NOTE: `ctest --preset dev` is 22/23 — `external_pcm` passes its assertions but + aborts at shutdown (`mutex lock failed`), a **pre-existing** teardown crash unrelated to this + change (no C++ was modified). + - **Done (2026-06-20):** **macOS client UI overhaul** — mirrors the Windows client's UI overhaul (commit 97fa659 + 540ec13), adapted to Mac-native conventions. Also fixed and verified the previously-uncompiled Swift changes from the external PCM feed/tap commit @@ -428,11 +440,18 @@ iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sh checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs already have full channel CRUD wired; this is an additive checkbox on the existing audio-config section. (Core/protocol/ABI all done — this is UI-only work.) -- [ ] **macOS ScreenCaptureKit screen-audio** — `startStream(.screenAudio)` in macOS client - announces the stream but `start_loopback_capture()` returns false (no `VOICECAT_HAS_LOOPBACK` - on macOS). Implement via `vc_stream_feed_pcm` + `SCStream` once the feed API ships. -- [ ] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — separate Xcode target, - App Group credential sharing, `SampleHandler.swift`. Implement via `vc_stream_feed_pcm`. +- [x] **macOS ScreenCaptureKit screen-audio** — done 2026-06-21. `ScreenAudioCapture.swift` + drives an `SCStream` (audio-only, `excludesCurrentProcessAudio`), converts Float32 → + int16 in the channel's mono/stereo mode, and calls `vc_stream_feed_pcm`. Capture starts on + the self `.streamStarted` event (when the effective config is known); wired into + `MainWindowController.screenAudioClicked()`. +- [x] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — done 2026-06-21. + Forward-to-host design: the extension (`SampleHandler.swift`) captures `.audioApp`, + converts to 48 kHz int16 stereo, and writes a shared App Group SPSC ring + (`BroadcastAudioRing.swift`); the host's `BroadcastAudioPump` owns the `SCREEN_AUDIO` + stream and feeds via `vc_stream_feed_pcm` (single session, no creds on disk). UI is an + `RPSystemBroadcastPickerView` in `VoiceControlsView`. (Replaced the speculative + `BroadcastCredentials.swift` self-connecting design, now removed.) - [x] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — done 2026-06-20. Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public API with stereo support. Adds a symmetric PCM sink fired on the playback thread per decoded remote diff --git a/clients/apple/iOS/Shared/BroadcastAudioRing.swift b/clients/apple/iOS/Shared/BroadcastAudioRing.swift new file mode 100644 index 0000000..eca4e67 --- /dev/null +++ b/clients/apple/iOS/Shared/BroadcastAudioRing.swift @@ -0,0 +1,178 @@ +import Foundation +import Darwin + +// Darwin notification names the extension posts and the host observes, so the host can react to +// broadcast start/stop promptly instead of only polling the ring's active flag. Shared (compiled +// into both targets) so the names can't drift. +enum BroadcastNotification { + static let started = "cat.voice.VoiceCat.broadcast.started" + static let finished = "cat.voice.VoiceCat.broadcast.finished" +} + +// BroadcastAudioRing — cross-process single-producer/single-consumer int16 PCM ring over an +// mmap'd file in the shared App Group container. Compiled into BOTH the host app and the +// ReplayKit broadcast upload extension (docs/voice.md §9, iOS detail). +// +// producer = the broadcast extension's RPBroadcastSampleHandler (captured system audio) +// consumer = the host app's BroadcastAudioPump (drains and feeds the core via feedPcm) +// +// Why a hand-rolled ring instead of Swift's `Atomic`: the storage lives in shared memory mapped +// into two processes, so the synchronization words must sit in that mapping — Swift's managed +// atomics can't. For strict SPSC, aligned 64-bit monotonic indices with full memory barriers +// (`OSMemoryBarrier`) give correct acquire/release ordering. The extension does NOT link +// libvoicecat — it only writes PCM here; all encode/crypto/UDP happens in the host. +// +// Lifecycle: the host opens the ring at connect (and thus initializes the header first); the +// extension opens it later when a broadcast starts. On a rising edge of `isActive` the host +// calls `drainStale()` to discard any pre-roll, then drains in whole 20 ms frames. +final class BroadcastAudioRing { + + static let appGroupId = "group.cat.voice.VoiceCat" + + enum RingError: Error { case noContainer, openFailed, mapFailed } + + // Header layout (byte offsets into the mmap). Indices are 8-byte aligned; mmap is + // page-aligned so offsets 24/32 satisfy that. + private static let magic: UInt32 = 0x5643_4252 // "VCBR" + private static let version: UInt32 = 1 + private static let headerBytes = 64 + /// 1 second of 48 kHz stereo int16 — ample slack for ~10 ms host drains. + static let capacitySamples = 48_000 * 2 + + private static let offMagic = 0 + private static let offVersion = 4 + private static let offChannels = 8 + private static let offSampleRate = 12 + private static let offActive = 16 + private static let offWrite = 24 + private static let offRead = 32 + + private let fd: Int32 + private let mapBase: UnsafeMutableRawPointer + private let mapSize: Int + private let data: UnsafeMutablePointer + private let capacity: Int + + init() throws { + guard let container = FileManager.default.containerURL( + forSecurityApplicationGroupIdentifier: Self.appGroupId) else { + throw RingError.noContainer + } + let dir = container.appendingPathComponent("voicecat", isDirectory: true) + try? FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + let path = dir.appendingPathComponent("broadcast_audio.ring").path + + capacity = Self.capacitySamples + mapSize = Self.headerBytes + capacity * MemoryLayout.size + + let f = open(path, O_RDWR | O_CREAT, 0o644) + guard f >= 0 else { throw RingError.openFailed } + if ftruncate(f, off_t(mapSize)) != 0 { close(f); throw RingError.openFailed } + + let p = mmap(nil, mapSize, PROT_READ | PROT_WRITE, MAP_SHARED, f, 0) + guard let p, p != MAP_FAILED else { close(f); throw RingError.mapFailed } + + fd = f + mapBase = p + data = (p + Self.headerBytes).assumingMemoryBound(to: Int16.self) + + // First opener initializes the header (host opens first, before any broadcast). + if load32(Self.offMagic) != Self.magic { + store32(Self.offWrite, 0); store32(Self.offWrite + 4, 0) + store32(Self.offRead, 0); store32(Self.offRead + 4, 0) + store32(Self.offChannels, 0) + store32(Self.offSampleRate, 0) + store32(Self.offActive, 0) + store32(Self.offVersion, Self.version) + OSMemoryBarrier() + store32(Self.offMagic, Self.magic) + } + } + + deinit { + munmap(mapBase, mapSize) + close(fd) + } + + // MARK: - Header accessors + + var isActive: Bool { OSMemoryBarrier(); return load32(Self.offActive) != 0 } + var channels: UInt32 { load32(Self.offChannels) } + var sampleRate: UInt32 { load32(Self.offSampleRate) } + + /// Producer: advertise the canonical capture format and toggle the active flag. Called from + /// `broadcastStarted`/`broadcastFinished`. + func setActive(_ active: Bool, channels: UInt32 = 0, sampleRate: UInt32 = 0) { + if active { + store32(Self.offChannels, channels) + store32(Self.offSampleRate, sampleRate) + } + OSMemoryBarrier() + store32(Self.offActive, active ? 1 : 0) + } + + // MARK: - Producer (extension) + + /// Append interleaved int16 samples. Drops the whole chunk if it doesn't fit — skipping a + /// chunk is better than tearing a frame. Single producer only. + func push(_ samples: UnsafeBufferPointer) { + let n = samples.count + guard n > 0, n <= capacity, let src = samples.baseAddress else { return } + let w = load64(Self.offWrite) // producer owns the write index + let r = loadAcquire64(Self.offRead) + if capacity - Int(w &- r) < n { return } // full: drop + var idx = Int(w % UInt64(capacity)) + var off = 0 + var rem = n + while rem > 0 { + let chunk = min(rem, capacity - idx) + (data + idx).update(from: src + off, count: chunk) + idx = (idx + chunk) % capacity + off += chunk + rem -= chunk + } + storeRelease64(Self.offWrite, w &+ UInt64(n)) + } + + // MARK: - Consumer (host) + + /// Read up to `out.count` interleaved int16 samples. Returns the number read. Single + /// consumer only. + func read(into out: UnsafeMutableBufferPointer) -> Int { + guard let dst = out.baseAddress else { return 0 } + let r = load64(Self.offRead) // consumer owns the read index + let w = loadAcquire64(Self.offWrite) + let available = Int(w &- r) + if available <= 0 { return 0 } + let n = min(available, out.count) + var idx = Int(r % UInt64(capacity)) + var off = 0 + var rem = n + while rem > 0 { + let chunk = min(rem, capacity - idx) + (dst + off).update(from: data + idx, count: chunk) + idx = (idx + chunk) % capacity + off += chunk + rem -= chunk + } + storeRelease64(Self.offRead, r &+ UInt64(n)) + return n + } + + /// Consumer: discard everything currently buffered (catch the read index up to write). + func drainStale() { storeRelease64(Self.offRead, loadAcquire64(Self.offWrite)) } + + // MARK: - Memory-ordered accessors + + private func ptr32(_ off: Int) -> UnsafeMutablePointer { + (mapBase + off).assumingMemoryBound(to: UInt32.self) + } + private func ptr64(_ off: Int) -> UnsafeMutablePointer { + (mapBase + off).assumingMemoryBound(to: UInt64.self) + } + private func load32(_ off: Int) -> UInt32 { ptr32(off).pointee } + private func store32(_ off: Int, _ v: UInt32) { ptr32(off).pointee = v } + private func load64(_ off: Int) -> UInt64 { ptr64(off).pointee } + private func loadAcquire64(_ off: Int) -> UInt64 { let v = ptr64(off).pointee; OSMemoryBarrier(); return v } + private func storeRelease64(_ off: Int, _ v: UInt64) { OSMemoryBarrier(); ptr64(off).pointee = v } +} diff --git a/clients/apple/iOS/VoiceCatBroadcast/Info.plist b/clients/apple/iOS/VoiceCatBroadcast/Info.plist new file mode 100644 index 0000000..fb2db3f --- /dev/null +++ b/clients/apple/iOS/VoiceCatBroadcast/Info.plist @@ -0,0 +1,31 @@ + + + + + CFBundleDisplayName + VoiceCat Screen Audio + CFBundleExecutable + $(EXECUTABLE_NAME) + CFBundleIdentifier + $(PRODUCT_BUNDLE_IDENTIFIER) + CFBundleInfoDictionaryVersion + 6.0 + CFBundleName + $(PRODUCT_NAME) + CFBundlePackageType + $(PRODUCT_BUNDLE_PACKAGE_TYPE) + CFBundleShortVersionString + 0.0.1 + CFBundleVersion + 1 + NSExtension + + NSExtensionPointIdentifier + com.apple.broadcast-services-upload + NSExtensionPrincipalClass + $(PRODUCT_MODULE_NAME).SampleHandler + RPBroadcastProcessMode + BroadcastUpload + + + diff --git a/clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift b/clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift new file mode 100644 index 0000000..70e2a4d --- /dev/null +++ b/clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift @@ -0,0 +1,88 @@ +import ReplayKit +import AVFoundation + +// SampleHandler — the ReplayKit broadcast upload extension entry point (docs/voice.md §9, iOS). +// +// This runs in a SEPARATE process with a ~50 MB memory cap. It captures system/app audio +// (`.audioApp`), drops video and mic buffers, converts each chunk to the core's canonical +// format (48 kHz, int16, stereo interleaved) with AVAudioConverter, and writes it into the +// App Group shared-memory ring. The host app's BroadcastAudioPump drains the ring and feeds the +// already-connected VoiceCatClient — so all Opus/AEAD/UDP work happens in the host, and the +// extension stays tiny and well inside the memory budget (no libvoicecat here). +class SampleHandler: RPBroadcastSampleHandler { + + private var ring: BroadcastAudioRing? + private var converter: AVAudioConverter? + private var inputFormat: AVAudioFormat? + private let outputFormat = AVAudioFormat(commonFormat: .pcmFormatInt16, + sampleRate: 48_000, channels: 2, interleaved: true)! + + override func broadcastStarted(withSetupInfo setupInfo: [String: NSObject]?) { + ring = try? BroadcastAudioRing() + ring?.setActive(true, channels: 2, sampleRate: 48_000) + postDarwin(BroadcastNotification.started) + } + + override func broadcastFinished() { + ring?.setActive(false) + postDarwin(BroadcastNotification.finished) + ring = nil + } + + override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, + with sampleBufferType: RPSampleBufferType) { + // App/system audio only — drop video (the memory hog) and the device mic (the host + // already captures and sends the user's voice). + guard sampleBufferType == .audioApp, let ring else { return } + guard let input = makeInputBuffer(sampleBuffer), + let conv = converter(for: input.format) else { return } + + let ratio = outputFormat.sampleRate / input.format.sampleRate + let capacity = AVAudioFrameCount(Double(input.frameLength) * ratio) + 1024 + guard let output = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: capacity) else { return } + + var supplied = false + var error: NSError? + let status = conv.convert(to: output, error: &error) { _, outStatus in + if supplied { outStatus.pointee = .noDataNow; return nil } + supplied = true + outStatus.pointee = .haveData + return input + } + guard status != .error, output.frameLength > 0, + let mData = output.audioBufferList.pointee.mBuffers.mData else { return } + + // Interleaved int16 → one buffer of frameLength * channels samples. + let count = Int(output.frameLength) * Int(outputFormat.channelCount) + ring.push(UnsafeBufferPointer(start: mData.assumingMemoryBound(to: Int16.self), count: count)) + } + + // MARK: - Helpers + + /// Wrap the ReplayKit CMSampleBuffer's PCM in an AVAudioPCMBuffer matching its native format. + private func makeInputBuffer(_ sb: CMSampleBuffer) -> AVAudioPCMBuffer? { + guard let fmtDesc = CMSampleBufferGetFormatDescription(sb), + var asbd = CMAudioFormatDescriptionGetStreamBasicDescription(fmtDesc)?.pointee, + let fmt = AVAudioFormat(streamDescription: &asbd) else { return nil } + let frames = AVAudioFrameCount(CMSampleBufferGetNumSamples(sb)) + guard frames > 0, let buf = AVAudioPCMBuffer(pcmFormat: fmt, frameCapacity: frames) else { return nil } + buf.frameLength = frames + let status = CMSampleBufferCopyPCMDataIntoAudioBufferList( + sb, at: 0, frameCount: Int32(frames), into: buf.mutableAudioBufferList) + return status == noErr ? buf : nil + } + + /// Reuse the converter while the input format is stable; rebuild if ReplayKit changes it. + private func converter(for inFmt: AVAudioFormat) -> AVAudioConverter? { + if let converter, inputFormat == inFmt { return converter } + inputFormat = inFmt + converter = AVAudioConverter(from: inFmt, to: outputFormat) + return converter + } + + private func postDarwin(_ name: String) { + CFNotificationCenterPostNotification( + CFNotificationCenterGetDarwinNotifyCenter(), + CFNotificationName(name as CFString), nil, nil, true) + } +} diff --git a/clients/apple/iOS/VoiceCatBroadcast/VoiceCatBroadcast.entitlements b/clients/apple/iOS/VoiceCatBroadcast/VoiceCatBroadcast.entitlements new file mode 100644 index 0000000..63c7480 --- /dev/null +++ b/clients/apple/iOS/VoiceCatBroadcast/VoiceCatBroadcast.entitlements @@ -0,0 +1,10 @@ + + + + + com.apple.security.application-groups + + group.cat.voice.VoiceCat + + + diff --git a/clients/apple/iOS/VoiceCatiOS.xcodeproj/project.pbxproj b/clients/apple/iOS/VoiceCatiOS.xcodeproj/project.pbxproj index 5141f2d..a3cee86 100644 --- a/clients/apple/iOS/VoiceCatiOS.xcodeproj/project.pbxproj +++ b/clients/apple/iOS/VoiceCatiOS.xcodeproj/project.pbxproj @@ -13,7 +13,6 @@ BBBB00000000000000000033 /* AudioSessionManager.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000001A /* AudioSessionManager.swift */; }; BBBB00000000000000000034 /* ServerListStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000001B /* ServerListStore.swift */; }; BBBB00000000000000000035 /* SavedServer.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000001C /* SavedServer.swift */; }; - BBBB00000000000000000036 /* BroadcastCredentials.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000001D /* BroadcastCredentials.swift */; }; BBBB00000000000000000037 /* ServerListView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000001E /* ServerListView.swift */; }; BBBB00000000000000000038 /* AddServerView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000001F /* AddServerView.swift */; }; BBBB00000000000000000039 /* ServerIdentityView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB00000000000000000020 /* ServerIdentityView.swift */; }; @@ -33,8 +32,31 @@ BBBB00000000000000000047 /* SettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000002E /* SettingsView.swift */; }; BBBB0000000000000000004B /* IOSAudioRouter.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000002F /* IOSAudioRouter.swift */; }; BBBB00000000000000000048 /* VoiceCatCore in Frameworks */ = {isa = PBXBuildFile; productRef = BBBB0000000000000000004A /* VoiceCatCore */; }; + CCCC00000000000000000010 /* BroadcastAudioPump.swift in Sources */ = {isa = PBXBuildFile; fileRef = CCCC00000000000000000002 /* BroadcastAudioPump.swift */; }; + CCCC00000000000000000011 /* BroadcastAudioRing.swift in Sources */ = {isa = PBXBuildFile; fileRef = CCCC00000000000000000001 /* BroadcastAudioRing.swift */; }; + CCCC00000000000000000012 /* SampleHandler.swift in Sources */ = {isa = PBXBuildFile; fileRef = CCCC00000000000000000003 /* SampleHandler.swift */; }; + CCCC00000000000000000013 /* BroadcastAudioRing.swift in Sources */ = {isa = PBXBuildFile; fileRef = CCCC00000000000000000001 /* BroadcastAudioRing.swift */; }; + CCCC00000000000000000014 /* VoiceCatBroadcast.appex in Embed Foundation Extensions */ = {isa = PBXBuildFile; fileRef = CCCC00000000000000000006 /* VoiceCatBroadcast.appex */; settings = {ATTRIBUTES = (RemoveHeadersOnCopy, ); }; }; /* End PBXBuildFile section */ +/* Begin PBXContainerItemProxy section */ + CCCC00000000000000000036 /* PBXContainerItemProxy */ = { + isa = PBXContainerItemProxy; + containerPortal = BBBB00000000000000000001 /* Project object */; + proxyType = 1; + remoteGlobalIDString = CCCC00000000000000000030 /* VoiceCatBroadcast */; + remoteInfo = VoiceCatBroadcast; + }; +/* End PBXContainerItemProxy section */ + +/* Begin PBXTargetDependency section */ + CCCC00000000000000000035 /* PBXTargetDependency */ = { + isa = PBXTargetDependency; + target = CCCC00000000000000000030 /* VoiceCatBroadcast */; + targetProxy = CCCC00000000000000000036 /* PBXContainerItemProxy */; + }; +/* End PBXTargetDependency section */ + /* Begin PBXFileReference section */ BBBB00000000000000000012 /* VoiceCatiOS.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = VoiceCatiOS.app; sourceTree = BUILT_PRODUCTS_DIR; }; BBBB00000000000000000015 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; @@ -45,7 +67,6 @@ BBBB0000000000000000001A /* AudioSessionManager.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AudioSessionManager.swift; sourceTree = ""; }; BBBB0000000000000000001B /* ServerListStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerListStore.swift; sourceTree = ""; }; BBBB0000000000000000001C /* SavedServer.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SavedServer.swift; sourceTree = ""; }; - BBBB0000000000000000001D /* BroadcastCredentials.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BroadcastCredentials.swift; sourceTree = ""; }; BBBB0000000000000000001E /* ServerListView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerListView.swift; sourceTree = ""; }; BBBB0000000000000000001F /* AddServerView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AddServerView.swift; sourceTree = ""; }; BBBB00000000000000000020 /* ServerIdentityView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerIdentityView.swift; sourceTree = ""; }; @@ -64,8 +85,28 @@ BBBB0000000000000000002D /* AccountsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AccountsView.swift; sourceTree = ""; }; BBBB0000000000000000002E /* SettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsView.swift; sourceTree = ""; }; BBBB0000000000000000002F /* IOSAudioRouter.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = IOSAudioRouter.swift; sourceTree = ""; }; + CCCC00000000000000000001 /* BroadcastAudioRing.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BroadcastAudioRing.swift; sourceTree = ""; }; + CCCC00000000000000000002 /* BroadcastAudioPump.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BroadcastAudioPump.swift; sourceTree = ""; }; + CCCC00000000000000000003 /* SampleHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SampleHandler.swift; sourceTree = ""; }; + CCCC00000000000000000004 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; + CCCC00000000000000000005 /* VoiceCatBroadcast.entitlements */ = {isa = PBXFileReference; lastKnownFileType = text.plist.entitlements; path = VoiceCatBroadcast.entitlements; sourceTree = ""; }; + CCCC00000000000000000006 /* VoiceCatBroadcast.appex */ = {isa = PBXFileReference; explicitFileType = "wrapper.app-extension"; includeInIndex = 0; path = VoiceCatBroadcast.appex; sourceTree = BUILT_PRODUCTS_DIR; }; /* End PBXFileReference section */ +/* Begin PBXCopyFilesBuildPhase section */ + CCCC00000000000000000037 /* Embed Foundation Extensions */ = { + isa = PBXCopyFilesBuildPhase; + buildActionMask = 2147483647; + dstPath = ""; + dstSubfolderSpec = 13; + files = ( + CCCC00000000000000000014 /* VoiceCatBroadcast.appex in Embed Foundation Extensions */, + ); + name = "Embed Foundation Extensions"; + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXCopyFilesBuildPhase section */ + /* Begin PBXFrameworksBuildPhase section */ BBBB00000000000000000011 /* Frameworks */ = { isa = PBXFrameworksBuildPhase; @@ -82,6 +123,8 @@ isa = PBXGroup; children = ( BBBB00000000000000000003 /* VoiceCatiOS */, + CCCC00000000000000000021 /* VoiceCatBroadcast */, + CCCC00000000000000000020 /* Shared */, BBBB00000000000000000007 /* Products */, ); sourceTree = ""; @@ -98,12 +141,30 @@ BBBB0000000000000000002F /* IOSAudioRouter.swift */, BBBB0000000000000000001B /* ServerListStore.swift */, BBBB0000000000000000001C /* SavedServer.swift */, - BBBB0000000000000000001D /* BroadcastCredentials.swift */, + CCCC00000000000000000002 /* BroadcastAudioPump.swift */, BBBB00000000000000000006 /* Views */, ); path = VoiceCatiOS; sourceTree = ""; }; + CCCC00000000000000000020 /* Shared */ = { + isa = PBXGroup; + children = ( + CCCC00000000000000000001 /* BroadcastAudioRing.swift */, + ); + path = Shared; + sourceTree = ""; + }; + CCCC00000000000000000021 /* VoiceCatBroadcast */ = { + isa = PBXGroup; + children = ( + CCCC00000000000000000003 /* SampleHandler.swift */, + CCCC00000000000000000004 /* Info.plist */, + CCCC00000000000000000005 /* VoiceCatBroadcast.entitlements */, + ); + path = VoiceCatBroadcast; + sourceTree = ""; + }; BBBB00000000000000000006 /* Views */ = { isa = PBXGroup; children = ( @@ -132,6 +193,7 @@ isa = PBXGroup; children = ( BBBB00000000000000000012 /* VoiceCatiOS.app */, + CCCC00000000000000000006 /* VoiceCatBroadcast.appex */, ); name = Products; sourceTree = ""; @@ -146,10 +208,12 @@ BBBB0000000000000000000F /* Sources */, BBBB00000000000000000010 /* Resources */, BBBB00000000000000000011 /* Frameworks */, + CCCC00000000000000000037 /* Embed Foundation Extensions */, ); buildRules = ( ); dependencies = ( + CCCC00000000000000000035 /* PBXTargetDependency */, ); name = VoiceCatiOS; packageProductDependencies = ( @@ -159,6 +223,21 @@ productReference = BBBB00000000000000000012 /* VoiceCatiOS.app */; productType = "com.apple.product-type.application"; }; + CCCC00000000000000000030 /* VoiceCatBroadcast */ = { + isa = PBXNativeTarget; + buildConfigurationList = CCCC00000000000000000032 /* Build configuration list for PBXNativeTarget "VoiceCatBroadcast" */; + buildPhases = ( + CCCC00000000000000000031 /* Sources */, + ); + buildRules = ( + ); + dependencies = ( + ); + name = VoiceCatBroadcast; + productName = VoiceCatBroadcast; + productReference = CCCC00000000000000000006 /* VoiceCatBroadcast.appex */; + productType = "com.apple.product-type.app-extension"; + }; /* End PBXNativeTarget section */ /* Begin PBXProject section */ @@ -186,6 +265,7 @@ projectRoot = ""; targets = ( BBBB00000000000000000008 /* VoiceCatiOS */, + CCCC00000000000000000030 /* VoiceCatBroadcast */, ); }; /* End PBXProject section */ @@ -212,7 +292,8 @@ BBBB0000000000000000004B /* IOSAudioRouter.swift in Sources */, BBBB00000000000000000034 /* ServerListStore.swift in Sources */, BBBB00000000000000000035 /* SavedServer.swift in Sources */, - BBBB00000000000000000036 /* BroadcastCredentials.swift in Sources */, + CCCC00000000000000000010 /* BroadcastAudioPump.swift in Sources */, + CCCC00000000000000000011 /* BroadcastAudioRing.swift in Sources */, BBBB00000000000000000037 /* ServerListView.swift in Sources */, BBBB00000000000000000038 /* AddServerView.swift in Sources */, BBBB00000000000000000039 /* ServerIdentityView.swift in Sources */, @@ -233,6 +314,15 @@ ); runOnlyForDeploymentPostprocessing = 0; }; + CCCC00000000000000000031 /* Sources */ = { + isa = PBXSourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + CCCC00000000000000000012 /* SampleHandler.swift in Sources */, + CCCC00000000000000000013 /* BroadcastAudioRing.swift in Sources */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; /* End PBXSourcesBuildPhase section */ /* Begin XCBuildConfiguration section */ @@ -408,6 +498,57 @@ }; name = Release; }; + CCCC00000000000000000033 /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + CODE_SIGN_ENTITLEMENTS = VoiceCatBroadcast/VoiceCatBroadcast.entitlements; + CODE_SIGN_STYLE = Automatic; + CURRENT_PROJECT_VERSION = 1; + DEVELOPMENT_TEAM = FJV8L966W4; + GENERATE_INFOPLIST_FILE = NO; + INFOPLIST_FILE = VoiceCatBroadcast/Info.plist; + IPHONEOS_DEPLOYMENT_TARGET = 18.0; + LD_RUNPATH_SEARCH_PATHS = ( + "$(inherited)", + "@executable_path/Frameworks", + "@executable_path/../../Frameworks", + ); + MARKETING_VERSION = 0.0.1; + PRODUCT_BUNDLE_IDENTIFIER = cat.voice.VoiceCatiOS.broadcast; + PRODUCT_NAME = "$(TARGET_NAME)"; + SKIP_INSTALL = YES; + SWIFT_OPTIMIZATION_LEVEL = "-Onone"; + SWIFT_VERSION = 5.9; + TARGETED_DEVICE_FAMILY = "1,2"; + }; + name = Debug; + }; + CCCC00000000000000000034 /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + CODE_SIGN_ENTITLEMENTS = VoiceCatBroadcast/VoiceCatBroadcast.entitlements; + CODE_SIGN_STYLE = Automatic; + CURRENT_PROJECT_VERSION = 1; + DEVELOPMENT_TEAM = FJV8L966W4; + GENERATE_INFOPLIST_FILE = NO; + INFOPLIST_FILE = VoiceCatBroadcast/Info.plist; + IPHONEOS_DEPLOYMENT_TARGET = 18.0; + LD_RUNPATH_SEARCH_PATHS = ( + "$(inherited)", + "@executable_path/Frameworks", + "@executable_path/../../Frameworks", + ); + MARKETING_VERSION = 0.0.1; + PRODUCT_BUNDLE_IDENTIFIER = cat.voice.VoiceCatiOS.broadcast; + PRODUCT_NAME = "$(TARGET_NAME)"; + SKIP_INSTALL = YES; + SWIFT_COMPILATION_MODE = wholemodule; + SWIFT_OPTIMIZATION_LEVEL = "-O"; + SWIFT_VERSION = 5.9; + TARGETED_DEVICE_FAMILY = "1,2"; + }; + name = Release; + }; /* End XCBuildConfiguration section */ /* Begin XCConfigurationList section */ @@ -429,6 +570,15 @@ defaultConfigurationIsVisible = 0; defaultConfigurationName = Release; }; + CCCC00000000000000000032 /* Build configuration list for PBXNativeTarget "VoiceCatBroadcast" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + CCCC00000000000000000033 /* Debug */, + CCCC00000000000000000034 /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Release; + }; /* End XCConfigurationList section */ /* Begin XCLocalSwiftPackageReference section */ diff --git a/clients/apple/iOS/VoiceCatiOS/AppState.swift b/clients/apple/iOS/VoiceCatiOS/AppState.swift index b47a490..4fb6314 100644 --- a/clients/apple/iOS/VoiceCatiOS/AppState.swift +++ b/clients/apple/iOS/VoiceCatiOS/AppState.swift @@ -153,7 +153,6 @@ final class AppState { guard let client = connectingClient else { break } let perms = client.getPermissions() let newSession = SessionState(client: client, selfUserId: ev.userId, permissions: perms) - ServerListStore.shared.writeBroadcastCredentials(server: server, nickname: nil) connectingClient = nil isConnecting = false connectStatus = "" diff --git a/clients/apple/iOS/VoiceCatiOS/BroadcastAudioPump.swift b/clients/apple/iOS/VoiceCatiOS/BroadcastAudioPump.swift new file mode 100644 index 0000000..db9729c --- /dev/null +++ b/clients/apple/iOS/VoiceCatiOS/BroadcastAudioPump.swift @@ -0,0 +1,159 @@ +import Foundation + +// BroadcastAudioPump — host side of iOS screen-audio sharing (docs/voice.md §9, iOS detail). +// +// Drains the App Group shared-memory ring written by the broadcast upload extension and hands +// whole 20 ms frames to a feed closure (which calls VoiceCatClient.feedPcm). The host app owns +// the SCREEN_AUDIO stream, so screen audio appears as a second stream of the SAME user — exactly +// like the Windows/macOS desktop-audio share, and a single session (no separate connection). +// +// It reacts to the extension's Darwin notifications for prompt start/stop, and the drain timer +// also watches the ring's active flag as a safety net if a notification is missed. The ring +// always carries stereo; we downmix to mono when the stream's effective config is mono. +// +// Not @MainActor: the drain runs on a background queue (feedPcm is thread-safe). The start/stop +// callbacks are dispatched to main so they can drive @MainActor SessionState. +final class BroadcastAudioPump { + + /// The host should start the SCREEN_AUDIO stream (a broadcast became active). + var onBroadcastStarted: (() -> Void)? + /// The host should stop the SCREEN_AUDIO stream (the broadcast ended). + var onBroadcastFinished: (() -> Void)? + + private static let frameSamplesPerChannel = 960 // 20 ms @ 48 kHz + + private let queue = DispatchQueue(label: "cat.voice.broadcast.pump") + private var ring: BroadcastAudioRing? + private var timer: DispatchSourceTimer? + private var feed: ((UnsafePointer, Int, UInt32) -> Void)? + private var streamChannels = 1 + private var ringChannels = 2 + private var pending: [Int16] = [] // interleaved at ringChannels width + private var scratch = [Int16](repeating: 0, count: 8192) + private var started = false + + // MARK: - Lifecycle (host connect/disconnect) + + func start() { + guard !started else { return } + ring = try? BroadcastAudioRing() + started = true + registerDarwin() + // Host reconnected while a broadcast is still running — pick it up. + if ring?.isActive == true { onBroadcastStarted?() } + } + + func stop() { + guard started else { return } + started = false + unregisterDarwin() + endFeeding() + ring = nil + } + + // MARK: - Feeding (driven by the host once the stream is live) + + /// Begin draining the ring into `feed`. Called after the SCREEN_AUDIO stream's + /// `.streamStarted` event, when its effective channel count is known. + func beginFeeding(streamChannels: UInt32, + feed: @escaping (UnsafePointer, Int, UInt32) -> Void) { + queue.async { + self.streamChannels = max(1, min(2, Int(streamChannels))) + self.ringChannels = max(1, Int(self.ring?.channels ?? 2)) + self.feed = feed + self.pending.removeAll(keepingCapacity: true) + self.ring?.drainStale() // discard pre-roll buffered before we were ready + self.startTimer() + } + } + + func endFeeding() { + queue.async { + self.timer?.cancel() + self.timer = nil + self.feed = nil + self.pending.removeAll(keepingCapacity: true) + } + } + + // MARK: - Drain + + private func startTimer() { + let t = DispatchSource.makeTimerSource(queue: queue) + t.schedule(deadline: .now(), repeating: .milliseconds(10), leeway: .milliseconds(2)) + t.setEventHandler { [weak self] in self?.drain() } + timer = t + t.resume() + } + + private func drain() { + guard let ring, let feed else { return } + // Safety net: broadcast ended but we missed the Darwin note. + if !ring.isActive { + DispatchQueue.main.async { [weak self] in self?.onBroadcastFinished?() } + return + } + while true { + let got = scratch.withUnsafeMutableBufferPointer { ring.read(into: $0) } + if got == 0 { break } + pending.append(contentsOf: scratch[0.., Int, UInt32) -> Void) { + let n = Self.frameSamplesPerChannel + let rc = ringChannels + let sc = streamChannels + let inFrame = n * rc + while pending.count >= inFrame { + if sc == rc { + pending.withUnsafeBufferPointer { feed($0.baseAddress!, n, UInt32(sc)) } + } else if sc == 1 && rc == 2 { + var mono = [Int16](repeating: 0, count: n) + pending.withUnsafeBufferPointer { buf in + let p = buf.baseAddress! + for i in 0...fromOpaque(observer).takeUnretainedValue() + let raw = name.rawValue as String + DispatchQueue.main.async { + if raw == BroadcastNotification.started { pump.onBroadcastStarted?() } + else if raw == BroadcastNotification.finished { pump.onBroadcastFinished?() } + } + } + CFNotificationCenterAddObserver(center, observer, callback, + BroadcastNotification.started as CFString, nil, .deliverImmediately) + CFNotificationCenterAddObserver(center, observer, callback, + BroadcastNotification.finished as CFString, nil, .deliverImmediately) + } + + private func unregisterDarwin() { + CFNotificationCenterRemoveEveryObserver( + CFNotificationCenterGetDarwinNotifyCenter(), + Unmanaged.passUnretained(self).toOpaque()) + } + + deinit { if started { unregisterDarwin() } } +} diff --git a/clients/apple/iOS/VoiceCatiOS/BroadcastCredentials.swift b/clients/apple/iOS/VoiceCatiOS/BroadcastCredentials.swift deleted file mode 100644 index d6ccfa5..0000000 --- a/clients/apple/iOS/VoiceCatiOS/BroadcastCredentials.swift +++ /dev/null @@ -1,21 +0,0 @@ -import Foundation - -struct BroadcastCredentials: Codable { - var host: String - var port: Int - var authMode: String - var username: String - var tofuPinsPath: String - - static func loadFromAppGroup() -> BroadcastCredentials? { - let groupId = "group.cat.voice.VoiceCat" - guard let container = FileManager.default.containerURL( - forSecurityApplicationGroupIdentifier: groupId) - else { return nil } - let url = container - .appendingPathComponent("voicecat", isDirectory: true) - .appendingPathComponent("broadcast_credentials.json") - guard let data = try? Data(contentsOf: url) else { return nil } - return try? JSONDecoder().decode(BroadcastCredentials.self, from: data) - } -} diff --git a/clients/apple/iOS/VoiceCatiOS/ServerListStore.swift b/clients/apple/iOS/VoiceCatiOS/ServerListStore.swift index 816c974..018528c 100644 --- a/clients/apple/iOS/VoiceCatiOS/ServerListStore.swift +++ b/clients/apple/iOS/VoiceCatiOS/ServerListStore.swift @@ -89,18 +89,4 @@ final class ServerListStore { SecItemDelete(query as CFDictionary) } - // MARK: - Broadcast credentials (shared with ReplayKit extension) - - func writeBroadcastCredentials(server: SavedServer, nickname: String?) { - let creds = BroadcastCredentials( - host: server.host, - port: Int(server.port), - authMode: server.authMode.rawValue, - username: server.authMode == .password ? server.savedUsername : (nickname ?? ""), - tofuPinsPath: tofuStorePath - ) - guard let data = try? JSONEncoder().encode(creds) else { return } - let url = voicecatDir.appendingPathComponent("broadcast_credentials.json") - try? data.write(to: url, options: .atomic) - } } diff --git a/clients/apple/iOS/VoiceCatiOS/SessionState.swift b/clients/apple/iOS/VoiceCatiOS/SessionState.swift index 2c48b31..48dfd32 100644 --- a/clients/apple/iOS/VoiceCatiOS/SessionState.swift +++ b/clients/apple/iOS/VoiceCatiOS/SessionState.swift @@ -29,6 +29,8 @@ struct VoiceState { var level: Float = 0.0 var currentDeviceId: String? var localStreamId: UInt32 = 0 + var screenSharing = false + var screenStreamId: UInt32 = 0 } // MARK: - SessionState @@ -49,6 +51,10 @@ final class SessionState { var accounts: [Account] = [] var devices: [Device] = [] + /// Host side of iOS screen-audio sharing — drains the broadcast extension's App Group ring + /// and feeds the SCREEN_AUDIO stream this session owns. See BroadcastAudioPump. + private let broadcastPump = BroadcastAudioPump() + init(client: VoiceCatClient, selfUserId: UInt32, permissions: Permissions) { self.client = client self.selfUserId = selfUserId @@ -64,9 +70,13 @@ final class SessionState { client.onLevel = { [weak self] _, rms in Task { @MainActor [weak self] in self?.voiceState.level = rms } } + broadcastPump.onBroadcastStarted = { [weak self] in self?.startScreenShare() } + broadcastPump.onBroadcastFinished = { [weak self] in self?.stopScreenShare() } + broadcastPump.start() } deinit { + broadcastPump.stop() MainActor.assumeIsolated { AudioSessionManager.shared.client = nil } @@ -100,6 +110,19 @@ final class SessionState { let who = users.first(where: { $0.id == ev.userId })?.nickname ?? "user \(ev.userId)" addActivity(talking ? "\(who) started talking" : "\(who) stopped talking") case .streamStarted: + // Our own SCREEN_AUDIO stream is live — begin draining the broadcast ring into it, + // in the stream's effective channel mode (downmix to mono if the channel is mono). + if ev.userId == selfUserId && ev.streamId == voiceState.screenStreamId { + let sid = voiceState.screenStreamId + let (r, cfg) = client.getStreamAudioConfig(userId: selfUserId, streamId: sid) + let channels: UInt32 = (r == .ok && cfg?.stereo == true) ? 2 : 1 + let c = client + broadcastPump.beginFeeding(streamChannels: channels) { pcm, samples, ch in + c.feedPcm(streamId: sid, pcm: pcm, samplesPerChannel: samples, channels: ch) + } + addActivity("Sharing screen audio (\(channels == 2 ? "stereo" : "mono"))") + break + } // A remote user started a stream — ensure the audio session is active so we can // hear them even if we haven't joined voice ourselves. if ev.userId != selfUserId { @@ -236,6 +259,41 @@ final class SessionState { // disconnecting from the server (see AppState.disconnect / .disconnected event). } + // MARK: - Screen audio share + + /// Called when the broadcast extension becomes active. Announces the SCREEN_AUDIO stream; + /// feeding begins on the resulting `.streamStarted` event (see handleEvent). The actual + /// system-audio capture happens in the ReplayKit upload extension (a separate process). + private func startScreenShare() { + guard voiceState.screenStreamId == 0 else { return } + guard currentChannelId != 0 else { + addActivity("Screen audio ignored — join a channel first") + return + } + let (result, streamId) = client.startStream( + StreamDescriptor(kind: .screenAudio, deviceId: nil, label: "Screen audio")) + if result == .ok { + voiceState.screenStreamId = streamId + voiceState.screenSharing = true + addActivity("Screen audio share starting…") + } else { + addActivity("Failed to start screen audio: \(result.description)") + } + } + + /// Called when the broadcast ends (or on disconnect). Stops feeding and the stream. + private func stopScreenShare() { + broadcastPump.endFeeding() + if voiceState.screenStreamId != 0 { + client.stopStream(voiceState.screenStreamId) + voiceState.screenStreamId = 0 + } + if voiceState.screenSharing { + voiceState.screenSharing = false + addActivity("Stopped sharing screen audio") + } + } + func setMute(_ muted: Bool, deafened: Bool) { client.setSelfMute(micMuted: muted, deafened: deafened) voiceState.selfMuted = muted diff --git a/clients/apple/iOS/VoiceCatiOS/Views/VoiceControlsView.swift b/clients/apple/iOS/VoiceCatiOS/Views/VoiceControlsView.swift index 5cc269e..4e63011 100644 --- a/clients/apple/iOS/VoiceCatiOS/Views/VoiceControlsView.swift +++ b/clients/apple/iOS/VoiceCatiOS/Views/VoiceControlsView.swift @@ -1,5 +1,6 @@ import SwiftUI import VoiceCatCore +import ReplayKit struct VoiceControlsView: View { @Bindable var session: SessionState @@ -58,6 +59,14 @@ struct VoiceControlsView: View { .disabled(!session.voiceState.micActive) .accessibilityLabel(session.voiceState.selfDeafened ? "Undeafen" : "Deafen") + // Share screen audio (ReplayKit system broadcast picker). The picker launches the + // broadcast upload extension; the host's BroadcastAudioPump then owns + feeds the + // SCREEN_AUDIO stream. Tinted while sharing. + BroadcastPickerButton(isSharing: session.voiceState.screenSharing) + .frame(width: 32, height: 32) + .accessibilityLabel(session.voiceState.screenSharing + ? "Stop sharing screen audio" : "Share screen audio") + // Disconnect Button(role: .destructive) { session.stopMicStream() @@ -108,6 +117,26 @@ private struct PTTButton: View { } } +// MARK: - Broadcast picker + +/// Wraps `RPSystemBroadcastPickerView` (which contains its own button) and points it at our +/// broadcast upload extension. Tapping it shows the system broadcast picker; the user starts the +/// broadcast and our extension launches. +private struct BroadcastPickerButton: UIViewRepresentable { + let isSharing: Bool + + func makeUIView(context: Context) -> RPSystemBroadcastPickerView { + let picker = RPSystemBroadcastPickerView(frame: CGRect(x: 0, y: 0, width: 32, height: 32)) + picker.preferredExtension = "cat.voice.VoiceCatiOS.broadcast" + picker.showsMicrophoneButton = false + return picker + } + + func updateUIView(_ uiView: RPSystemBroadcastPickerView, context: Context) { + uiView.tintColor = isSharing ? .systemGreen : .label + } +} + // MARK: - Level Meter private struct LevelMeterView: View { diff --git a/clients/apple/macOS/VoiceCatMac.xcodeproj/project.pbxproj b/clients/apple/macOS/VoiceCatMac.xcodeproj/project.pbxproj index d948201..45a9dcb 100644 --- a/clients/apple/macOS/VoiceCatMac.xcodeproj/project.pbxproj +++ b/clients/apple/macOS/VoiceCatMac.xcodeproj/project.pbxproj @@ -30,6 +30,7 @@ AAAA00000000000000000046 /* PrivateMessageWindowController.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA00000000000000000045 /* PrivateMessageWindowController.swift */; }; AAAA00000000000000000048 /* UserPickerSheet.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA00000000000000000047 /* UserPickerSheet.swift */; }; AAAA0000000000000000004A /* SettingsWindowController.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA00000000000000000049 /* SettingsWindowController.swift */; }; + AAAA0000000000000000004C /* ScreenAudioCapture.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA0000000000000000004B /* ScreenAudioCapture.swift */; }; /* End PBXBuildFile section */ /* Begin PBXFileReference section */ @@ -42,6 +43,7 @@ AAAA00000000000000000018 /* ServerListStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerListStore.swift; sourceTree = ""; }; AAAA00000000000000000019 /* ConnectWindowController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConnectWindowController.swift; sourceTree = ""; }; AAAA0000000000000000001A /* MainWindowController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MainWindowController.swift; sourceTree = ""; }; + AAAA0000000000000000004B /* ScreenAudioCapture.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenAudioCapture.swift; sourceTree = ""; }; AAAA0000000000000000001B /* AddServerSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AddServerSheet.swift; sourceTree = ""; }; AAAA0000000000000000001C /* ServerIdentitySheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerIdentitySheet.swift; sourceTree = ""; }; AAAA0000000000000000001D /* PasswordPromptSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PasswordPromptSheet.swift; sourceTree = ""; }; @@ -89,12 +91,21 @@ AAAA00000000000000000015 /* main.swift */, AAAA00000000000000000016 /* AppDelegate.swift */, AAAA00000000000000000004 /* Models */, + AAAA0000000000000000004D /* Audio */, AAAA00000000000000000005 /* Windows */, AAAA00000000000000000006 /* Sheets */, ); path = VoiceCatMac; sourceTree = ""; }; + AAAA0000000000000000004D /* Audio */ = { + isa = PBXGroup; + children = ( + AAAA0000000000000000004B /* ScreenAudioCapture.swift */, + ); + path = Audio; + sourceTree = ""; + }; AAAA00000000000000000004 /* Models */ = { isa = PBXGroup; children = ( @@ -231,6 +242,7 @@ AAAA00000000000000000046 /* PrivateMessageWindowController.swift in Sources */, AAAA00000000000000000048 /* UserPickerSheet.swift in Sources */, AAAA0000000000000000004A /* SettingsWindowController.swift in Sources */, + AAAA0000000000000000004C /* ScreenAudioCapture.swift in Sources */, ); runOnlyForDeploymentPostprocessing = 0; }; diff --git a/clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift b/clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift new file mode 100644 index 0000000..629b812 --- /dev/null +++ b/clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift @@ -0,0 +1,158 @@ +import AVFoundation +import ScreenCaptureKit + +// ScreenAudioCapture — macOS system/desktop audio capture for the SCREEN_AUDIO stream. +// +// The macOS analog of the Windows WASAPI loopback path (docs/voice.md §9). ScreenCaptureKit +// (macOS 13+) captures whatever the system is playing; we convert each audio CMSampleBuffer +// (Float32) → int16 interleaved and push 20 ms frames (960 samples/channel @ 48 kHz) into the +// core via `vc_stream_feed_pcm` (exposed as `VoiceCatClient.feedPcm`). The core then runs the +// same Opus-encode → media-AEAD → UDP path as any other stream — only the *source* is +// platform-specific (architecture.md §4). +// +// Audio-only: we request a 2×2 video plane at 1 fps purely because SCStream needs a video +// configuration, and we never add a `.screen` output — only `.audio`. `excludesCurrentProcess +// Audio` prevents the self-echo loop of re-capturing our own incoming voice mix. +// +// `feedPcm` is thread-safe (any thread), so we forward straight from the sample-handler queue. +final class ScreenAudioCapture: NSObject, SCStreamOutput, SCStreamDelegate { + + /// Receives a full 20 ms frame: (interleaved int16 PCM, samplesPerChannel = 960, channels). + typealias PcmHandler = (UnsafePointer, Int, UInt32) -> Void + + enum CaptureError: Error { case noDisplay } + + private static let frameSamplesPerChannel = 960 // 20 ms @ 48 kHz + + private let onPcm: PcmHandler + private let channels: Int // 1 (mono) or 2 (stereo interleaved), matches the stream's mode + private let sampleQueue = DispatchQueue(label: "cat.voice.screenaudio.samples") + private var stream: SCStream? + + /// Interleaved int16 carry-over between callbacks (ScreenCaptureKit buffers don't align to + /// 20 ms), drained in whole `frameSamplesPerChannel * channels` chunks. Only touched on + /// `sampleQueue`. + private var pending: [Int16] = [] + + init(channels: UInt32, onPcm: @escaping PcmHandler) { + self.channels = max(1, min(2, Int(channels))) + self.onPcm = onPcm + super.init() + } + + /// Begin capture. Throws if Screen Recording permission is denied (the first + /// `SCShareableContent.current` access is what surfaces the TCC prompt) or no display exists. + func start() async throws { + let content = try await SCShareableContent.current + guard let display = content.displays.first else { throw CaptureError.noDisplay } + + let filter = SCContentFilter(display: display, excludingWindows: []) + + let config = SCStreamConfiguration() + config.capturesAudio = true + config.excludesCurrentProcessAudio = true + config.sampleRate = 48000 + config.channelCount = channels + // SCStream requires a video config even when we only consume audio — keep it minimal. + config.width = 2 + config.height = 2 + config.minimumFrameInterval = CMTime(value: 1, timescale: 1) // ~1 fps + config.queueDepth = 6 + + let stream = SCStream(filter: filter, configuration: config, delegate: self) + try stream.addStreamOutput(self, type: .audio, sampleHandlerQueue: sampleQueue) + try await stream.startCapture() + self.stream = stream + } + + /// Stop capture and release the stream. Safe to call multiple times. + func stop() { + guard let stream else { return } + self.stream = nil + Task { try? await stream.stopCapture() } + } + + // MARK: - SCStreamOutput + + func stream(_ stream: SCStream, didOutputSampleBuffer sampleBuffer: CMSampleBuffer, + of type: SCStreamOutputType) { + guard type == .audio, CMSampleBufferDataIsReady(sampleBuffer) else { return } + guard let fmt = sampleBuffer.formatDescription, + let asbd = fmt.audioStreamBasicDescription else { return } + // ScreenCaptureKit always delivers Float32 PCM; bail on anything unexpected. + guard asbd.mFormatFlags & kAudioFormatFlagIsFloat != 0 else { return } + + let nonInterleaved = asbd.mFormatFlags & kAudioFormatFlagIsNonInterleaved != 0 + let srcChannels = max(1, Int(asbd.mChannelsPerFrame)) + + try? sampleBuffer.withAudioBufferList { ablPtr, _ in + convert(ablPtr, nonInterleaved: nonInterleaved, srcChannels: srcChannels) + } + } + + // MARK: - Conversion (called on sampleQueue) + + private func convert(_ abl: UnsafeMutableAudioBufferListPointer, + nonInterleaved: Bool, srcChannels: Int) { + guard let first = abl.first, first.mData != nil else { return } + + let out = channels + var interleaved: [Int16] + + if nonInterleaved { + // One buffer per source channel; each is `frames` Float32 samples. + let frames = Int(first.mDataByteSize) / MemoryLayout.size + if frames == 0 { return } + let ch0 = first.mData!.assumingMemoryBound(to: Float.self) + let ch1: UnsafePointer? = (abl.count > 1) + ? UnsafePointer(abl[1].mData!.assumingMemoryBound(to: Float.self)) : nil + interleaved = [Int16](repeating: 0, count: frames * out) + for i in 0..= 2 ? r : l) + } else { + interleaved[i] = Self.f2i(srcChannels >= 2 ? (l + r) * 0.5 : l) + } + } + } else { + // Single interleaved Float32 buffer, srcChannels wide. + let total = Int(first.mDataByteSize) / MemoryLayout.size + let frames = total / srcChannels + if frames == 0 { return } + let src = first.mData!.assumingMemoryBound(to: Float.self) + interleaved = [Int16](repeating: 0, count: frames * out) + for i in 0..= 2 ? src[i * srcChannels + 1] : l + if out == 2 { + interleaved[i * 2] = Self.f2i(l) + interleaved[i * 2 + 1] = Self.f2i(r) + } else { + interleaved[i] = Self.f2i(srcChannels >= 2 ? (l + r) * 0.5 : l) + } + } + } + + emit(interleaved) + } + + /// Accumulate interleaved int16 and fire `onPcm` for every whole 20 ms frame. + private func emit(_ interleaved: [Int16]) { + pending.append(contentsOf: interleaved) + let full = Self.frameSamplesPerChannel * channels + while pending.count >= full { + pending.withUnsafeBufferPointer { buf in + onPcm(buf.baseAddress!, Self.frameSamplesPerChannel, UInt32(channels)) + } + pending.removeFirst(full) + } + } + + private static func f2i(_ f: Float) -> Int16 { + let v = max(-1.0, min(1.0, f)) * 32767.0 + return Int16(v.rounded()) + } +} diff --git a/clients/apple/macOS/VoiceCatMac/Windows/MainWindowController.swift b/clients/apple/macOS/VoiceCatMac/Windows/MainWindowController.swift index a70ce8c..19b6074 100644 --- a/clients/apple/macOS/VoiceCatMac/Windows/MainWindowController.swift +++ b/clients/apple/macOS/VoiceCatMac/Windows/MainWindowController.swift @@ -30,6 +30,7 @@ final class MainWindowController: NSWindowController, NSWindowDelegate { canMoveUsers: false, canAdminAccounts: false, isAdmin: false) internal var micStreamId: UInt32 = 0 private var screenStreamId: UInt32 = 0 + private var screenCapture: ScreenAudioCapture? internal var pttKeyCode: UInt16 = 0x60 // F8 private var pttMonitor: Any? private var serverMuted = false @@ -477,6 +478,10 @@ final class MainWindowController: NSWindowController, NSWindowDelegate { } case .streamStarted: + // Our own SCREEN_AUDIO stream is now live on the server — begin capture. + if event.userId == selfUserId && event.streamId == screenStreamId { + startScreenCapture() + } guard let u = users[event.userId], u.channelId == currentChannelId else { break } let streams = client.listUserStreams(event.userId) let kind = streams.first(where: { $0.id == event.streamId })?.kind @@ -601,6 +606,7 @@ final class MainWindowController: NSWindowController, NSWindowDelegate { channelTree = []; channelOutlineView.reloadData() displayedUsers = []; userTableView.reloadData() users.removeAll(); talkingUsers.removeAll() + stopScreenCapture() currentChannelId = 0; micStreamId = 0; screenStreamId = 0 composeField.isEnabled = false; sendButton.isEnabled = false joinVoiceButton?.isEnabled = false @@ -708,26 +714,70 @@ final class MainWindowController: NSWindowController, NSWindowDelegate { @objc private func screenAudioClicked() { if screenStreamId == 0 { + // Announce the stream now; ScreenCaptureKit capture starts once the server's + // StreamAnnounceResult lands (the .streamStarted event), when the effective audio + // config — and thus the channel count to capture — is known. See startScreenCapture. let (result, streamId) = client.startStream(StreamDescriptor(kind: .screenAudio, deviceId: nil, label: "Desktop audio")) if result == .ok { screenStreamId = streamId - shareScreenButton?.title = "Stop Screen Audio" - shareScreenButton?.image = NSImage(systemSymbolName: "rectangle.on.rectangle.angled.fill", - accessibilityDescription: "Stop Screen Audio") - addActivity("Started sharing screen audio") + setShareScreenButton(active: true) + addActivity("Starting screen audio share…") } else { addActivity("Failed to start screen audio: \(result)") } } else { + stopScreenCapture() client.stopStream(screenStreamId) screenStreamId = 0 - shareScreenButton?.title = "Share Screen Audio" - shareScreenButton?.image = NSImage(systemSymbolName: "rectangle.on.rectangle.angled", - accessibilityDescription: "Share Screen Audio") + setShareScreenButton(active: false) addActivity("Stopped sharing screen audio") } } + private func setShareScreenButton(active: Bool) { + shareScreenButton?.title = active ? "Stop Screen Audio" : "Share Screen Audio" + shareScreenButton?.image = NSImage( + systemSymbolName: active ? "rectangle.on.rectangle.angled.fill" : "rectangle.on.rectangle.angled", + accessibilityDescription: active ? "Stop Screen Audio" : "Share Screen Audio") + } + + /// Start ScreenCaptureKit capture for our own SCREEN_AUDIO stream. Called from the + /// `.streamStarted` event handler, where the effective audio config is available. Captures + /// in the channel's mode (stereo when the channel is stereo) and feeds 20 ms PCM frames. + private func startScreenCapture() { + guard screenStreamId != 0, screenCapture == nil else { return } + let (cfgResult, cfg) = client.getStreamAudioConfig(userId: selfUserId, streamId: screenStreamId) + let channels: UInt32 = (cfgResult == .ok && cfg?.stereo == true) ? 2 : 1 + + let streamId = screenStreamId + let capture = ScreenAudioCapture(channels: channels) { [weak self] pcm, samples, ch in + self?.client.feedPcm(streamId: streamId, pcm: pcm, samplesPerChannel: samples, channels: ch) + } + screenCapture = capture + + Task { @MainActor in + do { + try await capture.start() + addActivity("Started sharing screen audio (\(channels == 2 ? "stereo" : "mono"))") + } catch { + // Most commonly: Screen Recording permission denied. Roll back the stream. + screenCapture = nil + if screenStreamId != 0 { + client.stopStream(screenStreamId) + screenStreamId = 0 + setShareScreenButton(active: false) + } + addActivity("Screen audio capture failed — grant Screen Recording in System " + + "Settings ▸ Privacy & Security, then try again. (\(error))") + } + } + } + + private func stopScreenCapture() { + screenCapture?.stop() + screenCapture = nil + } + @objc private func muteChanged() { let muted = muteButton?.state == .on let deafened = deafenButton?.state == .on @@ -999,6 +1049,7 @@ final class MainWindowController: NSWindowController, NSWindowDelegate { } } client.setPushToTalk(false) + stopScreenCapture() if screenStreamId != 0 { client.stopStream(screenStreamId) } if micStreamId != 0 { client.stopStream(micStreamId) } client.onLevel = nil diff --git a/docs/voice.md b/docs/voice.md index eea36f5..7db71ea 100644 --- a/docs/voice.md +++ b/docs/voice.md @@ -274,23 +274,31 @@ normal stream; only the *source* is platform-specific. | Platform | Mechanism | Notes | |----------|-----------|-------| | **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. | -| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | OS requires screen-recording permission; capture happens in the main app. Swift converts each `CMSampleBuffer` (Float32) → int16 and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). | -| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). The extension calls `vc_stream_feed_pcm` to drive the encode path without a hardware device. | +| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). | +| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). | ### iOS detail +The extension **captures**, the host app **sends**. Unlike a self-connecting extension, this +keeps a **single session** — the screen-audio share appears as a second stream of the *same* +user (exactly like macOS/Windows), and no credentials are ever persisted to disk. + - The user starts a broadcast from Control Center's screen-record button; we surface it via - `RPSystemBroadcastPickerView` from inside the app for one-tap start. -- The **Broadcast Upload Extension** receives `RPSampleBufferType.audioApp` (system/app - audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video - buffers entirely — video is what blows the **~50 MB** extension memory budget, so an - audio-only consumer stays comfortably inside it. -- The extension is a *separate process*. It reads the active session token and server - endpoint from a shared **App Group** container, opens a UDP media channel, and calls - `vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the - Opus encode + AEAD + send path directly, with no `ma_device` or audio hardware. The host - app announces the stream over its control channel (`StreamAnnounce`) so the server and - peers learn about it. -- Mic + voice continue to run in the **host app**; only the system-audio share lives in the - extension. When the broadcast stops (`broadcastFinished`), the extension sends a final - frame with the `last-frame-before-stop` flag and the host app emits `StreamStop`. + `RPSystemBroadcastPickerView` from inside the app (`VoiceControlsView`) for one-tap start. +- The **Broadcast Upload Extension** (`clients/apple/iOS/VoiceCatBroadcast/SampleHandler.swift`) + receives `RPSampleBufferType.audioApp` (system/app audio), `.audioMic`, and `.video`. We + consume **`.audioApp`** only and drop video + mic — video is what blows the **~50 MB** + extension memory budget, so an audio-only consumer stays comfortably inside it. The extension + does **not** link `libvoicecat`. +- The extension converts each chunk to the core's canonical format (48 kHz int16 stereo, via + `AVAudioConverter`) and writes it into a lock-free single-producer/single-consumer ring in a + shared **App Group** mmap'd file (`clients/apple/iOS/Shared/BroadcastAudioRing.swift`). It + posts Darwin notifications on start/stop so the host reacts promptly. +- The **host app** owns the stream: its `BroadcastAudioPump` announces the `SCREEN_AUDIO` + stream over the control channel (`StreamAnnounce`), drains the ring, and calls + `vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the Opus + encode + AEAD + send path. It downmixes to mono when the channel's effective config is mono. +- Mic + voice also run in the host app. When the broadcast stops (`broadcastFinished`), the + extension clears the ring's active flag (and posts a Darwin notification); the host stops + feeding and emits `StreamStop`. The host must be alive to relay — always true while in a + call (the app declares the `audio` background mode).