feat(macos): per-app audio selection for screen sharing
Let users choose what the SCREEN_AUDIO stream captures before sharing: share everything, only selected apps, or all except selected apps, plus a first-class "Exclude screen reader (VoiceOver) audio" toggle. ScreenCaptureKit filters audio per application, so ScreenAudioCapture now takes a ScreenAudioSelection and builds the matching SCContentFilter (including:/excludingApplications:). New ScreenSharePickerSheet lists running apps from SCShareableContent. iOS left untouched -- ReplayKit only delivers the mixed system stream, so per-app filtering is impossible there.
This commit is contained in:
12
PROGRESS.md
12
PROGRESS.md
@@ -61,6 +61,18 @@ up instantly. Newest status at the top.
|
||||
aborts at shutdown (`mutex lock failed`), a **pre-existing** teardown crash unrelated to this
|
||||
change (no C++ was modified).
|
||||
|
||||
- **Done (2026-06-21):** **macOS per-app screen-audio selection.** Before sharing, a new
|
||||
`ScreenSharePickerSheet` lets the user choose scope — share Everything / Only selected apps /
|
||||
All except selected apps — plus a first-class **"Exclude screen reader (VoiceOver) audio"**
|
||||
toggle. `ScreenAudioCapture` now takes a `ScreenAudioSelection` and builds the matching
|
||||
`SCContentFilter` (`including:` / `excludingApplications:`); app list comes from
|
||||
`SCShareableContent`. macOS `xcodebuild` Debug BUILD SUCCEEDED. iOS deliberately untouched —
|
||||
ReplayKit only delivers the mixed system stream, so per-app/VoiceOver filtering is impossible
|
||||
there (documented in voice.md §9). **Still to verify on-device:** which process actually
|
||||
carries VoiceOver speech (VoiceOver app vs. `com.apple.speech.speechsynthesisd`) — the exclude
|
||||
set covers both candidates in `ScreenAudioCapture.screenReaderBundleIDs`; confirm exclusion
|
||||
actually silences it in a real share.
|
||||
|
||||
- **Done (2026-06-20):** **macOS client UI overhaul** — mirrors the Windows client's UI
|
||||
overhaul (commit 97fa659 + 540ec13), adapted to Mac-native conventions. Also fixed and
|
||||
verified the previously-uncompiled Swift changes from the external PCM feed/tap commit
|
||||
|
||||
@@ -31,6 +31,7 @@
|
||||
AAAA00000000000000000048 /* UserPickerSheet.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA00000000000000000047 /* UserPickerSheet.swift */; };
|
||||
AAAA0000000000000000004A /* SettingsWindowController.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA00000000000000000049 /* SettingsWindowController.swift */; };
|
||||
AAAA0000000000000000004C /* ScreenAudioCapture.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA0000000000000000004B /* ScreenAudioCapture.swift */; };
|
||||
AAAA0000000000000000004E /* ScreenSharePickerSheet.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA0000000000000000004F /* ScreenSharePickerSheet.swift */; };
|
||||
/* End PBXBuildFile section */
|
||||
|
||||
/* Begin PBXFileReference section */
|
||||
@@ -57,6 +58,7 @@
|
||||
AAAA00000000000000000044 /* PttKeyCaptureSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PttKeyCaptureSheet.swift; sourceTree = "<group>"; };
|
||||
AAAA00000000000000000045 /* PrivateMessageWindowController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PrivateMessageWindowController.swift; sourceTree = "<group>"; };
|
||||
AAAA00000000000000000047 /* UserPickerSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = UserPickerSheet.swift; sourceTree = "<group>"; };
|
||||
AAAA0000000000000000004F /* ScreenSharePickerSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenSharePickerSheet.swift; sourceTree = "<group>"; };
|
||||
AAAA00000000000000000049 /* SettingsWindowController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsWindowController.swift; sourceTree = "<group>"; };
|
||||
AAAA00000000000000000025 /* Security.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = Security.framework; path = System/Library/Frameworks/Security.framework; sourceTree = SDKROOT; };
|
||||
/* End PBXFileReference section */
|
||||
@@ -141,6 +143,7 @@
|
||||
AAAA00000000000000000024 /* PermissionsSheet.swift */,
|
||||
AAAA00000000000000000044 /* PttKeyCaptureSheet.swift */,
|
||||
AAAA00000000000000000047 /* UserPickerSheet.swift */,
|
||||
AAAA0000000000000000004F /* ScreenSharePickerSheet.swift */,
|
||||
);
|
||||
path = Sheets;
|
||||
sourceTree = "<group>";
|
||||
@@ -241,6 +244,7 @@
|
||||
AAAA00000000000000000043 /* PttKeyCaptureSheet.swift in Sources */,
|
||||
AAAA00000000000000000046 /* PrivateMessageWindowController.swift in Sources */,
|
||||
AAAA00000000000000000048 /* UserPickerSheet.swift in Sources */,
|
||||
AAAA0000000000000000004E /* ScreenSharePickerSheet.swift in Sources */,
|
||||
AAAA0000000000000000004A /* SettingsWindowController.swift in Sources */,
|
||||
AAAA0000000000000000004C /* ScreenAudioCapture.swift in Sources */,
|
||||
);
|
||||
|
||||
@@ -1,6 +1,25 @@
|
||||
import AVFoundation
|
||||
import ScreenCaptureKit
|
||||
|
||||
// Which apps' audio the SCREEN_AUDIO stream captures. ScreenCaptureKit filters audio at the
|
||||
// *application* level (not per-window), so the selection is expressed as bundle IDs. The
|
||||
// picker UI (ScreenSharePickerSheet) produces a `ScreenAudioSelection`; `start()` turns it
|
||||
// into the matching `SCContentFilter`.
|
||||
enum ScreenAudioScope: Equatable {
|
||||
case entireDesktop // whole display — the original behaviour
|
||||
case onlyApps([String]) // capture only these bundle IDs
|
||||
case allExcept([String]) // capture everything except these bundle IDs
|
||||
}
|
||||
|
||||
struct ScreenAudioSelection: Equatable {
|
||||
var scope: ScreenAudioScope = .entireDesktop
|
||||
/// Drop the macOS screen-reader (VoiceOver) speech from the shared mix. Meaningful for
|
||||
/// `.entireDesktop`/`.allExcept`; for `.onlyApps` the screen reader is already excluded.
|
||||
var excludeScreenReader: Bool = false
|
||||
|
||||
static let `default` = ScreenAudioSelection()
|
||||
}
|
||||
|
||||
// ScreenAudioCapture — macOS system/desktop audio capture for the SCREEN_AUDIO stream.
|
||||
//
|
||||
// The macOS analog of the Windows WASAPI loopback path (docs/voice.md §9). ScreenCaptureKit
|
||||
@@ -26,16 +45,27 @@ final class ScreenAudioCapture: NSObject, SCStreamOutput, SCStreamDelegate {
|
||||
|
||||
private let onPcm: PcmHandler
|
||||
private let channels: Int // 1 (mono) or 2 (stereo interleaved), matches the stream's mode
|
||||
private let selection: ScreenAudioSelection
|
||||
private let sampleQueue = DispatchQueue(label: "cat.voice.screenaudio.samples")
|
||||
private var stream: SCStream?
|
||||
|
||||
/// Bundle IDs whose audio carries the macOS screen-reader speech. VoiceOver itself plus the
|
||||
/// speech-synthesis daemon that actually renders the spoken audio — the speech is usually
|
||||
/// emitted by the daemon, not the VoiceOver app, so we exclude whichever are running.
|
||||
static let screenReaderBundleIDs: Set<String> = [
|
||||
"com.apple.VoiceOver",
|
||||
"com.apple.VoiceOver4",
|
||||
"com.apple.speech.speechsynthesisd",
|
||||
]
|
||||
|
||||
/// Interleaved int16 carry-over between callbacks (ScreenCaptureKit buffers don't align to
|
||||
/// 20 ms), drained in whole `frameSamplesPerChannel * channels` chunks. Only touched on
|
||||
/// `sampleQueue`.
|
||||
private var pending: [Int16] = []
|
||||
|
||||
init(channels: UInt32, onPcm: @escaping PcmHandler) {
|
||||
init(channels: UInt32, selection: ScreenAudioSelection, onPcm: @escaping PcmHandler) {
|
||||
self.channels = max(1, min(2, Int(channels)))
|
||||
self.selection = selection
|
||||
self.onPcm = onPcm
|
||||
super.init()
|
||||
}
|
||||
@@ -46,7 +76,8 @@ final class ScreenAudioCapture: NSObject, SCStreamOutput, SCStreamDelegate {
|
||||
let content = try await SCShareableContent.current
|
||||
guard let display = content.displays.first else { throw CaptureError.noDisplay }
|
||||
|
||||
let filter = SCContentFilter(display: display, excludingWindows: [])
|
||||
let filter = Self.makeFilter(selection: selection, display: display,
|
||||
apps: content.applications)
|
||||
|
||||
let config = SCStreamConfiguration()
|
||||
config.capturesAudio = true
|
||||
@@ -65,6 +96,39 @@ final class ScreenAudioCapture: NSObject, SCStreamOutput, SCStreamDelegate {
|
||||
self.stream = stream
|
||||
}
|
||||
|
||||
/// Turn a `ScreenAudioSelection` into an `SCContentFilter` against the running apps.
|
||||
/// ScreenCaptureKit filters audio per application, so we map bundle IDs → SCRunningApplication.
|
||||
private static func makeFilter(selection: ScreenAudioSelection, display: SCDisplay,
|
||||
apps: [SCRunningApplication]) -> SCContentFilter {
|
||||
func appsMatching(_ ids: Set<String>) -> [SCRunningApplication] {
|
||||
apps.filter { ids.contains($0.bundleIdentifier) }
|
||||
}
|
||||
|
||||
switch selection.scope {
|
||||
case .onlyApps(let bundleIDs):
|
||||
// Include-only already excludes everything else (the screen reader included), so the
|
||||
// excludeScreenReader flag is moot in this mode.
|
||||
return SCContentFilter(display: display,
|
||||
including: appsMatching(Set(bundleIDs)),
|
||||
exceptingWindows: [])
|
||||
|
||||
case .allExcept(let bundleIDs):
|
||||
var ids = Set(bundleIDs)
|
||||
if selection.excludeScreenReader { ids.formUnion(screenReaderBundleIDs) }
|
||||
return SCContentFilter(display: display,
|
||||
excludingApplications: appsMatching(ids),
|
||||
exceptingWindows: [])
|
||||
|
||||
case .entireDesktop:
|
||||
if selection.excludeScreenReader {
|
||||
return SCContentFilter(display: display,
|
||||
excludingApplications: appsMatching(screenReaderBundleIDs),
|
||||
exceptingWindows: [])
|
||||
}
|
||||
return SCContentFilter(display: display, excludingWindows: [])
|
||||
}
|
||||
}
|
||||
|
||||
/// Stop capture and release the stream. Safe to call multiple times.
|
||||
func stop() {
|
||||
guard let stream else { return }
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
import AppKit
|
||||
import ScreenCaptureKit
|
||||
|
||||
// ScreenSharePickerSheet — chooses *what* the SCREEN_AUDIO stream captures before sharing
|
||||
// starts. ScreenCaptureKit filters audio per application (not per window), so the user picks
|
||||
// a mode (everything / only-these / all-except-these) plus a set of apps, and a dedicated
|
||||
// toggle to drop their own screen-reader (VoiceOver) speech from the mix.
|
||||
//
|
||||
// Mirrors the modal-sheet pattern used by the rest of the macOS client (UserPickerSheet,
|
||||
// MoveUserSheet, …): an NSViewController presented via MainWindowController.presentSheet, with
|
||||
// an `onComplete` callback that returns the chosen `ScreenAudioSelection` (or nil on cancel).
|
||||
//
|
||||
// The app list comes from `SCShareableContent.current`, fetched asynchronously — that first
|
||||
// access is also what surfaces the Screen Recording (TCC) prompt, which is why the picker is
|
||||
// the natural place for it to appear, before any capture begins.
|
||||
final class ScreenSharePickerSheet: NSViewController, NSTableViewDataSource, NSTableViewDelegate {
|
||||
|
||||
/// Called with the chosen selection, or `nil` if the user cancelled.
|
||||
var onComplete: ((ScreenAudioSelection?) -> Void)?
|
||||
|
||||
private enum Mode: Int { case everything = 0, only = 1, except = 2 }
|
||||
|
||||
private struct AppEntry { let name: String; let bundleID: String; let icon: NSImage? }
|
||||
|
||||
private let initialSelection: ScreenAudioSelection
|
||||
private var mode: Mode
|
||||
private var excludeScreenReader: Bool
|
||||
private var checked: Set<String> // bundle IDs ticked in the app table
|
||||
|
||||
private var apps: [AppEntry] = []
|
||||
private let tableView = NSTableView()
|
||||
private var modeControl: NSSegmentedControl?
|
||||
private var screenReaderCheckbox: NSButton?
|
||||
private var statusLabel: NSTextField?
|
||||
|
||||
init(selection: ScreenAudioSelection) {
|
||||
self.initialSelection = selection
|
||||
switch selection.scope {
|
||||
case .entireDesktop: mode = .everything; checked = []
|
||||
case .onlyApps(let ids): mode = .only; checked = Set(ids)
|
||||
case .allExcept(let ids): mode = .except; checked = Set(ids)
|
||||
}
|
||||
self.excludeScreenReader = selection.excludeScreenReader
|
||||
super.init(nibName: nil, bundle: nil)
|
||||
}
|
||||
|
||||
required init?(coder: NSCoder) { fatalError() }
|
||||
|
||||
override func loadView() {
|
||||
view = NSView(frame: NSRect(x: 0, y: 0, width: 360, height: 420))
|
||||
}
|
||||
|
||||
override func viewDidLoad() {
|
||||
super.viewDidLoad()
|
||||
buildUI()
|
||||
loadApps()
|
||||
}
|
||||
|
||||
// MARK: - UI
|
||||
|
||||
private func buildUI() {
|
||||
let titleLabel = NSTextField(labelWithString: "Choose what to share:")
|
||||
titleLabel.font = .boldSystemFont(ofSize: 13)
|
||||
titleLabel.setAccessibilityLabel("Choose what to share")
|
||||
|
||||
let modeControl = NSSegmentedControl(
|
||||
labels: ["Everything", "Only selected", "All except selected"],
|
||||
trackingMode: .selectOne, target: self, action: #selector(modeChanged))
|
||||
modeControl.selectedSegment = mode.rawValue
|
||||
modeControl.segmentDistribution = .fillEqually
|
||||
modeControl.setAccessibilityLabel("Share mode")
|
||||
self.modeControl = modeControl
|
||||
|
||||
let col = NSTableColumn(identifier: NSUserInterfaceItemIdentifier("app"))
|
||||
tableView.addTableColumn(col)
|
||||
tableView.headerView = nil
|
||||
tableView.dataSource = self
|
||||
tableView.delegate = self
|
||||
tableView.rowHeight = 22
|
||||
tableView.setAccessibilityLabel("Application list")
|
||||
|
||||
let scroll = NSScrollView()
|
||||
scroll.documentView = tableView
|
||||
scroll.hasVerticalScroller = true
|
||||
scroll.borderType = .bezelBorder
|
||||
scroll.translatesAutoresizingMaskIntoConstraints = false
|
||||
scroll.setContentHuggingPriority(.defaultLow, for: .vertical)
|
||||
|
||||
let statusLabel = NSTextField(labelWithString: "Loading apps…")
|
||||
statusLabel.textColor = .secondaryLabelColor
|
||||
statusLabel.font = .systemFont(ofSize: 11)
|
||||
self.statusLabel = statusLabel
|
||||
|
||||
let screenReaderCheckbox = NSButton(checkboxWithTitle: "Exclude screen reader (VoiceOver) audio",
|
||||
target: self, action: #selector(screenReaderToggled))
|
||||
screenReaderCheckbox.state = excludeScreenReader ? .on : .off
|
||||
screenReaderCheckbox.toolTip = "Keep your VoiceOver speech out of the shared audio."
|
||||
self.screenReaderCheckbox = screenReaderCheckbox
|
||||
|
||||
let cancelButton = NSButton(title: "Cancel", target: self, action: #selector(cancelClicked))
|
||||
cancelButton.bezelStyle = .rounded
|
||||
cancelButton.keyEquivalent = "\u{1b}" // Esc
|
||||
|
||||
let shareButton = NSButton(title: "Share", target: self, action: #selector(shareClicked))
|
||||
shareButton.bezelStyle = .rounded
|
||||
shareButton.keyEquivalent = "\r"
|
||||
|
||||
let buttonRow = NSStackView(views: [NSView(), cancelButton, shareButton])
|
||||
buttonRow.orientation = .horizontal
|
||||
buttonRow.spacing = 8
|
||||
|
||||
let stack = NSStackView(views: [titleLabel, modeControl, scroll, statusLabel,
|
||||
screenReaderCheckbox, buttonRow])
|
||||
stack.orientation = .vertical
|
||||
stack.spacing = 8
|
||||
stack.edgeInsets = NSEdgeInsets(top: 16, left: 16, bottom: 16, right: 16)
|
||||
stack.translatesAutoresizingMaskIntoConstraints = false
|
||||
view.addSubview(stack)
|
||||
NSLayoutConstraint.activate([
|
||||
stack.topAnchor.constraint(equalTo: view.topAnchor),
|
||||
stack.leadingAnchor.constraint(equalTo: view.leadingAnchor),
|
||||
stack.trailingAnchor.constraint(equalTo: view.trailingAnchor),
|
||||
stack.bottomAnchor.constraint(equalTo: view.bottomAnchor),
|
||||
])
|
||||
|
||||
updateEnabledStates()
|
||||
}
|
||||
|
||||
/// Reflect the current mode: the app table only matters for only/except; the screen-reader
|
||||
/// toggle is moot for `.only` (include-only already excludes the screen reader).
|
||||
private func updateEnabledStates() {
|
||||
let listActive = (mode != .everything)
|
||||
tableView.isEnabled = listActive
|
||||
tableView.alphaValue = listActive ? 1.0 : 0.45
|
||||
screenReaderCheckbox?.isEnabled = (mode != .only)
|
||||
}
|
||||
|
||||
private func loadApps() {
|
||||
Task { @MainActor in
|
||||
do {
|
||||
let content = try await SCShareableContent.current
|
||||
var seen = Set<String>()
|
||||
var entries: [AppEntry] = []
|
||||
for app in content.applications {
|
||||
let bid = app.bundleIdentifier
|
||||
guard !bid.isEmpty, !seen.contains(bid) else { continue }
|
||||
// Hide our own app (its audio is already excluded) and the screen reader
|
||||
// (handled by its own toggle).
|
||||
if bid == Bundle.main.bundleIdentifier { continue }
|
||||
if ScreenAudioCapture.screenReaderBundleIDs.contains(bid) { continue }
|
||||
seen.insert(bid)
|
||||
let name = app.applicationName.isEmpty ? bid : app.applicationName
|
||||
let icon = NSRunningApplication
|
||||
.runningApplications(withBundleIdentifier: bid).first?.icon
|
||||
entries.append(AppEntry(name: name, bundleID: bid, icon: icon))
|
||||
}
|
||||
entries.sort { $0.name.localizedCaseInsensitiveCompare($1.name) == .orderedAscending }
|
||||
self.apps = entries
|
||||
self.statusLabel?.isHidden = true
|
||||
self.tableView.reloadData()
|
||||
} catch {
|
||||
self.statusLabel?.stringValue = "Screen Recording permission needed to list apps — "
|
||||
+ "grant it in System Settings ▸ Privacy & Security, then reopen this."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Actions
|
||||
|
||||
@objc private func modeChanged() {
|
||||
mode = Mode(rawValue: modeControl?.selectedSegment ?? 0) ?? .everything
|
||||
updateEnabledStates()
|
||||
}
|
||||
|
||||
@objc private func screenReaderToggled() {
|
||||
excludeScreenReader = (screenReaderCheckbox?.state == .on)
|
||||
}
|
||||
|
||||
@objc private func appCheckboxToggled(_ sender: NSButton) {
|
||||
let row = sender.tag
|
||||
guard row >= 0, row < apps.count else { return }
|
||||
let bid = apps[row].bundleID
|
||||
if sender.state == .on { checked.insert(bid) } else { checked.remove(bid) }
|
||||
}
|
||||
|
||||
@objc private func shareClicked() {
|
||||
let scope: ScreenAudioScope
|
||||
switch mode {
|
||||
case .everything: scope = .entireDesktop
|
||||
case .only: scope = .onlyApps(Array(checked))
|
||||
case .except: scope = .allExcept(Array(checked))
|
||||
}
|
||||
dismiss(nil)
|
||||
onComplete?(ScreenAudioSelection(scope: scope, excludeScreenReader: excludeScreenReader))
|
||||
}
|
||||
|
||||
@objc private func cancelClicked() {
|
||||
dismiss(nil)
|
||||
onComplete?(nil)
|
||||
}
|
||||
|
||||
// MARK: - NSTableViewDataSource / Delegate
|
||||
|
||||
func numberOfRows(in tableView: NSTableView) -> Int { apps.count }
|
||||
|
||||
func tableView(_ tableView: NSTableView, viewFor tableColumn: NSTableColumn?, row: Int) -> NSView? {
|
||||
let app = apps[row]
|
||||
let cell = AppCheckboxCell()
|
||||
cell.checkbox.title = app.name
|
||||
cell.checkbox.state = checked.contains(app.bundleID) ? .on : .off
|
||||
cell.checkbox.isEnabled = (mode != .everything)
|
||||
cell.checkbox.tag = row
|
||||
cell.checkbox.target = self
|
||||
cell.checkbox.action = #selector(appCheckboxToggled(_:))
|
||||
cell.iconView.image = app.icon
|
||||
return cell
|
||||
}
|
||||
|
||||
// Selecting a row shouldn't visually highlight — interaction is via the checkbox.
|
||||
func tableView(_ tableView: NSTableView, shouldSelectRow row: Int) -> Bool { false }
|
||||
}
|
||||
|
||||
// One row: a leading app icon and a checkbox titled with the app name.
|
||||
private final class AppCheckboxCell: NSTableCellView {
|
||||
let iconView = NSImageView()
|
||||
let checkbox = NSButton(checkboxWithTitle: "", target: nil, action: nil)
|
||||
|
||||
init() {
|
||||
super.init(frame: .zero)
|
||||
iconView.translatesAutoresizingMaskIntoConstraints = false
|
||||
checkbox.translatesAutoresizingMaskIntoConstraints = false
|
||||
addSubview(iconView)
|
||||
addSubview(checkbox)
|
||||
NSLayoutConstraint.activate([
|
||||
iconView.leadingAnchor.constraint(equalTo: leadingAnchor, constant: 4),
|
||||
iconView.centerYAnchor.constraint(equalTo: centerYAnchor),
|
||||
iconView.widthAnchor.constraint(equalToConstant: 16),
|
||||
iconView.heightAnchor.constraint(equalToConstant: 16),
|
||||
checkbox.leadingAnchor.constraint(equalTo: iconView.trailingAnchor, constant: 6),
|
||||
checkbox.trailingAnchor.constraint(equalTo: trailingAnchor, constant: -4),
|
||||
checkbox.centerYAnchor.constraint(equalTo: centerYAnchor),
|
||||
])
|
||||
}
|
||||
|
||||
required init?(coder: NSCoder) { fatalError() }
|
||||
}
|
||||
@@ -31,6 +31,8 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
|
||||
internal var micStreamId: UInt32 = 0
|
||||
private var screenStreamId: UInt32 = 0
|
||||
private var screenCapture: ScreenAudioCapture?
|
||||
// Last app/exclusion choice from the share picker; reused as the default next time.
|
||||
private var screenAudioSelection: ScreenAudioSelection = .default
|
||||
internal var pttKeyCode: UInt16 = 0x60 // F8
|
||||
private var pttMonitor: Any?
|
||||
private var serverMuted = false
|
||||
@@ -720,9 +722,29 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
|
||||
|
||||
@objc private func screenAudioClicked() {
|
||||
if screenStreamId == 0 {
|
||||
// Announce the stream now; ScreenCaptureKit capture starts once the server's
|
||||
// StreamAnnounceResult lands (the .streamStarted event), when the effective audio
|
||||
// config — and thus the channel count to capture — is known. See startScreenCapture.
|
||||
// Let the user pick what to share (apps to include/exclude, screen-reader audio)
|
||||
// before we announce anything. The picker remembers the previous choice.
|
||||
let sheet = ScreenSharePickerSheet(selection: screenAudioSelection)
|
||||
sheet.onComplete = { [weak self] selection in
|
||||
guard let self, let selection else { return } // nil = cancelled
|
||||
self.screenAudioSelection = selection
|
||||
self.beginScreenAudioShare()
|
||||
}
|
||||
presentSheet(sheet)
|
||||
} else {
|
||||
stopScreenCapture()
|
||||
client.stopStream(screenStreamId)
|
||||
screenStreamId = 0
|
||||
setShareScreenButton(active: false)
|
||||
addActivity("Stopped sharing screen audio")
|
||||
}
|
||||
}
|
||||
|
||||
/// Announce the SCREEN_AUDIO stream with the chosen selection in hand. ScreenCaptureKit
|
||||
/// capture starts once the server's StreamAnnounceResult lands (the .streamStarted event),
|
||||
/// when the effective audio config — and thus the channel count — is known. See
|
||||
/// startScreenCapture.
|
||||
private func beginScreenAudioShare() {
|
||||
let (result, streamId) = client.startStream(StreamDescriptor(kind: .screenAudio, deviceId: nil, label: "Desktop audio"))
|
||||
if result == .ok {
|
||||
screenStreamId = streamId
|
||||
@@ -731,13 +753,6 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
|
||||
} else {
|
||||
addActivity("Failed to start screen audio: \(result)")
|
||||
}
|
||||
} else {
|
||||
stopScreenCapture()
|
||||
client.stopStream(screenStreamId)
|
||||
screenStreamId = 0
|
||||
setShareScreenButton(active: false)
|
||||
addActivity("Stopped sharing screen audio")
|
||||
}
|
||||
}
|
||||
|
||||
private func setShareScreenButton(active: Bool) {
|
||||
@@ -760,7 +775,7 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
|
||||
let channels: UInt32 = (cfgResult == .ok && cfg?.stereo == true) ? 2 : 1
|
||||
|
||||
let streamId = screenStreamId
|
||||
let capture = ScreenAudioCapture(channels: channels) { [weak self] pcm, samples, ch in
|
||||
let capture = ScreenAudioCapture(channels: channels, selection: screenAudioSelection) { [weak self] pcm, samples, ch in
|
||||
self?.client.feedPcm(streamId: streamId, pcm: pcm, samplesPerChannel: samples, channels: ch)
|
||||
}
|
||||
screenCapture = capture
|
||||
@@ -768,7 +783,8 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
|
||||
Task { @MainActor in
|
||||
do {
|
||||
try await capture.start()
|
||||
addActivity("Started sharing screen audio (\(channels == 2 ? "stereo" : "mono"))")
|
||||
addActivity("Started sharing screen audio (\(channels == 2 ? "stereo" : "mono"))"
|
||||
+ "\(Self.scopeSuffix(for: screenAudioSelection))")
|
||||
} catch {
|
||||
// Most commonly: Screen Recording permission denied. Roll back the stream.
|
||||
screenCapture = nil
|
||||
@@ -788,6 +804,21 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
|
||||
screenCapture = nil
|
||||
}
|
||||
|
||||
/// A short human-readable description of the share scope for the activity log.
|
||||
private static func scopeSuffix(for selection: ScreenAudioSelection) -> String {
|
||||
var parts: [String] = []
|
||||
switch selection.scope {
|
||||
case .entireDesktop: break
|
||||
case .onlyApps(let ids): if !ids.isEmpty { parts.append("only \(ids.count) app(s)") }
|
||||
case .allExcept(let ids): if !ids.isEmpty { parts.append("excluding \(ids.count) app(s)") }
|
||||
}
|
||||
// For .onlyApps the screen reader is already excluded, so don't claim it twice.
|
||||
if selection.excludeScreenReader {
|
||||
if case .onlyApps = selection.scope {} else { parts.append("no screen reader") }
|
||||
}
|
||||
return parts.isEmpty ? "" : " — " + parts.joined(separator: ", ")
|
||||
}
|
||||
|
||||
@objc private func muteChanged() {
|
||||
let muted = muteButton?.state == .on
|
||||
let deafened = deafenButton?.state == .on
|
||||
|
||||
@@ -274,8 +274,24 @@ normal stream; only the *source* is platform-specific.
|
||||
| Platform | Mechanism | Notes |
|
||||
|----------|-----------|-------|
|
||||
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). **Supports per-app audio selection** — see below. |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). ReplayKit only ever delivers the *mixed* system stream as `.audioApp`, so **per-app filtering / VoiceOver exclusion is not possible on iOS** (it has no per-app granularity, unlike ScreenCaptureKit). |
|
||||
|
||||
### macOS detail — per-app audio selection
|
||||
|
||||
ScreenCaptureKit filters audio at the **application** level, so before sharing starts the user
|
||||
picks a scope in `ScreenSharePickerSheet` (`clients/apple/macOS/VoiceCatMac/Sheets/`):
|
||||
|
||||
- **Everything** — whole display, the original behaviour (`SCContentFilter(display:excludingWindows:)`).
|
||||
- **Only selected apps** — capture just the ticked apps (`init(display:including:exceptingWindows:)`).
|
||||
- **All except selected apps** — capture everything but the ticked apps
|
||||
(`init(display:excludingApplications:exceptingWindows:)`).
|
||||
|
||||
A dedicated **"Exclude screen reader (VoiceOver) audio"** toggle merges the screen-reader
|
||||
process(es) into the exclude set (`ScreenAudioCapture.screenReaderBundleIDs` — VoiceOver plus
|
||||
the speech-synthesis daemon that actually renders the spoken audio). The chosen
|
||||
`ScreenAudioSelection` is passed into `ScreenAudioCapture`, which builds the matching
|
||||
`SCContentFilter`. iOS/ReplayKit has no equivalent control (see the table note above).
|
||||
|
||||
### iOS detail
|
||||
|
||||
|
||||
Reference in New Issue
Block a user