9 Commits

Author SHA1 Message Date
fb73b694d0 fix(ios): VPIO silent playback + quiet speaker in stereo/studio presets
Some checks failed
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Two on-device bugs in the native iOS Voice-Processing path (Swift-only;
no core/ABI change).

1. Voice Chat (VPIO) silent playback: doStartMicStream() called
   audioRestart() BEFORE startStream, so when the engine was already
   running (a remote stream had started it) it reopened with
   external_capture=false and opened a hardware miniaudio capture device.
   The announce-result restart then early-returns (engine already running)
   so that device was never dropped and fought the AVAudioEngine VPIO unit,
   silencing playback. Now: setExternalPlayback first, then startStream
   (stores external_feed synchronously), THEN audioRestart() — the core
   reopens in full external mode (no hardware devices). Added VPIO
   diagnostics: graph/route formats at start, ring written/read totals at
   teardown.

2. Stereo Mic / Studio quiet earpiece: the .builtInMicBtA2dp presets omit
   .defaultToSpeaker (it breaks A2DP) and skip forceSpeaker, so with no
   Bluetooth connected output pinned to the quiet receiver. New
   IOSAudioRouter.applyA2dpSpeakerFallback() overrides to the built-in
   speaker when no external (A2DP/wired/AirPlay) output is present and
   clears the override when one is — called after activation and on
   device-change route changes.
2026-06-22 03:43:00 +02:00
6c17881cc0 feat(ios): real echo cancellation/NR via native Voice-Processing engine
iOS "voice chat" had echo and no noise suppression: real iOS AEC/NS/AGC
come only from Apple's Voice-Processing I/O unit (VPIO), but the core
plays/captures via miniaudio's plain RemoteIO units, so .voiceChat mode
alone never engaged AEC.

Core (ABI PATCH 1->2):
- vc_set_mixed_output_sink + vc_set_external_playback. In external mode the
  AudioEngine opens no hardware playback device; a mixer-timer thread drives
  on_playback (decode+mix) on a ~20ms cadence and ships the final mix to the
  sink. start() also skips the hardware capture device when the MIC stream is
  external_feed (AudioParams.external_capture).
- New white-box test test_external_playback (drives the timer with no hw).

iOS/Swift:
- StreamDescriptor.externalFeed; VoiceCatClient.setMixedOutputSink /
  setExternalPlayback wrappers.
- IOSVoiceProcessingEngine: AVAudioEngine + setVoiceProcessingEnabled; mic
  tap -> feedPcm, mixed-sink lock-free ring -> AVAudioSourceNode (both share
  the VPIO unit so AEC has its reference signal).
- IOSAudioRouter.currentConfigUsesVoiceProcessing scopes VPIO to the AEC
  presets; SessionState join/leave + reconcileVoicePath() switch paths;
  Voice Chat defaults to speaker; Settings surfaces AEC/NS state.

Known: pending on-device verification; a few bugs to fix afterward.
2026-06-22 02:38:01 +02:00
e806b698ec feat(windows): per-app screen-audio sharing (include/exclude)
Some checks failed
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Adds "only selected apps" and "all apps except selected" screen-audio
modes to the Windows client, alongside the existing entire-desktop path.

Per-app capture uses WASAPI process loopback (AUDCLNT_ACTIVATIONTYPE_
PROCESS_LOOPBACK) via ProcessLoopbackCapture, mixed by ProcessAudioMixer
and fed to the core through vc_stream_feed_pcm (external_feed=1 so the
core skips its own loopback device).

Init must pass AUDCLNT_STREAMFLAGS_LOOPBACK | EVENTCALLBACK |
AUTOCONVERTPCM; the LOOPBACK flag is what makes the virtual endpoint
deliver rendered audio (without it every buffer is flagged SILENT) and
AUTOCONVERTPCM resamples the app's native format to 48k s16.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 00:19:38 +02:00
4f89d2d32d feat(deploy): Docker + Linux server deployment
Some checks failed
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Multi-stage Dockerfile (builder → export → runtime) producing a 149 MB
Ubuntu 24.04 image, verified booting end-to-end on Docker Desktop.  vcpkg
fetched via shallow git fetch at the pinned baseline, release-only overlay
triplets (x64-linux, arm64-linux) to halve intermediate disk usage, and
buildtrees deleted within the RUN layer so they never land in the image or
the BuildKit cache.  Binary cache mount (VCPKG_BINARY_SOURCES) makes
subsequent rebuilds restore pre-built packages instead of recompiling.

Also adds:
- docker-compose.yml for one-command local deploy
- .dockerignore (excludes clients/, build/, .git/)
- .github/workflows/build-linux.yml — CI cross-build for amd64 + arm64
  with downloadable artifacts (primary path for building from Windows)
- scripts/build-linux-binaries.sh — local Docker binary extraction fallback
- deploy/linux/voicecat.service — hardened systemd unit for bare-metal
- cmake/voicecat-toolchain.cmake now auto-wires VCPKG_OVERLAY_TRIPLETS

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 19:51:43 +02:00
6071c8e238 fix(media): stop permanent voice loss after bad-network blip (protocol v2)
A bad UDP packet on a flaky link could permanently wedge the voice path,
unrecoverable even across app restarts. Three defects:

1. Anti-replay window was advanced from the UNAUTHENTICATED header seq
   before the AEAD tag was checked, and not rolled back on failure. One
   corrupted/forged frame shoved recv_highest_ far ahead, after which every
   legitimate frame was rejected as "too old" forever. Reorder to
   replay-check -> authenticate -> update (RFC 3711 3.3); the window now
   moves only after a successful tag check.

2. The wire seq was only the low 16 bits of the nonce counter (zero-extended
   on receive). After 65,536 frames the nonce desynced and all frames failed
   auth. Widen the voice frame seq u16 -> u64 (header 14 -> 20 bytes). The
   core owns all UDP framing, so Swift/C# clients need only a rebuild. This
   is a versioned wire change: VOICECAT_PROTOCOL_VERSION 1 -> 2, handshake
   rejects on mismatch.

3. Server leaked per-session UDP state on disconnect; unregister_session now
   frees udp_endpoints_/udp_tokens_/ssrc_to_session_.

Also add rate-limited dropped-frame logging to MediaRelay so a wedged media
path is observable. New regression tests in test_media_aead.cpp cover the
poison (fails on old code) and the 16-bit wrap. ctest --preset dev
-E external_pcm: 22/22 pass (external_pcm aborts on a pre-existing CoreAudio
shutdown race, unrelated).
2026-06-21 17:45:28 +02:00
5be6d8430d feat(ios): user-toggleable speakerphone output
Add a "speaker output" override so users can route audio to the built-in
speaker instead of the earpiece when no headphones/Bluetooth are connected.
Previously the receiver was the only fallback on the default Voice Chat preset.

The toggle inserts .defaultToSpeaker into the AVAudioSession category options
(skipped for the A2DP mode, where it would break Bluetooth routing). It yields
to connected BT/wired output and is orthogonal to preset matching. Exposed both
as a call-bar button in VoiceControlsView and a persisted Settings toggle.
2026-06-21 15:47:20 +02:00
0c6b1a36cf feat(ios): ad-hoc distribution scripts for pre-TestFlight testing
Add scripts/dist-ios-adhoc.sh and scripts/asc_api.py to build an ad-hoc
signed IPA and the OTA web-install files (manifest.plist + index.html)
for sharing the iOS client with registered devices before TestFlight.

- asc_api.py: App Store Connect API helper (ES256 JWT via cryptography,
  urllib) to register device UDIDs and list registered devices.
- dist-ios-adhoc.sh: registers UDIDs, builds the iOS device xcframework
  slice, archives + exports with method=release-testing using
  -allowProvisioningUpdates, and stages the install files into
  dist/ios-adhoc/.
- Document the workflow in clients/apple/README.md; ignore __pycache__.
2026-06-21 15:28:26 +02:00
c1e6f4f7ff feat(macos): per-app audio selection for screen sharing
Let users choose what the SCREEN_AUDIO stream captures before sharing:
share everything, only selected apps, or all except selected apps, plus a
first-class "Exclude screen reader (VoiceOver) audio" toggle.

ScreenCaptureKit filters audio per application, so ScreenAudioCapture now
takes a ScreenAudioSelection and builds the matching SCContentFilter
(including:/excludingApplications:). New ScreenSharePickerSheet lists
running apps from SCShareableContent. iOS left untouched -- ReplayKit only
delivers the mixed system stream, so per-app filtering is impossible there.
2026-06-21 13:35:01 +02:00
6b7f06a282 feat(clients): expose all channel codec params + guest nickname everywhere
Channel create/edit UIs only surfaced a subset of the core's vc_audio_config,
and DRED was exposed nowhere. While adding it, found a latent ABI mismatch:
both Swift AudioConfig and the C# VcAudioConfigNative blittable struct were one
int short of the native vc_audio_config (missing the trailing `dred`), so native
read past the managed struct in vc_create_channel/vc_edit_channel.

- core marshaling: thread `dred` through Swift (Models/Marshaling/toNative) and
  C# (Structs/Models/Marshaling/VoiceCatClient) -- fixes the ABI gap + enables it
- windows: add the one missing DRED checkbox to ChannelEditDialog
- macos: ChannelEditSheet now exposes application, sample rate, packet loss,
  complexity, and DRED (was stereo/bitrate/frame/FEC/DTX only)
- ios: rebuild ChannelEditView into a full create+edit form (all params); add
  SessionState.editChannel + an admin Edit swipe action (iOS had no edit UI)
- guest nickname: add a dedicated `nickname` to SavedServer on macOS+iOS
  (backward-compatible Codable), shown in Guest mode, wired into the guest auth
  path -- guests could not set a display name on either before (only Windows)

Verified: macOS + iOS (sim, arm64) xcodebuild BUILD SUCCEEDED; core ctest 22/23
(only external_pcm aborts on a pre-existing shutdown mutex race; no C++ changed).
2026-06-21 04:03:50 +02:00
72 changed files with 3808 additions and 166 deletions

24
.dockerignore Normal file
View File

@@ -0,0 +1,24 @@
# Git history — large and never needed inside the build context
.git/
# Previous build outputs
build/
# Native GUI client code (Swift/Xcode, C#/WinForms) — server build doesn't need these
clients/
# Documentation and prose — not compiled
docs/
*.md
AGENTS.md
PROGRESS.md
CLAUDE.md
# Editor / tooling config
.clang-format
.vscode/
.idea/
# OS noise
.DS_Store
Thumbs.db

87
.github/workflows/build-linux.yml vendored Normal file
View File

@@ -0,0 +1,87 @@
name: Build Linux Binaries
# Builds stripped voicecat-server + voicecat-admin for linux/amd64 and linux/arm64.
# Run manually from the Actions tab, or on any push to main.
# Artifacts are downloadable from the workflow run for ~90 days.
on:
workflow_dispatch: # manual trigger from the Actions tab
push:
branches: [main]
paths:
- 'core/**'
- 'server/**'
- 'tools/**'
- 'cmake/**'
- 'CMakeLists.txt'
- 'CMakePresets.json'
- 'vcpkg.json'
- 'Dockerfile'
- '.github/workflows/build-linux.yml'
jobs:
build:
strategy:
fail-fast: false
matrix:
include:
- arch: amd64
runner: ubuntu-24.04
vcpkg_triplet: x64-linux
- arch: arm64
runner: ubuntu-24.04-arm
vcpkg_triplet: arm64-linux
name: linux/${{ matrix.arch }}
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v4
- name: Cache vcpkg packages
uses: actions/cache@v4
with:
path: |
~/.cache/vcpkg
/usr/local/share/vcpkg/buildtrees
key: vcpkg-${{ matrix.vcpkg_triplet }}-${{ hashFiles('vcpkg.json') }}
restore-keys: |
vcpkg-${{ matrix.vcpkg_triplet }}-
- name: Install build dependencies
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential cmake ninja-build git curl zip unzip tar \
pkg-config autoconf autoconf-archive automake libtool nasm python3
- name: Set up vcpkg
run: |
VCPKG_COMMIT=$(jq -r '."builtin-baseline"' vcpkg.json)
git init /tmp/vcpkg
git -C /tmp/vcpkg remote add origin https://github.com/microsoft/vcpkg.git
git -C /tmp/vcpkg fetch --depth=1 origin "$VCPKG_COMMIT"
git -C /tmp/vcpkg checkout FETCH_HEAD
/tmp/vcpkg/bootstrap-vcpkg.sh -disableMetrics
echo "VCPKG_ROOT=/tmp/vcpkg" >> "$GITHUB_ENV"
echo "VCPKG_DISABLE_METRICS=1" >> "$GITHUB_ENV"
- name: Build (server-release preset)
run: |
cmake --preset server-release
cmake --build --preset server-release
- name: Collect binaries
run: |
mkdir -p dist
cp build/server-release/bin/voicecat-server dist/
cp build/server-release/bin/voicecat-admin dist/
file dist/voicecat-server dist/voicecat-admin
ls -lh dist/
- name: Upload binaries
uses: actions/upload-artifact@v4
with:
name: voicecat-linux-${{ matrix.arch }}
path: dist/
retention-days: 90

3
.gitignore vendored
View File

@@ -51,3 +51,6 @@ clients/apple/Package.resolved
# Test artifacts: TOFU pin store written by vc_client during headless tests
# (core/src/core/client.cpp falls back to this relative path when tofu_store_path is unset).
voicecat_tofu_pins.txt
# Python bytecode cache (e.g. scripts/asc_api.py)
__pycache__/

View File

@@ -8,7 +8,7 @@ and what's next* read [`PROGRESS.md`](PROGRESS.md); for *design* read [`docs/`](
> server-mute, channel CRUD, in-app account management, disconnect/keepalive/reaper. Windows
> WinForms C# client shipped (M4). **macOS AppKit client shipped** — `VoiceCatMac.xcodeproj`
> at `clients/apple/macOS/`. **iOS SwiftUI client shipped** — `VoiceCatiOS.xcodeproj` at
> `clients/apple/iOS/`. `ctest --preset dev` green — 23/23 tests.
> `clients/apple/iOS/`. `ctest --preset dev` green — 24/24 tests.
> External PCM feed/tap API (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) shipped.
> **Screen-audio sharing shipped on macOS (ScreenCaptureKit) and iOS (ReplayKit Broadcast
> Upload Extension → host App Group ring → `vc_stream_feed_pcm`).** See [`PROGRESS.md`](PROGRESS.md).

98
Dockerfile Normal file
View File

@@ -0,0 +1,98 @@
# syntax=docker/dockerfile:1
# ─────────────────────────────────────────────────────────────────────────────
# Stage 1 — Build
# ─────────────────────────────────────────────────────────────────────────────
FROM ubuntu:24.04 AS builder
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cmake \
ninja-build \
git \
curl \
zip \
unzip \
tar \
pkg-config \
ca-certificates \
autoconf \
autoconf-archive \
automake \
libtool \
nasm \
python3 \
&& rm -rf /var/lib/apt/lists/*
# Fetch vcpkg at the exact commit pinned in vcpkg.json builtin-baseline.
# vcpkg resolves baselines via `git show <sha>:versions/baseline.json`, so it
# needs a real .git repo — not a tarball. A single shallow fetch is fast (~30 MB)
# and gives vcpkg exactly what it needs.
ARG VCPKG_COMMIT=d46283cf33cf5de7bd88e12156ce03882be1f179
RUN git init /vcpkg \
&& git -C /vcpkg remote add origin https://github.com/microsoft/vcpkg.git \
&& git -C /vcpkg fetch --depth=1 origin "${VCPKG_COMMIT}" \
&& git -C /vcpkg checkout FETCH_HEAD \
&& /vcpkg/bootstrap-vcpkg.sh -disableMetrics
ENV VCPKG_ROOT=/vcpkg
ENV VCPKG_DISABLE_METRICS=1
WORKDIR /src
COPY . .
ARG TARGETARCH
# Three cache mounts:
# downloads — source tarballs (~200 MB); safe to share across arches
# vcpkg-cache — vcpkg binary cache (pre-built .zip archives per package ABI);
# restores packages in seconds on subsequent builds instead of
# recompiling. Scoped by arch so amd64/arm64 don't collide.
# buildtrees — NOT cached; deleted at end of layer so neither the Docker
# image nor the BuildKit cache accumulates several GB of
# intermediate build artifacts.
ENV VCPKG_BINARY_SOURCES="clear;files,/vcpkg-cache,readwrite"
RUN --mount=type=cache,target=/vcpkg/downloads \
--mount=type=cache,target=/vcpkg-cache,id=vc-bin-${TARGETARCH} \
cmake --preset server-release \
&& cmake --build --preset server-release \
&& rm -rf /vcpkg/buildtrees
# ─────────────────────────────────────────────────────────────────────────────
# Stage 2 — Export (binary-only, used by scripts/build-linux-binaries.sh)
# docker buildx build --target export --output type=local,dest=./dist/linux-amd64 .
# ─────────────────────────────────────────────────────────────────────────────
FROM scratch AS export
COPY --from=builder /src/build/server-release/bin/voicecat-server /voicecat-server
COPY --from=builder /src/build/server-release/bin/voicecat-admin /voicecat-admin
# ─────────────────────────────────────────────────────────────────────────────
# Stage 3 — Runtime (default stage — must be last)
# ─────────────────────────────────────────────────────────────────────────────
FROM ubuntu:24.04 AS runtime
ENV DEBIAN_FRONTEND=noninteractive
# ca-certificates is useful if the server ever makes outbound TLS calls; also
# satisfies any mbedTLS system-CA lookup at runtime.
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
RUN groupadd -r voicecat && useradd -r -g voicecat -s /sbin/nologin voicecat
COPY --from=builder /src/build/server-release/bin/voicecat-server /usr/local/bin/voicecat-server
COPY --from=builder /src/build/server-release/bin/voicecat-admin /usr/local/bin/voicecat-admin
RUN mkdir -p /data && chown voicecat:voicecat /data
USER voicecat
# Persistent state: Ed25519 identity key, self-signed TLS cert, SQLite database.
VOLUME ["/data"]
# Control (TLS 1.3) and media (ChaCha20-Poly1305) share one port number on TCP+UDP.
EXPOSE 8384/tcp
EXPOSE 8384/udp
ENTRYPOINT ["/usr/local/bin/voicecat-server"]
CMD ["--data-dir", "/data"]

View File

@@ -10,6 +10,130 @@ up instantly. Newest status at the top.
## ▶ Where we left off / next action
- **Awaiting on-device verification (2026-06-22):** **iOS real echo cancellation / noise suppression
via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS
AEC/NS/AGC come only from Apple's Voice-Processing I/O unit (VPIO), but the core uses miniaudio's
plain RemoteIO units — so `.voiceChat` mode alone never engaged AEC. Fix moves both mic capture and
playback to a native Swift `AVAudioEngine` (`setVoiceProcessingEnabled`) on the AEC presets, with the
core in external mode.
- **Core (done, builds + tests green):** new ABI `vc_set_mixed_output_sink` + `vc_set_external_playback`
(voicecat.h PATCH→2). `AudioEngine` gains a mixer-timer thread that drives `on_playback` (decode+mix)
on a ~20 ms cadence with NO hardware playback device and ships the final mix to the mixed-output
sink; `start()` also skips the hardware capture device when the MIC stream is `external_feed`
(`AudioParams.external_capture`). New white-box test `test_external_playback` (23/24;
pre-existing `external_pcm` teardown crash on Darwin 25.5 is UNRELATED — original tree crashes too).
- **Swift (done, builds):** `VoiceCatCore` wrappers (`externalFeed` on `StreamDescriptor`,
`setMixedOutputSink`, `setExternalPlayback`); new `IOSVoiceProcessingEngine.swift` (VPIO
`AVAudioEngine`: mic tap→`feedPcm`, mixed-sink lock-free ring→`AVAudioSourceNode`);
`IOSAudioRouter.currentConfigUsesVoiceProcessing` gates the path per preset; `SessionState`
join/leave + `reconcileVoicePath()` switch between VPIO and the miniaudio path; Voice Chat defaults
to speaker; SettingsView shows AEC/NS state. **Rebuild the xcframework** before building the app:
`clients/apple/scripts/build-xcframework.sh --all` (new ABI symbols). `xcodebuild` iOS sim Debug
BUILD SUCCEEDED.
- **Post-verification fixes (2026-06-22, Swift-only — no core/ABI change):** two on-device bugs fixed.
- **Voice Chat (VPIO) silent playback:** `SessionState.doStartMicStream()` called `audioRestart()`
BEFORE `startStream`, so when the engine was already running (a remote stream had started it) it
reopened with `external_capture=false` and opened a hardware miniaudio capture device; the
announce-result restart then early-returned (engine already running) so that device was never
dropped and fought the `AVAudioEngine` VPIO unit, silencing playback. Fix: set
`setExternalPlayback` first, then `startStream` (which stores `external_feed` synchronously), THEN
`audioRestart()` — the core reopens in full external mode (no hardware devices). Added VPIO
diagnostics (graph/route formats at start; ring written/read totals at teardown).
- **Stereo Mic / Studio quiet earpiece:** the `.builtInMicBtA2dp` presets omit `.defaultToSpeaker`
(it breaks A2DP) and skip `forceSpeaker`, so with no Bluetooth connected output pinned to the quiet
receiver. New `IOSAudioRouter.applyA2dpSpeakerFallback()` overrides to the built-in speaker when no
external (A2DP/wired/AirPlay) output is present, clears the override when one is — called after
activation and on device-change route changes (`AudioSessionManager`).
- **Next (user, on device):** two iPhones on speaker, Voice Chat preset → confirm (a) no echo, (b)
background noise suppressed, (c) speaker output by default AND remote audio is now audible; then
Stereo Mic / Studio with no BT → confirm loud speaker (not earpiece), and A2DP takes over when a BT
headset connects. Tune the mixer-timer/ring sizing if there's under/overrun.
- **Done (2026-06-21):** **Docker + Linux deployment + GitHub Actions cross-build.** Added the complete Linux server
deployment story (the only missing platform — Windows and macOS already have native
binaries):
- `Dockerfile` — multi-stage (builder: `ubuntu:24.04` + vcpkg + `cmake --preset
server-release`; runtime: `ubuntu:24.04`, non-root `voicecat` user, `/data` volume,
TCP+UDP 8384). vcpkg is fetched via the GitHub archive tarball at the exact
`builtin-baseline` commit (`d46283cf…`), avoiding a full git-history clone. BuildKit
cache mounts on `/vcpkg/downloads`, `/vcpkg/buildtrees`, `/vcpkg/packages` (scoped by
`TARGETARCH`) keep rebuilds fast. Both `voicecat-server` and `voicecat-admin` are
copied into the runtime image.
- `docker-compose.yml` — single-service compose file with `restart: unless-stopped`,
named volume `voicecat-data`, and port mappings for TCP+UDP 8384. `command:` shows
how to set `--name`.
- `.dockerignore` — excludes `.git/`, `build/`, `clients/` (Swift/C# code), `docs/`,
markdown, editor config; build context is just `core/`, `server/`, `tools/`, `cmake/`,
and the three root CMake/vcpkg files.
- `deploy/linux/voicecat.service` — hardened systemd unit (non-root, `ProtectSystem`,
`NoNewPrivileges`, `AmbientCapabilities=CAP_NET_BIND_SERVICE`) for bare-metal deploys.
- Multi-arch: `docker buildx build --platform linux/amd64,linux/arm64 .` works without
any triplet override — `cmake/voicecat-toolchain.cmake` auto-detects from the host
arch cmake sees inside the buildx container.
- Quick start: `docker compose up -d` (or `docker run -d -p 8384:8384/tcp -p
8384:8384/udp -v voicecat-data:/data voicecat`). First run auto-generates identity
+ cert + DB; check logs for fingerprint + admin password.
- **GitHub Actions** (`.github/workflows/build-linux.yml`): primary cross-platform
binary build path — amd64 uses `ubuntu-24.04`, arm64 uses `ubuntu-24.04-arm`
(native, not QEMU). Triggers on push to main (when C++/cmake files change) and
manually via `workflow_dispatch`. Downloads land as 90-day artifacts.
`scripts/build-linux-binaries.sh` is the local Docker fallback (needs ~1015 GB
free disk; suits Linux dev machines, not Windows Docker Desktop).
- **Done (2026-06-21):** **Fix permanent voice-loss bug + harden the UDP media path (protocol v2).**
Field report: two iOS users lost all audio mid-call after a bad-network blip and could not
recover even by restarting the apps. Root causes found in the UDP media path:
1. **Anti-replay window poisoned by unauthenticated packets (the trigger).**
`SodiumMediaCrypto::open()` advanced `recv_highest_` from the plaintext header `seq`
*before* verifying the AEAD tag and never rolled it back on failure. One corrupted/forged
frame (a bit-flip on flaky wifi) shoved the high-water mark far ahead, after which every
legitimate frame was rejected as "too old" — permanently. Fixed by reordering to
replay-check → authenticate → update (RFC 3711 §3.3): the window is now touched only after
a successful tag check. Regression test in `test_media_aead.cpp`
(`test_corrupted_seq_does_not_poison_window`) — fails on the old code, passes now.
2. **16-bit seq wrap with no rollover counter.** The wire header carried only the low 16 bits
of the nonce counter (zero-extended on receive); after 65,536 frames the reconstructed
nonce diverged and all frames failed auth. **Wire format widened to a full u64 seq**
(`voice_frame.h`: header 14 → 20 bytes, `seq` u16 → u64; `crypto.cpp`, `client.cpp`,
`media_relay.cpp` updated; `JitterBuffer::Frame::seq` widened). This is a **versioned wire
change → `VOICECAT_PROTOCOL_VERSION` 1 → 2**; the `Hello` handshake rejects on mismatch
(`conn_session.cpp`). The voice frame is parsed only in `core/`+`server/`+`tests/`, so the
Swift/C# clients need only a rebuild — no parser changes.
3. **Server leaked UDP state on disconnect.** `SessionRegistry::unregister_session()` now also
frees `udp_endpoints_`/`udp_tokens_`/`ssrc_to_session_` (scan-and-erase by session id).
4. **Diagnostics.** `MediaRelay` now emits rate-limited dropped-frame counters
(unmapped-endpoint / no-recv-crypto / open-failed) so a wedged media path is observable.
- **Verified:** `cmake --build --preset dev` clean; `ctest --preset dev -E external_pcm`
**22/22 pass** (incl. `m2_voice` e2e relay + the two new AEAD regressions). `external_pcm`
still aborts on the **pre-existing** CoreAudio shutdown mutex race (confirmed identical on a
clean baseline checkout under the same harness — unrelated to these changes). Docs updated:
`voice.md` §2 (header), `protocol.md` (v2 + negotiation), `security.md` (authenticate-then-advance).
- **Done (2026-06-21):** **Expose all channel codec params + guest nickname in every client.**
- **DRED everywhere + ABI fix.** `dred` (Opus 1.6 Deep REDundancy) existed in the C ABI
(`vc_audio_config.dred`) and proto but was absent from *both* client marshaling layers — a
latent ABI mismatch: Swift `AudioConfig` and the C# `VcAudioConfigNative` blittable struct
were each one `int` short of the native struct passed to `vc_create_channel`/`vc_edit_channel`.
Added `dred` through Swift (`Models.swift`, `Marshaling.swift`, `VoiceCatClient.toNative`) and
C# (`Structs.cs`, `Models.cs`, `Marshaling.cs`, `VoiceCatClient.cs`).
- **Windows:** added the one missing DRED checkbox to `ChannelEditDialog` (all other params
were already present).
- **macOS:** `ChannelEditSheet` now exposes the previously-hidden params — application profile,
sample rate, expected packet loss, complexity, and DRED (was only stereo/bitrate/frame/FEC/DTX).
- **iOS:** `ChannelEditView` was name+topic only; rebuilt into a full create **and edit** form
(General: name/topic/parent/password/max-users/sort-order; Audio: stereo/bitrate/sample-rate/
frame/application/packet-loss/complexity/FEC/DTX/DRED). Added `SessionState.editChannel` and an
"Edit" swipe action (admins) in `ChannelTreeView` + `ChannelBrowserView` (iOS previously had no
edit-channel UI at all). Note: the channel list doesn't carry the current audio config, so on
edit the audio fields start from codec defaults — same limitation as macOS/Windows.
- **Guest nickname.** Guests could not set a display name on iOS *or* macOS (the field was
absent/disabled; only Windows had it). Added a dedicated `nickname` to `SavedServer` on both
(backward-compatible Codable), a Nickname field shown in Guest mode (`AddServerView` /
`AddServerSheet`), and wired the guest auth path to use it (`AppState`, `ConnectWindowController`).
- **Verified:** `xcodebuild` Debug — macOS BUILD SUCCEEDED; iOS (sim, `ARCHS=arm64`) BUILD
SUCCEEDED. Core `ctest --preset dev` 22/23 (only `external_pcm` aborts on a pre-existing
shutdown mutex race; no C++ was changed). Windows C# not buildable on macOS — changes reviewed.
- **Done (2026-06-21):** **iOS iPhone-layout UX fixes.** (1) Channels are now a **drill-down**
on iPhone — new `ChannelBrowserView` (root list of top-level channels) → `ChannelDetailView`
(people in the channel + sub-channels + an explicit "Join Channel" button with password
@@ -36,6 +160,18 @@ up instantly. Newest status at the top.
aborts at shutdown (`mutex lock failed`), a **pre-existing** teardown crash unrelated to this
change (no C++ was modified).
- **Done (2026-06-21):** **macOS per-app screen-audio selection.** Before sharing, a new
`ScreenSharePickerSheet` lets the user choose scope — share Everything / Only selected apps /
All except selected apps — plus a first-class **"Exclude screen reader (VoiceOver) audio"**
toggle. `ScreenAudioCapture` now takes a `ScreenAudioSelection` and builds the matching
`SCContentFilter` (`including:` / `excludingApplications:`); app list comes from
`SCShareableContent`. macOS `xcodebuild` Debug BUILD SUCCEEDED. iOS deliberately untouched —
ReplayKit only delivers the mixed system stream, so per-app/VoiceOver filtering is impossible
there (documented in voice.md §9). **Still to verify on-device:** which process actually
carries VoiceOver speech (VoiceOver app vs. `com.apple.speech.speechsynthesisd`) — the exclude
set covers both candidates in `ScreenAudioCapture.screenReaderBundleIDs`; confirm exclusion
actually silences it in a real share.
- **Done (2026-06-20):** **macOS client UI overhaul** — mirrors the Windows client's UI
overhaul (commit 97fa659 + 540ec13), adapted to Mac-native conventions. Also fixed and
verified the previously-uncompiled Swift changes from the external PCM feed/tap commit

View File

@@ -128,3 +128,29 @@ The `Package.swift` declares:
the fat static lib is C++20, so the final executable must link libc++ (the LLVM C++ standard
library on macOS). vcpkg's static deps are already in the `.a`; macOS system frameworks
(CoreAudio/CoreFoundation) are auto-discovered by the linker.
## Ad-hoc distribution (iOS, pre-TestFlight)
To hand the iOS app to a handful of friends before TestFlight, use
[`scripts/dist-ios-adhoc.sh`](../../scripts/dist-ios-adhoc.sh). It registers each device's
UDID, builds an ad-hoc-signed `VoiceCatiOS.ipa`, and generates the `manifest.plist` +
`index.html` for an over-the-air (`itms-services://`) web install. Ad-hoc builds only run on
devices whose UDID is registered *before* signing, and stock iOS won't install a bare `.ipa`
without a sideloading tool — so the web-install page is the friend-friendly path.
```bash
# One-time: create an App Store Connect API "Team Key" (.p8, Admin/App Manager access) at
# App Store Connect → Users and Access → Integrations → App Store Connect API
export ASC_KEY_ID=ABC123 ASC_ISSUER_ID=1111-... ASC_KEY_PATH=~/.appstoreconnect/AuthKey_ABC123.p8
# Register a device + build + stage everything into dist/ios-adhoc/
scripts/dist-ios-adhoc.sh --udid <UDID> --name "Friend iPhone" \
--base-url https://example.com/voicecat
```
Then upload the three staged files (`VoiceCatiOS.ipa`, `manifest.plist`, `index.html`) to
that **HTTPS** folder and open `index.html` in Safari on a registered iPhone (iOS 18+).
Device UDID registration is automated via [`scripts/asc_api.py`](../../scripts/asc_api.py)
(`asc_api.py list` shows the registered devices against the 100-iOS-devices/year cap).
Requires a paid Apple Developer Program membership. Run `scripts/dist-ios-adhoc.sh --help`
for all flags.

View File

@@ -94,7 +94,7 @@ internal enum Marshaling {
AudioConfig(codec: c.codec, stereo: c.mode != 0, sampleRate: c.sample_rate,
bitrateBps: c.bitrate_bps, frameMs: c.frame_ms, application: c.application,
fec: c.fec != 0, expectedPacketLoss: c.expected_packet_loss,
dtx: c.dtx != 0, complexity: c.complexity)
dtx: c.dtx != 0, complexity: c.complexity, dred: c.dred != 0)
}
static func permissions(_ p: vc_permissions) -> Permissions {

View File

@@ -193,15 +193,16 @@ public struct AudioConfig: Sendable, Equatable {
public let expectedPacketLoss: UInt32 // % 0..100
public let dtx: Bool
public let complexity: UInt32 // 0..10
public let dred: Bool // Deep REDundancy (Opus 1.6), off by default
public init(codec: UInt32 = 0, stereo: Bool = false, sampleRate: UInt32 = 48000,
bitrateBps: UInt32 = 64000, frameMs: UInt32 = 20, application: UInt32 = 0,
fec: Bool = true, expectedPacketLoss: UInt32 = 5, dtx: Bool = false,
complexity: UInt32 = 10) {
complexity: UInt32 = 10, dred: Bool = false) {
self.codec = codec; self.stereo = stereo; self.sampleRate = sampleRate
self.bitrateBps = bitrateBps; self.frameMs = frameMs; self.application = application
self.fec = fec; self.expectedPacketLoss = expectedPacketLoss; self.dtx = dtx
self.complexity = complexity
self.complexity = complexity; self.dred = dred
}
}
@@ -211,9 +212,14 @@ public struct StreamDescriptor: Sendable, Equatable {
/// nil = default device for this kind.
public let deviceId: String?
public let label: String
/// When true the caller feeds PCM via `feedPcm` (e.g. the iOS VPIO mic path) and the core
/// skips opening a hardware capture device for this stream. Mirrors `vc_stream_desc.external_feed`.
public let externalFeed: Bool
public init(kind: VoiceCatStreamKind, deviceId: String? = nil, label: String) {
public init(kind: VoiceCatStreamKind, deviceId: String? = nil, label: String,
externalFeed: Bool = false) {
self.kind = kind; self.deviceId = deviceId; self.label = label
self.externalFeed = externalFeed
}
}

View File

@@ -41,6 +41,10 @@ import Foundation
/// `VcPcmSinkCallback` delegate.
public typealias VoiceCatPcmSinkCallback = vc_pcm_sink_cb
/// Swift-idiomatic alias for the C `vc_mixed_output_cb` function-pointer type from `voicecat.h`
/// the external mixed-output sink used by the iOS VPIO path (see `setMixedOutputSink`).
public typealias VoiceCatMixedOutputCallback = vc_mixed_output_cb
/// The Swift wrapper around `vc_client*`. Owns the native handle for its entire lifetime;
/// `deinit` destroys it. Events and level meters are delivered on the main queue via the
/// `onEvent` / `onLevel` closures.
@@ -292,6 +296,7 @@ public final class VoiceCatClient {
desc.kind = descriptor.kind.cValue
desc.device_id = deviceIdPtr.map { UnsafePointer($0) }
desc.label = UnsafePointer(labelPtr)
desc.external_feed = descriptor.externalFeed ? 1 : 0
let r = vc_stream_start(handle, &desc, &streamId)
return (VoiceCatResult(r), streamId)
@@ -355,6 +360,26 @@ public final class VoiceCatClient {
VoiceCatResult(vc_set_pcm_sink(handle, cb, user))
}
/// External mixed-output sink (iOS VPIO) receives the FINAL mixed remote audio as int16
/// PCM on the core's mixer-timer thread when external playback is enabled. The Swift VPIO
/// renderer copies this into its ring and plays it through the voice-processing output so
/// echo cancellation has its reference signal. Pass `nil` to disable. Mirrors
/// `vc_set_mixed_output_sink`. The callback MUST NOT block or allocate.
@discardableResult
public func setMixedOutputSink(_ cb: VoiceCatMixedOutputCallback?,
user: UnsafeMutableRawPointer?) -> VoiceCatResult {
VoiceCatResult(vc_set_mixed_output_sink(handle, cb, user))
}
/// Enable/disable external-playback mode (iOS VPIO). When enabled, the core opens NO hardware
/// playback device; it drives decode+mix on a timer and delivers the final mix via
/// `setMixedOutputSink`. Apply before the engine starts, or follow with `audioRestart()` to
/// apply to a running engine. Mirrors `vc_set_external_playback`.
@discardableResult
public func setExternalPlayback(_ enabled: Bool) -> VoiceCatResult {
VoiceCatResult(vc_set_external_playback(handle, enabled ? 1 : 0))
}
@discardableResult
public func setInputMode(_ mode: VoiceCatInputMode) -> VoiceCatResult {
VoiceCatResult(vc_set_input_mode(handle, mode.cValue))
@@ -586,6 +611,7 @@ extension AudioConfig {
n.expected_packet_loss = expectedPacketLoss
n.dtx = dtx ? 1 : 0
n.complexity = complexity
n.dred = dred ? 1 : 0
return n
}
}

View File

@@ -33,6 +33,7 @@
BBBB00000000000000000046 /* AccountsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000002D /* AccountsView.swift */; };
BBBB00000000000000000047 /* SettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000002E /* SettingsView.swift */; };
BBBB0000000000000000004B /* IOSAudioRouter.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB0000000000000000002F /* IOSAudioRouter.swift */; };
BBBB00000000000000000F02 /* IOSVoiceProcessingEngine.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBBB00000000000000000F01 /* IOSVoiceProcessingEngine.swift */; };
BBBB00000000000000000048 /* VoiceCatCore in Frameworks */ = {isa = PBXBuildFile; productRef = BBBB0000000000000000004A /* VoiceCatCore */; };
CCCC00000000000000000010 /* BroadcastAudioPump.swift in Sources */ = {isa = PBXBuildFile; fileRef = CCCC00000000000000000002 /* BroadcastAudioPump.swift */; };
CCCC00000000000000000011 /* BroadcastAudioRing.swift in Sources */ = {isa = PBXBuildFile; fileRef = CCCC00000000000000000001 /* BroadcastAudioRing.swift */; };
@@ -89,6 +90,7 @@
BBBB0000000000000000002D /* AccountsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AccountsView.swift; sourceTree = "<group>"; };
BBBB0000000000000000002E /* SettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsView.swift; sourceTree = "<group>"; };
BBBB0000000000000000002F /* IOSAudioRouter.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = IOSAudioRouter.swift; sourceTree = "<group>"; };
BBBB00000000000000000F01 /* IOSVoiceProcessingEngine.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = IOSVoiceProcessingEngine.swift; sourceTree = "<group>"; };
CCCC00000000000000000001 /* BroadcastAudioRing.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BroadcastAudioRing.swift; sourceTree = "<group>"; };
CCCC00000000000000000002 /* BroadcastAudioPump.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BroadcastAudioPump.swift; sourceTree = "<group>"; };
CCCC00000000000000000003 /* SampleHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SampleHandler.swift; sourceTree = "<group>"; };
@@ -143,6 +145,7 @@
BBBB00000000000000000019 /* SessionState.swift */,
BBBB0000000000000000001A /* AudioSessionManager.swift */,
BBBB0000000000000000002F /* IOSAudioRouter.swift */,
BBBB00000000000000000F01 /* IOSVoiceProcessingEngine.swift */,
BBBB0000000000000000001B /* ServerListStore.swift */,
BBBB0000000000000000001C /* SavedServer.swift */,
CCCC00000000000000000002 /* BroadcastAudioPump.swift */,
@@ -296,6 +299,7 @@
BBBB00000000000000000032 /* SessionState.swift in Sources */,
BBBB00000000000000000033 /* AudioSessionManager.swift in Sources */,
BBBB0000000000000000004B /* IOSAudioRouter.swift in Sources */,
BBBB00000000000000000F02 /* IOSVoiceProcessingEngine.swift in Sources */,
BBBB00000000000000000034 /* ServerListStore.swift in Sources */,
BBBB00000000000000000035 /* SavedServer.swift in Sources */,
CCCC00000000000000000010 /* BroadcastAudioPump.swift in Sources */,

View File

@@ -76,7 +76,7 @@ final class AppState {
// Auth is queued immediately the core serialises it behind TLS + TOFU.
switch server.authMode {
case .guest:
let nick = server.savedUsername.isEmpty ? "iOS User" : server.savedUsername
let nick = (server.nickname?.isEmpty == false) ? server.nickname! : "iOS User"
client.authenticateGuest(nick)
case .password:
let savedPw = ServerListStore.shared.loadPassword(tag: server.keychainTag)

View File

@@ -15,6 +15,12 @@ final class AudioSessionManager {
/// channel count (e.g. when switching stereo mono) without going through `SessionState`.
var activeMicStreamId: UInt32?
/// Set by `SessionState`. Invoked by `IOSAudioRouter` after an audio-config change so the
/// voice path (native VPIO vs the core's miniaudio path) can be restarted to match the new
/// preset/route when the mic is active. No-op when not in voice. See
/// `SessionState.reconcileVoicePath()` and `IOSVoiceProcessingEngine`.
var reconcileVoicePath: (() -> Void)?
/// Tracks whether WE activated the session. The session must be active whenever the
/// AudioEngine is running (for capture OR playback), so it is activated when any audio
/// needs to play (a remote stream started OR the user joins voice) and only deactivated
@@ -50,12 +56,10 @@ final class AudioSessionManager {
let session = AVAudioSession.sharedInstance()
try session.setActive(true, options: [])
isSessionActive = true
// For the A2DP output presets, make sure output isn't pinned to the built-in speaker.
// A2DP routing in .playAndRecord is fragile; clearing any speaker override after the
// session is live nudges iOS to honor the Bluetooth output route.
if IOSAudioRouter.shared.wantsA2dpOutput {
try? session.overrideOutputAudioPort(.none)
}
// For the A2DP output presets, pick the right output once the session is live: defer to
// a connected A2DP/wired/AirPlay route, but fall back to the loud built-in speaker (not
// the quiet earpiece) when nothing external is connected. See applyA2dpSpeakerFallback().
IOSAudioRouter.shared.applyA2dpSpeakerFallback()
let route = AVAudioSession.sharedInstance().currentRoute
let outputNames = route.outputs.map { $0.portName }.joined(separator: ", ")
let inputNames = route.inputs.map { $0.portName }.joined(separator: ", ")
@@ -154,6 +158,11 @@ final class AudioSessionManager {
if reason == .oldDeviceUnavailable || reason == .newDeviceAvailable {
logger.info("routeChange — external device change, re-applying config")
IOSAudioRouter.shared.applyConfiguration()
// Re-evaluate the A2DP-mode speaker fallback: a Bluetooth unplug should drop us onto
// the loud speaker (not the earpiece), and a replug should hand output back to A2DP.
if isSessionActive {
IOSAudioRouter.shared.applyA2dpSpeakerFallback()
}
}
IOSAudioRouter.shared.refreshRoutes()

View File

@@ -55,6 +55,10 @@ final class IOSAudioRouter: ObservableObject {
@Published var inputPorts: [IOSAudioInputPort] = []
@Published var outputRoutes: [IOSAudioOutputRoute] = []
@Published var bluetoothMode: BluetoothMode = .btHfpVoice
/// User-requested speaker fallback: when on, route to the built-in speaker instead of the
/// earpiece (receiver) when no headphones/Bluetooth are connected. Orthogonal to the
/// bluetooth mode and presets. Default off current behavior is unchanged for existing users.
@Published var forceSpeaker: Bool = false
@Published var micMode: MicMode = .standard
@Published var captureChannels: CaptureChannels = .mono
@Published var selectedInputPortId: String?
@@ -165,6 +169,7 @@ final class IOSAudioRouter: ObservableObject {
private let kDataSourceId = "cat.voice.audio.dataSourceId"
private let kPolarPattern = "cat.voice.audio.polarPattern"
private let kPreset = "cat.voice.audio.preset"
private let kForceSpeaker = "cat.voice.audio.forceSpeaker"
/// Re-entrancy guard: setCategory/setPreferredInput/etc. trigger route-change
/// notifications synchronously on the same thread. Without this guard,
@@ -302,6 +307,14 @@ final class IOSAudioRouter: ObservableObject {
return .custom
}
/// Whether the current configuration should use the native iOS Voice-Processing path (VPIO:
/// real AEC/NS/AGC via `IOSVoiceProcessingEngine`). True exactly when `applyConfiguration`
/// selects the `.voiceChat` AVAudioSession mode mono + standard processing + not A2DP
/// (A2DP / stereo / raw modes can't use VPIO, so they keep the core's miniaudio path).
var currentConfigUsesVoiceProcessing: Bool {
captureChannels == .mono && micMode == .standard && bluetoothMode != .builtInMicBtA2dp
}
// MARK: - Apply configuration
/// Apply the full audio configuration to AVAudioSession. Call this before the core
@@ -322,9 +335,11 @@ final class IOSAudioRouter: ObservableObject {
// .mixWithOthers is ALWAYS set it keeps other audio (notably VoiceOver, which a
// blind user needs to operate the phone) audible while our session is active. Never
// drop it.
// .defaultToSpeaker is set ONLY for the speaker preset. It forces output to the
// built-in speaker instead of the receiver but it also actively breaks A2DP
// routing in .playAndRecord, so it must NOT be set for the A2DP or HFP presets.
// .defaultToSpeaker is set for the speaker preset and, when the user enables the
// `forceSpeaker` toggle, for the HFP preset too it forces output to the built-in
// speaker instead of the receiver while still yielding to connected BT/wired output.
// It also actively breaks A2DP routing in .playAndRecord, so it must NEVER be set for
// the A2DP preset (forceSpeaker is intentionally ignored there).
// .allowAirPlay is added to the Bluetooth presets so AirPlay output also works.
var options: AVAudioSession.CategoryOptions = [.mixWithOthers]
switch bluetoothMode {
@@ -347,6 +362,14 @@ final class IOSAudioRouter: ObservableObject {
options.insert(.defaultToSpeaker)
}
// User-requested speaker fallback: route to the built-in speaker instead of the
// receiver when no headphones/BT are connected. Skipped for the A2DP mode because
// .defaultToSpeaker breaks A2DP routing (see note above). Redundant for
// builtInMicSpeaker, which already sets it.
if forceSpeaker && bluetoothMode != .builtInMicBtA2dp {
options.insert(.defaultToSpeaker)
}
// 2. Set category + mode, chosen per scenario:
// - Stereo capture: .default .voiceChat (the AEC/VPIO path) forces MONO, so stereo
// is only possible in a non-VPIO mode. .default supports multi-capsule stereo AND
@@ -514,6 +537,7 @@ final class IOSAudioRouter: ObservableObject {
selectedInputPortId = UserDefaults.standard.string(forKey: kInputPortId)
selectedDataSourceId = UserDefaults.standard.string(forKey: kDataSourceId)
selectedPolarPattern = UserDefaults.standard.string(forKey: kPolarPattern)
forceSpeaker = UserDefaults.standard.bool(forKey: kForceSpeaker)
}
/// Persist current selections to UserDefaults.
func savePreferences() {
@@ -523,6 +547,7 @@ final class IOSAudioRouter: ObservableObject {
UserDefaults.standard.set(selectedInputPortId, forKey: kInputPortId)
UserDefaults.standard.set(selectedDataSourceId, forKey: kDataSourceId)
UserDefaults.standard.set(selectedPolarPattern, forKey: kPolarPattern)
UserDefaults.standard.set(forceSpeaker, forKey: kForceSpeaker)
}
// MARK: - Selection setters (called from SettingsView pickers)
@@ -556,6 +581,17 @@ final class IOSAudioRouter: ObservableObject {
savePreferences()
applyConfiguration()
refreshRoutes()
// VPIO class or route may have changed restart the voice path if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
}
func setForceSpeaker(_ on: Bool) {
forceSpeaker = on
savePreferences()
applyConfiguration()
refreshRoutes()
// Route changed under a possibly-running VPIO engine reconcile if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
}
func selectMicMode(_ mode: MicMode) {
@@ -563,6 +599,8 @@ final class IOSAudioRouter: ObservableObject {
savePreferences()
applyConfiguration()
updateWarnings()
// StandardRaw flips the VPIO class reconcile if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
}
func selectCaptureChannels(_ channels: CaptureChannels) {
@@ -579,6 +617,8 @@ final class IOSAudioRouter: ObservableObject {
// route), then capture avoiding the race where stereo capture activation drops
// A2DP before the playback device has a chance to claim the route.
_ = AudioSessionManager.shared.client?.audioRestart()
// Monostereo flips the VPIO class (stereo can't use VPIO) reconcile if mic is active.
AudioSessionManager.shared.reconcileVoicePath?()
}
// MARK: - Presets
@@ -593,6 +633,10 @@ final class IOSAudioRouter: ObservableObject {
micMode = preset.micMode
captureChannels = preset.captureChannels
// Voice Chat is a phone-call experience default to the loud speaker so output doesn't
// land on the quiet earpiece (receiver). Still yields to connected BT/wired output.
if preset == .voiceChat { forceSpeaker = true }
if preset.usesBuiltInMic {
// Find the built-in mic port from available inputs and select it.
let session = AVAudioSession.sharedInstance()
@@ -624,6 +668,9 @@ final class IOSAudioRouter: ObservableObject {
// count is stored. Playback opens first (commits A2DP route), then capture.
_ = AudioSessionManager.shared.client?.audioRestart()
refreshRoutes()
// The preset may have flipped the VPIO class (and/or the route) restart the voice path
// if the mic is active so AEC/NS engage (or disengage) to match the new preset.
AudioSessionManager.shared.reconcileVoicePath?()
logger.info("applyPreset — \(preset.rawValue)")
}
@@ -642,9 +689,35 @@ final class IOSAudioRouter: ObservableObject {
showsA2dpNoAecWarning = (bluetoothMode == .builtInMicBtA2dp)
}
/// Whether the current configuration wants Bluetooth A2DP output. Used after session
/// activation to clear any lingering speaker override that would pin output to the speaker.
var wantsA2dpOutput: Bool { bluetoothMode == .builtInMicBtA2dp }
/// Route fallback for the A2DP-output presets (Stereo Mic / Studio / BT Headphones + Mono
/// Mic, all `.builtInMicBtA2dp`). These presets deliberately omit `.defaultToSpeaker` (it
/// breaks A2DP routing) and skip the `forceSpeaker` override, so when NO external output
/// (Bluetooth A2DP / wired / AirPlay) is connected `.playAndRecord` pins output to the quiet
/// built-in receiver (earpiece). This routes to the loud built-in speaker instead via a
/// post-activation `overrideOutputAudioPort(.speaker)` the documented "A2DP if connected,
/// else speaker" behavior. When an external output IS present we clear the override so A2DP /
/// headphones / AirPlay are honored. No-op outside `.builtInMicBtA2dp` mode (other modes pick
/// their route via category options). Must be called AFTER the session is active.
func applyA2dpSpeakerFallback() {
guard bluetoothMode == .builtInMicBtA2dp else { return }
let session = AVAudioSession.sharedInstance()
// Treat the built-in receiver and speaker as "internal"; anything else (A2DP, headphones,
// USB, AirPlay) is an external output we should defer to.
let hasExternalOutput = session.currentRoute.outputs.contains {
$0.portType != .builtInReceiver && $0.portType != .builtInSpeaker
}
do {
if hasExternalOutput {
try session.overrideOutputAudioPort(.none)
logger.info("A2DP mode — external output present, clearing speaker override")
} else {
try session.overrideOutputAudioPort(.speaker)
logger.info("A2DP mode — no external output, routing to built-in speaker")
}
} catch {
logger.error("A2DP speaker fallback failed: \(error.localizedDescription)")
}
}
/// The selected input port object, if any.
var selectedPort: IOSAudioInputPort? {

View File

@@ -0,0 +1,269 @@
import AVFoundation
import Darwin
import os
import VoiceCatCore
private let logger = Logger(subsystem: "cat.voice.VoiceCatiOS", category: "IOSVoiceProcessingEngine")
/// In-process single-producer/single-consumer int16 PCM ring for the VPIO playback path.
///
/// producer = the core's mixer-timer thread (the `vc_set_mixed_output_sink` callback)
/// consumer = the `AVAudioSourceNode` render thread
///
/// Heap-backed (not shared memory like `BroadcastAudioRing`), but the same discipline: aligned
/// 64-bit monotonic indices with `OSMemoryBarrier` for acquire/release ordering. Both the C
/// callback and the render block are real-time they only do index math + a memcpy here, never
/// lock or allocate.
final class PCMRing {
private let data: UnsafeMutablePointer<Int16>
private let capacity: Int
private var writeIdx: UInt64 = 0
private var readIdx: UInt64 = 0
init(capacitySamples: Int) {
capacity = capacitySamples
data = UnsafeMutablePointer<Int16>.allocate(capacity: capacitySamples)
data.initialize(repeating: 0, count: capacitySamples)
}
deinit { data.deallocate() }
/// Producer: append `count` interleaved int16 samples. Drops the chunk if it doesn't fit
/// (better to skip than tear). Single producer only (the core mixer-timer thread).
func write(_ src: UnsafePointer<Int16>, count: Int) {
guard count > 0, count <= capacity else { return }
let w = writeIdx
OSMemoryBarrier()
let r = readIdx
if capacity - Int(w &- r) < count { return } // full: drop
var idx = Int(w % UInt64(capacity))
var off = 0
var rem = count
while rem > 0 {
let chunk = min(rem, capacity - idx)
(data + idx).update(from: src + off, count: chunk)
idx = (idx + chunk) % capacity
off += chunk
rem -= chunk
}
OSMemoryBarrier()
writeIdx = w &+ UInt64(count)
}
/// Consumer: read up to `count` interleaved int16 samples into `dst`; returns the number
/// read (the rest is the caller's to silence-fill). Single consumer only (render thread).
func read(into dst: UnsafeMutablePointer<Int16>, count: Int) -> Int {
let r = readIdx
OSMemoryBarrier()
let w = writeIdx
let available = Int(w &- r)
if available <= 0 { return 0 }
let n = min(available, count)
var idx = Int(r % UInt64(capacity))
var off = 0
var rem = n
while rem > 0 {
let chunk = min(rem, capacity - idx)
(dst + off).update(from: data + idx, count: chunk)
idx = (idx + chunk) % capacity
off += chunk
rem -= chunk
}
OSMemoryBarrier()
readIdx = r &+ UInt64(n)
return n
}
/// Discard everything buffered call before (re)starting so stale pre-roll isn't played.
func reset() { OSMemoryBarrier(); readIdx = writeIdx }
/// Diagnostics: monotonic total samples written / read since the ring was created. The
/// indices are already cumulative, so these are free. Only read them when both threads are
/// quiesced (e.g. at teardown after the engine + mixer sink are stopped) they are not
/// synchronized for live cross-thread reads. Lets us tell "core never delivered PCM" (Bug 1
/// core path) apart from "PCM arrived but produced no sound" (AVAudioEngine output graph).
var debugTotalWritten: UInt64 { writeIdx }
var debugTotalRead: UInt64 { readIdx }
}
/// Native iOS voice-processing audio path (docs/voice.md §8 "iOS voice processing").
///
/// Real iOS echo cancellation / noise suppression / AGC come ONLY from Apple's Voice-Processing
/// I/O unit (VPIO), which `AVAudioEngine.setVoiceProcessingEnabled(true)` enables. For VPIO to
/// cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
/// played-back signal from the mic), so on the AEC presets this engine drives both directions and
/// the core runs in external mode (no hardware devices):
/// - **Mic core:** a tap on the VPIO input node 48 kHz int16 `client.feedPcm(micStreamId)`.
/// - **core speaker:** the core's mixed-output sink fills `ring`; an `AVAudioSourceNode` pulls
/// from it and renders through the VPIO output, giving AEC its reference signal.
///
/// Lifecycle is driven by `SessionState` join/leave. The Stereo Mic / Studio / A2DP presets keep
/// the core's miniaudio path instead (they want raw / stereo / no-AEC routing VPIO can't provide).
@MainActor
final class IOSVoiceProcessingEngine {
static let shared = IOSVoiceProcessingEngine()
private(set) var isRunning = false
private let engine = AVAudioEngine()
private var sourceNode: AVAudioSourceNode?
private weak var client: VoiceCatClient?
private var micStreamId: UInt32 = 0
// 48 kHz stereo Float32 (deinterleaved) the format the source node renders and the engine
// processes in. The core delivers 48 kHz stereo int16 via the mixed-output sink.
private let outFormat = AVAudioFormat(
commonFormat: .pcmFormatFloat32, sampleRate: 48000, channels: 2, interleaved: false)!
// Playback ring (mixed remote audio): ~0.5 s of 48 kHz stereo int16. Filled by the core's
// mixer-timer thread, drained by the source-node render thread.
private let ring = PCMRing(capacitySamples: 48000 * 2 / 2)
// Render-thread scratch for deinterleaving pre-allocated so the render block never allocates.
private let renderScratchFrames = 8192
private let renderScratch: UnsafeMutablePointer<Int16>
// Mic-feed converter (input-node format 48 kHz int16) and its target buffer. Owned here so
// the (background) tap block reuses them instead of allocating per callback.
private var micConverter: AVAudioConverter?
private var micTargetFormat: AVAudioFormat?
private init() {
renderScratch = UnsafeMutablePointer<Int16>.allocate(capacity: renderScratchFrames * 2)
renderScratch.initialize(repeating: 0, count: renderScratchFrames * 2)
}
/// Start the VPIO engine for an active mic stream. The caller must have already enabled
/// external playback on the core (`client.setExternalPlayback(true)` + `audioRestart()`) and
/// started the MIC stream with `externalFeed: true`.
func start(client: VoiceCatClient, micStreamId: UInt32, captureChannels: UInt32) {
guard !isRunning else { return }
self.client = client
self.micStreamId = micStreamId
ring.reset()
// Enable the voice-processing I/O unit (AEC/NS/AGC) on the shared input+output unit.
do {
try engine.inputNode.setVoiceProcessingEnabled(true)
} catch {
logger.error("setVoiceProcessingEnabled failed: \(error.localizedDescription) — AEC unavailable")
}
// Playback: source node pulls mixed PCM from the ring through the VPIO output.
let ring = self.ring
let scratch = self.renderScratch
let scratchFrames = self.renderScratchFrames
let src = AVAudioSourceNode(format: outFormat) { _, _, frameCount, ablPtr in
let frames = Int(frameCount)
let abl = UnsafeMutableAudioBufferListPointer(ablPtr)
let n = min(frames, scratchFrames)
let got = ring.read(into: scratch, count: n * 2) / 2 // interleaved stereo frames
// Deinterleave int16 Float32 per channel; silence-fill any underrun tail.
let scale: Float = 1.0 / 32768.0
for ch in 0..<abl.count {
guard let base = abl[ch].mData?.assumingMemoryBound(to: Float.self) else { continue }
for i in 0..<frames {
if i < got {
let s = scratch[i * 2 + min(ch, 1)]
base[i] = Float(s) * scale
} else {
base[i] = 0
}
}
}
return noErr
}
sourceNode = src
engine.attach(src)
engine.connect(src, to: engine.mainMixerNode, format: outFormat)
// Mic: tap the VPIO input node, convert to 48 kHz int16, feed the core.
let inFormat = engine.inputNode.outputFormat(forBus: 0)
let targetCh = max(1, min(2, captureChannels))
let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 48000,
channels: AVAudioChannelCount(targetCh), interleaved: true)
micTargetFormat = target
micConverter = (target != nil && inFormat.sampleRate > 0)
? AVAudioConverter(from: inFormat, to: target!) : nil
if micConverter == nil {
logger.error("mic converter unavailable (in=\(inFormat)) — mic will not transmit")
}
let c = client
let sid = micStreamId
let converter = micConverter
let tgt = micTargetFormat
engine.inputNode.installTap(onBus: 0, bufferSize: 960, format: inFormat) { buffer, _ in
guard let converter, let tgt else { return }
// Convert this tap buffer to 48 kHz int16. Output capacity scaled for any upsample.
let ratio = tgt.sampleRate / buffer.format.sampleRate
let outCap = AVAudioFrameCount(Double(buffer.frameLength) * ratio + 16)
guard let outBuf = AVAudioPCMBuffer(pcmFormat: tgt, frameCapacity: outCap) else { return }
var fed = false
let status = converter.convert(to: outBuf, error: nil) { _, outStatus in
if fed { outStatus.pointee = .noDataNow; return nil }
fed = true
outStatus.pointee = .haveData
return buffer
}
guard status != .error, outBuf.frameLength > 0,
let chData = outBuf.int16ChannelData else { return }
let spc = Int(outBuf.frameLength)
// int16 interleaved channelData[0] is the interleaved buffer for interleaved formats.
c.feedPcm(streamId: sid, pcm: chData[0], samplesPerChannel: spc, channels: targetCh)
}
// Wire the core's mixed-output sink into the ring (C function pointer, no captures).
let ringPtr = Unmanaged.passUnretained(self.ring).toOpaque()
client.setMixedOutputSink({ user, pcm, spc, ch, _ in
guard let user, let pcm else { return }
let ring = Unmanaged<PCMRing>.fromOpaque(user).takeUnretainedValue()
ring.write(pcm, count: spc * Int(ch))
}, user: ringPtr)
engine.prepare()
do {
try engine.start()
isRunning = true
// Diagnostics: capture the negotiated graph formats and the live output route so a
// silent-playback report can be triaged (format/rate mismatch vs. routing vs. the
// core not delivering PCM see the ring stats logged in teardown()).
let outFmt = engine.outputNode.outputFormat(forBus: 0)
let mixFmt = engine.mainMixerNode.outputFormat(forBus: 0)
let route = AVAudioSession.sharedInstance().currentRoute.outputs
.map { "\($0.portName)[\($0.portType.rawValue)]" }.joined(separator: ", ")
logger.info("""
VPIO engine started — inFormat=\(inFormat), captureCh=\(targetCh), \
outputNode=\(outFmt), mainMixer=\(mixFmt), outputRoute=[\(route)]
""")
} catch {
logger.error("VPIO engine start failed: \(error.localizedDescription)")
teardown()
}
}
/// Stop the VPIO engine. The caller is responsible for restoring the core's hardware playback
/// afterwards (`client.setExternalPlayback(false)` + `audioRestart()`).
func stop() {
guard isRunning else { return }
teardown()
logger.info("VPIO engine stopped")
}
private func teardown() {
client?.setMixedOutputSink(nil, user: nil)
engine.inputNode.removeTap(onBus: 0)
if engine.isRunning { engine.stop() }
// Diagnostics (threads now quiesced): how much mixed PCM the core delivered into the ring
// vs. how much the render thread consumed. written==0 the core never delivered (Bug 1
// core/lifecycle path); written>0 with no audible output the AVAudioEngine output graph.
logger.info("VPIO ring stats — written=\(self.ring.debugTotalWritten) read=\(self.ring.debugTotalRead) samples")
try? engine.inputNode.setVoiceProcessingEnabled(false)
if let src = sourceNode {
engine.detach(src)
sourceNode = nil
}
micConverter = nil
micTargetFormat = nil
ring.reset()
isRunning = false
}
}

View File

@@ -6,17 +6,22 @@ struct SavedServer: Codable, Identifiable, Equatable {
var port: UInt16
var authMode: AuthMode
var savedUsername: String
/// Free-form display name used when connecting as a guest. Distinct from the account
/// `savedUsername`. Empty falls back to a default. Optional for backward-compatible decoding.
var nickname: String?
var keychainTag: String
enum AuthMode: String, Codable { case guest, password }
init(id: UUID = UUID(), host: String, port: UInt16,
authMode: AuthMode = .guest, savedUsername: String = "", keychainTag: String = "") {
authMode: AuthMode = .guest, savedUsername: String = "",
nickname: String? = nil, keychainTag: String = "") {
self.id = id
self.host = host
self.port = port
self.authMode = authMode
self.savedUsername = savedUsername
self.nickname = nickname
self.keychainTag = keychainTag.isEmpty ? id.uuidString : keychainTag
}

View File

@@ -73,6 +73,9 @@ final class SessionState {
broadcastPump.onBroadcastStarted = { [weak self] in self?.startScreenShare() }
broadcastPump.onBroadcastFinished = { [weak self] in self?.stopScreenShare() }
broadcastPump.start()
// When IOSAudioRouter changes the audio config, restart the voice path if needed so the
// native VPIO engine (AEC/NS/AGC) engages or disengages to match the new preset/route.
AudioSessionManager.shared.reconcileVoicePath = { [weak self] in self?.reconcileVoicePath() }
}
deinit {
@@ -222,7 +225,24 @@ final class SessionState {
addActivity("AVAudioSession activate failed: \(error)")
return
}
let desc = StreamDescriptor(kind: .mic, deviceId: voiceState.currentDeviceId, label: "Mic")
// VPIO path: on the AEC presets, the native AVAudioEngine does AEC/NS/AGC and the core
// runs in external mode (no hardware mic/playback). The mic stream is started with
// externalFeed so the core skips the hardware capture device; setExternalPlayback makes
// it skip the hardware playback device and deliver the mix to IOSVoiceProcessingEngine.
//
// ORDER MATTERS: set the external-playback flag now, but defer audioRestart() until
// AFTER startStream (below) so the MIC LocalStream which carries external_feed=true
// already exists when ensure_audio_running() derives external_capture. Restarting before
// the stream exists makes the core reopen a hardware capture device that is never dropped
// (the announce-result restart early-returns because the engine is already running); that
// lingering miniaudio capture unit then fights the AVAudioEngine VPIO unit on the same
// .voiceChat session and silences VPIO playback.
let useVPIO = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
client.setExternalPlayback(useVPIO)
let desc = StreamDescriptor(kind: .mic, deviceId: voiceState.currentDeviceId, label: "Mic",
externalFeed: useVPIO)
let (result, streamId) = client.startStream(desc)
if result == .ok {
voiceState.micActive = true
@@ -241,17 +261,41 @@ final class SessionState {
if channels != 1 {
client.setCaptureChannels(streamId: streamId, channels: channels)
}
if useVPIO {
// The external-feed MIC stream now exists, so restart the core into full
// external mode (no hardware capture/playback, mixer-timer only) mic and
// speaker are owned entirely by the VPIO engine, which we start right after.
client.audioRestart()
IOSVoiceProcessingEngine.shared.start(
client: client, micStreamId: streamId, captureChannels: channels)
}
} else {
addActivity("Failed to start mic: \(result.description)")
if useVPIO { // revert external-playback mode so remote audio still plays
client.setExternalPlayback(false)
client.audioRestart()
}
}
}
func stopMicStream() {
// Tear down the VPIO engine first (removes the mic tap + unregisters the mixed sink),
// then stop the mic stream, then restore the core's hardware playback for any remaining
// remote audio. Order matters: the mic stream must be gone before audioRestart so the
// core opens a normal playback device (and no capture device there's no mic stream).
let wasVPIO = IOSVoiceProcessingEngine.shared.isRunning
if wasVPIO {
IOSVoiceProcessingEngine.shared.stop()
}
if voiceState.localStreamId != 0 {
client.stopStream(voiceState.localStreamId)
voiceState.localStreamId = 0
AudioSessionManager.shared.activeMicStreamId = nil
}
if wasVPIO {
client.setExternalPlayback(false)
client.audioRestart() // reopen hardware playback (no mic stream no hw capture)
}
voiceState.micActive = false
voiceState.level = 0
// Do NOT deactivate the AVAudioSession here the user may still want to hear
@@ -259,6 +303,19 @@ final class SessionState {
// disconnecting from the server (see AppState.disconnect / .disconnected event).
}
/// Restart the voice path when the audio config changes mid-call (driven by IOSAudioRouter).
/// If VPIO is involved on either the current or desired side, restart the mic so the native
/// voice-processing engine engages/disengages and re-binds to the new route. Pure miniaudio
/// config tweaks need no restart the core's own audioRestart (already issued) handles them.
private func reconcileVoicePath() {
guard voiceState.micActive else { return }
let want = IOSAudioRouter.shared.currentConfigUsesVoiceProcessing
let have = IOSVoiceProcessingEngine.shared.isRunning
guard want || have else { return }
stopMicStream()
doStartMicStream()
}
// MARK: - Screen audio share
/// Called when the broadcast extension becomes active. Announces the SCREEN_AUDIO stream;
@@ -342,13 +399,14 @@ final class SessionState {
client.setServerMute(userId, muted: muted, deafened: deafened)
}
func createChannel(name: String, topic: String) {
let info = ChannelEdit(id: 0, parentId: 0, name: name, topic: topic,
passwordProtected: false, password: nil,
maxUsers: 0, sortOrder: 0, audio: AudioConfig())
func createChannel(_ info: ChannelEdit) {
client.createChannel(info)
}
func editChannel(_ info: ChannelEdit) {
client.editChannel(info)
}
func deleteChannel(_ channelId: UInt32) {
client.deleteChannel(channelId)
}

View File

@@ -9,6 +9,7 @@ struct AddServerView: View {
@State private var host = ""
@State private var port = "7878"
@State private var authMode = SavedServer.AuthMode.guest
@State private var nickname = ""
@State private var username = ""
@State private var password = ""
@State private var savePassword = false
@@ -35,6 +36,13 @@ struct AddServerView: View {
.pickerStyle(.segmented)
.accessibilityLabel("Authentication mode")
if authMode == .guest {
TextField("Nickname (optional)", text: $nickname)
.autocorrectionDisabled()
.textInputAutocapitalization(.never)
.accessibilityLabel("Guest nickname, optional display name")
}
if authMode == .password {
TextField("Username", text: $username)
.textContentType(.username)
@@ -67,6 +75,7 @@ struct AddServerView: View {
port = "\(s.port)"
authMode = s.authMode
username = s.savedUsername
nickname = s.nickname ?? ""
}
}
}
@@ -76,17 +85,21 @@ struct AddServerView: View {
guard !trimmedHost.isEmpty, let portNum = UInt16(port) else { return }
let pw = (savePassword && authMode == .password && !password.isEmpty) ? password : nil
let trimmedNick = nickname.trimmingCharacters(in: .whitespaces)
let nick: String? = (authMode == .guest && !trimmedNick.isEmpty) ? trimmedNick : nil
if var s = editing {
s.host = trimmedHost
s.port = portNum
s.authMode = authMode
s.savedUsername = authMode == .password ? username : ""
s.nickname = nick
appState.updateServer(s, password: pw)
} else {
let s = SavedServer(host: trimmedHost, port: portNum,
authMode: authMode,
savedUsername: authMode == .password ? username : "")
savedUsername: authMode == .password ? username : "",
nickname: nick)
appState.addServer(s, password: pw)
}
dismiss()

View File

@@ -7,6 +7,7 @@ import VoiceCatCore
struct ChannelBrowserView: View {
@Bindable var session: SessionState
@State private var showCreateChannel = false
@State private var editChannel: Channel?
var body: some View {
NavigationStack {
@@ -26,6 +27,16 @@ struct ChannelBrowserView: View {
}
}
}
.swipeActions(edge: .leading) {
if session.permissions.isAdmin {
Button {
editChannel = ch
} label: {
Label("Edit", systemImage: "pencil")
}
.tint(.blue)
}
}
}
}
.navigationTitle("Channels")
@@ -44,6 +55,9 @@ struct ChannelBrowserView: View {
.sheet(isPresented: $showCreateChannel) {
ChannelEditView(channelId: nil, session: session)
}
.sheet(item: $editChannel) { ch in
ChannelEditView(channelId: ch.id, session: session)
}
}
}

View File

@@ -1,12 +1,34 @@
import SwiftUI
import VoiceCatCore
struct ChannelEditView: View {
let channelId: UInt32?
@Bindable var session: SessionState
@Environment(\.dismiss) private var dismiss
// General
@State private var name = ""
@State private var topic = ""
@State private var parentId: UInt32 = 0
@State private var passwordProtected = false
@State private var password = ""
@State private var maxUsers = "0"
@State private var sortOrder = "0"
// Audio (Opus). Note: the channel list does not carry the current audio config, so when
// editing an existing channel these start from the codec defaults (same as macOS/Windows).
@State private var stereo = false
@State private var bitrate = "64000"
@State private var sampleRate = "48000"
@State private var frameMs: UInt32 = 20
@State private var application: UInt32 = 0
@State private var packetLoss = "5"
@State private var complexity = 10
@State private var fec = true
@State private var dtx = false
@State private var dred = false
private var isEditing: Bool { channelId != nil }
var body: some View {
NavigationStack {
@@ -17,9 +39,57 @@ struct ChannelEditView: View {
.accessibilityLabel("Channel name")
TextField("Topic (optional)", text: $topic)
.accessibilityLabel("Channel topic, optional")
Picker("Parent", selection: $parentId) {
Text("(root)").tag(UInt32(0))
ForEach(parentOptions) { ch in
Text(ch.name).tag(ch.id)
}
}
.navigationTitle(channelId == nil ? "New Channel" : "Edit Channel")
.accessibilityLabel("Parent channel")
Toggle("Password protected", isOn: $passwordProtected)
if passwordProtected {
SecureField("Password (blank keeps existing)", text: $password)
.accessibilityLabel("Channel password")
}
TextField("Max users (0 = unlimited)", text: $maxUsers)
.keyboardType(.numberPad)
.accessibilityLabel("Maximum users, zero means unlimited")
TextField("Sort order", text: $sortOrder)
.keyboardType(.numberPad)
.accessibilityLabel("Sort order")
}
Section("Audio (Opus)") {
Toggle("Stereo", isOn: $stereo)
TextField("Bitrate (bps)", text: $bitrate)
.keyboardType(.numberPad)
.accessibilityLabel("Bitrate in bits per second")
TextField("Sample rate (Hz)", text: $sampleRate)
.keyboardType(.numberPad)
.accessibilityLabel("Sample rate in Hz")
Picker("Frame", selection: $frameMs) {
ForEach([UInt32(10), 20, 40, 60], id: \.self) { ms in
Text("\(ms) ms").tag(ms)
}
}
.accessibilityLabel("Opus frame duration")
Picker("Application", selection: $application) {
Text("VoIP").tag(UInt32(0))
Text("Audio").tag(UInt32(1))
Text("Low delay").tag(UInt32(2))
}
.accessibilityLabel("Opus application profile")
TextField("Expected packet loss %", text: $packetLoss)
.keyboardType(.numberPad)
.accessibilityLabel("Expected packet loss percent, 0 to 100")
Stepper("Complexity: \(complexity)", value: $complexity, in: 0...10)
.accessibilityLabel("Opus complexity, 0 to 10")
Toggle("FEC (forward error correction)", isOn: $fec)
Toggle("DTX (discontinuous transmission)", isOn: $dtx)
Toggle("DRED (deep redundancy)", isOn: $dred)
}
}
.navigationTitle(isEditing ? "Edit Channel" : "New Channel")
.navigationBarTitleDisplayMode(.inline)
.toolbar {
ToolbarItem(placement: .cancellationAction) {
@@ -27,12 +97,67 @@ struct ChannelEditView: View {
}
ToolbarItem(placement: .confirmationAction) {
Button("Save") {
session.createChannel(name: name, topic: topic)
save()
dismiss()
}
.disabled(name.trimmingCharacters(in: .whitespaces).isEmpty)
}
}
.onAppear(perform: loadIfEditing)
}
}
/// Channels offered as a parent. Excludes the channel being edited so it can't parent itself.
private var parentOptions: [Channel] {
session.channels
.filter { $0.id != channelId }
.sorted { $0.name < $1.name }
}
private func loadIfEditing() {
guard let id = channelId,
let ch = session.channels.first(where: { $0.id == id }) else { return }
name = ch.name
topic = ch.topic
parentId = ch.parentId
passwordProtected = ch.passwordProtected
maxUsers = "\(ch.maxUsers)"
}
private func save() {
let trimmedName = name.trimmingCharacters(in: .whitespaces)
guard !trimmedName.isEmpty else { return }
let audio = AudioConfig(
stereo: stereo,
sampleRate: UInt32(sampleRate) ?? 48000,
bitrateBps: UInt32(bitrate) ?? 64000,
frameMs: frameMs,
application: application,
fec: fec,
expectedPacketLoss: min(UInt32(packetLoss) ?? 5, 100),
dtx: dtx,
complexity: UInt32(complexity),
dred: dred
)
let pw: String? = passwordProtected ? (password.isEmpty ? nil : password) : nil
let info = ChannelEdit(
id: channelId ?? 0,
parentId: parentId,
name: trimmedName,
topic: topic,
passwordProtected: passwordProtected,
password: pw,
maxUsers: UInt32(maxUsers) ?? 0,
sortOrder: UInt32(sortOrder) ?? 0,
audio: audio
)
if isEditing {
session.editChannel(info)
} else {
session.createChannel(info)
}
}
}

View File

@@ -12,6 +12,7 @@ struct ChannelNode: Identifiable {
struct ChannelTreeView: View {
@Bindable var session: SessionState
@State private var showCreateChannel = false
@State private var editChannel: Channel?
@State private var channelPassword = ""
@State private var passwordChannelId: UInt32?
@@ -34,6 +35,16 @@ struct ChannelTreeView: View {
}
}
}
.swipeActions(edge: .leading) {
if session.permissions.isAdmin {
Button {
editChannel = node.channel
} label: {
Label("Edit", systemImage: "pencil")
}
.tint(.blue)
}
}
}
.listStyle(.sidebar)
.toolbar {
@@ -59,6 +70,9 @@ struct ChannelTreeView: View {
.sheet(isPresented: $showCreateChannel) {
ChannelEditView(channelId: nil, session: session)
}
.sheet(item: $editChannel) { ch in
ChannelEditView(channelId: ch.id, session: session)
}
.alert("Channel Password", isPresented: Binding(
get: { passwordChannelId != nil },
set: { if !$0 { passwordChannelId = nil; channelPassword = "" } }

View File

@@ -23,6 +23,30 @@ struct SettingsView: View {
}
.accessibilityLabel("Audio preset")
Toggle("Speaker output", isOn: Binding(
get: { router.forceSpeaker },
set: { router.setForceSpeaker($0) }
))
.accessibilityLabel("Speaker output")
.accessibilityHint("Routes audio to the speaker instead of the earpiece when no headphones are connected.")
// Surface the voice-processing state. On the AEC presets the native iOS
// Voice-Processing unit (VPIO) does echo cancellation, noise suppression and
// automatic gain control; the other presets (stereo/studio/A2DP) can't use it.
if router.currentConfigUsesVoiceProcessing {
Label("Echo cancellation & noise suppression on (iOS voice processing)",
systemImage: "waveform.badge.mic")
.font(.caption)
.foregroundStyle(.secondary)
.accessibilityLabel("Echo cancellation and noise suppression are on")
} else {
Label("No echo cancellation in this preset (stereo / studio / A2DP)",
systemImage: "waveform.slash")
.font(.caption)
.foregroundStyle(.secondary)
.accessibilityLabel("Echo cancellation is off in this preset")
}
if !router.hasBluetoothDevice && !router.hasWiredHeadset {
Text("Connect Bluetooth headphones or a wired headset for more presets.")
.font(.caption)

View File

@@ -4,6 +4,7 @@ import ReplayKit
struct VoiceControlsView: View {
@Bindable var session: SessionState
@StateObject private var router = IOSAudioRouter.shared
var body: some View {
HStack(spacing: 20) {
@@ -37,6 +38,19 @@ struct VoiceControlsView: View {
Spacer()
// Speaker output toggle force the built-in speaker instead of the earpiece when
// no headphones/BT are connected. Mirrors the persisted Settings Audio toggle.
Button {
router.setForceSpeaker(!router.forceSpeaker)
} label: {
Image(systemName: router.forceSpeaker ? "speaker.wave.2.fill" : "speaker.fill")
.font(.title3)
.foregroundStyle(router.forceSpeaker ? Color.accentColor : .primary)
}
.accessibilityLabel(router.forceSpeaker
? "Speaker on — turn off to use the earpiece"
: "Speaker off — turn on for speakerphone")
// Self mute (disabled when not in voice)
Button {
session.setMute(!session.voiceState.selfMuted, deafened: session.voiceState.selfDeafened)

View File

@@ -31,6 +31,7 @@
AAAA00000000000000000048 /* UserPickerSheet.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA00000000000000000047 /* UserPickerSheet.swift */; };
AAAA0000000000000000004A /* SettingsWindowController.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA00000000000000000049 /* SettingsWindowController.swift */; };
AAAA0000000000000000004C /* ScreenAudioCapture.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA0000000000000000004B /* ScreenAudioCapture.swift */; };
AAAA0000000000000000004E /* ScreenSharePickerSheet.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAAA0000000000000000004F /* ScreenSharePickerSheet.swift */; };
/* End PBXBuildFile section */
/* Begin PBXFileReference section */
@@ -57,6 +58,7 @@
AAAA00000000000000000044 /* PttKeyCaptureSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PttKeyCaptureSheet.swift; sourceTree = "<group>"; };
AAAA00000000000000000045 /* PrivateMessageWindowController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PrivateMessageWindowController.swift; sourceTree = "<group>"; };
AAAA00000000000000000047 /* UserPickerSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = UserPickerSheet.swift; sourceTree = "<group>"; };
AAAA0000000000000000004F /* ScreenSharePickerSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenSharePickerSheet.swift; sourceTree = "<group>"; };
AAAA00000000000000000049 /* SettingsWindowController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsWindowController.swift; sourceTree = "<group>"; };
AAAA00000000000000000025 /* Security.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = Security.framework; path = System/Library/Frameworks/Security.framework; sourceTree = SDKROOT; };
/* End PBXFileReference section */
@@ -141,6 +143,7 @@
AAAA00000000000000000024 /* PermissionsSheet.swift */,
AAAA00000000000000000044 /* PttKeyCaptureSheet.swift */,
AAAA00000000000000000047 /* UserPickerSheet.swift */,
AAAA0000000000000000004F /* ScreenSharePickerSheet.swift */,
);
path = Sheets;
sourceTree = "<group>";
@@ -241,6 +244,7 @@
AAAA00000000000000000043 /* PttKeyCaptureSheet.swift in Sources */,
AAAA00000000000000000046 /* PrivateMessageWindowController.swift in Sources */,
AAAA00000000000000000048 /* UserPickerSheet.swift in Sources */,
AAAA0000000000000000004E /* ScreenSharePickerSheet.swift in Sources */,
AAAA0000000000000000004A /* SettingsWindowController.swift in Sources */,
AAAA0000000000000000004C /* ScreenAudioCapture.swift in Sources */,
);

View File

@@ -1,6 +1,25 @@
import AVFoundation
import ScreenCaptureKit
// Which apps' audio the SCREEN_AUDIO stream captures. ScreenCaptureKit filters audio at the
// *application* level (not per-window), so the selection is expressed as bundle IDs. The
// picker UI (ScreenSharePickerSheet) produces a `ScreenAudioSelection`; `start()` turns it
// into the matching `SCContentFilter`.
enum ScreenAudioScope: Equatable {
case entireDesktop // whole display the original behaviour
case onlyApps([String]) // capture only these bundle IDs
case allExcept([String]) // capture everything except these bundle IDs
}
struct ScreenAudioSelection: Equatable {
var scope: ScreenAudioScope = .entireDesktop
/// Drop the macOS screen-reader (VoiceOver) speech from the shared mix. Meaningful for
/// `.entireDesktop`/`.allExcept`; for `.onlyApps` the screen reader is already excluded.
var excludeScreenReader: Bool = false
static let `default` = ScreenAudioSelection()
}
// ScreenAudioCapture macOS system/desktop audio capture for the SCREEN_AUDIO stream.
//
// The macOS analog of the Windows WASAPI loopback path (docs/voice.md §9). ScreenCaptureKit
@@ -26,16 +45,27 @@ final class ScreenAudioCapture: NSObject, SCStreamOutput, SCStreamDelegate {
private let onPcm: PcmHandler
private let channels: Int // 1 (mono) or 2 (stereo interleaved), matches the stream's mode
private let selection: ScreenAudioSelection
private let sampleQueue = DispatchQueue(label: "cat.voice.screenaudio.samples")
private var stream: SCStream?
/// Bundle IDs whose audio carries the macOS screen-reader speech. VoiceOver itself plus the
/// speech-synthesis daemon that actually renders the spoken audio the speech is usually
/// emitted by the daemon, not the VoiceOver app, so we exclude whichever are running.
static let screenReaderBundleIDs: Set<String> = [
"com.apple.VoiceOver",
"com.apple.VoiceOver4",
"com.apple.speech.speechsynthesisd",
]
/// Interleaved int16 carry-over between callbacks (ScreenCaptureKit buffers don't align to
/// 20 ms), drained in whole `frameSamplesPerChannel * channels` chunks. Only touched on
/// `sampleQueue`.
private var pending: [Int16] = []
init(channels: UInt32, onPcm: @escaping PcmHandler) {
init(channels: UInt32, selection: ScreenAudioSelection, onPcm: @escaping PcmHandler) {
self.channels = max(1, min(2, Int(channels)))
self.selection = selection
self.onPcm = onPcm
super.init()
}
@@ -46,7 +76,8 @@ final class ScreenAudioCapture: NSObject, SCStreamOutput, SCStreamDelegate {
let content = try await SCShareableContent.current
guard let display = content.displays.first else { throw CaptureError.noDisplay }
let filter = SCContentFilter(display: display, excludingWindows: [])
let filter = Self.makeFilter(selection: selection, display: display,
apps: content.applications)
let config = SCStreamConfiguration()
config.capturesAudio = true
@@ -65,6 +96,39 @@ final class ScreenAudioCapture: NSObject, SCStreamOutput, SCStreamDelegate {
self.stream = stream
}
/// Turn a `ScreenAudioSelection` into an `SCContentFilter` against the running apps.
/// ScreenCaptureKit filters audio per application, so we map bundle IDs SCRunningApplication.
private static func makeFilter(selection: ScreenAudioSelection, display: SCDisplay,
apps: [SCRunningApplication]) -> SCContentFilter {
func appsMatching(_ ids: Set<String>) -> [SCRunningApplication] {
apps.filter { ids.contains($0.bundleIdentifier) }
}
switch selection.scope {
case .onlyApps(let bundleIDs):
// Include-only already excludes everything else (the screen reader included), so the
// excludeScreenReader flag is moot in this mode.
return SCContentFilter(display: display,
including: appsMatching(Set(bundleIDs)),
exceptingWindows: [])
case .allExcept(let bundleIDs):
var ids = Set(bundleIDs)
if selection.excludeScreenReader { ids.formUnion(screenReaderBundleIDs) }
return SCContentFilter(display: display,
excludingApplications: appsMatching(ids),
exceptingWindows: [])
case .entireDesktop:
if selection.excludeScreenReader {
return SCContentFilter(display: display,
excludingApplications: appsMatching(screenReaderBundleIDs),
exceptingWindows: [])
}
return SCContentFilter(display: display, excludingWindows: [])
}
}
/// Stop capture and release the stream. Safe to call multiple times.
func stop() {
guard let stream else { return }

View File

@@ -11,6 +11,9 @@ struct SavedServer: Codable, Identifiable {
var port: UInt16
var authMode: AuthMode
var savedUsername: String?
/// Free-form display name used when connecting as a guest. Distinct from the account
/// `savedUsername`. Empty/nil falls back to the system full name.
var nickname: String?
var keychainTag: String?
var displayString: String {

View File

@@ -13,6 +13,7 @@ final class AddServerSheet: NSViewController {
return f
}()
private let authPicker = NSPopUpButton()
private let nicknameField = NSTextField()
private let usernameField = NSTextField()
private let passwordField = NSSecureTextField()
private let savePwCheckbox = NSButton(checkboxWithTitle: "Save password in Keychain", target: nil, action: nil)
@@ -35,6 +36,7 @@ final class AddServerSheet: NSViewController {
hostField.stringValue = s.host
portField.stringValue = "\(s.port)"
authPicker.selectItem(withTitle: s.authMode == .guest ? "Guest" : "Account")
nicknameField.stringValue = s.nickname ?? ""
usernameField.stringValue = s.savedUsername ?? ""
updateAuthVisibility()
}
@@ -47,6 +49,7 @@ final class AddServerSheet: NSViewController {
let hostLabel = NSTextField(labelWithString: "Host:")
let portLabel = NSTextField(labelWithString: "Port:")
let authLabel = NSTextField(labelWithString: "Auth:")
let nickLabel = NSTextField(labelWithString: "Nickname:")
let userLabel = NSTextField(labelWithString: "Username:")
let pwLabel = NSTextField(labelWithString: "Password:")
@@ -60,6 +63,9 @@ final class AddServerSheet: NSViewController {
authPicker.target = self; authPicker.action = #selector(authChanged)
authPicker.setAccessibilityLabel("Authentication mode")
nicknameField.placeholderString = "leave blank to use your system name"
nicknameField.setAccessibilityLabel("Guest nickname (display name)")
usernameField.placeholderString = "username"
usernameField.setAccessibilityLabel("Username")
@@ -79,6 +85,7 @@ final class AddServerSheet: NSViewController {
[hostLabel, hostField],
[portLabel, portField],
[authLabel, authPicker],
[nickLabel, nicknameField],
[userLabel, usernameField],
[pwLabel, passwordField],
[NSView(), savePwCheckbox],
@@ -110,6 +117,7 @@ final class AddServerSheet: NSViewController {
private func updateAuthVisibility() {
let isAccount = authPicker.titleOfSelectedItem == "Account"
nicknameField.isEnabled = !isAccount
usernameField.isEnabled = isAccount
passwordField.isEnabled = isAccount
savePwCheckbox.isEnabled = isAccount
@@ -130,6 +138,8 @@ final class AddServerSheet: NSViewController {
server.host = host; server.port = port
server.authMode = isGuest ? .guest : .password
server.savedUsername = isGuest ? nil : usernameField.stringValue.trimmingCharacters(in: .whitespaces)
let nick = nicknameField.stringValue.trimmingCharacters(in: .whitespaces)
server.nickname = nick.isEmpty ? nil : nick
if !isGuest && savePwCheckbox.state == .on {
let pw = passwordField.stringValue

View File

@@ -25,7 +25,18 @@ final class ChannelEditSheet: NSViewController {
}()
private let fecCheckbox = NSButton(checkboxWithTitle: "FEC", target: nil, action: nil)
private let dtxCheckbox = NSButton(checkboxWithTitle: "DTX", target: nil, action: nil)
private let dredCheckbox = NSButton(checkboxWithTitle: "DRED", target: nil, action: nil)
private let frameMsPicker = NSPopUpButton()
private let applicationPicker = NSPopUpButton()
private let sampleRateField: NSTextField = {
let f = NSTextField(); f.stringValue = "48000"; return f
}()
private let packetLossField: NSTextField = {
let f = NSTextField(); f.stringValue = "5"; return f
}()
private let complexityField: NSTextField = {
let f = NSTextField(); f.stringValue = "10"; return f
}()
init(channels: [Channel], editing: ChannelEdit?) {
self.channels = channels
@@ -72,14 +83,24 @@ final class ChannelEditSheet: NSViewController {
maxUsersField.setAccessibilityLabel("Max users (0 = unlimited)")
sortOrderField.setAccessibilityLabel("Sort order")
for ms in ["20", "40", "60"] { frameMsPicker.addItem(withTitle: "\(ms) ms") }
for ms in ["10", "20", "40", "60"] { frameMsPicker.addItem(withTitle: "\(ms) ms") }
frameMsPicker.selectItem(withTitle: "20 ms")
frameMsPicker.setAccessibilityLabel("Opus frame duration")
applicationPicker.addItem(withTitle: "VoIP")
applicationPicker.addItem(withTitle: "Audio")
applicationPicker.addItem(withTitle: "Low delay")
applicationPicker.setAccessibilityLabel("Opus application profile")
fecCheckbox.state = .on
stereoCheckbox.setAccessibilityLabel("Stereo audio")
bitrateField.setAccessibilityLabel("Bitrate in bits per second")
sampleRateField.setAccessibilityLabel("Sample rate in Hz")
packetLossField.setAccessibilityLabel("Expected packet loss percent (0 to 100)")
complexityField.setAccessibilityLabel("Opus complexity (0 to 10)")
fecCheckbox.setAccessibilityLabel("Forward error correction")
dtxCheckbox.setAccessibilityLabel("Discontinuous transmission")
dredCheckbox.setAccessibilityLabel("Deep redundancy (DRED)")
let generalGrid = NSGridView(views: [
[NSTextField(labelWithString: "Name:"), nameField],
@@ -94,9 +115,13 @@ final class ChannelEditSheet: NSViewController {
let audioGrid = NSGridView(views: [
[NSTextField(labelWithString: "Bitrate:"), bitrateField],
[NSTextField(labelWithString: "Sample rate:"), sampleRateField],
[NSTextField(labelWithString: "Frame:"), frameMsPicker],
[NSTextField(labelWithString: "Application:"), applicationPicker],
[NSTextField(labelWithString: "Packet loss %:"), packetLossField],
[NSTextField(labelWithString: "Complexity:"), complexityField],
[stereoCheckbox, fecCheckbox],
[dtxCheckbox, NSView()],
[dtxCheckbox, dredCheckbox],
])
audioGrid.rowSpacing = 8; audioGrid.columnSpacing = 8
audioGrid.column(at: 0).xPlacement = .trailing
@@ -149,8 +174,13 @@ final class ChannelEditSheet: NSViewController {
sortOrderField.stringValue = "\(e.sortOrder)"
stereoCheckbox.state = e.audio.stereo ? .on : .off
bitrateField.stringValue = "\(e.audio.bitrateBps)"
sampleRateField.stringValue = "\(e.audio.sampleRate)"
packetLossField.stringValue = "\(e.audio.expectedPacketLoss)"
complexityField.stringValue = "\(e.audio.complexity)"
fecCheckbox.state = e.audio.fec ? .on : .off
dtxCheckbox.state = e.audio.dtx ? .on : .off
dredCheckbox.state = e.audio.dred ? .on : .off
applicationPicker.selectItem(at: Int(min(e.audio.application, 2)))
let frameStr = "\(e.audio.frameMs) ms"
if let item = frameMsPicker.item(withTitle: frameStr) { frameMsPicker.select(item) }
}
@@ -167,15 +197,24 @@ final class ChannelEditSheet: NSViewController {
let maxUsers = UInt32(maxUsersField.stringValue) ?? 0
let sortOrder = UInt32(sortOrderField.stringValue) ?? 0
let bitrate = UInt32(bitrateField.stringValue) ?? 64000
let sampleRate = UInt32(sampleRateField.stringValue) ?? 48000
let packetLoss = min(UInt32(packetLossField.stringValue) ?? 5, 100)
let complexity = min(UInt32(complexityField.stringValue) ?? 10, 10)
let frameMsStr = frameMsPicker.titleOfSelectedItem?.replacingOccurrences(of: " ms", with: "") ?? "20"
let frameMs = UInt32(frameMsStr) ?? 20
let application = UInt32(max(applicationPicker.indexOfSelectedItem, 0))
let audio = AudioConfig(
stereo: stereoCheckbox.state == .on,
sampleRate: sampleRate,
bitrateBps: bitrate,
frameMs: frameMs,
application: application,
fec: fecCheckbox.state == .on,
dtx: dtxCheckbox.state == .on
expectedPacketLoss: packetLoss,
dtx: dtxCheckbox.state == .on,
complexity: complexity,
dred: dredCheckbox.state == .on
)
let pwProtected = pwCheckbox.state == .on
let pw: String? = pwProtected ? (pwField.stringValue.isEmpty ? nil : pwField.stringValue) : nil

View File

@@ -0,0 +1,246 @@
import AppKit
import ScreenCaptureKit
// ScreenSharePickerSheet chooses *what* the SCREEN_AUDIO stream captures before sharing
// starts. ScreenCaptureKit filters audio per application (not per window), so the user picks
// a mode (everything / only-these / all-except-these) plus a set of apps, and a dedicated
// toggle to drop their own screen-reader (VoiceOver) speech from the mix.
//
// Mirrors the modal-sheet pattern used by the rest of the macOS client (UserPickerSheet,
// MoveUserSheet, ): an NSViewController presented via MainWindowController.presentSheet, with
// an `onComplete` callback that returns the chosen `ScreenAudioSelection` (or nil on cancel).
//
// The app list comes from `SCShareableContent.current`, fetched asynchronously that first
// access is also what surfaces the Screen Recording (TCC) prompt, which is why the picker is
// the natural place for it to appear, before any capture begins.
final class ScreenSharePickerSheet: NSViewController, NSTableViewDataSource, NSTableViewDelegate {
/// Called with the chosen selection, or `nil` if the user cancelled.
var onComplete: ((ScreenAudioSelection?) -> Void)?
private enum Mode: Int { case everything = 0, only = 1, except = 2 }
private struct AppEntry { let name: String; let bundleID: String; let icon: NSImage? }
private let initialSelection: ScreenAudioSelection
private var mode: Mode
private var excludeScreenReader: Bool
private var checked: Set<String> // bundle IDs ticked in the app table
private var apps: [AppEntry] = []
private let tableView = NSTableView()
private var modeControl: NSSegmentedControl?
private var screenReaderCheckbox: NSButton?
private var statusLabel: NSTextField?
init(selection: ScreenAudioSelection) {
self.initialSelection = selection
switch selection.scope {
case .entireDesktop: mode = .everything; checked = []
case .onlyApps(let ids): mode = .only; checked = Set(ids)
case .allExcept(let ids): mode = .except; checked = Set(ids)
}
self.excludeScreenReader = selection.excludeScreenReader
super.init(nibName: nil, bundle: nil)
}
required init?(coder: NSCoder) { fatalError() }
override func loadView() {
view = NSView(frame: NSRect(x: 0, y: 0, width: 360, height: 420))
}
override func viewDidLoad() {
super.viewDidLoad()
buildUI()
loadApps()
}
// MARK: - UI
private func buildUI() {
let titleLabel = NSTextField(labelWithString: "Choose what to share:")
titleLabel.font = .boldSystemFont(ofSize: 13)
titleLabel.setAccessibilityLabel("Choose what to share")
let modeControl = NSSegmentedControl(
labels: ["Everything", "Only selected", "All except selected"],
trackingMode: .selectOne, target: self, action: #selector(modeChanged))
modeControl.selectedSegment = mode.rawValue
modeControl.segmentDistribution = .fillEqually
modeControl.setAccessibilityLabel("Share mode")
self.modeControl = modeControl
let col = NSTableColumn(identifier: NSUserInterfaceItemIdentifier("app"))
tableView.addTableColumn(col)
tableView.headerView = nil
tableView.dataSource = self
tableView.delegate = self
tableView.rowHeight = 22
tableView.setAccessibilityLabel("Application list")
let scroll = NSScrollView()
scroll.documentView = tableView
scroll.hasVerticalScroller = true
scroll.borderType = .bezelBorder
scroll.translatesAutoresizingMaskIntoConstraints = false
scroll.setContentHuggingPriority(.defaultLow, for: .vertical)
let statusLabel = NSTextField(labelWithString: "Loading apps…")
statusLabel.textColor = .secondaryLabelColor
statusLabel.font = .systemFont(ofSize: 11)
self.statusLabel = statusLabel
let screenReaderCheckbox = NSButton(checkboxWithTitle: "Exclude screen reader (VoiceOver) audio",
target: self, action: #selector(screenReaderToggled))
screenReaderCheckbox.state = excludeScreenReader ? .on : .off
screenReaderCheckbox.toolTip = "Keep your VoiceOver speech out of the shared audio."
self.screenReaderCheckbox = screenReaderCheckbox
let cancelButton = NSButton(title: "Cancel", target: self, action: #selector(cancelClicked))
cancelButton.bezelStyle = .rounded
cancelButton.keyEquivalent = "\u{1b}" // Esc
let shareButton = NSButton(title: "Share", target: self, action: #selector(shareClicked))
shareButton.bezelStyle = .rounded
shareButton.keyEquivalent = "\r"
let buttonRow = NSStackView(views: [NSView(), cancelButton, shareButton])
buttonRow.orientation = .horizontal
buttonRow.spacing = 8
let stack = NSStackView(views: [titleLabel, modeControl, scroll, statusLabel,
screenReaderCheckbox, buttonRow])
stack.orientation = .vertical
stack.spacing = 8
stack.edgeInsets = NSEdgeInsets(top: 16, left: 16, bottom: 16, right: 16)
stack.translatesAutoresizingMaskIntoConstraints = false
view.addSubview(stack)
NSLayoutConstraint.activate([
stack.topAnchor.constraint(equalTo: view.topAnchor),
stack.leadingAnchor.constraint(equalTo: view.leadingAnchor),
stack.trailingAnchor.constraint(equalTo: view.trailingAnchor),
stack.bottomAnchor.constraint(equalTo: view.bottomAnchor),
])
updateEnabledStates()
}
/// Reflect the current mode: the app table only matters for only/except; the screen-reader
/// toggle is moot for `.only` (include-only already excludes the screen reader).
private func updateEnabledStates() {
let listActive = (mode != .everything)
tableView.isEnabled = listActive
tableView.alphaValue = listActive ? 1.0 : 0.45
screenReaderCheckbox?.isEnabled = (mode != .only)
}
private func loadApps() {
Task { @MainActor in
do {
let content = try await SCShareableContent.current
var seen = Set<String>()
var entries: [AppEntry] = []
for app in content.applications {
let bid = app.bundleIdentifier
guard !bid.isEmpty, !seen.contains(bid) else { continue }
// Hide our own app (its audio is already excluded) and the screen reader
// (handled by its own toggle).
if bid == Bundle.main.bundleIdentifier { continue }
if ScreenAudioCapture.screenReaderBundleIDs.contains(bid) { continue }
seen.insert(bid)
let name = app.applicationName.isEmpty ? bid : app.applicationName
let icon = NSRunningApplication
.runningApplications(withBundleIdentifier: bid).first?.icon
entries.append(AppEntry(name: name, bundleID: bid, icon: icon))
}
entries.sort { $0.name.localizedCaseInsensitiveCompare($1.name) == .orderedAscending }
self.apps = entries
self.statusLabel?.isHidden = true
self.tableView.reloadData()
} catch {
self.statusLabel?.stringValue = "Screen Recording permission needed to list apps — "
+ "grant it in System Settings ▸ Privacy & Security, then reopen this."
}
}
}
// MARK: - Actions
@objc private func modeChanged() {
mode = Mode(rawValue: modeControl?.selectedSegment ?? 0) ?? .everything
updateEnabledStates()
}
@objc private func screenReaderToggled() {
excludeScreenReader = (screenReaderCheckbox?.state == .on)
}
@objc private func appCheckboxToggled(_ sender: NSButton) {
let row = sender.tag
guard row >= 0, row < apps.count else { return }
let bid = apps[row].bundleID
if sender.state == .on { checked.insert(bid) } else { checked.remove(bid) }
}
@objc private func shareClicked() {
let scope: ScreenAudioScope
switch mode {
case .everything: scope = .entireDesktop
case .only: scope = .onlyApps(Array(checked))
case .except: scope = .allExcept(Array(checked))
}
dismiss(nil)
onComplete?(ScreenAudioSelection(scope: scope, excludeScreenReader: excludeScreenReader))
}
@objc private func cancelClicked() {
dismiss(nil)
onComplete?(nil)
}
// MARK: - NSTableViewDataSource / Delegate
func numberOfRows(in tableView: NSTableView) -> Int { apps.count }
func tableView(_ tableView: NSTableView, viewFor tableColumn: NSTableColumn?, row: Int) -> NSView? {
let app = apps[row]
let cell = AppCheckboxCell()
cell.checkbox.title = app.name
cell.checkbox.state = checked.contains(app.bundleID) ? .on : .off
cell.checkbox.isEnabled = (mode != .everything)
cell.checkbox.tag = row
cell.checkbox.target = self
cell.checkbox.action = #selector(appCheckboxToggled(_:))
cell.iconView.image = app.icon
return cell
}
// Selecting a row shouldn't visually highlight interaction is via the checkbox.
func tableView(_ tableView: NSTableView, shouldSelectRow row: Int) -> Bool { false }
}
// One row: a leading app icon and a checkbox titled with the app name.
private final class AppCheckboxCell: NSTableCellView {
let iconView = NSImageView()
let checkbox = NSButton(checkboxWithTitle: "", target: nil, action: nil)
init() {
super.init(frame: .zero)
iconView.translatesAutoresizingMaskIntoConstraints = false
checkbox.translatesAutoresizingMaskIntoConstraints = false
addSubview(iconView)
addSubview(checkbox)
NSLayoutConstraint.activate([
iconView.leadingAnchor.constraint(equalTo: leadingAnchor, constant: 4),
iconView.centerYAnchor.constraint(equalTo: centerYAnchor),
iconView.widthAnchor.constraint(equalToConstant: 16),
iconView.heightAnchor.constraint(equalToConstant: 16),
checkbox.leadingAnchor.constraint(equalTo: iconView.trailingAnchor, constant: 6),
checkbox.trailingAnchor.constraint(equalTo: trailingAnchor, constant: -4),
checkbox.centerYAnchor.constraint(equalTo: centerYAnchor),
])
}
required init?(coder: NSCoder) { fatalError() }
}

View File

@@ -216,7 +216,7 @@ final class ConnectWindowController: NSWindowController, NSWindowDelegate {
switch server.authMode {
case .guest:
let nick = server.savedUsername?.isEmpty == false ? server.savedUsername! : NSFullUserName()
let nick = server.nickname?.isEmpty == false ? server.nickname! : NSFullUserName()
newClient.authenticateGuest(nick)
case .password:
let username = server.savedUsername ?? ""
@@ -259,7 +259,7 @@ final class ConnectWindowController: NSWindowController, NSWindowDelegate {
case .authResult:
if event.result == .ok {
let nickname = server.authMode == .guest
? (server.savedUsername?.isEmpty == false ? server.savedUsername! : NSFullUserName())
? (server.nickname?.isEmpty == false ? server.nickname! : NSFullUserName())
: (server.savedUsername ?? "")
authSucceeded(client: client!, selfUserId: event.userId, nickname: nickname)
} else {

View File

@@ -31,6 +31,8 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
internal var micStreamId: UInt32 = 0
private var screenStreamId: UInt32 = 0
private var screenCapture: ScreenAudioCapture?
// Last app/exclusion choice from the share picker; reused as the default next time.
private var screenAudioSelection: ScreenAudioSelection = .default
internal var pttKeyCode: UInt16 = 0x60 // F8
private var pttMonitor: Any?
private var serverMuted = false
@@ -720,9 +722,29 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
@objc private func screenAudioClicked() {
if screenStreamId == 0 {
// Announce the stream now; ScreenCaptureKit capture starts once the server's
// StreamAnnounceResult lands (the .streamStarted event), when the effective audio
// config and thus the channel count to capture is known. See startScreenCapture.
// Let the user pick what to share (apps to include/exclude, screen-reader audio)
// before we announce anything. The picker remembers the previous choice.
let sheet = ScreenSharePickerSheet(selection: screenAudioSelection)
sheet.onComplete = { [weak self] selection in
guard let self, let selection else { return } // nil = cancelled
self.screenAudioSelection = selection
self.beginScreenAudioShare()
}
presentSheet(sheet)
} else {
stopScreenCapture()
client.stopStream(screenStreamId)
screenStreamId = 0
setShareScreenButton(active: false)
addActivity("Stopped sharing screen audio")
}
}
/// Announce the SCREEN_AUDIO stream with the chosen selection in hand. ScreenCaptureKit
/// capture starts once the server's StreamAnnounceResult lands (the .streamStarted event),
/// when the effective audio config and thus the channel count is known. See
/// startScreenCapture.
private func beginScreenAudioShare() {
let (result, streamId) = client.startStream(StreamDescriptor(kind: .screenAudio, deviceId: nil, label: "Desktop audio"))
if result == .ok {
screenStreamId = streamId
@@ -731,13 +753,6 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
} else {
addActivity("Failed to start screen audio: \(result)")
}
} else {
stopScreenCapture()
client.stopStream(screenStreamId)
screenStreamId = 0
setShareScreenButton(active: false)
addActivity("Stopped sharing screen audio")
}
}
private func setShareScreenButton(active: Bool) {
@@ -760,7 +775,7 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
let channels: UInt32 = (cfgResult == .ok && cfg?.stereo == true) ? 2 : 1
let streamId = screenStreamId
let capture = ScreenAudioCapture(channels: channels) { [weak self] pcm, samples, ch in
let capture = ScreenAudioCapture(channels: channels, selection: screenAudioSelection) { [weak self] pcm, samples, ch in
self?.client.feedPcm(streamId: streamId, pcm: pcm, samplesPerChannel: samples, channels: ch)
}
screenCapture = capture
@@ -768,7 +783,8 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
Task { @MainActor in
do {
try await capture.start()
addActivity("Started sharing screen audio (\(channels == 2 ? "stereo" : "mono"))")
addActivity("Started sharing screen audio (\(channels == 2 ? "stereo" : "mono"))"
+ "\(Self.scopeSuffix(for: screenAudioSelection))")
} catch {
// Most commonly: Screen Recording permission denied. Roll back the stream.
screenCapture = nil
@@ -788,6 +804,21 @@ final class MainWindowController: NSWindowController, NSWindowDelegate {
screenCapture = nil
}
/// A short human-readable description of the share scope for the activity log.
private static func scopeSuffix(for selection: ScreenAudioSelection) -> String {
var parts: [String] = []
switch selection.scope {
case .entireDesktop: break
case .onlyApps(let ids): if !ids.isEmpty { parts.append("only \(ids.count) app(s)") }
case .allExcept(let ids): if !ids.isEmpty { parts.append("excluding \(ids.count) app(s)") }
}
// For .onlyApps the screen reader is already excluded, so don't claim it twice.
if selection.excludeScreenReader {
if case .onlyApps = selection.scope {} else { parts.append("no screen reader") }
}
return parts.isEmpty ? "" : "" + parts.joined(separator: ", ")
}
@objc private func muteChanged() {
let muted = muteButton?.state == .on
let deafened = deafenButton?.state == .on

View File

@@ -0,0 +1,205 @@
using System.Diagnostics;
using System.Runtime.InteropServices;
namespace VoiceCat.App.Audio;
// ── Scope types (mirror macOS ScreenAudioScope / ScreenAudioSelection) ────────
public abstract record AppAudioScope;
public sealed record EntireDesktop : AppAudioScope;
public sealed record OnlyApps(IReadOnlyList<int> Pids, IReadOnlyList<string> Names) : AppAudioScope;
public sealed record AllExceptApps(IReadOnlyList<int> Pids, IReadOnlyList<string> Names) : AppAudioScope;
public sealed record AudioAppInfo(int Pid, string DisplayName);
// ── Enumerator ─────────────────────────────────────────────────────────────────
public static class AudioSessionEnumerator
{
// Returns all user-facing apps: visible-window processes (primary, like macOS
// SCShareableContent.current) plus any background audio-session-only processes
// (e.g. Spotify in mini-player). Excludes VoiceCat itself and system processes.
public static IReadOnlyList<AudioAppInfo> GetAudioApps()
{
var seen = new HashSet<int>();
var result = new List<AudioAppInfo>();
int selfPid = Environment.ProcessId;
// ── 1. Visible-window processes (EnumWindows) ─────────────────────────
// Same set macOS ScreenCaptureKit exposes: all apps with at least one
// visible top-level window. Shows apps even when not currently producing audio.
EnumWindows((hWnd, _) =>
{
if (!IsWindowVisible(hWnd)) return true;
GetWindowThreadProcessId(hWnd, out uint pid);
if (pid == 0 || pid == (uint)selfPid || !seen.Add((int)pid)) return true;
try
{
var proc = Process.GetProcessById((int)pid);
string name = proc.MainWindowTitle.Length > 0
? $"{proc.ProcessName} — {proc.MainWindowTitle}"
: proc.ProcessName;
if (!string.IsNullOrEmpty(proc.ProcessName))
result.Add(new AudioAppInfo((int)pid, name));
}
catch { /* process exited between EnumWindows and GetProcessById */ }
return true; // continue enumeration
}, IntPtr.Zero);
// ── 2. Background audio-session processes (WASAPI, supplement) ────────
// Catches apps that produce audio but have no visible window (screen reader,
// background music player, etc.). Silently skipped if WASAPI is unavailable.
AppendAudioSessionApps(seen, selfPid, result);
return result.OrderBy(a => a.DisplayName, StringComparer.OrdinalIgnoreCase).ToList();
}
// ── EnumWindows P/Invoke ──────────────────────────────────────────────────
private delegate bool EnumWindowsProc(IntPtr hWnd, IntPtr lParam);
[DllImport("user32.dll")]
private static extern bool EnumWindows(EnumWindowsProc lpEnumFunc, IntPtr lParam);
[DllImport("user32.dll")]
private static extern bool IsWindowVisible(IntPtr hWnd);
[DllImport("user32.dll")]
private static extern uint GetWindowThreadProcessId(IntPtr hWnd, out uint lpdwProcessId);
// ── WASAPI audio session supplement ──────────────────────────────────────
private static void AppendAudioSessionApps(HashSet<int> seen, int selfPid,
List<AudioAppInfo> result)
{
IMMDeviceEnumerator? enumerator = null;
IMMDevice? device = null;
IAudioSessionManager2? manager = null;
IAudioSessionEnumerator? sessions = null;
try
{
enumerator = (IMMDeviceEnumerator)Activator.CreateInstance(
Type.GetTypeFromCLSID(new Guid("BCDE0395-E52F-467C-8E3D-C4579291692E"))!)!;
enumerator.GetDefaultAudioEndpoint(0 /*eRender*/, 1 /*eMultimedia*/, out device);
var mgr2Iid = new Guid("77AA99A0-1BD6-484F-8BC7-2C654C9A9B6F");
device.Activate(ref mgr2Iid, 0x17 /*CLSCTX_ALL*/, IntPtr.Zero, out object mgr);
manager = (IAudioSessionManager2)mgr;
manager.GetSessionEnumerator(out sessions);
sessions.GetCount(out int count);
for (int i = 0; i < count; i++)
{
IAudioSessionControl? ctrl = null;
try
{
sessions.GetSession(i, out ctrl);
var ctrl2 = (IAudioSessionControl2)ctrl;
ctrl2.GetProcessId(out uint pid);
int ipid = (int)pid;
if (pid == 0 || ipid == selfPid || !seen.Add(ipid)) continue;
try
{
var proc = Process.GetProcessById(ipid);
if (!string.IsNullOrEmpty(proc.ProcessName))
result.Add(new AudioAppInfo(ipid, proc.ProcessName));
}
catch { /* exited */ }
}
catch { /* stale session */ }
finally { if (ctrl != null) Marshal.ReleaseComObject(ctrl); }
}
}
catch { /* no audio device or WASAPI unavailable — ignore */ }
finally
{
if (sessions != null) Marshal.ReleaseComObject(sessions);
if (manager != null) Marshal.ReleaseComObject(manager);
if (device != null) Marshal.ReleaseComObject(device);
if (enumerator != null) Marshal.ReleaseComObject(enumerator);
}
}
// ── COM interface declarations ─────────────────────────────────────────────
[ComImport, Guid("A95664D2-9614-4F35-A746-DE8DB63617E6"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IMMDeviceEnumerator
{
[PreserveSig] int EnumAudioEndpoints(int dataFlow, int stateMask, out IntPtr devices);
[PreserveSig] int GetDefaultAudioEndpoint(int dataFlow, int role, out IMMDevice endpoint);
[PreserveSig] int GetDevice([MarshalAs(UnmanagedType.LPWStr)] string id, out IMMDevice device);
[PreserveSig] int RegisterEndpointNotificationCallback(IntPtr client);
[PreserveSig] int UnregisterEndpointNotificationCallback(IntPtr client);
}
[ComImport, Guid("D666063F-1587-4E43-81F1-B948E807363F"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IMMDevice
{
[PreserveSig] int Activate(ref Guid iid, int clsCtx, IntPtr activationParams,
[MarshalAs(UnmanagedType.IUnknown)] out object ppInterface);
[PreserveSig] int OpenPropertyStore(int stgmAccess, out IntPtr propStore);
[PreserveSig] int GetId([MarshalAs(UnmanagedType.LPWStr)] out string id);
[PreserveSig] int GetState(out int state);
}
[ComImport, Guid("77AA99A0-1BD6-484F-8BC7-2C654C9A9B6F"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IAudioSessionManager2
{
[PreserveSig] int GetAudioSessionControl(ref Guid audioSessionGuid, int streamFlags,
out IAudioSessionControl session);
[PreserveSig] int GetSimpleAudioVolume(ref Guid audioSessionGuid, int streamFlags,
out IntPtr audioVolume);
[PreserveSig] int GetSessionEnumerator(out IAudioSessionEnumerator sessionEnum);
[PreserveSig] int RegisterSessionNotification(IntPtr notification);
[PreserveSig] int UnregisterSessionNotification(IntPtr notification);
[PreserveSig] int RegisterDuckNotification([MarshalAs(UnmanagedType.LPWStr)] string sessionID,
IntPtr notification);
[PreserveSig] int UnregisterDuckNotification(IntPtr notification);
}
[ComImport, Guid("E2F5BB11-0570-40CA-ACDD-3AA01277DEE8"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IAudioSessionEnumerator
{
[PreserveSig] int GetCount(out int sessionCount);
[PreserveSig] int GetSession(int sessionIndex, out IAudioSessionControl session);
}
[ComImport, Guid("F4B1A599-7266-4319-A8CA-E70ACB11E8CD"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IAudioSessionControl
{
[PreserveSig] int GetState(out int state);
[PreserveSig] int GetDisplayName([MarshalAs(UnmanagedType.LPWStr)] out string name);
[PreserveSig] int SetDisplayName([MarshalAs(UnmanagedType.LPWStr)] string name,
ref Guid eventContext);
[PreserveSig] int GetIconPath([MarshalAs(UnmanagedType.LPWStr)] out string iconPath);
[PreserveSig] int SetIconPath([MarshalAs(UnmanagedType.LPWStr)] string iconPath,
ref Guid eventContext);
[PreserveSig] int GetGroupingParam(out Guid groupingParam);
[PreserveSig] int SetGroupingParam(ref Guid groupingParam, ref Guid eventContext);
[PreserveSig] int RegisterAudioSessionNotification(IntPtr notification);
[PreserveSig] int UnregisterAudioSessionNotification(IntPtr notification);
}
[ComImport, Guid("BFB7FF88-7239-4FC9-8FA2-07C950BE9C6D"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IAudioSessionControl2 : IAudioSessionControl
{
[PreserveSig] int GetSessionIdentifier([MarshalAs(UnmanagedType.LPWStr)] out string id);
[PreserveSig] int GetSessionInstanceIdentifier([MarshalAs(UnmanagedType.LPWStr)] out string id);
[PreserveSig] int GetProcessId(out uint pid);
[PreserveSig] int IsSystemSoundsSession();
[PreserveSig] int SetDuckingPreference(bool optOut);
}
}

View File

@@ -0,0 +1,146 @@
using VoiceCat.Interop;
// Owns N ProcessLoopbackCapture instances, mixes their PCM every 20 ms, and feeds
// the result to the core via vc_stream_feed_pcm. Used for per-app audio sharing.
namespace VoiceCat.App.Audio;
public sealed class ProcessAudioMixer : IDisposable
{
private const int SampleRate = 48000;
private const int FrameSamples = 960;
private const int Channels = 2; // stereo; captures fall back to mono if needed
private readonly List<ProcessLoopbackCapture> _captures = [];
// Per-capture latest frame, protected by _frameLock.
private readonly object _frameLock = new();
private List<short[]> _latestFrames = [];
private int _activeChannels = Channels;
private Thread? _mixThread;
private volatile bool _running;
private VoiceCatClient? _client;
private uint _streamId;
public void Start(AppAudioScope scope, VoiceCatClient client, uint streamId,
IReadOnlyList<AudioAppInfo> allApps)
{
if (_running) return;
_client = client;
_streamId = streamId;
var pids = ResolvePids(scope, allApps);
if (pids.Count == 0)
{
// nothing to capture — scope resolved to empty set
return;
}
lock (_frameLock)
{
_latestFrames = new List<short[]>(new short[pids.Count][]);
_activeChannels = Channels;
}
for (int i = 0; i < pids.Count; i++)
{
int captureIndex = i;
var cap = new ProcessLoopbackCapture(pids[i], ProcessLoopbackCapture.Mode.Include);
cap.PcmFrameReady += (pcm, spc, ch) => OnCaptureFrame(captureIndex, pcm, ch);
_captures.Add(cap);
}
foreach (var c in _captures) c.Start();
_running = true;
_mixThread = new Thread(MixLoop) { IsBackground = true, Name = "ProcessAudioMixer" };
_mixThread.Start();
}
public void Stop()
{
_running = false;
_mixThread?.Join(500);
foreach (var c in _captures) { c.Stop(); c.Dispose(); }
_captures.Clear();
}
public void Dispose() => Stop();
// ── Capture callback ──────────────────────────────────────────────────────
private void OnCaptureFrame(int index, short[] pcm, int channels)
{
lock (_frameLock)
{
// Upmix mono → stereo interleave if the capture fell back to mono.
if (channels == 1 && _activeChannels == 2)
pcm = MonoToStereo(pcm);
if (index < _latestFrames.Count)
_latestFrames[index] = pcm;
}
}
// ── Mix loop (20 ms timer) ────────────────────────────────────────────────
private void MixLoop()
{
// Use a target period close to 20 ms; small under-shoot avoids accumulating drift.
const int periodMs = 19;
while (_running)
{
Thread.Sleep(periodMs);
if (!_running) break;
short[] mix;
lock (_frameLock)
{
int len = FrameSamples * _activeChannels;
mix = new short[len];
foreach (var frame in _latestFrames)
{
if (frame == null) continue;
int frameLen = Math.Min(frame.Length, len);
for (int i = 0; i < frameLen; i++)
{
int sum = mix[i] + frame[i];
// Saturating clamp
mix[i] = (short)Math.Clamp(sum, short.MinValue, short.MaxValue);
}
}
}
_client?.StreamFeedPcm(_streamId, mix, FrameSamples, (uint)_activeChannels);
}
}
// ── Helpers ───────────────────────────────────────────────────────────────
// For "AllExcept": enumerate all running audio apps and exclude the specified ones.
// For "OnlyApps": use their PIDs directly.
private static List<int> ResolvePids(AppAudioScope scope, IReadOnlyList<AudioAppInfo> allApps)
{
return scope switch
{
OnlyApps o => [.. o.Pids],
AllExceptApps a =>
allApps
.Where(app => !a.Pids.Contains(app.Pid))
.Select(app => app.Pid)
.ToList(),
_ => [],
};
}
private static short[] MonoToStereo(short[] mono)
{
var stereo = new short[mono.Length * 2];
for (int i = 0; i < mono.Length; i++)
{
stereo[i * 2] = mono[i];
stereo[i * 2 + 1] = mono[i];
}
return stereo;
}
}

View File

@@ -0,0 +1,434 @@
using System.Runtime.InteropServices;
// Single-process WASAPI loopback capture via AUDIOCLIENT_ACTIVATION_PARAMS
// (Windows 10 2004+ / Build 19041+).
//
// Threading: ALL WASAPI init runs on the capture thread (MTA). If called from the
// WinForms UI thread (STA), ActivateAudioInterfaceAsync fires ActivateCompleted on
// an MTA pool thread; COM marshals that back to the STA pump — but the STA thread is
// blocked on CompletionEvent.Wait → deadlock. MTA capture thread avoids this.
//
// COM QI policy: the COM objects returned by the process-loopback activation path
// reject QueryInterface for their own IIDs under .NET's RCW mechanism. Every call
// to IAudioClient and IAudioCaptureClient is therefore dispatched via raw vtable
// pointer arithmetic, bypassing .NET COM interop entirely.
namespace VoiceCat.App.Audio;
public sealed class ProcessLoopbackCapture : IDisposable
{
public enum Mode { Include, Exclude }
// Fired on the capture thread every 20 ms (960 samples/channel @ 48 kHz, interleaved s16).
public event Action<short[], int /*samplesPerChannel*/, int /*channels*/>? PcmFrameReady;
private const int SampleRate = 48000;
private const int FrameSamples = 960; // 20 ms
private const string LoopbackDevicePath = "VAD\\Process_Loopback";
private readonly int _pid;
private readonly Mode _mode;
// Raw COM pointers — managed via explicit AddRef/Release, no RCW wrapping.
private IntPtr _audioClientPtr; // IAudioClient*
private IntPtr _captureClientPtr; // IAudioCaptureClient*
private AutoResetEvent? _bufferEvent;
private Thread? _captureThread;
private volatile bool _running;
private int _channels;
// Accumulator: assembles driver-callback-sized fragments into FrameSamples chunks.
private short[] _accumBuf = [];
private int _accumCount;
// Init-done signal: Set() by the capture thread after ActivateClient() completes.
private readonly ManualResetEventSlim _initDone = new(false);
private bool _initOk;
public ProcessLoopbackCapture(int pid, Mode mode)
{
_pid = pid;
_mode = mode;
}
/// <summary>Starts capture. Blocks until WASAPI activation completes (typically &lt;100 ms).
/// Returns false if the process cannot be captured.</summary>
public bool Start()
{
if (_running) return false;
_running = true;
_captureThread = new Thread(CaptureThreadProc)
{
IsBackground = true,
Name = $"ProcLoopback:{_pid}",
};
_captureThread.Start();
bool ok = _initDone.Wait(5000) && _initOk;
if (!ok) _running = false;
return ok;
}
public void Stop()
{
_running = false;
_bufferEvent?.Set();
_captureThread?.Join(500);
if (_audioClientPtr != IntPtr.Zero) AC_Stop(_audioClientPtr);
}
public void Dispose()
{
Stop();
ComRelease(ref _captureClientPtr);
ComRelease(ref _audioClientPtr);
_bufferEvent?.Dispose();
_initDone.Dispose();
}
// ── Capture thread (MTA) ──────────────────────────────────────────────────
private void CaptureThreadProc()
{
_initOk = ActivateAndStart();
_initDone.Set();
if (!_initOk) return;
CaptureLoop();
}
private bool ActivateAndStart()
{
if (!ActivateClient()) return false;
_bufferEvent = new AutoResetEvent(false);
if (AC_SetEventHandle(_audioClientPtr, _bufferEvent.SafeWaitHandle.DangerousGetHandle()) < 0)
return false;
return AC_Start(_audioClientPtr) >= 0;
}
// ── Activation ───────────────────────────────────────────────────────────
private unsafe bool ActivateClient()
{
var activationParams = new AudioClientActivationParams
{
ActivationType = 1, // AUDCLNT_ACTIVATIONTYPE_PROCESS_LOOPBACK
TargetProcessId = (uint)_pid,
ProcessLoopbackMode = _mode == Mode.Include ? 0u : 1u,
};
var handler = new ActivationCompletionHandler();
var audioClientIid = new Guid("1CB9AD4C-DBFA-4c32-B178-C2F568A703B2");
IntPtr opPtr;
{
AudioClientActivationParams* pParams = &activationParams;
// PROPVARIANT (VT_BLOB) x64: vt(2)+res(6)+cbSize(4)+pad(4)+pBlobData(8) = 24 B.
var pv = stackalloc byte[24];
*(ushort*)(pv + 0) = 65;
*(uint*) (pv + 8) = (uint)sizeof(AudioClientActivationParams);
*(nint*) (pv + 16) = (nint)pParams;
int hr = ActivateAudioInterfaceAsync(
LoopbackDevicePath, ref audioClientIid,
(IntPtr)pv, handler, out opPtr);
if (hr < 0) return false;
}
if (!handler.CompletionEvent.Wait(3000))
{
ComRelease(ref opPtr);
return false;
}
// Call GetActivateResult via vtable (slot 3) — avoids QI on the async-op object.
if (!Vtable_GetActivateResult(handler.OperationPtr, out int activateHr, out IntPtr activatedPtr))
{
handler.ReleaseOp();
ComRelease(ref opPtr);
return false;
}
handler.ReleaseOp();
ComRelease(ref opPtr);
if (activateHr < 0 || activatedPtr == IntPtr.Zero) return false;
// Store the raw IAudioClient* — do NOT create an RCW; use vtable dispatch instead.
_audioClientPtr = activatedPtr;
// activatedPtr already has ref count from GetActivateResult; don't double-release.
return InitializeStream();
}
// IActivateAudioInterfaceAsyncOperation vtable slot 3: GetActivateResult(HRESULT*, IUnknown**)
private static unsafe bool Vtable_GetActivateResult(IntPtr op,
out int activateHr, out IntPtr activatedPtr)
{
activateHr = unchecked((int)0x80004005);
activatedPtr = IntPtr.Zero;
if (op == IntPtr.Zero) return false;
void** vtable = *(void***)op.ToPointer();
var fn = (delegate* unmanaged[Stdcall]<IntPtr, int*, IntPtr*, int>)vtable[3];
fixed (int* pHr = &activateHr)
fixed (IntPtr* pPtr = &activatedPtr)
return fn(op, pHr, pPtr) >= 0;
}
private unsafe bool InitializeStream()
{
// Try s16 stereo first; fall back to s16 mono.
foreach (int ch in new[] { 2, 1 })
{
var fmt = new WaveFormatEx
{
wFormatTag = 1, // WAVE_FORMAT_PCM
nChannels = (ushort)ch,
nSamplesPerSec = SampleRate,
wBitsPerSample = 16,
nBlockAlign = (ushort)(ch * 2),
nAvgBytesPerSec = (uint)(SampleRate * ch * 2),
cbSize = 0,
};
// Process-loopback requires LOOPBACK (deliver rendered audio) + EVENTCALLBACK +
// AUTOCONVERTPCM (resample the app's native format to our requested s16/48k). Without
// LOOPBACK every buffer comes back AUDCLNT_BUFFERFLAGS_SILENT; without AUTOCONVERTPCM
// the requested format is rejected. Matches the MS ApplicationLoopback sample.
// AUDCLNT_SHAREMODE_SHARED = 0
// AUDCLNT_STREAMFLAGS_LOOPBACK = 0x00020000
// AUDCLNT_STREAMFLAGS_EVENTCALLBACK = 0x00040000
// AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM = 0x80000000
const uint streamFlags = 0x00020000u | 0x00040000u | 0x80000000u;
int hr = AC_Initialize(_audioClientPtr, 0, streamFlags,
2_000_000 /*200ms hns*/, 0, &fmt, null);
if (hr >= 0)
{
_channels = ch;
_accumBuf = new short[FrameSamples * ch];
_accumCount = 0;
break;
}
if (ch == 1) return false;
}
var captureIid = new Guid("C8ADBD64-E71E-48a0-A4DE-185C395CD317");
int getHr = AC_GetService(_audioClientPtr, ref captureIid, out _captureClientPtr);
return getHr >= 0 && _captureClientPtr != IntPtr.Zero;
}
// ── Capture loop ─────────────────────────────────────────────────────────
private void CaptureLoop()
{
while (_running)
{
_bufferEvent!.WaitOne(100);
if (!_running) break;
while (_running)
{
int hr = CC_GetNextPacketSize(_captureClientPtr, out uint packetSize);
if (hr < 0 || packetSize == 0) break;
hr = CC_GetBuffer(_captureClientPtr, out IntPtr dataPtr, out uint framesAvailable,
out uint flags);
if (hr < 0) break;
bool silent = (flags & 2) != 0; // AUDCLNT_BUFFERFLAGS_SILENT
if (framesAvailable > 0)
{
if (silent) AccumulateSilence((int)framesAvailable);
else AccumulatePcm(dataPtr, (int)framesAvailable);
}
CC_ReleaseBuffer(_captureClientPtr, framesAvailable);
}
}
}
private unsafe void AccumulatePcm(IntPtr data, int frames)
{
var src = (short*)data.ToPointer();
int total = frames * _channels;
int idx = 0;
while (idx < total)
{
int space = _accumBuf.Length - _accumCount;
int copy = Math.Min(total - idx, space);
fixed (short* dst = _accumBuf)
Buffer.MemoryCopy(src + idx, dst + _accumCount, copy * 2L, copy * 2L);
_accumCount += copy;
idx += copy;
if (_accumCount == _accumBuf.Length)
FlushFrame();
}
}
private void AccumulateSilence(int frames)
{
int total = frames * _channels;
int idx = 0;
while (idx < total)
{
int space = _accumBuf.Length - _accumCount;
int fill = Math.Min(total - idx, space);
Array.Clear(_accumBuf, _accumCount, fill);
_accumCount += fill;
idx += fill;
if (_accumCount == _accumBuf.Length)
FlushFrame();
}
}
private void FlushFrame()
{
var copy = new short[_accumBuf.Length];
_accumBuf.AsSpan().CopyTo(copy);
PcmFrameReady?.Invoke(copy, FrameSamples, _channels);
_accumCount = 0;
}
// ── IAudioClient vtable helpers (raw dispatch, no RCW / no QI) ───────────
//
// Vtable layout (IUnknown base: 0=QI 1=AddRef 2=Release; then IAudioClient methods):
// 3=Initialize 4=GetBufferSize 5=GetStreamLatency 6=GetCurrentPadding
// 7=IsFormatSupported 8=GetMixFormat 9=GetDevicePeriod
// 10=Start 11=Stop 12=Reset 13=SetEventHandle 14=GetService
private static unsafe int AC_Initialize(IntPtr ac, int shareMode, uint streamFlags,
long hnsBufferDuration, long hnsPeriodicity, WaveFormatEx* pFormat, Guid* pSession)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, int, uint, long, long, WaveFormatEx*, Guid*, int>)
(*(void***)ac)[3];
return fn(ac, shareMode, streamFlags, hnsBufferDuration, hnsPeriodicity, pFormat, pSession);
}
private static unsafe int AC_Start(IntPtr ac)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, int>)(*(void***)ac)[10];
return fn(ac);
}
private static unsafe int AC_Stop(IntPtr ac)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, int>)(*(void***)ac)[11];
return fn(ac);
}
private static unsafe int AC_SetEventHandle(IntPtr ac, IntPtr eventHandle)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, IntPtr, int>)(*(void***)ac)[13];
return fn(ac, eventHandle);
}
private static unsafe int AC_GetService(IntPtr ac, ref Guid riid, out IntPtr ppv)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, Guid*, IntPtr*, int>)(*(void***)ac)[14];
fixed (Guid* pIid = &riid)
fixed (IntPtr* pPpv = &ppv)
return fn(ac, pIid, pPpv);
}
// ── IAudioCaptureClient vtable helpers ────────────────────────────────────
//
// Vtable (IUnknown: 0-2; then): 3=GetBuffer 4=ReleaseBuffer 5=GetNextPacketSize
private static unsafe int CC_GetBuffer(IntPtr cc, out IntPtr ppData,
out uint pNumFrames, out uint pdwFlags)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, IntPtr*, uint*, uint*, ulong*, ulong*, int>)
(*(void***)cc)[3];
ulong devPos = 0, qpcPos = 0;
fixed (IntPtr* p0 = &ppData)
fixed (uint* p1 = &pNumFrames)
fixed (uint* p2 = &pdwFlags)
return fn(cc, p0, p1, p2, &devPos, &qpcPos);
}
private static unsafe int CC_ReleaseBuffer(IntPtr cc, uint numFrames)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, uint, int>)(*(void***)cc)[4];
return fn(cc, numFrames);
}
private static unsafe int CC_GetNextPacketSize(IntPtr cc, out uint pNumFrames)
{
var fn = (delegate* unmanaged[Stdcall]<IntPtr, uint*, int>)(*(void***)cc)[5];
fixed (uint* p = &pNumFrames)
return fn(cc, p);
}
// ── COM utilities ─────────────────────────────────────────────────────────
private static unsafe void ComRelease(ref IntPtr ptr)
{
if (ptr == IntPtr.Zero) return;
var fn = (delegate* unmanaged[Stdcall]<IntPtr, uint>)(*(void***)ptr)[2]; // IUnknown::Release
fn(ptr);
ptr = IntPtr.Zero;
}
// ── P/Invoke & structs ────────────────────────────────────────────────────
[DllImport("Mmdevapi.dll", CharSet = CharSet.Unicode)]
private static extern int ActivateAudioInterfaceAsync(
string deviceInterfacePath,
ref Guid riid,
IntPtr activationParams,
[MarshalAs(UnmanagedType.Interface)] IActivateAudioInterfaceCompletionHandler completionHandler,
out IntPtr activationOperation);
[StructLayout(LayoutKind.Sequential)]
private struct AudioClientActivationParams
{
public int ActivationType; // AUDCLNT_ACTIVATIONTYPE_PROCESS_LOOPBACK = 1
public uint TargetProcessId;
public uint ProcessLoopbackMode; // INCLUDE=0, EXCLUDE=1
}
[StructLayout(LayoutKind.Sequential, Pack = 2)]
private struct WaveFormatEx
{
public ushort wFormatTag;
public ushort nChannels;
public uint nSamplesPerSec;
public uint nAvgBytesPerSec;
public ushort nBlockAlign;
public ushort wBitsPerSample;
public ushort cbSize;
}
// ── Completion handler CCW ────────────────────────────────────────────────
//
// Only this object still uses .NET COM interop (as a CCW). The activateOperation
// parameter is IntPtr to avoid QI on the incoming async-op pointer.
[ComVisible(true), ClassInterface(ClassInterfaceType.None)]
private sealed class ActivationCompletionHandler : IActivateAudioInterfaceCompletionHandler
{
public readonly ManualResetEventSlim CompletionEvent = new(false);
public IntPtr OperationPtr { get; private set; }
public void ActivateCompleted(IntPtr activateOperation)
{
OperationPtr = activateOperation;
if (OperationPtr != IntPtr.Zero) Marshal.AddRef(OperationPtr);
CompletionEvent.Set();
}
public void ReleaseOp()
{
if (OperationPtr == IntPtr.Zero) return;
Marshal.Release(OperationPtr);
OperationPtr = IntPtr.Zero;
}
}
[ComImport, Guid("41D949AB-9862-444A-80F6-C261334DA5EB"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
private interface IActivateAudioInterfaceCompletionHandler
{
void ActivateCompleted(IntPtr activateOperation);
}
}

View File

@@ -0,0 +1,166 @@
using VoiceCat.App.Audio;
namespace VoiceCat.App.Forms;
/// <summary>
/// Modal dialog for selecting which apps' audio to share.
/// Returns the chosen <see cref="AppAudioScope"/> via <see cref="ChosenScope"/>,
/// or <see cref="DialogResult.Cancel"/> if the user dismissed without sharing.
/// </summary>
public sealed class AppAudioPickerDialog : Form
{
private readonly RadioButton _rdoAll;
private readonly RadioButton _rdoOnly;
private readonly RadioButton _rdoExcept;
private readonly ListView _appList;
private readonly Label _lblApps;
// Snapshot taken when the dialog opens (refresh on open, not on every check change).
private IReadOnlyList<AudioAppInfo> _apps = [];
public AppAudioScope? ChosenScope { get; private set; }
public AppAudioPickerDialog()
{
// ── Radio buttons ───────────────────────────────────────────────────
_rdoAll = new RadioButton
{
Text = "&Entire desktop",
Checked = true,
Location = new Point(12, 12),
Size = new Size(360, 20),
TabIndex = 0,
};
_rdoOnly = new RadioButton
{
Text = "&Only selected apps",
Location = new Point(12, 36),
Size = new Size(360, 20),
TabIndex = 1,
};
_rdoExcept = new RadioButton
{
Text = "All apps e&xcept selected",
Location = new Point(12, 60),
Size = new Size(360, 20),
TabIndex = 2,
};
_rdoAll.CheckedChanged += OnModeChanged;
_rdoOnly.CheckedChanged += OnModeChanged;
_rdoExcept.CheckedChanged += OnModeChanged;
// ── App list ────────────────────────────────────────────────────────
_lblApps = new Label
{
Text = "Apps with active audio sessions:",
AutoSize = true,
Location = new Point(12, 90),
Visible = false,
TabIndex = 3,
};
_appList = new ListView
{
Location = new Point(12, 112),
Size = new Size(360, 180),
CheckBoxes = true,
View = View.List,
Visible = false,
TabIndex = 4,
FullRowSelect = true,
};
// ── Buttons ─────────────────────────────────────────────────────────
var btnShare = new Button
{
Text = "&Share",
DialogResult = DialogResult.OK,
Location = new Point(216, 308),
Size = new Size(75, 27),
TabIndex = 5,
};
var btnCancel = new Button
{
Text = "&Cancel",
DialogResult = DialogResult.Cancel,
Location = new Point(297, 308),
Size = new Size(75, 27),
TabIndex = 6,
};
btnShare.Click += OnShareClick;
AcceptButton = btnShare;
CancelButton = btnCancel;
AutoScaleMode = AutoScaleMode.Font;
ClientSize = new Size(384, 348);
Controls.AddRange([_rdoAll, _rdoOnly, _rdoExcept, _lblApps, _appList, btnShare, btnCancel]);
FormBorderStyle = FormBorderStyle.FixedDialog;
MaximizeBox = false;
MinimizeBox = false;
StartPosition = FormStartPosition.CenterParent;
Text = "Share App Audio";
}
protected override void OnLoad(EventArgs e)
{
base.OnLoad(e);
RefreshAppList();
}
private void RefreshAppList()
{
_apps = AudioSessionEnumerator.GetAudioApps();
_appList.Items.Clear();
foreach (var app in _apps)
_appList.Items.Add(new ListViewItem($"{app.DisplayName} (PID {app.Pid})") { Tag = app.Pid });
}
private void OnModeChanged(object? sender, EventArgs e)
{
bool showList = _rdoOnly.Checked || _rdoExcept.Checked;
_lblApps.Visible = showList;
_appList.Visible = showList;
if (showList && _appList.Items.Count == 0)
RefreshAppList();
}
private void OnShareClick(object? sender, EventArgs e)
{
if (_rdoAll.Checked)
{
ChosenScope = new EntireDesktop();
return;
}
var checkedPids = new List<int>();
var checkedNames = new List<string>();
foreach (ListViewItem item in _appList.CheckedItems)
{
if (item.Tag is int pid)
{
checkedPids.Add(pid);
checkedNames.Add(item.Text);
}
}
if (checkedPids.Count == 0)
{
MessageBox.Show(
"Select at least one app, or choose 'Entire desktop'.",
"No apps selected",
MessageBoxButtons.OK,
MessageBoxIcon.Warning);
DialogResult = DialogResult.None; // prevent close
return;
}
ChosenScope = _rdoOnly.Checked
? new OnlyApps(checkedPids, checkedNames)
: new AllExceptApps(checkedPids, checkedNames);
}
/// <summary>All apps that were visible in the list when the user clicked Share.</summary>
public IReadOnlyList<AudioAppInfo> VisibleApps => _apps;
}

View File

@@ -28,6 +28,7 @@ public sealed class ChannelEditDialog : Form
private CheckBox _chkFec = null!;
private NumericUpDown _numExpectedLoss = null!;
private CheckBox _chkDtx = null!;
private CheckBox _chkDred = null!;
private NumericUpDown _numComplexity = null!;
public ChannelEditInfo? Result { get; private set; }
@@ -183,7 +184,7 @@ public sealed class ChannelEditDialog : Form
private void BuildAudioPage(TabPage page, AudioConfigInfo? audio)
{
audio ??= new AudioConfigInfo(0, false, 48000, 0, 20, 0, true, 0, false, 10);
audio ??= new AudioConfigInfo(0, false, 48000, 0, 20, 0, true, 0, false, 10, false);
int y = 16;
int labelWidth = 150;
@@ -314,6 +315,17 @@ public sealed class ChannelEditDialog : Form
TabIndex = 18,
};
page.Controls.Add(_chkDtx);
y += 28;
_chkDred = new CheckBox
{
Text = "D&RED (deep redundancy)",
Location = new Point(inputX, y),
AutoSize = true,
Checked = audio.Dred,
TabIndex = 19,
};
page.Controls.Add(_chkDred);
}
private static void AddLabel(Control parent, string text, int x, int y, int width)
@@ -362,7 +374,8 @@ public sealed class ChannelEditDialog : Form
Fec: _chkFec.Checked,
ExpectedPacketLoss: (uint)_numExpectedLoss.Value,
Dtx: _chkDtx.Checked,
Complexity: (uint)_numComplexity.Value);
Complexity: (uint)_numComplexity.Value,
Dred: _chkDred.Checked);
Result = new ChannelEditInfo(
Id: _editingId,

View File

@@ -1,3 +1,4 @@
using VoiceCat.App.Audio;
using VoiceCat.Interop;
namespace VoiceCat.App.Forms;
@@ -22,6 +23,7 @@ public partial class MainForm : Form
// Voice state
private uint _micStreamId; // 0 = not started
private uint _screenStreamId; // 0 = not sharing screen audio
private ProcessAudioMixer? _screenMixer; // non-null only in per-app capture mode
private Keys _pttKey = Keys.F8;
private bool _serverMuted;
private bool _serverDeafened;
@@ -442,6 +444,7 @@ public partial class MainForm : Form
_currentChannelId = 0;
_micStreamId = 0;
_screenStreamId = 0;
_screenMixer?.Stop(); _screenMixer?.Dispose(); _screenMixer = null;
txtCompose.Enabled = false;
btnSend.Enabled = false;
tsbJoinVoice.Enabled = false;
@@ -622,28 +625,72 @@ public partial class MainForm : Form
{
if (_screenStreamId == 0)
{
var (result, streamId) = _client.StartStream(VcStreamKind.ScreenAudio, "Desktop audio");
if (result == VcResult.Ok)
{
_screenStreamId = streamId;
tsbScreenShare.Text = "Stop Screen Audio";
_miScreenShare.Text = "Stop Screen &Audio";
AddActivity("Started sharing screen audio");
StartScreenAudio();
}
else
{
StopScreenAudio();
}
}
private void StartScreenAudio()
{
using var picker = new AppAudioPickerDialog();
if (picker.ShowDialog(this) != DialogResult.OK || picker.ChosenScope == null) return;
var scope = picker.ChosenScope;
if (scope is EntireDesktop)
{
// Existing whole-device WASAPI loopback path — core handles it.
var (result, streamId) = _client.StartStream(VcStreamKind.ScreenAudio, "Desktop audio");
if (result != VcResult.Ok)
{
AddActivity($"Failed to start screen audio: {result}");
return;
}
_screenStreamId = streamId;
AddActivity("Sharing screen audio: entire desktop");
}
else
{
// Per-app path: suppress core loopback, C# mixer feeds PCM.
var (result, streamId) = _client.StartStreamExternalFeed(VcStreamKind.ScreenAudio, "App audio");
if (result != VcResult.Ok)
{
AddActivity($"Failed to start screen audio: {result}");
return;
}
_screenStreamId = streamId;
_screenMixer = new ProcessAudioMixer();
_screenMixer.Start(scope, _client, streamId, picker.VisibleApps);
string desc = scope switch
{
OnlyApps o => $"only {o.Pids.Count} app(s)",
AllExceptApps a => $"all except {a.Pids.Count} app(s)",
_ => "apps",
};
AddActivity($"Sharing screen audio: {desc}");
}
tsbScreenShare.Text = "Stop Screen Audio";
_miScreenShare.Text = "Stop Screen &Audio";
}
private void StopScreenAudio()
{
_screenMixer?.Stop();
_screenMixer?.Dispose();
_screenMixer = null;
_client.StopStream(_screenStreamId);
_screenStreamId = 0;
tsbScreenShare.Text = "Share Screen Audio";
_miScreenShare.Text = "Share Screen &Audio";
AddActivity("Stopped sharing screen audio");
}
}
private void TrkOutputVolume_Scroll(object? sender, EventArgs e) =>
_client.SetOutputVolume(trkOutputVolume.Value / 100f);
@@ -876,7 +923,7 @@ public partial class MainForm : Form
var editInfo = new ChannelEditInfo(
channel.Id, channel.ParentId, channel.Name, channel.Topic,
channel.PasswordProtected, null, channel.MaxUsers, 0,
new AudioConfigInfo(0, false, 48000, 0, 20, 0, true, 0, false, 10));
new AudioConfigInfo(0, false, 48000, 0, 20, 0, true, 0, false, 10, false));
using var dlg = new ChannelEditDialog(_channels, editInfo);
if (dlg.ShowDialog(this) != DialogResult.OK || dlg.Result is null) return;
@@ -1019,7 +1066,7 @@ public partial class MainForm : Form
_client.EventReceived -= OnEvent;
foreach (var win in _pmWindows.Values.ToList()) win.Close();
_pmWindows.Clear();
if (_screenStreamId != 0) _client.StopStream(_screenStreamId);
if (_screenStreamId != 0) { _screenMixer?.Stop(); _screenMixer?.Dispose(); _screenMixer = null; _client.StopStream(_screenStreamId); }
if (_micStreamId != 0) _client.StopStream(_micStreamId);
_client.Disconnect();
_client.Dispose();

View File

@@ -18,6 +18,9 @@
WinForms' own analyzer (WFO0003) flags manifest-based DPI settings as superseded by
this property in modern .NET. -->
<ApplicationHighDpiMode>PerMonitorV2</ApplicationHighDpiMode>
<!-- WASAPI process-loopback COM interop in Audio/ProcessLoopbackCapture.cs uses unsafe
pointer arithmetic for PROPVARIANT/AUDIOCLIENT_ACTIVATION_PARAMS marshaling. -->
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<!-- System.Security.Cryptography.ProtectedData (PasswordProtector.cs's DPAPI wrapper) ships

View File

@@ -188,7 +188,7 @@ public sealed class VoiceCatClientSmokeTests : IDisposable
Password: null,
MaxUsers: 42,
SortOrder: 0,
Audio: new AudioConfigInfo(0, true, 48000, 64000, 20, 1, true, 5, false, 10))));
Audio: new AudioConfigInfo(0, true, 48000, 64000, 20, 1, true, 5, false, 10, false))));
Assert.True(PumpUntil(client,
() => events.Any(e => e.Type == VcEventType.GenericResult && e.Result == VcResult.Ok), 3000),
"CreateChannel did not succeed");
@@ -208,7 +208,7 @@ public sealed class VoiceCatClientSmokeTests : IDisposable
null,
100,
0,
new AudioConfigInfo(0, true, 48000, 64000, 20, 1, true, 5, false, 10))));
new AudioConfigInfo(0, true, 48000, 64000, 20, 1, true, 5, false, 10, false))));
events.Clear();
Assert.True(PumpUntil(client,
() => events.Any(e => e.Type == VcEventType.GenericResult && e.Result == VcResult.Ok), 3000),

View File

@@ -96,7 +96,8 @@ internal static class Marshaling
native.Fec != 0,
native.ExpectedPacketLoss,
native.Dtx != 0,
native.Complexity);
native.Complexity,
native.Dred != 0);
public static PermissionsInfo ToManaged(in VcPermissionsNative native) => new(
native.CanCreateTempChannel != 0,

View File

@@ -71,4 +71,5 @@ public sealed record AudioConfigInfo(
bool Fec,
uint ExpectedPacketLoss,
bool Dtx,
uint Complexity);
uint Complexity,
bool Dred);

View File

@@ -46,6 +46,9 @@ internal struct VcStreamDescNative
public IntPtr DeviceId; // unused by vc_stream_start today — device selection is a
// separate vc_set_input_device call; always IntPtr.Zero here.
public IntPtr Label;
// Mirrors vc_stream_desc::external_feed. When 1, the core skips its own WASAPI loopback
// and the caller feeds PCM via StreamFeedPcm (per-app capture path on Windows).
public int ExternalFeed;
}
[StructLayout(LayoutKind.Sequential)]
@@ -61,6 +64,7 @@ internal struct VcAudioConfigNative
public uint ExpectedPacketLoss;
public int Dtx;
public uint Complexity;
public int Dred;
}
[StructLayout(LayoutKind.Sequential)]

View File

@@ -186,6 +186,26 @@ public sealed class VoiceCatClient : IDisposable
}
}
/// <summary>
/// Like <see cref="StartStream"/> but sets <c>external_feed = 1</c> so the core skips its
/// own WASAPI loopback. The caller is responsible for feeding PCM via
/// <see cref="StreamFeedPcm"/>. Used by the Windows per-app capture path.
/// </summary>
public (VcResult Result, uint StreamId) StartStreamExternalFeed(VcStreamKind kind, string label)
{
nint labelPtr = Marshal.StringToCoTaskMemUTF8(label);
try
{
var desc = new VcStreamDescNative { Kind = kind, DeviceId = 0, Label = labelPtr, ExternalFeed = 1 };
var r = NativeMethods.vc_stream_start(_handle.DangerousGetHandle(), in desc, out uint streamId);
return (r, streamId);
}
finally
{
Marshal.FreeCoTaskMem(labelPtr);
}
}
public VcResult StopStream(uint streamId) =>
NativeMethods.vc_stream_stop(_handle.DangerousGetHandle(), streamId);
@@ -346,6 +366,7 @@ public sealed class VoiceCatClient : IDisposable
ExpectedPacketLoss = info.Audio.ExpectedPacketLoss,
Dtx = info.Audio.Dtx ? 1 : 0,
Complexity = info.Audio.Complexity,
Dred = info.Audio.Dred ? 1 : 0,
}
};
}

View File

@@ -0,0 +1,7 @@
set(VCPKG_TARGET_ARCHITECTURE arm64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)
set(VCPKG_CMAKE_SYSTEM_NAME Linux)
# Only build release configurations — halves buildtree disk usage.
# The server always ships a Release build; debug deps are never needed.
set(VCPKG_BUILD_TYPE release)

View File

@@ -0,0 +1,7 @@
set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)
set(VCPKG_CMAKE_SYSTEM_NAME Linux)
# Only build release configurations — halves buildtree disk usage.
# The server always ships a Release build; debug deps are never needed.
set(VCPKG_BUILD_TYPE release)

View File

@@ -68,5 +68,15 @@ if(NOT DEFINED VCPKG_TARGET_TRIPLET)
unset(_voicecat_target)
endif()
# ── Overlay triplets — project-local overrides take precedence ────────────────
# Enables custom triplets (e.g. release-only x64-linux/arm64-linux, iOS slices)
# without needing to fork vcpkg's built-in ones. Cross-compile presets that
# already set VCPKG_OVERLAY_TRIPLETS explicitly (apple-ios, apple-ios-sim) keep
# their own value via the DEFINED guard.
if(NOT DEFINED VCPKG_OVERLAY_TRIPLETS)
set(VCPKG_OVERLAY_TRIPLETS "${CMAKE_CURRENT_LIST_DIR}/vcpkg-overlays/triplets"
CACHE STRING "vcpkg overlay triplets directory")
endif()
# ── Hand off to the real vcpkg toolchain ──────────────────────────────────────
include("$ENV{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake")

View File

@@ -46,10 +46,12 @@ extern "C" {
/* ── Version ──────────────────────────────────────────────────────────────── */
#define VOICECAT_VERSION_MAJOR 0
#define VOICECAT_VERSION_MINOR 0
#define VOICECAT_VERSION_PATCH 1
#define VOICECAT_VERSION_PATCH 2 /* +vc_set_mixed_output_sink / vc_set_external_playback (iOS VPIO) */
/* The control-protocol version this build speaks (docs/protocol.md §4). */
#define VOICECAT_PROTOCOL_VERSION 1
/* The control-protocol version this build speaks (docs/protocol.md §4).
* v2 widened the UDP voice frame seq field u16 → u64 (docs/voice.md §2); a v2 server
* and a v1 client cannot interoperate, so the Hello handshake rejects on mismatch. */
#define VOICECAT_PROTOCOL_VERSION 2
/* ── Result codes ─────────────────────────────────────────────────────────── */
typedef enum vc_result {
@@ -205,6 +207,12 @@ typedef struct vc_stream_desc {
vc_stream_kind kind;
const char* device_id; /* NULL = default device for this kind */
const char* label; /* human label, e.g. "Microphone" */
/* If 1, the caller will feed PCM via vc_stream_feed_pcm; the core will NOT start its own
* WASAPI loopback capture. Only meaningful for VC_STREAM_SCREEN_AUDIO on Windows. Callers
* that brace-initialize this struct (tests, macOS) get 0 = auto-start loopback — no ABI
* break. See clients/windows/VoiceCat.App/Audio/ProcessAudioMixer.cs for the Windows
* per-app capture implementation that sets this. */
int external_feed;
} vc_stream_desc;
/* The effective Opus configuration in use for a stream — for a stream you own, this is
@@ -452,6 +460,43 @@ typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
uint32_t channels, uint32_t sample_rate);
VC_API vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
/* ── External playback (iOS VPIO / echo cancellation) ───────────────────────────
* On iOS, real echo cancellation + noise suppression + AGC are provided ONLY by Apple's
* Voice-Processing I/O audio unit (VPIO), which the Swift AVAudioEngine layer owns. For VPIO
* to cancel echo, the remote-audio playback must go through the SAME VPIO unit as the mic
* capture (VPIO subtracts the played-back signal from the mic). So in that topology the core
* must NOT open/drive its own hardware playback device — its output would bypass VPIO, giving
* it no reference signal and producing echo. Instead, enable external playback: the core keeps
* decoding + mixing every remote stream on a steady ~20 ms cadence and delivers the FINAL
* MIXED PCM (post output-volume, all streams summed) to this sink, which the Swift layer
* renders through the VPIO output.
*
* cb(user, pcm, samples_per_channel, channels, sample_rate)
*
* pcm : final mixed int16 PCM, interleaved when channels == 2.
* samples_per_channel : samples per channel for this block (960 @ 20 ms / 48 kHz).
* channels : the engine's playback channel count (2 = stereo).
* sample_rate : always 48000.
*
* The callback fires on the core's mixer-timer thread (NOT a hardware audio thread). It fires
* steadily even with no remote streams (a silent block), so the renderer has a continuous
* clock. The callback MUST NOT block, lock, or allocate — copy into a lock-free ring and
* return. Independent of vc_set_pcm_sink (the per-stream tap), which still works. Pass cb=NULL
* to disable (default: disabled). */
typedef void (*vc_mixed_output_cb)(void* user, const int16_t* pcm,
size_t samples_per_channel, uint32_t channels,
uint32_t sample_rate);
VC_API vc_result vc_set_mixed_output_sink(vc_client* c, vc_mixed_output_cb cb, void* user);
/* Enable/disable external-playback mode (default: disabled = normal hardware playback). When
* enabled, the core does NOT open a hardware playback device; decode+mix runs on an internal
* ~20 ms timer and the result is delivered via vc_set_mixed_output_sink. To also bypass the
* hardware mic (feeding VPIO-processed mic PCM instead), start the MIC stream with
* vc_stream_desc.external_feed=1 and push frames via vc_stream_feed_pcm — the core then skips
* the hardware capture device too. Apply BEFORE the engine starts, or follow with
* vc_audio_restart() to apply to a running engine. `enable` is a bool (0/1). */
VC_API vc_result vc_set_external_playback(vc_client* c, int enable);
/* ── Text ─────────────────────────────────────────────────────────────────── */
VC_API vc_result vc_send_text(vc_client* c, vc_text_scope scope, uint32_t target_id,
const char* utf8);

View File

@@ -216,6 +216,10 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
}
ma_context* ctx = context_inited_ ? &context_ : nullptr;
// External playback (iOS VPIO): skip the hardware playback device entirely — a timer
// thread drives the mixer and the final mix goes to the Swift VPIO renderer (launched
// after the capture block below).
if (!external_playback_) {
// ── Playback device (opened first on iOS: commits the output route — e.g. A2DP —
// before the capture device starts. Starting stereo capture can trigger an iOS audio
// route reconfiguration; opening playback first ensures A2DP is already committed
@@ -252,7 +256,11 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
}
}
}
} // end if (!external_playback_)
// External capture (iOS VPIO / external feed): skip the hardware mic device — PCM is fed
// via inject_capture / vc_stream_feed_pcm. capture_cb_ (set above) still fires for fed frames.
if (!params_.external_capture) {
// ── Capture device (opened after playback so the output route is already committed) ──
ma_device_id cap_id{};
bool have_cap_id = !p.capture_device_id.empty() &&
@@ -277,6 +285,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
ma_device_uninit(&capture_device_);
}
}
} // end if (!params_.external_capture)
// External playback: launch the mixer-timer thread now that the engine is configured. It
// drives on_playback() (decode + mix all remote streams) every frame_ms and ships the final
// mix to mixed_sink_ for the Swift VPIO renderer. No hardware playback device exists.
if (external_playback_) {
mixer_scratch_.assign(
static_cast<size_t>(frame_samples_) * params_.playback_channels, 0);
mixer_timer_stop_.store(false, std::memory_order_release);
mixer_timer_thread_ = std::thread([this] { run_mixer_timer(); });
}
#endif // VOICECAT_HAS_AUDIO
return true;
@@ -322,6 +341,9 @@ void AudioEngine::stop() {
if (!running_.exchange(false)) return;
#ifdef VOICECAT_HAS_AUDIO
// External playback: stop + join the mixer-timer thread before tearing down state it reads.
mixer_timer_stop_.store(true, std::memory_order_release);
if (mixer_timer_thread_.joinable()) mixer_timer_thread_.join();
if (capture_started_) {
ma_device_stop(&capture_device_);
ma_device_uninit(&capture_device_);
@@ -338,6 +360,12 @@ void AudioEngine::stop() {
bool AudioEngine::suspend() {
#ifdef VOICECAT_HAS_AUDIO
if (!running_.load(std::memory_order_acquire)) return true;
// External playback: pause the mixer timer (there is no playback device to stop). This
// matches the VPIO renderer going down during an AVAudioSession interruption.
if (external_playback_) {
mixer_timer_stop_.store(true, std::memory_order_release);
if (mixer_timer_thread_.joinable()) mixer_timer_thread_.join();
}
bool ok = true;
if (capture_started_) ok &= (ma_device_stop(&capture_device_) == MA_SUCCESS);
if (playback_started_) ok &= (ma_device_stop(&playback_device_) == MA_SUCCESS);
@@ -350,6 +378,11 @@ bool AudioEngine::suspend() {
bool AudioEngine::resume() {
#ifdef VOICECAT_HAS_AUDIO
if (!running_.load(std::memory_order_acquire)) return true;
// External playback: relaunch the mixer timer (mixer_scratch_ is still sized from start()).
if (external_playback_ && !mixer_timer_thread_.joinable()) {
mixer_timer_stop_.store(false, std::memory_order_release);
mixer_timer_thread_ = std::thread([this] { run_mixer_timer(); });
}
bool ok = true;
if (capture_started_) ok &= (ma_device_start(&capture_device_) == MA_SUCCESS);
if (playback_started_) ok &= (ma_device_start(&playback_device_) == MA_SUCCESS);
@@ -515,6 +548,11 @@ void AudioEngine::set_pcm_sink(PcmSink cb, void* user) {
pcm_sink_.store(cb, std::memory_order_release);
}
void AudioEngine::set_mixed_output_sink(MixedSink cb, void* user) {
mixed_sink_user_.store(user, std::memory_order_relaxed);
mixed_sink_.store(cb, std::memory_order_release);
}
#ifdef VOICECAT_HAS_AUDIO
void AudioEngine::capture_data_cb(ma_device* dev, void* /*out*/,
@@ -707,6 +745,32 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
#endif
}
// External-playback timer (iOS VPIO): with no hardware playback device to "pull" frames, this
// dedicated thread drives the mixer on a steady cadence. It is NOT a real-time audio thread (a
// plain timed worker, same class as vc_client::run_talk_timer), but it must still not allocate
// in the loop because on_playback() takes streams_mu_ via try_lock and runs the RT-safe decode
// path — so mixer_scratch_ is pre-sized in start(). Uses a deadline-based sleep to bound drift.
void AudioEngine::run_mixer_timer() {
using clock = std::chrono::steady_clock;
const uint32_t ch = params_.playback_channels;
const uint32_t spc = static_cast<uint32_t>(frame_samples_);
const auto period = std::chrono::milliseconds(params_.frame_ms);
auto next = clock::now();
while (!mixer_timer_stop_.load(std::memory_order_acquire)) {
on_playback(mixer_scratch_.data(), spc);
if (auto sink = mixed_sink_.load(std::memory_order_relaxed)) {
sink(mixed_sink_user_.load(std::memory_order_relaxed), mixer_scratch_.data(), spc, ch,
params_.sample_rate);
}
next += period;
// If we fell badly behind (e.g. the thread was descheduled), reset the deadline rather
// than spin to catch up — the VPIO renderer rides its own clock + jitter ring.
auto now = clock::now();
if (next < now) next = now + period;
std::this_thread::sleep_until(next);
}
}
#ifdef VOICECAT_HAS_LOOPBACK
void AudioEngine::loopback_data_cb(ma_device* dev, void* /*out*/, const void* in,
ma_uint32 frame_count) {

View File

@@ -21,6 +21,7 @@
#include <memory>
#include <mutex>
#include <optional>
#include <thread>
#include <unordered_map>
#include <vector>
@@ -42,7 +43,7 @@ namespace voicecat::audio {
class JitterBuffer {
public:
struct Frame {
uint16_t seq;
uint64_t seq;
uint32_t timestamp;
bool fec_present;
std::vector<uint8_t> payload;
@@ -92,6 +93,10 @@ struct AudioParams {
uint32_t frame_ms = 20;
std::string capture_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
std::string playback_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
// External capture: the mic is fed via inject_capture/vc_stream_feed_pcm (e.g. iOS VPIO),
// so start() skips opening the hardware capture device. Derived from the MIC LocalStream's
// external_feed flag in vc_client::ensure_audio_running().
bool external_capture = false;
};
// One enumerated device, returned by AudioEngine::enumerate_devices(). `id` is an internal,
@@ -202,6 +207,18 @@ class AudioEngine {
using PcmSink = void(*)(void*, uint32_t, uint32_t, const int16_t*, size_t, uint32_t, uint32_t);
void set_pcm_sink(PcmSink cb, void* user);
// External mixed-output sink (iOS VPIO). Receives the FINAL mixed PCM (post output-volume,
// all remote streams summed) on the mixer-timer thread when external playback is enabled.
// Matching signature to vc_mixed_output_cb (cast at the C-ABI boundary). Pass nullptr to
// disable. Thread-safe (atomic store; the timer read is relaxed-load).
using MixedSink = void(*)(void*, const int16_t*, size_t, uint32_t, uint32_t);
void set_mixed_output_sink(MixedSink cb, void* user);
// External-playback mode (iOS VPIO): when enabled, start() does NOT open a hardware
// playback device; a timer thread drives the mixer (on_playback) on a ~20 ms cadence and
// ships the final mix to the mixed-output sink. Set before start() (or apply via restart).
void set_external_playback(bool enable) { external_playback_ = enable; }
// External PCM feed overload: stereo-aware variant of inject_capture. samples_per_channel
// is samples per channel; total samples written = samples_per_channel * channels.
void inject_capture(int kind, const int16_t* pcm, size_t samples_per_channel, int channels);
@@ -336,6 +353,14 @@ class AudioEngine {
bool capture_started_ = false;
bool playback_started_ = false;
// External-playback mode (iOS VPIO): no hardware playback device; this timer thread drives
// on_playback() on a ~20 ms cadence and delivers the final mix to mixed_sink_. mixer_scratch_
// is pre-sized in start() (frame_samples_ * playback_channels) so the loop never allocates.
std::thread mixer_timer_thread_;
std::atomic<bool> mixer_timer_stop_{false};
std::vector<int16_t> mixer_scratch_;
void run_mixer_timer();
#ifdef VOICECAT_HAS_LOOPBACK
// Desktop-audio loopback capture (SCREEN_AUDIO) — own lifecycle, decoupled from
// capture_device_/playback_device_ start/stop (a screen-share can start/stop independently
@@ -491,6 +516,13 @@ class AudioEngine {
std::atomic<PcmSink> pcm_sink_{nullptr};
std::atomic<void*> pcm_sink_user_{nullptr};
// External mixed-output sink + mode flag (iOS VPIO). mixed_sink_ is written by
// set_mixed_output_sink (any thread); read by run_mixer_timer via relaxed load.
// external_playback_ is read in start() to gate hardware-playback-device creation.
std::atomic<MixedSink> mixed_sink_{nullptr};
std::atomic<void*> mixed_sink_user_{nullptr};
bool external_playback_ = false;
#ifdef VOICECAT_HAS_OPUS
::OpusDREDDecoder* dred_dec_ = nullptr; // shared DRED decoder; null if unsupported
#endif

View File

@@ -298,7 +298,7 @@ void vc_client::run_io(std::string host, uint16_t port) {
voicecat::v1::Envelope env;
env.set_request_id(next_req_id_++);
auto* hello = env.mutable_client_hello();
hello->set_proto_version(1);
hello->set_proto_version(2);
hello->set_client_name(cfg_.client_name ? cfg_.client_name : "vccli");
hello->set_client_version(cfg_.client_version ? cfg_.client_version : "0.1.0");
auto frame = make_frame(env);
@@ -1047,7 +1047,7 @@ void vc_client::on_capture_frame(int kind, const int16_t* pcm, int samples, int
voicecat::net::VoiceFrame hdr;
hdr.ssrc = ls.ssrc;
hdr.seq = static_cast<uint16_t>(media_send_crypto_->peek_send_counter());
hdr.seq = media_send_crypto_->peek_send_counter();
hdr.timestamp = ls.timestamp;
ls.timestamp += static_cast<uint32_t>(samples);
@@ -1086,8 +1086,14 @@ void vc_client::ensure_audio_running() {
if (it != local_streams_.end()) {
p.capture_device_id = it->second.capture_device_id;
p.capture_channels = it->second.capture_channels;
// iOS VPIO: when the mic is fed externally, skip the hardware capture device — the
// Swift AVAudioEngine VPIO path feeds processed mic PCM via vc_stream_feed_pcm.
p.external_capture = it->second.external_feed;
}
}
// iOS VPIO: skip the hardware playback device and drive the mixer on a timer, delivering the
// final mix to the mixed-output sink for the Swift VPIO renderer (vc_set_external_playback).
audio_engine_.set_external_playback(external_playback_.load(std::memory_order_acquire));
audio_engine_.start(p, [this](int kind, const int16_t* pcm, int samples, int channels) {
on_capture_frame(kind, pcm, samples, channels);
});
@@ -1162,6 +1168,7 @@ vc_result vc_client::stream_start(const vc_stream_desc& desc, uint32_t* out_stre
uint32_t sid = next_local_stream_id_++;
ls.stream_id = sid;
ls.pending = true;
ls.external_feed = (desc.external_feed != 0);
if (out_stream_id) *out_stream_id = sid;
req_id = next_req_id_++;
@@ -1196,6 +1203,7 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
bool ok_to_emit = false;
int kind;
int loopback_channels = 1; // only meaningful for SCREEN_AUDIO; set under the lock
bool loopback_external = false;
{
std::lock_guard lk(local_streams_mu_);
auto pit = pending_announce_kind_.find(req_id);
@@ -1240,11 +1248,12 @@ void vc_client::handle_stream_announce_result(uint64_t req_id,
// the lock alongside the rest of the LocalStream setup; used below after unlock.
if (kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO)) {
loopback_channels = ls.effective_params.stereo ? 2 : 1;
loopback_external = ls.external_feed;
}
}
ensure_audio_running();
if (kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO)) {
if (kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO) && !loopback_external) {
audio_engine_.start_loopback_capture(kind, loopback_channels);
}
@@ -1261,6 +1270,7 @@ vc_result vc_client::stream_stop(uint32_t stream_id) {
if (state_net_.load(std::memory_order_acquire) != VC_STATE_CONNECTED) return VC_ERR_NOT_CONNECTED;
int stopped_kind = -1;
bool stopped_external = false;
{
std::lock_guard lk(local_streams_mu_);
for (auto& [k, ls] : local_streams_) {
@@ -1268,11 +1278,12 @@ vc_result vc_client::stream_stop(uint32_t stream_id) {
}
LocalStream* ls = find_local_stream_by_id(stream_id);
if (!ls || !ls->active.load(std::memory_order_acquire)) return VC_ERR_INVALID_ARG;
stopped_external = ls->external_feed;
ls->active.store(false, std::memory_order_release);
ls->encoder.destroy();
}
if (stopped_kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO)) {
if (stopped_kind == static_cast<int>(VC_STREAM_SCREEN_AUDIO) && !stopped_external) {
audio_engine_.stop_loopback_capture();
}
@@ -1513,6 +1524,20 @@ vc_result vc_client::set_pcm_sink(vc_pcm_sink_cb cb, void* user) {
return VC_OK;
}
vc_result vc_client::set_mixed_output_sink(vc_mixed_output_cb cb, void* user) {
audio_engine_.set_mixed_output_sink(
reinterpret_cast<voicecat::audio::AudioEngine::MixedSink>(cb), user);
return VC_OK;
}
vc_result vc_client::set_external_playback(bool enable) {
external_playback_.store(enable, std::memory_order_release);
// Stored on the engine too; takes effect on the next start()/vc_audio_restart() (matching
// the vc_set_capture_channels "apply on next restart" contract).
audio_engine_.set_external_playback(enable);
return VC_OK;
}
vc_result vc_client::test_inject_capture(uint32_t stream_id, const int16_t* pcm, size_t samples) {
return stream_feed_pcm(stream_id, pcm, samples, 1);
}
@@ -1893,6 +1918,12 @@ vc_result vc_client::stream_feed_pcm(uint32_t, const int16_t*, size_t, uint32_t)
vc_result vc_client::set_pcm_sink(vc_pcm_sink_cb, void*) {
return VC_ERR_NOT_IMPLEMENTED;
}
vc_result vc_client::set_mixed_output_sink(vc_mixed_output_cb, void*) {
return VC_ERR_NOT_IMPLEMENTED;
}
vc_result vc_client::set_external_playback(bool) {
return VC_ERR_NOT_IMPLEMENTED;
}
vc_result vc_client::test_inject_capture(uint32_t, const int16_t*, size_t) {
return VC_ERR_NOT_IMPLEMENTED;
}

View File

@@ -89,6 +89,10 @@ struct vc_client {
// External PCM sink (see voicecat.h: vc_set_pcm_sink). Delegates to AudioEngine.
vc_result set_pcm_sink(vc_pcm_sink_cb cb, void* user);
// External mixed-output sink + external-playback mode (iOS VPIO; see voicecat.h).
vc_result set_mixed_output_sink(vc_mixed_output_cb cb, void* user);
vc_result set_external_playback(bool enable);
// TEST-ONLY (see voicecat.h) — deprecated alias for stream_feed_pcm(..., channels=1).
vc_result test_inject_capture(uint32_t stream_id, const int16_t* pcm, size_t samples);
@@ -246,6 +250,11 @@ struct vc_client {
// VC_STREAM_MIC. Set via vc_set_capture_channels; read by ensure_audio_running() to
// configure AudioParams.capture_channels before the device opens. Defaults to 1 (mono).
uint32_t capture_channels = 1;
// If true, the caller is feeding PCM via vc_stream_feed_pcm — skip start/stop of the
// core's WASAPI loopback device. Set from vc_stream_desc::external_feed at stream_start
// time and checked in handle_stream_announce_result / stream_stop.
bool external_feed = false;
};
mutable std::mutex local_streams_mu_;
std::unordered_map<int, LocalStream> local_streams_; // keyed by vc_stream_kind
@@ -282,6 +291,11 @@ struct vc_client {
std::atomic<vc_input_mode> current_input_mode_{VC_INPUT_VOICE_ACTIVATION};
std::atomic<bool> ptt_active_{false};
std::atomic<float> vad_threshold_{0.025f}; // remembered across mode switches
// External-playback mode (iOS VPIO): when true, ensure_audio_running() configures the
// AudioEngine to skip its hardware playback device and drive the mixer on a timer instead,
// delivering the final mix to the mixed-output sink. Set via vc_set_external_playback.
std::atomic<bool> external_playback_{false};
std::unique_ptr<voicecat::audio::ApmProcessor> mic_vad_;
// teardown_voice() is called both from run_io()'s own cleanup (on the io_thread_, when

View File

@@ -353,22 +353,26 @@ long SodiumMediaCrypto::open(const uint8_t* sealed, size_t len, const uint8_t* a
if (len < crypto_aead_chacha20poly1305_ietf_ABYTES) return -1;
if (out_cap < len - crypto_aead_chacha20poly1305_ietf_ABYTES) return -1;
// Reconstruct 64-bit counter from aad[8..9] (seq, big-endian u16).
// For M2, we zero-extend the 16-bit seq; TODO: add ROC for long sessions.
if (aad_len < 10) return -1;
uint64_t counter = (static_cast<uint64_t>(aad[8]) << 8) | aad[9];
// Read the full 64-bit nonce counter directly from aad[8..15] (seq, big-endian
// u64). Protocol v2 carries the full counter on the wire, so the nonce is exact —
// no reconstruction/rollover guessing needed.
if (aad_len < 16) return -1;
uint64_t counter = (static_cast<uint64_t>(aad[8]) << 56) |
(static_cast<uint64_t>(aad[9]) << 48) |
(static_cast<uint64_t>(aad[10]) << 40) |
(static_cast<uint64_t>(aad[11]) << 32) |
(static_cast<uint64_t>(aad[12]) << 24) |
(static_cast<uint64_t>(aad[13]) << 16) |
(static_cast<uint64_t>(aad[14]) << 8) |
static_cast<uint64_t>(aad[15]);
// ── Anti-replay check ────────────────────────────────────────────────────
if (!recv_initialized_) {
recv_highest_ = counter;
recv_window_ = 1; // bit0 = highest itself
recv_initialized_ = true;
} else {
if (counter > recv_highest_) {
uint64_t shift = counter - recv_highest_;
recv_window_ = (shift >= 64) ? 0 : (recv_window_ << shift);
recv_highest_ = counter;
}
// ── Anti-replay: REJECT-ONLY checks (no state mutation) ───────────────────
// The counter comes from the UNAUTHENTICATED header, so we must NOT advance the
// window before the AEAD tag is verified — otherwise a single corrupted/forged
// packet would shove recv_highest_ far ahead and reject every later legitimate
// packet as "too old", permanently wedging the stream. Order per RFC 3711 §3.3:
// replay-check → authenticate → update.
if (recv_initialized_ && counter <= recv_highest_) {
uint64_t offset = recv_highest_ - counter;
if (offset >= 64) return -1; // too old
if (recv_window_ & (UINT64_C(1) << offset)) return -1; // replay
@@ -382,11 +386,21 @@ long SodiumMediaCrypto::open(const uint8_t* sealed, size_t len, const uint8_t* a
out, &plain_len, nullptr, sealed, static_cast<unsigned long long>(len),
aad, static_cast<unsigned long long>(aad_len),
nonce, key_.data()) != 0)
return -1;
return -1; // auth failure — leave the replay window untouched
// Mark this counter as accepted in the window.
uint64_t offset = recv_highest_ - counter;
recv_window_ |= (UINT64_C(1) << offset);
// ── Authenticated: now it's safe to advance the window ────────────────────
if (!recv_initialized_) {
recv_highest_ = counter;
recv_window_ = 1; // bit0 = highest itself
recv_initialized_ = true;
} else if (counter > recv_highest_) {
uint64_t shift = counter - recv_highest_;
recv_window_ = (shift >= 64) ? 0 : (recv_window_ << shift);
recv_window_ |= 1; // bit0 = the new highest
recv_highest_ = counter;
} else {
recv_window_ |= (UINT64_C(1) << (recv_highest_ - counter));
}
return static_cast<long>(plain_len);
}

View File

@@ -163,7 +163,9 @@ class SodiumMediaCrypto final : public MediaCrypto {
long seal(const uint8_t* plain, size_t len, const uint8_t* aad, size_t aad_len,
uint8_t* out, size_t out_cap) override;
// open(): reconstructs counter from aad[8..9] (seq field), checks anti-replay.
// open(): reads the full 64-bit counter from aad[8..15] (seq field), checks
// anti-replay, then decrypts. The replay window is advanced ONLY after the AEAD
// tag verifies, so a corrupted/forged packet cannot poison it (RFC 3711 §3.3).
long open(const uint8_t* sealed, size_t len, const uint8_t* aad, size_t aad_len,
uint8_t* out, size_t out_cap) override;

View File

@@ -29,7 +29,7 @@ inline constexpr uint8_t kFlagLast = 0x08; // last frame before stream st
inline constexpr uint16_t kCodecOpus = 0;
// Size of the serialized header (bytes before the payload).
inline constexpr size_t kVoiceHeaderSize = 14;
inline constexpr size_t kVoiceHeaderSize = 20;
/*
* Wire layout (big-endian):
@@ -37,24 +37,28 @@ inline constexpr size_t kVoiceHeaderSize = 14;
* [1] flags u8
* [2..3] codec u16
* [4..7] ssrc u32
* [8..9] seq u16 (low 16 bits of monotonic send counter)
* [10..13] timestamp u32 (sample clock @48 kHz)
* [14+] payload (AEAD-encrypted Opus packet)
* [8..15] seq u64 (full monotonic send counter — the AEAD nonce counter)
* [16..19] timestamp u32 (sample clock @48 kHz)
* [20+] payload (AEAD-encrypted Opus packet)
*
* The 14-byte header is the AEAD AAD (authenticated, not encrypted).
* The 20-byte header is the AEAD AAD (authenticated, not encrypted).
* The payload region is the AEAD ciphertext + 16-byte Poly1305 MAC.
*
* Protocol v2 widened `seq` from u16 to u64: the receiver reconstructs the AEAD
* nonce counter directly from this field, so the full 64-bit counter must be on the
* wire (a 16-bit field wrapped after 65,536 frames and desynced the nonce).
*/
struct VoiceFrame {
uint8_t type = kFrameVoice;
uint8_t flags = 0;
uint16_t codec = kCodecOpus;
uint32_t ssrc = 0;
uint16_t seq = 0;
uint64_t seq = 0;
uint32_t timestamp = 0;
std::vector<uint8_t> payload; // Opus bytes (pre-AEAD on send; post-AEAD on recv)
};
// Serialize the 14-byte header into buf[0..13]. buf must be at least kVoiceHeaderSize bytes.
// Serialize the 20-byte header into buf[0..19]. buf must be at least kVoiceHeaderSize bytes.
inline void serialize_header(const VoiceFrame& f, uint8_t* buf) {
buf[0] = f.type;
buf[1] = f.flags;
@@ -64,15 +68,21 @@ inline void serialize_header(const VoiceFrame& f, uint8_t* buf) {
buf[5] = static_cast<uint8_t>(f.ssrc >> 16);
buf[6] = static_cast<uint8_t>(f.ssrc >> 8);
buf[7] = static_cast<uint8_t>(f.ssrc & 0xFF);
buf[8] = static_cast<uint8_t>(f.seq >> 8);
buf[9] = static_cast<uint8_t>(f.seq & 0xFF);
buf[10] = static_cast<uint8_t>(f.timestamp >> 24);
buf[11] = static_cast<uint8_t>(f.timestamp >> 16);
buf[12] = static_cast<uint8_t>(f.timestamp >> 8);
buf[13] = static_cast<uint8_t>(f.timestamp & 0xFF);
buf[8] = static_cast<uint8_t>(f.seq >> 56);
buf[9] = static_cast<uint8_t>(f.seq >> 48);
buf[10] = static_cast<uint8_t>(f.seq >> 40);
buf[11] = static_cast<uint8_t>(f.seq >> 32);
buf[12] = static_cast<uint8_t>(f.seq >> 24);
buf[13] = static_cast<uint8_t>(f.seq >> 16);
buf[14] = static_cast<uint8_t>(f.seq >> 8);
buf[15] = static_cast<uint8_t>(f.seq & 0xFF);
buf[16] = static_cast<uint8_t>(f.timestamp >> 24);
buf[17] = static_cast<uint8_t>(f.timestamp >> 16);
buf[18] = static_cast<uint8_t>(f.timestamp >> 8);
buf[19] = static_cast<uint8_t>(f.timestamp & 0xFF);
}
// Parse the 14-byte header from buf. Returns false if len < kVoiceHeaderSize.
// Parse the 20-byte header from buf. Returns false if len < kVoiceHeaderSize.
inline bool parse_header(const uint8_t* buf, size_t len, VoiceFrame& out) {
if (len < kVoiceHeaderSize) return false;
out.type = buf[0];
@@ -82,11 +92,18 @@ inline bool parse_header(const uint8_t* buf, size_t len, VoiceFrame& out) {
(static_cast<uint32_t>(buf[5]) << 16) |
(static_cast<uint32_t>(buf[6]) << 8) |
static_cast<uint32_t>(buf[7]);
out.seq = static_cast<uint16_t>((buf[8] << 8) | buf[9]);
out.timestamp = (static_cast<uint32_t>(buf[10]) << 24) |
(static_cast<uint32_t>(buf[11]) << 16) |
(static_cast<uint32_t>(buf[12]) << 8) |
static_cast<uint32_t>(buf[13]);
out.seq = (static_cast<uint64_t>(buf[8]) << 56) |
(static_cast<uint64_t>(buf[9]) << 48) |
(static_cast<uint64_t>(buf[10]) << 40) |
(static_cast<uint64_t>(buf[11]) << 32) |
(static_cast<uint64_t>(buf[12]) << 24) |
(static_cast<uint64_t>(buf[13]) << 16) |
(static_cast<uint64_t>(buf[14]) << 8) |
static_cast<uint64_t>(buf[15]);
out.timestamp = (static_cast<uint32_t>(buf[16]) << 24) |
(static_cast<uint32_t>(buf[17]) << 16) |
(static_cast<uint32_t>(buf[18]) << 8) |
static_cast<uint32_t>(buf[19]);
return true;
}

View File

@@ -158,6 +158,16 @@ vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user) {
return c->set_pcm_sink(cb, user);
}
vc_result vc_set_mixed_output_sink(vc_client* c, vc_mixed_output_cb cb, void* user) {
if (c == nullptr) return VC_ERR_INVALID_ARG;
return c->set_mixed_output_sink(cb, user);
}
vc_result vc_set_external_playback(vc_client* c, int enable) {
if (c == nullptr) return VC_ERR_INVALID_ARG;
return c->set_external_playback(enable != 0);
}
vc_result vc_set_capture_channels(vc_client* c, uint32_t stream_id, uint32_t channels) {
if (c == nullptr) return VC_ERR_INVALID_ARG;
return c->set_capture_channels(stream_id, channels);

View File

@@ -0,0 +1,23 @@
[Unit]
Description=VoiceCat Voice & Text Server
Documentation=https://github.com/org/voicecat
After=network.target
[Service]
Type=simple
User=voicecat
Group=voicecat
ExecStart=/usr/local/bin/voicecat-server --data-dir /var/lib/voicecat
Restart=on-failure
RestartSec=5s
# Allow binding port 8384 without running as root
AmbientCapabilities=CAP_NET_BIND_SERVICE
# Harden the process
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/voicecat
PrivateTmp=true
[Install]
WantedBy=multi-user.target

16
docker-compose.yml Normal file
View File

@@ -0,0 +1,16 @@
services:
voicecat:
build: .
image: ghcr.io/org/voicecat:latest
container_name: voicecat
restart: unless-stopped
ports:
- "8384:8384/tcp"
- "8384:8384/udp"
volumes:
- voicecat-data:/data
# Override CMD to customise the server name; add --no-guests to require accounts.
command: ["--data-dir", "/data", "--name", "My VoiceCat Server"]
volumes:
voicecat-data:

View File

@@ -153,6 +153,9 @@ Notes:
- **Version negotiation.** Each side sends `proto_version` (integer) and a `features`
string list. The effective version is `min(client, server)`; the effective feature set
is the intersection. A client that doesn't understand a feature simply never uses it.
The **current `proto_version` is 2**. v2 widened the UDP voice frame `seq` field from
u16 to u64 (voice.md §2) — a wire-format change with no backward compatibility on the
media path, so the server rejects any peer not on v2 rather than min-negotiating down.
- **Auth over TLS.** Passwords cross the wire only inside TLS 1.3 and are verified against
an Argon2id hash at rest (see security.md). `auth_methods` in `ServerHello` advertises
whether `guest` is enabled.

View File

@@ -89,8 +89,13 @@ the design depends on that.
- **Nonce discipline:** `nonce = direction_bit ‖ ssrc ‖ monotonic_packet_counter`. The
counter never repeats under one key; the session **rekeys** (re-derives via the exporter
with a bumped epoch) well before counter exhaustion or on a time/byte budget.
- **Anti-replay:** a sliding-window replay filter per ssrc (à la IPsec) keyed on the packet
counter. Replays and out-of-window packets are dropped before decode.
- **Anti-replay:** a 64-bit sliding-window replay filter keyed on the packet counter (à la
IPsec). The window is **advanced only after the AEAD tag verifies** (RFC 3711 §3.3 order:
replay-check → authenticate → update). The counter is read from the unauthenticated
header, so advancing the high-water mark *before* authentication would let a single
corrupted or forged packet jump it far ahead, after which every legitimate packet is
rejected as "too old" — a permanent denial of the whole stream. Failed-auth packets leave
the window untouched. Replays and out-of-window packets are dropped before decode.
## 3. UDP session binding

View File

@@ -35,13 +35,16 @@ macOS, **and iOS** (via a ReplayKit broadcast extension).
A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.
The header is **20 bytes** (protocol v2; v1 was 14 bytes with a u16 seq — see note below).
```
0 1 2 3 4 5 6 7 8 ...
0 1 2 3 4 5 6 7 8 ............ 15
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
│ type │flags │ codec │ ssrc (u32)
├──────┴──────┴──────┴────────────────────────────────────────────┤
│ seq (u16) │ timestamp (u32, in samples @48k) │ payload ... │
└─────────────┴──────────────────────────────────────────┴──────────────┘
│ type │flags │ codec │ ssrc (u32) seq (u64) ──▶
├──────┴──────┴──────┴────────────────────────┴─────────────────────┤
◀── seq (u64) ──┤ timestamp (u32 @48k) │ payload ...
└──────────────────┴────────────────────────────────────┴──────────────┘
bytes [8..15] = seq (u64) [16..19] = timestamp (u32)
type u8 1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present
@@ -49,11 +52,19 @@ flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present
codec u16 0 = OPUS (room for future codecs)
ssrc u32 media-plane stream id. Client sends its own ssrc; the server
validates it against the bound session and relays unchanged.
seq u16 per-ssrc sequence number, wraps; drives loss detection + reorder
seq u64 full monotonic send counter. This IS the AEAD nonce counter, so the
receiver derives the nonce directly from it — no rollover guessing.
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
payload one Opus packet (the encoder's output for one frame)
```
> **Why u64 (protocol v2).** v1 carried only the low 16 bits of the counter and the
> receiver zero-extended them to rebuild the AEAD nonce. After 65,536 frames the seq
> wrapped, the reconstructed nonce diverged from the sealing nonce, and **every frame
> failed authentication permanently** (no rollover counter). v2 puts the full 64-bit
> counter on the wire so the nonce is always exact. A v2 server and a v1 client cannot
> interoperate; the `Hello` handshake rejects on `proto_version` mismatch.
This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's
full machinery. The **server relays the payload unmodified** — it only reads the header to
route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once
@@ -204,6 +215,23 @@ Each receiver keeps an **adaptive jitter buffer per ssrc**.
`AVAudioSession`, then restarts the devices (`vc_audio_restart`) so they reopen against the
new route — mirroring TeamTalk5's `closeSoundDevices`/`initSoundInputDevice`/
`initSoundOutputDevice` pattern.
- **iOS voice processing (AEC/NS/AGC) — native VPIO path.** Real iOS echo cancellation, noise
suppression and AGC are provided ONLY by Apple's **Voice-Processing I/O audio unit (VPIO)**,
*not* by the `AVAudioSession` mode alone. The core uses miniaudio's plain `RemoteIO` audio
units, which never engage VPIO — so `.voiceChat` mode by itself yields no AEC. For VPIO to
cancel echo it must own BOTH the mic capture and the remote-audio playback (it subtracts the
played-back signal from the mic), so on the AEC presets (Voice Chat / Bluetooth Headset HFP /
Wired Headset) the Swift layer runs a native `AVAudioEngine` with
`inputNode.setVoiceProcessingEnabled(true)` and the core runs in **external mode**:
- **Mic:** the MIC stream is started with `vc_stream_desc.external_feed=1`; the VPIO input tap
feeds processed mic PCM via `vc_stream_feed_pcm`. The core skips its hardware capture device
(`AudioParams.external_capture`).
- **Playback:** `vc_set_external_playback(1)` makes the core skip its hardware playback device;
a mixer-timer thread drives decode+mix on a ~20 ms cadence and delivers the FINAL mixed PCM
via `vc_set_mixed_output_sink`. The Swift engine renders that through the VPIO output, so
VPIO has its echo-cancellation reference signal.
The Stereo Mic / Studio / A2DP presets keep the miniaudio path (they want raw / stereo /
no-AEC routing that VPIO can't provide — VPIO forces mono).
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
@@ -274,8 +302,24 @@ normal stream; only the *source* is platform-specific.
| Platform | Mechanism | Notes |
|----------|-----------|-------|
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | **Implemented.** Captures in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when the channel is mono — so a stereo music/screen-share channel gets genuine stereo end-to-end (no downmix). Whole-device capture, not process-specific — it inherently captures this app's own incoming voice mix along with everything else playing (an accepted self-echo-loop characteristic of desktop-audio capture, not a bug). Windows 10 2004+'s process-specific loopback (`AUDIOCLIENT_ACTIVATION_PARAMS`) would avoid this but miniaudio doesn't expose it — a future enhancement. |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). **Supports per-app audio selection** — see below. |
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). ReplayKit only ever delivers the *mixed* system stream as `.audioApp`, so **per-app filtering / VoiceOver exclusion is not possible on iOS** (it has no per-app granularity, unlike ScreenCaptureKit). |
### macOS detail — per-app audio selection
ScreenCaptureKit filters audio at the **application** level, so before sharing starts the user
picks a scope in `ScreenSharePickerSheet` (`clients/apple/macOS/VoiceCatMac/Sheets/`):
- **Everything** — whole display, the original behaviour (`SCContentFilter(display:excludingWindows:)`).
- **Only selected apps** — capture just the ticked apps (`init(display:including:exceptingWindows:)`).
- **All except selected apps** — capture everything but the ticked apps
(`init(display:excludingApplications:exceptingWindows:)`).
A dedicated **"Exclude screen reader (VoiceOver) audio"** toggle merges the screen-reader
process(es) into the exclude set (`ScreenAudioCapture.screenReaderBundleIDs` — VoiceOver plus
the speech-synthesis daemon that actually renders the spoken audio). The chosen
`ScreenAudioSelection` is passed into `ScreenAudioCapture`, which builds the matching
`SCContentFilter`. iOS/ReplayKit has no equivalent control (see the table note above).
### iOS detail

175
scripts/asc_api.py Executable file
View File

@@ -0,0 +1,175 @@
#!/usr/bin/env python3
"""asc_api.py — minimal App Store Connect API helper for ad-hoc device registration.
Used by scripts/dist-ios-adhoc.sh to register friends' device UDIDs before building an
ad-hoc IPA. Talks to the App Store Connect API directly: signs an ES256 JWT with your
Team Key .p8 (via the `cryptography` package — no PyJWT/requests needed) and calls the
REST endpoints with urllib from the standard library.
Subcommands:
register register one device UDID (idempotent — an already-registered UDID is OK)
list list all registered devices (and the count, against the 100/year cap)
Auth is the same for both, supplied via flags (the wrapper script passes them from the
ASC_KEY_ID / ASC_ISSUER_ID / ASC_KEY_PATH env vars):
--key-id the Key ID of the App Store Connect API key
--issuer-id the Issuer ID (Users and Access -> Integrations)
--key path to the AuthKey_XXXXXXXXXX.p8 file
Examples:
python3 scripts/asc_api.py list \
--key-id ABC123 --issuer-id 11111111-2222-... --key ~/.appstoreconnect/AuthKey_ABC123.p8
python3 scripts/asc_api.py register --udid 00008110-0011... --name "My iPhone" \
--key-id ABC123 --issuer-id 11111111-2222-... --key ~/.appstoreconnect/AuthKey_ABC123.p8
The .p8 is created at App Store Connect -> Users and Access -> Integrations ->
App Store Connect API -> Team Keys, with Admin or App Manager access. Keep it out of the
repo.
"""
import argparse
import base64
import json
import sys
import time
import urllib.error
import urllib.request
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec, utils
API_BASE = "https://api.appstoreconnect.apple.com"
def _b64url(data: bytes) -> str:
"""base64url without padding, as JWT requires."""
return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii")
def make_jwt(key_id: str, issuer_id: str, key_path: str) -> str:
"""Build a short-lived ES256 JWT for the App Store Connect API."""
with open(key_path, "rb") as fh:
private_key = serialization.load_pem_private_key(fh.read(), password=None)
if not isinstance(private_key, ec.EllipticCurvePrivateKey):
sys.exit(f"error: {key_path} is not an EC private key (.p8 from App Store Connect)")
now = int(time.time())
header = {"alg": "ES256", "kid": key_id, "typ": "JWT"}
# exp must be <= 20 minutes out; 10 minutes is comfortable.
payload = {"iss": issuer_id, "iat": now, "exp": now + 600, "aud": "appstoreconnect-v1"}
signing_input = f"{_b64url(json.dumps(header).encode())}.{_b64url(json.dumps(payload).encode())}"
der_sig = private_key.sign(signing_input.encode("ascii"), ec.ECDSA(hashes.SHA256()))
# JWS wants raw r||s (two 32-byte big-endian ints), not the ASN.1/DER openssl emits.
r, s = utils.decode_dss_signature(der_sig)
raw_sig = r.to_bytes(32, "big") + s.to_bytes(32, "big")
return f"{signing_input}.{_b64url(raw_sig)}"
def _request(method: str, path: str, token: str, body: dict | None = None):
"""Perform an authenticated API call. Returns (status_code, parsed_json|None)."""
url = path if path.startswith("http") else f"{API_BASE}{path}"
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Authorization", f"Bearer {token}")
if data is not None:
req.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(req) as resp:
raw = resp.read()
return resp.status, (json.loads(raw) if raw else None)
except urllib.error.HTTPError as exc:
raw = exc.read()
try:
parsed = json.loads(raw) if raw else None
except json.JSONDecodeError:
parsed = {"_raw": raw.decode("utf-8", "replace")}
return exc.code, parsed
def _errors_text(payload) -> str:
if isinstance(payload, dict) and payload.get("errors"):
return "; ".join(
f"{e.get('title', '')}: {e.get('detail', '')}".strip(": ")
for e in payload["errors"]
)
return json.dumps(payload)
def cmd_register(args, token: str) -> int:
body = {
"data": {
"type": "devices",
"attributes": {
"name": args.name or args.udid,
"platform": "IOS",
"udid": args.udid,
},
}
}
status, payload = _request("POST", "/v1/devices", token, body)
if status in (200, 201):
print(f" registered: {args.udid} ({args.name or args.udid})")
return 0
# A UDID that already exists comes back as a 409 conflict, or a 422 with an error
# detail mentioning the device already exists. Either way it's fine — idempotent.
text = _errors_text(payload)
if status == 409 or "already exist" in text.lower() or "already been taken" in text.lower():
print(f" already registered: {args.udid}")
return 0
print(f"error: failed to register {args.udid} (HTTP {status}): {text}", file=sys.stderr)
return 1
def cmd_list(args, token: str) -> int:
path = "/v1/devices?limit=200&sort=name"
rows = []
while path:
status, payload = _request("GET", path, token)
if status != 200:
print(f"error: list failed (HTTP {status}): {_errors_text(payload)}", file=sys.stderr)
return 1
for d in payload.get("data", []):
a = d.get("attributes", {})
rows.append((a.get("platform", "?"), a.get("status", "?"),
a.get("udid", "?"), a.get("name", "")))
path = (payload.get("links") or {}).get("next")
ios = [r for r in rows if r[0] == "IOS"]
for platform, dev_status, udid, name in rows:
print(f" [{platform:7}] {dev_status:8} {udid} {name}")
print(f"\n {len(rows)} device(s) total, {len(ios)} iOS (cap is 100 iOS/membership year)")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
sub = parser.add_subparsers(dest="command", required=True)
def add_auth(p):
p.add_argument("--key-id", required=True)
p.add_argument("--issuer-id", required=True)
p.add_argument("--key", required=True, help="path to AuthKey_*.p8")
p_reg = sub.add_parser("register", help="register one device UDID (idempotent)")
add_auth(p_reg)
p_reg.add_argument("--udid", required=True)
p_reg.add_argument("--name", default=None)
p_list = sub.add_parser("list", help="list registered devices")
add_auth(p_list)
args = parser.parse_args()
token = make_jwt(args.key_id, args.issuer_id, args.key)
if args.command == "register":
return cmd_register(args, token)
if args.command == "list":
return cmd_list(args, token)
return 2
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,88 @@
#!/usr/bin/env bash
# build-linux-binaries.sh — build stripped Linux server binaries locally using Docker.
#
# PRIMARY path: push to main (or trigger manually from the GitHub Actions tab) and
# download the artifacts from .github/workflows/build-linux.yml — no local disk
# pressure, both amd64 and arm64 handled in the cloud.
#
# LOCAL path (this script): uses Docker + buildx with BuildKit cache. Requires
# ~1015 GB of free disk for the vcpkg build cache. Fine on a Linux dev machine;
# on Windows/macOS prefer the GitHub Actions path to avoid filling your Docker VM disk.
#
# Usage:
# ./scripts/build-linux-binaries.sh # build both arches
# ./scripts/build-linux-binaries.sh amd64 # build one arch only
# ./scripts/build-linux-binaries.sh arm64
#
# Output:
# dist/linux-amd64/{voicecat-server,voicecat-admin}
# dist/linux-arm64/{voicecat-server,voicecat-admin}
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
DIST_DIR="${REPO_ROOT}/dist"
# ── arg parsing ────────────────────────────────────────────────────────────────
case "${1:-both}" in
amd64) ARCHS=("amd64") ;;
arm64) ARCHS=("arm64") ;;
both) ARCHS=("amd64" "arm64") ;;
*) echo "Usage: $0 [amd64|arm64|both]" >&2; exit 1 ;;
esac
# ── pre-flight ─────────────────────────────────────────────────────────────────
if ! command -v docker &>/dev/null; then
echo "Error: docker not found." >&2
echo "Install Docker Desktop: https://docs.docker.com/get-docker/" >&2
echo "Or use GitHub Actions (push to main and download artifacts)." >&2
exit 1
fi
if ! docker buildx version &>/dev/null; then
echo "Error: docker buildx not available." >&2
echo "Docker Desktop ships with buildx. On Linux: docker buildx install" >&2
exit 1
fi
echo "Note: first run compiles vcpkg packages from source (~10-15 GB build cache)."
echo "On Windows/macOS, consider using GitHub Actions instead (Actions → Build Linux Binaries → Run workflow)."
echo ""
# ── ensure a builder with multi-platform support ───────────────────────────────
BUILDER="voicecat-builder"
if ! docker buildx inspect "${BUILDER}" &>/dev/null; then
echo "→ Creating buildx builder '${BUILDER}' (docker-container driver)..."
docker buildx create --name "${BUILDER}" --driver docker-container --bootstrap
fi
docker buildx use "${BUILDER}"
# ── build each arch ────────────────────────────────────────────────────────────
for ARCH in "${ARCHS[@]}"; do
PLATFORM="linux/${ARCH}"
OUT_DIR="${DIST_DIR}/linux-${ARCH}"
mkdir -p "${OUT_DIR}"
echo "══ Building ${PLATFORM} ══════════════════════════════════════════════════"
if [[ "${ARCH}" == "arm64" ]] && [[ "$(uname -m)" != "aarch64" ]]; then
echo " (Running under QEMU on a non-arm64 host — will be slow)"
fi
docker buildx build \
--platform "${PLATFORM}" \
--target export \
--output "type=local,dest=${OUT_DIR}" \
--progress plain \
"${REPO_ROOT}"
echo "${PLATFORM} binaries:"
ls -lh "${OUT_DIR}/"
echo ""
done
echo "════════════════════════════════════════════════════════════════════════════"
echo "Done. Binaries in dist/:"
for ARCH in "${ARCHS[@]}"; do
ls -lh "${DIST_DIR}/linux-${ARCH}/"
done

285
scripts/dist-ios-adhoc.sh Executable file
View File

@@ -0,0 +1,285 @@
#!/usr/bin/env bash
#
# dist-ios-adhoc.sh — build an ad-hoc IPA of the iOS client and the files needed to
# install it on registered devices from an HTTPS web page (itms-services OTA install).
#
# This is for handing the app to a handful of friends BEFORE TestFlight. Ad-hoc builds
# only run on devices whose UDID is registered in your Apple Developer account, so the
# script registers UDIDs (via the App Store Connect API) before it signs.
#
# Pipeline:
# 1. Register each --udid with App Store Connect (idempotent; skip with --skip-register).
# 2. Build VoiceCatCore.xcframework (iOS device slice).
# 3. xcodebuild archive (Release, generic/platform=iOS), letting Xcode auto-create the
# ad-hoc Distribution cert + profile via -allowProvisioningUpdates + the API key.
# 4. xcodebuild -exportArchive with method=release-testing (Xcode's modern name for
# "ad-hoc") -> VoiceCatiOS.ipa.
# 5. Generate manifest.plist (OTA install manifest) + index.html (install page).
# 6. Stage VoiceCatiOS.ipa + manifest.plist + index.html into dist/ios-adhoc/.
#
# You then upload those three files to your HTTPS host and open index.html on the iPhone.
#
# Prerequisites (one-time, not scriptable):
# - Paid Apple Developer Program membership (Team ID is already set in the project).
# - An App Store Connect API "Team Key" (.p8) with Admin or App Manager access:
# App Store Connect -> Users and Access -> Integrations -> App Store Connect API.
# Note its Key ID and Issuer ID. Keep the .p8 out of the repo.
# - Each friend's device UDID (read it in Finder with the iPhone connected to a Mac).
#
# Auth — supply via env vars (or the matching flags):
# ASC_KEY_ID App Store Connect API Key ID (--key-id)
# ASC_ISSUER_ID App Store Connect API Issuer ID (--issuer-id)
# ASC_KEY_PATH path to AuthKey_XXXXXXXXXX.p8 (--key)
#
# Usage:
# scripts/dist-ios-adhoc.sh --udid <UDID> --name "Friend iPhone" \
# --base-url https://example.com/voicecat
# scripts/dist-ios-adhoc.sh --udids-file friends.csv --base-url https://example.com/vc
# scripts/dist-ios-adhoc.sh --skip-register --base-url https://example.com/vc # rebuild only
# scripts/dist-ios-adhoc.sh --dist /out --no-configure
# scripts/dist-ios-adhoc.sh -h|--help
#
# Flags:
# --udid <UDID> device to register (repeatable). Pair with an optional --name.
# --name <label> label for the immediately-preceding --udid (default: the UDID).
# --udids-file <file> CSV of "udid,name" lines (one device per line; blank/# lines skipped).
# --base-url <url> public HTTPS folder you'll upload the output to. Used to fill
# manifest.plist + index.html. If omitted, a __BASE_URL__ placeholder
# is written and you must edit both files before they work.
# --skip-register don't touch App Store Connect; just rebuild/export/stage.
# --key-id / --issuer-id / --key override the ASC_* env vars.
# --dist <path> output root (default <repo>/dist). Files land in <dist>/ios-adhoc/.
# --no-configure skip the cmake configure step in the xcframework build.
# -h, --help this help.
#
# macOS only. Requires Xcode, VCPKG_ROOT (or a configured build/apple-dev cache), and the
# Python `cryptography` package (already present in a standard macOS Python 3).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VC_SCRIPT_NAME="dist-ios-adhoc"
source "$SCRIPT_DIR/common.sh"
# ── Project constants (see clients/apple/iOS/VoiceCatiOS.xcodeproj) ───────────────
TEAM_ID="FJV8L966W4"
APP_BUNDLE_ID="cat.voice.VoiceCatiOS"
SCHEME="VoiceCatiOS"
APP_TITLE="VoiceCat"
XCODEPROJ="$VC_REPO_ROOT/clients/apple/iOS/VoiceCatiOS.xcodeproj"
# ── Args ──────────────────────────────────────────────────────────────────────────
KEY_ID="${ASC_KEY_ID:-}"
ISSUER_ID="${ASC_ISSUER_ID:-}"
KEY_PATH="${ASC_KEY_PATH:-}"
BASE_URL=""
SKIP_REGISTER=false
DO_CONFIGURE=true
UDIDS=() # parallel arrays: UDIDS[i] / UDID_NAMES[i]
UDID_NAMES=()
while [[ $# -gt 0 ]]; do
case "$1" in
--udid) UDIDS+=("$2"); UDID_NAMES+=("$2"); shift 2 ;;
--name) [[ ${#UDID_NAMES[@]} -gt 0 ]] || vc_die "--name must follow a --udid"
UDID_NAMES[${#UDID_NAMES[@]}-1]="$2"; shift 2 ;;
--udids-file) [[ -f "$2" ]] || vc_die "udids file not found: $2"
while IFS=',' read -r u n _; do
u="${u// /}"
[[ -z "$u" || "$u" == \#* ]] && continue
UDIDS+=("$u"); UDID_NAMES+=("${n:-$u}")
done < "$2"
shift 2 ;;
--base-url) BASE_URL="${2%/}"; shift 2 ;;
--skip-register) SKIP_REGISTER=true; shift ;;
--key-id) KEY_ID="$2"; shift 2 ;;
--issuer-id) ISSUER_ID="$2"; shift 2 ;;
--key) KEY_PATH="$2"; shift 2 ;;
--dist) vc_set_dist "$2"; shift 2 ;;
--no-configure) DO_CONFIGURE=false; shift ;;
-h|--help) vc_print_help "$0"; exit 0 ;;
*) vc_die "unknown arg: $1 (try --help)" ;;
esac
done
vc_require_macos
# Auth is needed for device registration AND for -allowProvisioningUpdates (Xcode uses the
# same key to mint the ad-hoc Distribution cert/profile).
[[ -n "$KEY_ID" ]] || vc_die "missing API Key ID (set ASC_KEY_ID or pass --key-id)"
[[ -n "$ISSUER_ID" ]] || vc_die "missing Issuer ID (set ASC_ISSUER_ID or pass --issuer-id)"
[[ -n "$KEY_PATH" ]] || vc_die "missing .p8 path (set ASC_KEY_PATH or pass --key)"
[[ -f "$KEY_PATH" ]] || vc_die "API key file not found: $KEY_PATH"
python3 -c "import cryptography" 2>/dev/null \
|| vc_die "Python 'cryptography' package not available: pip3 install cryptography"
if [[ -z "$BASE_URL" ]]; then
BASE_URL="__BASE_URL__"
vc_log "no --base-url given: manifest.plist + index.html will use the __BASE_URL__"
vc_log "placeholder — edit both files to your real HTTPS folder before uploading."
fi
OUT="$(vc_dist_subdir ios-adhoc)"
ARCHIVE="$OUT/VoiceCatiOS.xcarchive"
EXPORT_DIR="$OUT/export"
DERIVED="$OUT/DerivedData"
IPA="$OUT/VoiceCatiOS.ipa"
# ── Step 1: register devices ───────────────────────────────────────────────────────
if $SKIP_REGISTER; then
vc_step "skipping device registration (--skip-register)"
elif [[ ${#UDIDS[@]} -eq 0 ]]; then
vc_step "no --udid/--udids-file given — skipping device registration"
vc_log "(devices must already be registered, or this IPA won't install for them)"
else
vc_step "register ${#UDIDS[@]} device(s) with App Store Connect"
for i in "${!UDIDS[@]}"; do
python3 "$SCRIPT_DIR/asc_api.py" register \
--key-id "$KEY_ID" --issuer-id "$ISSUER_ID" --key "$KEY_PATH" \
--udid "${UDIDS[$i]}" --name "${UDID_NAMES[$i]}"
done
vc_ok "device registration complete"
fi
# ── Step 2: xcframework (iOS device slice) ──────────────────────────────────────────
vc_step "build VoiceCatCore.xcframework (iOS device slice)"
vc_resolve_vcpkg_root "apple-dev"
XCFW_ARGS=( --preset apple-ios )
$DO_CONFIGURE || XCFW_ARGS+=( --no-configure )
"$VC_REPO_ROOT/clients/apple/scripts/build-xcframework.sh" "${XCFW_ARGS[@]}"
XCFW="$VC_REPO_ROOT/clients/apple/VoiceCatCore.xcframework"
[[ -d "$XCFW" ]] || vc_die "xcframework not found at $XCFW"
vc_ok "xcframework ready -> $XCFW"
# ── Step 3: archive ─────────────────────────────────────────────────────────────────
vc_step "xcodebuild archive ($SCHEME / Release / generic iOS)"
xcodebuild archive \
-project "$XCODEPROJ" \
-scheme "$SCHEME" \
-configuration Release \
-destination 'generic/platform=iOS' \
-archivePath "$ARCHIVE" \
-derivedDataPath "$DERIVED" \
-allowProvisioningUpdates \
-authenticationKeyPath "$KEY_PATH" \
-authenticationKeyID "$KEY_ID" \
-authenticationKeyIssuerID "$ISSUER_ID"
[[ -d "$ARCHIVE" ]] || vc_die "archive not produced at $ARCHIVE"
vc_ok "archived -> $ARCHIVE"
# ── Step 4: export ad-hoc IPA ───────────────────────────────────────────────────────
# method "release-testing" is Xcode 15.3+/26's name for the old "ad-hoc" method.
# manageAppGroups lets automatic signing carry the group.cat.voice.VoiceCat capability
# (used by the ReplayKit broadcast extension) into the generated profiles.
EXPORT_PLIST="$OUT/ExportOptions.plist"
cat > "$EXPORT_PLIST" <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>method</key> <string>release-testing</string>
<key>teamID</key> <string>$TEAM_ID</string>
<key>signingStyle</key> <string>automatic</string>
<key>manageAppGroups</key> <true/>
<key>stripSwiftSymbols</key> <true/>
</dict>
</plist>
PLIST
vc_step "xcodebuild -exportArchive (method=release-testing / ad-hoc)"
xcodebuild -exportArchive \
-archivePath "$ARCHIVE" \
-exportPath "$EXPORT_DIR" \
-exportOptionsPlist "$EXPORT_PLIST" \
-allowProvisioningUpdates \
-authenticationKeyPath "$KEY_PATH" \
-authenticationKeyID "$KEY_ID" \
-authenticationKeyIssuerID "$ISSUER_ID"
EXPORTED_IPA="$(find "$EXPORT_DIR" -maxdepth 1 -name '*.ipa' | head -n1)"
[[ -n "$EXPORTED_IPA" ]] || vc_die "no .ipa found in $EXPORT_DIR"
mv "$EXPORTED_IPA" "$IPA"
vc_ok "exported -> $IPA ($(du -h "$IPA" | cut -f1))"
# ── Step 5: read version + generate manifest.plist & index.html ─────────────────────
APP_IN_ARCHIVE="$(find "$ARCHIVE/Products/Applications" -maxdepth 1 -name '*.app' | head -n1)"
SHORT_VER="$(/usr/libexec/PlistBuddy -c 'Print :CFBundleShortVersionString' \
"$APP_IN_ARCHIVE/Info.plist" 2>/dev/null || echo "1.0")"
MANIFEST="$OUT/manifest.plist"
cat > "$MANIFEST" <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>items</key>
<array>
<dict>
<key>assets</key>
<array>
<dict>
<key>kind</key> <string>software-package</string>
<key>url</key> <string>$BASE_URL/VoiceCatiOS.ipa</string>
</dict>
</array>
<key>metadata</key>
<dict>
<key>bundle-identifier</key> <string>$APP_BUNDLE_ID</string>
<key>bundle-version</key> <string>$SHORT_VER</string>
<key>kind</key> <string>software</string>
<key>title</key> <string>$APP_TITLE</string>
</dict>
</dict>
</array>
</dict>
</plist>
PLIST
INDEX="$OUT/index.html"
cat > "$INDEX" <<HTML
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Install $APP_TITLE</title>
<style>
body { font: 17px -apple-system, system-ui, sans-serif; margin: 0; padding: 2.5rem 1.5rem;
color: #111; background: #f5f5f7; -webkit-text-size-adjust: 100%; }
main { max-width: 30rem; margin: 0 auto; }
h1 { font-size: 1.6rem; }
.btn { display: block; text-align: center; text-decoration: none; margin: 1.5rem 0;
padding: 0.9rem 1rem; border-radius: 0.8rem; background: #0a84ff; color: #fff;
font-weight: 600; }
.note { font-size: 0.9rem; color: #555; line-height: 1.5; }
code { background: #e5e5ea; padding: 0.1rem 0.3rem; border-radius: 0.3rem; }
</style>
</head>
<body>
<main>
<h1>$APP_TITLE — test build</h1>
<p>Tap below on your <strong>iPhone</strong> to install (version $SHORT_VER).</p>
<a class="btn" href="itms-services://?action=download-manifest&amp;url=$BASE_URL/manifest.plist">
Install $APP_TITLE
</a>
<p class="note">
Only works on devices whose UDID was registered for this build, on iOS 18 or later.
If nothing happens, make sure you opened this page in <strong>Safari</strong> and that
it is served over <strong>HTTPS</strong>. After installing, the app launches normally —
no "trust developer" step is needed.
</p>
</main>
</body>
</html>
HTML
# ── Done ────────────────────────────────────────────────────────────────────────────
vc_ok "staged install files in $OUT:"
printf ' %s\n' "VoiceCatiOS.ipa" "manifest.plist" "index.html"
echo
if [[ "$BASE_URL" == "__BASE_URL__" ]]; then
vc_log "NEXT: replace __BASE_URL__ in manifest.plist and index.html with your HTTPS"
vc_log " folder URL, then upload all three files there."
else
vc_log "NEXT: upload the three files above to $BASE_URL/"
vc_log " then open $BASE_URL/index.html in Safari on a registered iPhone."
fi

View File

@@ -257,13 +257,13 @@ void ConnSession::send_generic_result(uint64_t req_id, bool ok, uint32_t code,
// ── Handlers ─────────────────────────────────────────────────────────────────
void ConnSession::handle_client_hello(uint64_t req_id, const voicecat::v1::ClientHello& msg) {
if (msg.proto_version() != 1) {
if (msg.proto_version() != 2) { // protocol v2: 64-bit voice seq (docs/voice.md §2)
send_disconnect_and_close(1, "unsupported protocol version");
return;
}
auto env = make_env(req_id);
auto* hello = env.mutable_server_hello();
hello->set_proto_version(1);
hello->set_proto_version(2);
hello->set_server_name("VoiceCat Server");
hello->set_server_version("0.1.0");
if (allow_guests_) hello->add_auth_methods("guest");

View File

@@ -3,7 +3,9 @@
#ifdef VOICECAT_HAS_NET
#include <array>
#include <chrono>
#include <cstdio>
#include <string_view>
#include "conn_session.h"
#include "crypto/crypto.h"
@@ -35,6 +37,24 @@ uint16_t MediaRelay::media_port() const {
return static_cast<uint16_t>(udp_.local_endpoint().port());
}
void MediaRelay::note_drop(const char* reason) {
if (std::string_view(reason) == "unmapped-endpoint") ++drop_no_endpoint_;
else if (std::string_view(reason) == "no-recv-crypto") ++drop_no_crypto_;
else ++drop_open_failed_;
// Rate-limit the summary to at most once every 5s so a flood can't spam the log.
auto now_ms = std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now().time_since_epoch()).count();
if (now_ms - last_drop_log_ms_ < 5000) return;
last_drop_log_ms_ = now_ms;
std::fprintf(stderr,
"[media] dropped frames — unmapped-endpoint=%llu no-recv-crypto=%llu "
"open-failed(auth/replay)=%llu\n",
static_cast<unsigned long long>(drop_no_endpoint_),
static_cast<unsigned long long>(drop_no_crypto_),
static_cast<unsigned long long>(drop_open_failed_));
}
void MediaRelay::on_udp_frame(const uint8_t* data, size_t len,
asio::ip::udp::endpoint sender) {
if (len < 1) return;
@@ -59,10 +79,10 @@ void MediaRelay::on_udp_frame(const uint8_t* data, size_t len,
// Resolve sender session.
auto sender_session = registry_->find_by_udp_endpoint(sender);
if (!sender_session) return;
if (!sender_session) { note_drop("unmapped-endpoint"); return; }
auto* recv_crypto = sender_session->recv_crypto();
if (!recv_crypto) return;
if (!recv_crypto) { note_drop("no-recv-crypto"); return; }
// AAD = 14-byte header (authenticated, not encrypted).
const uint8_t* aad = data;
@@ -73,7 +93,7 @@ void MediaRelay::on_udp_frame(const uint8_t* data, size_t len,
long plain_len = recv_crypto->open(sealed, sealed_len, aad, voicecat::net::kVoiceHeaderSize,
plain_buf_.data(), plain_buf_.size());
if (plain_len < 0) return; // auth failure or replay
if (plain_len < 0) { note_drop("open-failed"); return; } // auth failure or replay
// Parse the voice frame header to find the source ssrc/channel.
voicecat::net::VoiceFrame hdr{};
@@ -111,8 +131,14 @@ void MediaRelay::on_udp_frame(const uint8_t* data, size_t len,
// so the header (which is the authenticated AAD) carries the matching counter.
std::memcpy(seal_buf_.data(), data, voicecat::net::kVoiceHeaderSize);
const uint64_t send_ctr = send_crypto->peek_send_counter();
seal_buf_[8] = static_cast<uint8_t>((send_ctr >> 8) & 0xFF);
seal_buf_[9] = static_cast<uint8_t>(send_ctr & 0xFF);
seal_buf_[8] = static_cast<uint8_t>((send_ctr >> 56) & 0xFF);
seal_buf_[9] = static_cast<uint8_t>((send_ctr >> 48) & 0xFF);
seal_buf_[10] = static_cast<uint8_t>((send_ctr >> 40) & 0xFF);
seal_buf_[11] = static_cast<uint8_t>((send_ctr >> 32) & 0xFF);
seal_buf_[12] = static_cast<uint8_t>((send_ctr >> 24) & 0xFF);
seal_buf_[13] = static_cast<uint8_t>((send_ctr >> 16) & 0xFF);
seal_buf_[14] = static_cast<uint8_t>((send_ctr >> 8) & 0xFF);
seal_buf_[15] = static_cast<uint8_t>(send_ctr & 0xFF);
uint8_t* out_payload = seal_buf_.data() + voicecat::net::kVoiceHeaderSize;
long sealed_out = send_crypto->seal(

View File

@@ -49,10 +49,20 @@ class MediaRelay {
private:
void on_udp_frame(const uint8_t* data, size_t len, asio::ip::udp::endpoint sender);
// Count a dropped inbound voice frame (by reason) and emit a rate-limited summary
// to stderr. Runs on the io thread, so plain counters are safe.
void note_drop(const char* reason);
asio::io_context& io_;
std::shared_ptr<SessionRegistry> registry_;
voicecat::net::UdpMediaChannel udp_;
// Diagnostics: dropped-frame counters so a wedged media path is observable.
uint64_t drop_no_endpoint_ = 0; // voice from an unmapped UDP endpoint
uint64_t drop_no_crypto_ = 0; // session has no recv_crypto yet
uint64_t drop_open_failed_ = 0; // AEAD auth failure or replay reject
int64_t last_drop_log_ms_ = 0;
// Scratch buffer for re-encrypted payloads (size = max_frame + 16 MAC)
static constexpr size_t kMaxPayload = 1500;
std::vector<uint8_t> seal_buf_ = std::vector<uint8_t>(kMaxPayload + 16, uint8_t{0});

View File

@@ -100,6 +100,22 @@ void SessionRegistry::unregister_session(uint64_t session_id) {
std::unique_lock lk(mu_);
sessions_.erase(session_id);
session_permissions_.erase(session_id);
// Free the per-session UDP/media state too. These maps are keyed by
// endpoint/token/ssrc (not session id), so scan-and-erase by value. Leaving them
// behind leaks entries and lets a stale endpoint/token resolve toward a dead
// session across reconnects (e.g. a wifi-handoff rebind from a new port).
auto erase_by_value = [session_id](auto& map) {
for (auto it = map.begin(); it != map.end();) {
if (it->second == session_id)
it = map.erase(it);
else
++it;
}
};
erase_by_value(udp_endpoints_);
erase_by_value(udp_tokens_);
erase_by_value(ssrc_to_session_);
}
uint32_t SessionRegistry::add_user(uint64_t session_id, const voicecat::v1::User& user) {

View File

@@ -69,6 +69,14 @@ if(VOICECAT_USE_VCPKG_DEPS)
target_include_directories(test_plc_cap PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME plc_cap COMMAND test_plc_cap)
# External playback (iOS VPIO): the mixer-timer thread drives decode+mix with NO hardware
# device and delivers the final mix to the mixed-output sink. White-box AudioEngine test.
add_executable(test_external_playback test_external_playback.cpp)
target_link_libraries(test_external_playback PRIVATE voicecat::voicecat)
target_compile_features(test_external_playback PRIVATE cxx_std_20)
target_include_directories(test_external_playback PRIVATE ${VC_TEST_INTERNAL_INCLUDES})
add_test(NAME external_playback COMMAND test_external_playback)
# M2 exit criterion: two headless clients relay encrypted Opus frames via the SFU.
add_executable(test_m2_voice test_m2_voice.cpp)
target_link_libraries(test_m2_voice PRIVATE voicecat::server)

View File

@@ -0,0 +1,146 @@
/*
* test_external_playback — verifies AudioEngine's external-playback mode (iOS VPIO path).
*
* When external playback is enabled, the engine opens NO hardware playback device; a mixer-timer
* thread drives decode+mix on a ~20ms cadence and delivers the FINAL mixed PCM to the
* mixed-output sink (the Swift AVAudioEngine VPIO renderer consumes this). This test asserts:
* 1. The mixed sink fires steadily on the timer thread (count grows over time) with the right
* format (48kHz, stereo), and carries real energy while a stream is being decoded.
* 2. The per-stream pcm_sink still fires concurrently (both taps coexist).
* 3. With no remote streams, the mixed sink KEEPS firing (silent-but-present blocks) so the
* renderer has a continuous clock.
*
* White-box: constructs AudioEngine directly (no server, no audio hardware needed) — the timer
* thread drives the mixer with no ma_device, which is the core new behavior under test.
*/
#include <atomic>
#include <chrono>
#include <cmath>
#include <cstdio>
#include <thread>
#include <vector>
#if defined(VOICECAT_HAS_AUDIO) && defined(VOICECAT_HAS_OPUS)
#include "audio/audio_engine.h"
#include "codec/opus_codec.h"
static int g_failures = 0;
#define CHECK(cond) \
do { \
if (!(cond)) { \
std::printf("FAIL [%s:%d]: %s\n", __FILE__, __LINE__, #cond); \
++g_failures; \
} \
} while (0)
// Shared state written by the sink callbacks (timer thread) and read by main.
struct MixedSinkState {
std::atomic<int> calls{0};
std::atomic<int64_t> max_energy{0};
std::atomic<uint32_t> last_channels{0};
std::atomic<uint32_t> last_sample_rate{0};
};
static MixedSinkState g_mixed;
static std::atomic<int> g_pcm_sink_calls{0};
static void mixed_cb(void* user, const int16_t* pcm, size_t spc, uint32_t ch, uint32_t sr) {
auto* s = static_cast<MixedSinkState*>(user);
s->calls.fetch_add(1, std::memory_order_relaxed);
s->last_channels.store(ch, std::memory_order_relaxed);
s->last_sample_rate.store(sr, std::memory_order_relaxed);
int64_t e = 0;
for (size_t i = 0; i < spc * ch; ++i) e += std::abs(static_cast<int>(pcm[i]));
int64_t prev = s->max_energy.load(std::memory_order_relaxed);
while (e > prev && !s->max_energy.compare_exchange_weak(prev, e, std::memory_order_relaxed)) {
}
}
static void pcm_cb(void*, uint32_t, uint32_t, const int16_t*, size_t, uint32_t, uint32_t) {
g_pcm_sink_calls.fetch_add(1, std::memory_order_relaxed);
}
int main() {
voicecat::audio::AudioEngine engine;
engine.set_external_playback(true);
engine.set_mixed_output_sink(&mixed_cb, &g_mixed);
engine.set_pcm_sink(&pcm_cb, nullptr);
voicecat::audio::AudioParams p;
p.sample_rate = 48000;
p.capture_channels = 1;
p.playback_channels = 2;
p.frame_ms = 20;
CHECK(engine.start(p)); // no hardware device opened — the timer thread drives the mixer
voicecat::codec::OpusParams op;
op.sample_rate = 48000;
op.frame_ms = 20;
op.stereo = false;
int frame_samples = voicecat::codec::opus_frame_samples(op); // 960
voicecat::codec::OpusEncoder enc;
CHECK(enc.init(op));
std::vector<int16_t> sine(static_cast<size_t>(frame_samples));
for (int i = 0; i < frame_samples; ++i) {
float t = static_cast<float>(i) / 48000.0f;
sine[i] = static_cast<int16_t>(std::sin(2.0f * 3.14159265f * 440.0f * t) * 20000.0f);
}
uint8_t opus_buf[1500];
int opus_len = enc.encode(sine.data(), frame_samples, opus_buf, sizeof(opus_buf));
CHECK(opus_len > 0);
const uint32_t ssrc = 1;
engine.init_recv_stream(ssrc, op, /*user_id=*/7, /*stream_id=*/3);
// ── Phase 1: feed ~600ms of real frames; the timer must decode + mix them. ──────────
uint32_t ts = 0;
for (int i = 0; i < 30; ++i) { // 30 * 20ms = 600ms of audio
voicecat::audio::JitterBuffer::Frame f;
f.seq = static_cast<uint64_t>(i);
f.timestamp = ts;
f.fec_present = false;
f.payload.assign(opus_buf, opus_buf + opus_len);
engine.push_recv_frame(ssrc, std::move(f));
ts += static_cast<uint32_t>(frame_samples);
}
std::this_thread::sleep_for(std::chrono::milliseconds(500));
int active_calls = g_mixed.calls.load(std::memory_order_relaxed);
std::printf("external_playback: phase1 mixed-sink calls=%d max_energy=%lld pcm_sink=%d\n",
active_calls, static_cast<long long>(g_mixed.max_energy.load()),
g_pcm_sink_calls.load());
// ~25 blocks expected at a 20ms cadence over 500ms; allow generous slack for scheduler/debug.
CHECK(active_calls >= 10);
CHECK(g_mixed.max_energy.load(std::memory_order_relaxed) > 0); // real decoded audio in the mix
CHECK(g_mixed.last_channels.load(std::memory_order_relaxed) == 2);
CHECK(g_mixed.last_sample_rate.load(std::memory_order_relaxed) == 48000);
CHECK(g_pcm_sink_calls.load(std::memory_order_relaxed) > 0); // per-stream tap coexists
// ── Phase 2: remove the stream; the mixed sink must KEEP firing (silent blocks). ─────
engine.remove_stream(ssrc);
int before = g_mixed.calls.load(std::memory_order_relaxed);
std::this_thread::sleep_for(std::chrono::milliseconds(200));
int after = g_mixed.calls.load(std::memory_order_relaxed);
std::printf("external_playback: phase2 silent blocks delivered=%d\n", after - before);
CHECK(after - before >= 5); // continuous clock even with nothing to play
engine.stop(); // joins the mixer-timer thread
enc.destroy();
if (g_failures == 0) {
std::printf("external_playback: all checks passed\n");
return 0;
}
std::printf("external_playback: %d failure(s)\n", g_failures);
return 1;
}
#else
int main() {
std::printf("external_playback: SKIP (VOICECAT_HAS_AUDIO or VOICECAT_HAS_OPUS not defined)\n");
return 0;
}
#endif

View File

@@ -212,7 +212,7 @@ struct TestClient {
{
v1::Envelope env;
env.set_request_id(1);
env.mutable_client_hello()->set_proto_version(1);
env.mutable_client_hello()->set_proto_version(2);
env.mutable_client_hello()->set_client_name(label);
if (!tcp_send_envelope(*tls, env)) return false;
}
@@ -450,10 +450,10 @@ int main() {
size_t payload_len = pcm.size() * sizeof(int16_t);
#endif
// Build 14-byte header (AAD).
// Build the voice frame header (AAD).
VoiceFrame hdr;
hdr.ssrc = A.assigned_ssrc;
hdr.seq = static_cast<uint16_t>(i);
hdr.seq = static_cast<uint64_t>(i);
hdr.timestamp = static_cast<uint32_t>(i * kFrameSamples);
uint8_t header_bytes[kVoiceHeaderSize];
serialize_header(hdr, header_bytes);

View File

@@ -24,8 +24,8 @@ static int g_failures = 0;
++g_failures; \
}} while (0)
// Build a synthetic 14-byte AAD (voice frame header).
static std::vector<uint8_t> make_aad(uint16_t seq) {
// Build a synthetic voice-frame-header AAD.
static std::vector<uint8_t> make_aad(uint64_t seq) {
VoiceFrame f;
f.ssrc = 0xCAFEBABE;
f.seq = seq;
@@ -132,7 +132,7 @@ static void test_multiple_packets() {
std::vector<uint8_t> plain(60, 0x99);
for (uint16_t seq = 0; seq < 10; ++seq) {
for (uint64_t seq = 0; seq < 10; ++seq) {
auto aad = make_aad(seq);
std::vector<uint8_t> cipher(plain.size() + crypto_aead_chacha20poly1305_ietf_ABYTES);
long sealed_len = sender.seal(plain.data(), plain.size(),
@@ -149,8 +149,8 @@ static void test_multiple_packets() {
}
}
// Build a 14-byte AAD (voice frame header) with a given ssrc + seq.
static std::vector<uint8_t> make_aad_ssrc(uint32_t ssrc, uint16_t seq) {
// Build a voice-frame-header AAD with a given ssrc + seq.
static std::vector<uint8_t> make_aad_ssrc(uint32_t ssrc, uint64_t seq) {
VoiceFrame f;
f.ssrc = ssrc;
f.seq = seq;
@@ -159,6 +159,12 @@ static std::vector<uint8_t> make_aad_ssrc(uint32_t ssrc, uint16_t seq) {
return aad;
}
// Overwrite the 8-byte big-endian seq field (header bytes [8..15]) in an AAD buffer.
static void set_aad_seq(std::vector<uint8_t>& aad, uint64_t seq) {
for (int i = 0; i < 8; ++i)
aad[8 + i] = static_cast<uint8_t>((seq >> (56 - 8 * i)) & 0xFF);
}
// Simulate one server relay hop for a single frame, sender → recipient R.
// - sender seals with its send key, setting header seq = its own send counter (client contract).
// - server opens with the sender's key, then re-seals with R's send key.
@@ -169,7 +175,7 @@ static bool relay_one(SodiumMediaCrypto& sender_send, SodiumMediaCrypto& server_
SodiumMediaCrypto& r_send, SodiumMediaCrypto& r_recv,
uint32_t ssrc, const std::vector<uint8_t>& plain, bool rewrite_seq) {
// Client A→server: seq carries the sender's send counter.
auto in_aad = make_aad_ssrc(ssrc, static_cast<uint16_t>(sender_send.peek_send_counter()));
auto in_aad = make_aad_ssrc(ssrc, sender_send.peek_send_counter());
std::vector<uint8_t> cipher(plain.size() + crypto_aead_chacha20poly1305_ietf_ABYTES);
long sealed = sender_send.seal(plain.data(), plain.size(), in_aad.data(), in_aad.size(),
cipher.data(), cipher.size());
@@ -185,11 +191,7 @@ static bool relay_one(SodiumMediaCrypto& sender_send, SodiumMediaCrypto& server_
// Server re-seals to R. Header passes through except seq, which (when fixed) is set to R's
// own send counter so R's open() reconstructs the matching nonce.
std::vector<uint8_t> out_aad = in_aad; // copy header verbatim
if (rewrite_seq) {
uint64_t ctr = r_send.peek_send_counter();
out_aad[8] = static_cast<uint8_t>((ctr >> 8) & 0xFF);
out_aad[9] = static_cast<uint8_t>(ctr & 0xFF);
}
if (rewrite_seq) set_aad_seq(out_aad, r_send.peek_send_counter());
std::vector<uint8_t> relay_cipher(recovered.size() + crypto_aead_chacha20poly1305_ietf_ABYTES);
long resealed = r_send.seal(recovered.data(), static_cast<size_t>(opened),
out_aad.data(), out_aad.size(),
@@ -251,6 +253,79 @@ static void test_relay_interleaved_reseal() {
}
}
// Regression for the bad-wifi wedge: the anti-replay window must NOT be advanced by a
// packet that fails authentication. A single corrupted/forged frame carrying a huge seq
// used to shove recv_highest_ far ahead (before the AEAD tag was checked), after which
// every legitimate frame was rejected as "too old" — permanent silence. open() now
// advances the window only after a successful tag check (RFC 3711 §3.3).
static void test_corrupted_seq_does_not_poison_window() {
uint8_t key[crypto_aead_chacha20poly1305_ietf_KEYBYTES];
crypto_generichash(key, sizeof(key),
reinterpret_cast<const uint8_t*>("poison-key"), 10, nullptr, 0);
SodiumMediaCrypto sender(key);
SodiumMediaCrypto receiver(key);
std::vector<uint8_t> plain(64, 0x5A);
std::vector<uint8_t> recovered(plain.size());
auto seal_at_current = [&](std::vector<uint8_t>& aad_out, std::vector<uint8_t>& cipher_out) {
aad_out = make_aad_ssrc(0xABCD, sender.peek_send_counter());
cipher_out.assign(plain.size() + crypto_aead_chacha20poly1305_ietf_ABYTES, 0);
long s = sender.seal(plain.data(), plain.size(), aad_out.data(), aad_out.size(),
cipher_out.data(), cipher_out.size());
CHECK(s > 0);
};
// 1. A normal frame (counter 0) decrypts. recv_highest_ = 0.
std::vector<uint8_t> aad0, cipher0;
seal_at_current(aad0, cipher0); // sender counter 0 → 1
CHECK(receiver.open(cipher0.data(), cipher0.size(), aad0.data(), aad0.size(),
recovered.data(), recovered.size()) == static_cast<long>(plain.size()));
// 2. A frame whose header seq has been corrupted to a huge value: it fails auth
// (the AAD no longer matches what was sealed) and must NOT move the window.
std::vector<uint8_t> aad1, cipher1;
seal_at_current(aad1, cipher1); // sender counter 1 → 2
std::vector<uint8_t> forged_aad = aad1;
set_aad_seq(forged_aad, 0x0000FFFFFFFFFFFFULL); // bit-flip-style corruption
CHECK(receiver.open(cipher1.data(), cipher1.size(), forged_aad.data(), forged_aad.size(),
recovered.data(), recovered.size()) < 0);
// 3. The next legitimate frame (counter 2) must still decrypt. On the old code this
// returned "too old" because step 2 had poisoned recv_highest_.
std::vector<uint8_t> aad2, cipher2;
seal_at_current(aad2, cipher2); // sender counter 2 → 3
CHECK(receiver.open(cipher2.data(), cipher2.size(), aad2.data(), aad2.size(),
recovered.data(), recovered.size()) == static_cast<long>(plain.size()));
}
// Regression for the 16-bit seq wrap: with a full 64-bit wire counter, sealing/opening
// across the old u16 boundary (65,535 → 65,536) must keep decrypting. On the old code the
// nonce desynced at the wrap and every frame failed auth permanently.
static void test_seq_past_16bit_boundary() {
uint8_t key[crypto_aead_chacha20poly1305_ietf_KEYBYTES];
crypto_generichash(key, sizeof(key),
reinterpret_cast<const uint8_t*>("wrap-key"), 8, nullptr, 0);
SodiumMediaCrypto sender(key);
SodiumMediaCrypto receiver(key);
std::vector<uint8_t> plain(48, 0x6B);
std::vector<uint8_t> recovered(plain.size());
std::vector<uint8_t> cipher(plain.size() + crypto_aead_chacha20poly1305_ietf_ABYTES);
bool all_ok = true;
for (uint64_t i = 0; i < 70000; ++i) { // crosses 65,536
auto aad = make_aad_ssrc(0x1234, sender.peek_send_counter());
long s = sender.seal(plain.data(), plain.size(), aad.data(), aad.size(),
cipher.data(), cipher.size());
if (s < 0) { all_ok = false; break; }
long o = receiver.open(cipher.data(), static_cast<size_t>(s), aad.data(), aad.size(),
recovered.data(), recovered.size());
if (o != static_cast<long>(plain.size())) { all_ok = false; break; }
}
CHECK(all_ok);
}
int main() {
if (sodium_init() < 0) {
std::printf("FAIL: sodium_init failed\n");
@@ -262,6 +337,8 @@ int main() {
test_tamper_detection();
test_multiple_packets();
test_relay_interleaved_reseal();
test_corrupted_seq_does_not_poison_window();
test_seq_past_16bit_boundary();
if (g_failures == 0) {
std::printf("media_aead: all tests passed\n");

View File

@@ -1,5 +1,5 @@
/*
* test_voice_frame — serialize/parse round-trips for the 14-byte UDP media header.
* test_voice_frame — serialize/parse round-trips for the 20-byte UDP media header.
*/
#include <cassert>
#include <cstdio>
@@ -24,7 +24,7 @@ static void test_header_round_trip() {
f.flags = kFlagMarker | kFlagFecPresent;
f.codec = kCodecOpus;
f.ssrc = 0xDEADBEEF;
f.seq = 0xAB12;
f.seq = 0x0123456789ABCDEFULL; // full 64-bit range (protocol v2)
f.timestamp = 0x12345678;
uint8_t buf[kVoiceHeaderSize];
@@ -97,18 +97,19 @@ static void test_parse_too_short() {
static void test_big_endian_layout() {
VoiceFrame f;
f.ssrc = 0x01020304;
f.seq = 0x0506;
f.timestamp = 0x0708090A;
f.seq = 0x05060708090A0B0CULL;
f.timestamp = 0x0D0E0F10;
uint8_t buf[kVoiceHeaderSize];
serialize_header(f, buf);
// ssrc at [4..7]
CHECK(buf[4] == 0x01 && buf[5] == 0x02 && buf[6] == 0x03 && buf[7] == 0x04);
// seq at [8..9]
CHECK(buf[8] == 0x05 && buf[9] == 0x06);
// timestamp at [10..13]
CHECK(buf[10] == 0x07 && buf[11] == 0x08 && buf[12] == 0x09 && buf[13] == 0x0A);
// seq (u64) at [8..15]
CHECK(buf[8] == 0x05 && buf[9] == 0x06 && buf[10] == 0x07 && buf[11] == 0x08 &&
buf[12] == 0x09 && buf[13] == 0x0A && buf[14] == 0x0B && buf[15] == 0x0C);
// timestamp at [16..19]
CHECK(buf[16] == 0x0D && buf[17] == 0x0E && buf[18] == 0x0F && buf[19] == 0x10);
}
int main() {