Retire legacy implementations and flatten managed layout
This commit is contained in:
+12
-23
@@ -2,37 +2,26 @@
|
||||
|
||||
VoiceCat is a self-hosted channel-based voice and text system. Control traffic uses TLS 1.3;
|
||||
media uses authenticated encrypted UDP derived from the TLS session. There is no plaintext
|
||||
mode and no central service.
|
||||
mode or central service.
|
||||
|
||||
The .NET 10 implementation is the source of truth. Some detailed documents predate the managed
|
||||
rewrite and are being corrected as code changes touch them. When prose conflicts with current
|
||||
managed code or tests, follow the managed implementation and fix the document in the same
|
||||
change.
|
||||
|
||||
## Current documents
|
||||
Current references:
|
||||
|
||||
- [architecture.md](architecture.md) — components, ownership, concurrency, and native boundary.
|
||||
- [protocol.md](protocol.md) — protobuf control messages and fixed UDP media header.
|
||||
- [security.md](security.md) — TLS, TOFU, media keys, authentication, and threat model.
|
||||
- [voice.md](voice.md) — streams, Opus, loss handling, jitter, mixing, and screen audio.
|
||||
- [api-dotnet.md](api-dotnet.md) — managed APIs and ownership contracts.
|
||||
- [api-dotnet.md](api-dotnet.md) — managed API and ownership contracts.
|
||||
- [building.md](building.md) — development, platform builds, tests, and publishing.
|
||||
- [deployment.md](deployment.md) — server configuration and packaging.
|
||||
- [ios-deploy.md](ios-deploy.md) — managed iOS physical-device build and deployment.
|
||||
- [deployment.md](deployment.md) — server packaging and operations.
|
||||
- [ios-deploy.md](ios-deploy.md) — physical-device iOS build and deployment.
|
||||
- [tech-stack.md](tech-stack.md) — supported dependencies and licensing.
|
||||
- [broadcast-ring-format.md](broadcast-ring-format.md) — frozen iOS extension/host ring ABI.
|
||||
- [roadmap.md](roadmap.md) — only current release gates and intentionally deferred features.
|
||||
- [broadcast-ring-format.md](broadcast-ring-format.md) — frozen extension/host ring ABI.
|
||||
|
||||
The completed C++-to-.NET migration plan was removed. Git history preserves that work without
|
||||
making every future agent load an obsolete implementation diary.
|
||||
`proto/voicecat.proto` is the control-plane wire contract. Managed tests are the executable
|
||||
behavior contract. Keep prose current with those sources rather than documenting historical
|
||||
implementations.
|
||||
|
||||
## Durable rules
|
||||
Durable rules:
|
||||
|
||||
- `proto/voicecat.proto` is the control-plane schema.
|
||||
- Encryption is mandatory.
|
||||
- No GPL or LGPL dependencies.
|
||||
- GPL and LGPL dependencies are forbidden.
|
||||
- Real-time audio callbacks never allocate, lock, block, or perform I/O.
|
||||
- The server relays encoded media; it does not mix or transcode.
|
||||
- Text is ephemeral in v1.
|
||||
- Accounts are administrator-provisioned; guests are an operator choice.
|
||||
- The server relays encoded media; it does not mix or transcode it.
|
||||
- Wire, database, and shared-ring changes are explicitly versioned.
|
||||
|
||||
+1
-1
@@ -132,7 +132,7 @@ erasure of every runtime/library copy.
|
||||
|
||||
`VoiceCat.Codec` and `VoiceCat.Dsp` call the desktop `voicecat_media` native library
|
||||
through source-generated `LibraryImport`. It links pinned Opus 1.5.2 and the existing
|
||||
vendored RNNoise; it has no dependency on the retired native core or its C ABI. Fixed C signatures
|
||||
vendored RNNoise. Fixed C signatures
|
||||
wrap Opus controls so P/Invoke never calls C varargs. SafeHandle owns every native
|
||||
encoder, decoder, DRED parser/state, and denoiser, including failed initialization.
|
||||
|
||||
|
||||
@@ -71,6 +71,3 @@ need direct native entry points. These are platform adapters, not a second core.
|
||||
- SQLite is the server's persistent store; schema changes require explicit migrations.
|
||||
- Client profiles and TOFU pins are local platform data.
|
||||
- The ReplayKit ring layout is separately versioned and frozen.
|
||||
|
||||
Unsupported C++ and Swift applications may remain temporarily during repository cleanup, but
|
||||
they are not dependencies, compatibility targets, or design authorities.
|
||||
|
||||
+24
-70
@@ -1,88 +1,42 @@
|
||||
# Building and testing
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- .NET SDK selected by `dotnet/global.json`
|
||||
- CMake and a C compiler for the Opus/RNNoise shim
|
||||
- PowerShell for the cross-platform build scripts
|
||||
- Xcode plus the pinned .NET macOS/iOS workloads for Apple clients
|
||||
|
||||
Dependencies and NuGet lock files are committed. GPL/LGPL dependencies are forbidden.
|
||||
|
||||
## Managed core, server, CLI, and tests
|
||||
VoiceCat requires the .NET SDK selected by `global.json`, PowerShell, CMake, and a C compiler.
|
||||
Apple clients additionally require macOS, Xcode, and the pinned .NET macOS/iOS workloads.
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
./dotnet/build-native.ps1
|
||||
dotnet restore dotnet/VoiceCat.slnx --locked-mode
|
||||
dotnet build dotnet/VoiceCat.slnx -c Release --no-restore
|
||||
dotnet test dotnet/VoiceCat.slnx -c Release --no-build
|
||||
./dotnet/check-licenses.ps1
|
||||
```powershell
|
||||
./scripts/build-native.ps1
|
||||
dotnet restore VoiceCat.slnx --locked-mode
|
||||
dotnet build VoiceCat.slnx -c Release --no-restore
|
||||
dotnet test VoiceCat.slnx -c Release --no-build
|
||||
./scripts/check-licenses.ps1
|
||||
```
|
||||
|
||||
The native script builds `native/media`, fetches checksum-pinned Opus 1.5.2, compiles vendored
|
||||
RNNoise, and stages the resulting library and notices in `dotnet/artifacts/native`.
|
||||
`build-native.ps1` builds the narrow Opus/RNNoise shim in `native/media` and stages it under
|
||||
`artifacts/native`. NuGet dependencies and lock files are committed; GPL/LGPL dependencies are
|
||||
forbidden and the approved license set is enforced by `scripts/check-licenses.ps1`.
|
||||
|
||||
Equivalent direct native build:
|
||||
|
||||
```bash
|
||||
cmake -S native/media -B dotnet/artifacts/native-build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build dotnet/artifacts/native-build --target voicecat_media --parallel 2
|
||||
cmake --install dotnet/artifacts/native-build --component DotnetMedia \
|
||||
--prefix dotnet/artifacts/native
|
||||
```
|
||||
|
||||
## Run locally
|
||||
|
||||
```bash
|
||||
dotnet run --project dotnet/src/VoiceCat.Server -- --data-dir ./voicecat-data
|
||||
dotnet run --project dotnet/src/VoiceCat.Cli -- \
|
||||
--host 127.0.0.1 --port 8384 --nickname Alice --trust-first
|
||||
```
|
||||
|
||||
Both commands support `--help`. The CLI also supports deterministic two-process text and tone
|
||||
checks used by the managed test suite.
|
||||
|
||||
## Windows client
|
||||
Run the server and CLI locally with:
|
||||
|
||||
```powershell
|
||||
./dotnet/build-native.ps1
|
||||
dotnet restore clients/windows/VoiceCat.slnx --locked-mode
|
||||
dotnet build clients/windows/VoiceCat.slnx -c Release --no-restore
|
||||
./clients/windows/publish-client.ps1
|
||||
dotnet run --project src/VoiceCat.Server -- --data-dir ./voicecat-data
|
||||
dotnet run --project src/VoiceCat.Cli -- --host 127.0.0.1 --nickname Local --trust-first
|
||||
```
|
||||
|
||||
The supported app references `VoiceCat.Managed` and the managed core. Published output must
|
||||
contain `voicecat_media.dll` and must not contain the retired `voicecat.dll`.
|
||||
Build the Windows client with `clients/windows/VoiceCat.slnx` and publish it with
|
||||
`clients/windows/publish-client.ps1`.
|
||||
|
||||
## Apple clients
|
||||
|
||||
On Apple Silicon with the SDK/workload versions documented in
|
||||
`clients/apple/dotnet/README.md`:
|
||||
On macOS, stage the native libraries and build the Apple clients with:
|
||||
|
||||
```bash
|
||||
./dotnet/build-native.ps1
|
||||
./dotnet/build-native-ios.sh
|
||||
dotnet restore clients/apple/dotnet/VoiceCat.Apple.slnx
|
||||
dotnet build clients/apple/dotnet/VoiceCat.Apple.slnx -c Debug --no-restore
|
||||
./scripts/build-native.ps1
|
||||
./scripts/build-native-ios.sh
|
||||
dotnet restore clients/apple/VoiceCat.Apple.slnx
|
||||
dotnet build clients/apple/VoiceCat.Apple.slnx -c Debug --no-restore
|
||||
```
|
||||
|
||||
The iOS build invokes the standalone project in `native/apple/broadcast` and embeds its appex.
|
||||
Use `clients/apple/dotnet/build-ios-device.sh` and `deploy-ios-device.sh` for signed device
|
||||
builds. Use `clients/apple/dotnet/publish-macos.sh --dry-run` for an ad-hoc validated macOS
|
||||
bundle; its environment variables enable Developer ID signing and notarization.
|
||||
See `clients/apple/README.md` and `docs/ios-deploy.md` for signing and device workflows.
|
||||
|
||||
## Server publishing
|
||||
|
||||
`dotnet/publish-server.ps1` produces locked self-contained Windows and Linux artifacts in one
|
||||
command. Pass `-Runtime win-x64` or `-Runtime linux-x64` to publish only one target.
|
||||
`Dockerfile`, Compose configuration, and systemd packaging use the managed server. Validate a
|
||||
published binary with its TLS `--health-check`, preferably including `--expect-fingerprint`.
|
||||
|
||||
## CI and release expectations
|
||||
|
||||
CI builds the native shim, managed solution, tests, licenses, and managed Apple clients. The
|
||||
old C++ implementation is not a conformance target. Release validation additionally includes
|
||||
real devices, screen readers, sustained calls, signing/notarization, container execution, and
|
||||
a server soak; see `roadmap.md`.
|
||||
Publish self-contained server binaries with `scripts/publish-server.ps1`. Generated native,
|
||||
client, and server artifacts live under `artifacts/`.
|
||||
|
||||
+19
-213
@@ -1,229 +1,35 @@
|
||||
# Deployment & Self-Hosting
|
||||
# Server deployment
|
||||
|
||||
## Managed server deployment checkpoint
|
||||
|
||||
The .NET server preserves protocol v2 and the native schema/credentials. Publish a
|
||||
self-contained executable (no installed .NET runtime required):
|
||||
Publish a locked, self-contained server from the repository root:
|
||||
|
||||
```powershell
|
||||
# Publishes both win-x64 and linux-x64 by default.
|
||||
./dotnet/publish-server.ps1
|
||||
# Publish just one target when needed.
|
||||
./dotnet/publish-server.ps1 -Runtime linux-x64
|
||||
./dotnet/artifacts/server/win-x64/VoiceCat.Server.exe --help
|
||||
./dotnet/artifacts/server/win-x64/VoiceCat.Server.exe account add Operator --admin --data-dir ./voicecat-data
|
||||
./dotnet/artifacts/server/win-x64/VoiceCat.Server.exe --data-dir ./voicecat-data --allow-guests false
|
||||
./scripts/publish-server.ps1
|
||||
./scripts/publish-server.ps1 -Runtime linux-x64
|
||||
./artifacts/server/win-x64/VoiceCat.Server.exe --help
|
||||
```
|
||||
|
||||
Account add/reset use a hidden password prompt, redirected standard input, or
|
||||
`VOICECAT_ADMIN_PASSWORD`. Passwords are never accepted as command arguments or logged.
|
||||
Account delete/list work against the same database, including while the server runs.
|
||||
Provisioning grants administrator access only through the local command's `--admin`;
|
||||
in-band account creation remains non-admin. Restrict access to the data directory.
|
||||
|
||||
Defaults are `0.0.0.0:8384` TCP+UDP, guest access enabled, 64 connections, 15-second TLS
|
||||
handshakes, 45-second idle expiry and 15-second sweeps. Override with flags or environment:
|
||||
|
||||
| Flag | Environment variable |
|
||||
|---|---|
|
||||
| `--data-dir` | `VOICECAT_DATA_DIR` |
|
||||
| `--bind` | `VOICECAT_BIND_ADDRESS` |
|
||||
| `--port` | `VOICECAT_BIND_PORT` |
|
||||
| `--name` | `VOICECAT_SERVER_NAME` |
|
||||
| `--allow-guests` | `VOICECAT_ALLOW_GUESTS` |
|
||||
| `--max-connections` | `VOICECAT_MAX_CONNECTIONS` |
|
||||
| `--handshake-seconds` | `VOICECAT_HANDSHAKE_TIMEOUT_SECONDS` |
|
||||
| `--idle-seconds` | `VOICECAT_IDLE_TIMEOUT_SECONDS` |
|
||||
| `--reaper-seconds` | `VOICECAT_REAPER_INTERVAL_SECONDS` |
|
||||
| `--auth-burst` | `VOICECAT_AUTH_BURST` |
|
||||
| `--auth-refill-seconds` | `VOICECAT_AUTH_REFILL_SECONDS` |
|
||||
|
||||
Command arguments override environment values. `--print-config` validates and prints JSON
|
||||
without creating files; `--print-fingerprint` prints the persisted leaf-certificate SHA-256
|
||||
pin. Startup emits one JSON `ready` event with both fingerprints and actual TCP/UDP ports.
|
||||
Bind accepts IP literals; IPv6 listeners are IPv6-only. Open/forward both protocols.
|
||||
An exclusive data-directory instance lock prevents duplicate managed server processes.
|
||||
Ctrl+C and Unix SIGINT/SIGTERM stop all transport tasks; shutdown has a ten-second deadline.
|
||||
Fatal listener/media/reaper failure exits the host rather than leaving a broken listener.
|
||||
`--health-check HOST:PORT` performs a real protocol TLS 1.3 handshake and emits JSON.
|
||||
Automation can add `--expect-fingerprint SHA256` to verify the persisted certificate.
|
||||
|
||||
Password authentication is limited before Argon2 by source address and username across
|
||||
connections: burst 5, refill one attempt per ten seconds. Starting at three failed attempts,
|
||||
backoff grows from one to thirty seconds. Success clears backoff but does not restore
|
||||
tokens. State is bounded to 4096 keys; idle entries retire after ten minutes when full.
|
||||
Throttle and credential failures share the generic auth error. Limits are process-local.
|
||||
|
||||
The publish script produces Windows x64 and Linux x64 self-contained outputs in one command
|
||||
by default. It uses separate per-RID lock files so deployment and development restore graphs
|
||||
remain reproducible. Pass `-Runtime` to publish only one target.
|
||||
The Linux smoke starts the published ELF, validates TLS health and the pin, checks mode 0700
|
||||
data creation, sends SIGTERM and requires a clean exit:
|
||||
|
||||
```bash
|
||||
sh deploy/linux/smoke.sh dotnet/artifacts/server/linux-x64/VoiceCat.Server
|
||||
```
|
||||
|
||||
`deploy/linux/voicecat.service` runs under a systemd dynamic user with a private state
|
||||
directory, no capabilities and filesystem/kernel hardening. Install a published binary and
|
||||
the unit from `deploy/linux/` as root, or use the helper there. The default Dockerfile now
|
||||
builds the managed server with locked packages and runs it as UID 1654 in Microsoft's
|
||||
chiseled .NET runtime-dependencies image. Compose drops all capabilities, sets the root
|
||||
filesystem read-only and persists only `/data`.
|
||||
|
||||
For operational endurance against an already running server:
|
||||
The default endpoint is TCP and UDP port 8384. The server creates its TLS identity and SQLite
|
||||
database in the data directory. Account passwords are accepted only through a hidden prompt,
|
||||
redirected standard input, or `VOICECAT_ADMIN_PASSWORD`; they are never command arguments.
|
||||
|
||||
```powershell
|
||||
./dotnet/soak-server.ps1 -HostName 127.0.0.1 -Port 8384 -Minutes 30 -Pairs 4
|
||||
./artifacts/server/win-x64/VoiceCat.Server.exe account add Operator --admin --data-dir ./voicecat-data
|
||||
./artifacts/server/win-x64/VoiceCat.Server.exe --data-dir ./voicecat-data --allow-guests false
|
||||
```
|
||||
|
||||
Every cycle creates independent managed processes that exchange text and decoded voice.
|
||||
A short 16-session published-server soak is part of this checkpoint. A release candidate
|
||||
still needs the long soak on its target host. TOML/reload and signed image publication remain
|
||||
before broad production rollout.
|
||||
The checked-in `Dockerfile` builds the managed Linux server. `docker-compose.yml` runs it as a
|
||||
non-root user with a read-only filesystem, persistent `/data`, and TCP/UDP 8384 exposed.
|
||||
|
||||
The product goal: someone looks at this and thinks *"oh, I (or my agent) can stand this up
|
||||
in a few minutes."* Everything below is in service of that. Three install paths, all
|
||||
**zero-config and encrypted by default**.
|
||||
|
||||
## 1. The three paths
|
||||
|
||||
### A. Docker (recommended)
|
||||
For a host installation, `deploy/linux/install.sh` installs a published Linux binary and the
|
||||
systemd unit. Validate a published binary with:
|
||||
|
||||
```bash
|
||||
docker build -t voicecat:local .
|
||||
docker run -d --name voicecat \
|
||||
-p 8384:8384/tcp \ # control (TLS 1.3)
|
||||
-p 8384:8384/udp \ # media (encrypted)
|
||||
-v voicecat-data:/data \
|
||||
--read-only --tmpfs /tmp:rw,noexec,nosuid,size=16m \
|
||||
--cap-drop ALL --security-opt no-new-privileges \
|
||||
voicecat:local
|
||||
sh deploy/linux/smoke.sh artifacts/server/linux-x64/VoiceCat.Server
|
||||
```
|
||||
|
||||
That's the whole thing. On first start it generates its Ed25519 identity + self-signed
|
||||
cert, creates the SQLite database under `/data`, prints the **server fingerprint** (for
|
||||
clients to verify), and listens. Control and media share one port number on TCP+UDP to keep
|
||||
firewall rules trivial.
|
||||
The server exposes `--health-check host:port` for TLS-aware probes. Before release, run the
|
||||
container and the concurrent client soak:
|
||||
|
||||
A `docker-compose.yml` is provided for people who prefer it, but it isn't required.
|
||||
|
||||
### B. Single static binary
|
||||
|
||||
```bash
|
||||
# download for your OS, then:
|
||||
./voicecat-server # uses ./voicecat-data/ , prints fingerprint, runs
|
||||
```powershell
|
||||
./scripts/soak-server.ps1 -HostName 127.0.0.1 -Port 8384 -Minutes 30 -Pairs 4
|
||||
```
|
||||
|
||||
The server is a **single statically linked executable** (mbedTLS, libsodium, opus, sqlite,
|
||||
etc. linked in — all permissive licenses). No runtime, no shared libraries to install, no
|
||||
package manager. Linux (primary), macOS, and Windows builds.
|
||||
|
||||
### C. From source
|
||||
|
||||
```bash
|
||||
git clone … && cd voice-cat
|
||||
cmake --preset server-release # vcpkg fetches & pins all deps; auto-triplet (Linux/macOS/Windows)
|
||||
cmake --build --preset server-release
|
||||
./build/server-release/bin/voicecat-server
|
||||
```
|
||||
|
||||
One `cmake` invocation; vcpkg (manifest mode) resolves the dependency graph reproducibly.
|
||||
The `server-release` preset produces an optimized, **stripped** binary (`-s` linker flag) —
|
||||
smaller executables suitable for distribution. Works on Linux (primary), macOS, and Windows;
|
||||
the vcpkg triplet is auto-resolved by [`cmake/voicecat-toolchain.cmake`](../cmake/voicecat-toolchain.cmake).
|
||||
No system packages to chase.
|
||||
|
||||
## 2. Zero-config defaults
|
||||
|
||||
The server runs with **no config file at all**. Sensible defaults:
|
||||
|
||||
| Setting | Default |
|
||||
|---------|---------|
|
||||
| Encryption | On, always (not configurable off) |
|
||||
| TLS cert / identity | Auto-generated on first run, persisted to the data dir |
|
||||
| Database | Embedded SQLite in the data dir (no external DB) |
|
||||
| Guests | Enabled (so the very first connect "just works"); easily disabled |
|
||||
| Ports | `8384` TCP + UDP |
|
||||
| A default channel | One "Lobby" voice/text channel created on first run |
|
||||
| Opus policy | 48 kHz, 20 ms frames, mono/VOIP defaults; per-channel overrides allowed |
|
||||
| Argon2id cost | Auto-tuned to the host on first run |
|
||||
|
||||
Override only what you care about, via env vars or an optional `server.toml`:
|
||||
|
||||
```toml
|
||||
# server.toml — every key is optional
|
||||
server_name = "Cats United"
|
||||
allow_guests = false
|
||||
bind_port = 8384
|
||||
data_dir = "/data"
|
||||
|
||||
[tls] # only if you want a real CA cert; otherwise self-signed
|
||||
cert_file = "/data/fullchain.pem"
|
||||
key_file = "/data/privkey.pem"
|
||||
|
||||
[opus.defaults] # default Opus policy for new channels
|
||||
mode = "mono"
|
||||
bitrate_bps = 24000
|
||||
frame_ms = 20
|
||||
fec = true
|
||||
dtx = true
|
||||
|
||||
[opus.limits] # server-enforced ceilings (bound bandwidth)
|
||||
max_bitrate_bps = 128000 # channels can't be configured above this
|
||||
```
|
||||
|
||||
Every key also has an `VOICECAT_*` env var form, which is what the Docker path uses.
|
||||
|
||||
## 3. Connecting (client side)
|
||||
|
||||
- **Pure direct-connect.** Enter `host:port` (and a nickname or account). There is no central
|
||||
directory or server browser — you connect to a server you know.
|
||||
- **Saved server list.** The client keeps a local list of saved servers (host:port, pinned
|
||||
fingerprint, nickname/credentials per server) so you can store several and pick one to
|
||||
join. This lives entirely in the client.
|
||||
- On first connect the client shows the server's **fingerprint** and pins it (TOFU). No
|
||||
accounts or certs needed to try it; if the operator disabled guests, the client prompts for
|
||||
the username/password an admin gave you.
|
||||
|
||||
That's the entire flow: run the server, share `host:port` + fingerprint, friends save it and
|
||||
connect.
|
||||
|
||||
## 3a. Provisioning accounts (admin)
|
||||
|
||||
Accounts are **admin-provisioned** — there is no self-serve registration. Two equivalent ways,
|
||||
both writing the same SQLite store:
|
||||
|
||||
```bash
|
||||
# CLI against the server's data dir or a running server
|
||||
voicecat-admin account add <username> # prompts for / generates a password
|
||||
voicecat-admin account reset <username>
|
||||
voicecat-admin account del <username>
|
||||
voicecat-admin account list
|
||||
```
|
||||
|
||||
…or from the **in-app admin interface** (a user with the admin permission), which sends the
|
||||
privileged `CreateAccount`/`ResetPassword`/`DeleteAccount` control messages over TLS
|
||||
(protocol.md §3). Guests need no provisioning; they just pick a nickname (if guests are
|
||||
enabled).
|
||||
|
||||
## 4. Why it stays this easy (design constraints that protect the goal)
|
||||
|
||||
- **No external services.** No separate database, no Redis, no TURN/STUN server, no reverse
|
||||
proxy required. SQLite is embedded; media is plain UDP.
|
||||
- **No certificate chore.** Self-signed + Ed25519 TOFU means encryption needs zero operator
|
||||
action. A domain owner *can* drop in a Let's Encrypt cert, but never *has* to.
|
||||
- **One port pair.** TCP+UDP on the same number; one firewall/port-forward rule.
|
||||
- **Static linking + permissive licenses.** The binary has no install-time dependencies and
|
||||
can be redistributed (including closed-source) without copyleft obligations.
|
||||
- **Agent-friendly.** The run command is a single line with no interactive prompts, the
|
||||
server logs its fingerprint and listen address in machine-readable form, and `--help` /
|
||||
`--print-config` expose everything an automation needs. Health endpoint for liveness checks.
|
||||
|
||||
## 5. Operational niceties (planned, not blocking v1)
|
||||
|
||||
- Signed, multi-architecture image publication.
|
||||
- Graceful reload of `server.toml` on `SIGHUP`.
|
||||
- `voicecat-admin` (see §3a) also handles bans and channel admin, talking to the same SQLite
|
||||
file or a running server.
|
||||
- Prebuilt images for `linux/amd64` + `linux/arm64` (Raspberry Pi / cheap VPS friendly).
|
||||
|
||||
+9
-9
@@ -1,7 +1,7 @@
|
||||
# Deploying the managed iOS app
|
||||
|
||||
VoiceCat's supported iOS client is the .NET 10 UIKit application in
|
||||
`clients/apple/dotnet/VoiceCat.iOS`. The checked-in scripts build its native Opus/RNNoise
|
||||
`clients/apple/VoiceCat.iOS`. The checked-in scripts build its native Opus/RNNoise
|
||||
dependency, compile and sign the managed app and Swift ReplayKit extension, stage the bundle,
|
||||
install it, and launch it on a paired physical device.
|
||||
|
||||
@@ -15,7 +15,7 @@ install it, and launch it on a paired physical device.
|
||||
List available devices:
|
||||
|
||||
```bash
|
||||
clients/apple/dotnet/deploy-ios-device.sh --list
|
||||
clients/apple/deploy-ios-device.sh --list
|
||||
```
|
||||
|
||||
## Build, install, and launch
|
||||
@@ -25,8 +25,8 @@ build systems, and Xcode needs the team to select the extension profile.
|
||||
|
||||
```bash
|
||||
export VOICECAT_DEVELOPMENT_TEAM=FJV8L966W4
|
||||
clients/apple/dotnet/build-ios-device.sh --configuration Debug
|
||||
clients/apple/dotnet/deploy-ios-device.sh \
|
||||
clients/apple/build-ios-device.sh --configuration Debug
|
||||
clients/apple/deploy-ios-device.sh \
|
||||
--device "Talon’s iPhone" \
|
||||
--configuration Debug \
|
||||
--no-build
|
||||
@@ -61,12 +61,12 @@ account can otherwise make provisioning updates fail even though offline signing
|
||||
|
||||
### Stale CMake source path
|
||||
|
||||
After moving the native shim from `dotnet/native` to `native/media`, an old CMake cache can
|
||||
report a source-directory mismatch. Move the generated directories aside and rebuild:
|
||||
An old CMake cache can report a source-directory mismatch. Move the generated directories
|
||||
aside and rebuild:
|
||||
|
||||
```bash
|
||||
mv dotnet/artifacts/native-build-ios-arm64-cmake /tmp/
|
||||
mv dotnet/artifacts/native-build-iossimulator-arm64-cmake /tmp/
|
||||
mv artifacts/native-build-ios-arm64-cmake /tmp/
|
||||
mv artifacts/native-build-iossimulator-arm64-cmake /tmp/
|
||||
```
|
||||
|
||||
### Locked restore reports changed runtime identifiers
|
||||
@@ -76,7 +76,7 @@ them after runtime changes, then verify that dependency versions did not change:
|
||||
|
||||
```bash
|
||||
/usr/local/share/dotnet/dotnet restore \
|
||||
clients/apple/dotnet/VoiceCat.iOS/VoiceCat.iOS.csproj \
|
||||
clients/apple/VoiceCat.iOS/VoiceCat.iOS.csproj \
|
||||
-p:VoiceCatIosStatic=true \
|
||||
--force-evaluate
|
||||
```
|
||||
|
||||
@@ -1,357 +0,0 @@
|
||||
# Control Protocol
|
||||
|
||||
The control plane runs over **TCP, wrapped in TLS 1.3**. It carries everything that is not
|
||||
real-time media: handshake, authentication, channel/user/presence state, text chat, and
|
||||
voice *signaling* (announcing that a media stream is starting/stopping). Real-time voice
|
||||
travels separately over UDP — see [voice.md](voice.md).
|
||||
|
||||
## 1. Framing
|
||||
|
||||
Inside the TLS stream, messages are length-prefixed:
|
||||
|
||||
```
|
||||
┌──────────────┬───────────────────────────────────────────────┐
|
||||
│ u32 length │ protobuf-encoded Envelope (length bytes) │
|
||||
│ (big-endian)│ │
|
||||
└──────────────┴───────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- `length` is the byte count of the payload that follows (not including the 4 length
|
||||
bytes). Hard cap (e.g. 16 MiB) to bound memory; oversized frame → protocol error +
|
||||
disconnect.
|
||||
- The payload is a single **`Envelope`** protobuf message. We do **not** add our own type
|
||||
byte; the type is the `oneof` discriminator inside the Envelope, which keeps the framing
|
||||
trivial and lets protobuf own all forward/backward compatibility.
|
||||
|
||||
TLS already provides record framing, integrity, and ordering; we only add message
|
||||
boundaries on top of the TLS byte stream.
|
||||
|
||||
## 2. Why Protocol Buffers for control
|
||||
|
||||
- Schema-driven codegen for the supported **C#** implementation → no hand-rolled
|
||||
parsers, no drift between client and server.
|
||||
- **Forward/backward compatible by construction**: unknown fields are preserved/ignored,
|
||||
new fields and new `oneof` arms are additive. This is exactly the "extensible protocol"
|
||||
requirement.
|
||||
- Compact enough for a control plane (text/state, not media). We use **`oneof`** envelopes
|
||||
rather than `Any` so the wire stays tight and the switch is exhaustive.
|
||||
|
||||
> Media frames do **not** use protobuf — they use a fixed binary header (see voice.md),
|
||||
> because per-packet protobuf overhead and allocation are unacceptable on the RT path.
|
||||
|
||||
## 3. The Envelope
|
||||
|
||||
```proto
|
||||
syntax = "proto3";
|
||||
package voicecat.v1;
|
||||
|
||||
message Envelope {
|
||||
// Monotonic per-connection id set by the sender of a request; echoed in the
|
||||
// matching response so async callers can correlate. 0 for unsolicited events.
|
||||
uint64 request_id = 1;
|
||||
|
||||
oneof body {
|
||||
// ── Session / handshake ───────────────────────────────
|
||||
ClientHello client_hello = 10;
|
||||
ServerHello server_hello = 11;
|
||||
AuthRequest auth_request = 12;
|
||||
AuthResult auth_result = 13;
|
||||
Disconnect disconnect = 14;
|
||||
Ping ping = 15;
|
||||
Pong pong = 16;
|
||||
|
||||
// ── State sync ────────────────────────────────────────
|
||||
ServerStateSnapshot server_state = 20;
|
||||
ChannelEvent channel_event = 21; // created/updated/deleted
|
||||
UserEvent user_event = 22; // joined/left/updated
|
||||
SubscribeRequest subscribe = 23;
|
||||
|
||||
// ── Channel operations ────────────────────────────────
|
||||
JoinChannelRequest join_channel = 30;
|
||||
JoinChannelResult join_channel_result= 31;
|
||||
LeaveChannelRequest leave_channel = 32;
|
||||
CreateChannelRequest create_channel = 33;
|
||||
EditChannelRequest edit_channel = 34;
|
||||
DeleteChannelRequest delete_channel = 35;
|
||||
MoveUserRequest move_user = 36;
|
||||
GenericResult generic_result = 37; // ack/err for the above
|
||||
|
||||
// ── Voice signaling (media is on UDP) ─────────────────
|
||||
StreamAnnounce stream_announce = 40;
|
||||
StreamAnnounceResult stream_announce_result = 41;
|
||||
StreamStop stream_stop = 42;
|
||||
StreamStateUpdate stream_state = 43; // talking/muted indicator
|
||||
UdpBinding udp_binding = 44; // token to bind the UDP 5-tuple
|
||||
SubscribeVoiceRequest subscribe_voice = 45; // join the voice plane
|
||||
UnsubscribeVoiceRequest unsubscribe_voice = 46; // leave the voice plane
|
||||
VoiceSubscriptionResult voice_subscription_result = 47; // ack with subscribed flag
|
||||
|
||||
// ── Text ──────────────────────────────────────────────
|
||||
TextMessage text_message = 50;
|
||||
TextMessageAck text_message_ack = 51;
|
||||
TypingIndicator typing = 52;
|
||||
|
||||
// ── Moderation / permissions ──────────────────────────
|
||||
KickRequest kick = 60;
|
||||
BanRequest ban = 61;
|
||||
SetPermissionRequest set_permission = 62;
|
||||
ServerMuteRequest server_mute = 63;
|
||||
|
||||
// ── Admin account management (privileged) ─────────────
|
||||
// Accounts are admin-provisioned (no self-serve registration in v1).
|
||||
// These ride the same TLS control channel and require an admin permission.
|
||||
CreateAccountRequest create_account = 70;
|
||||
ResetPasswordRequest reset_password = 71;
|
||||
DeleteAccountRequest delete_account = 72;
|
||||
ListAccountsRequest list_accounts = 73;
|
||||
ListAccountsResult list_accounts_result = 74;
|
||||
|
||||
// ── Extension escape hatch ────────────────────────────
|
||||
Extension extension = 200; // {string ns; bytes payload;}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Reserved tag ranges keep future families from colliding: **10–19** session, **20–29** state,
|
||||
**30–39** channels, **40–49** voice signaling, **50–59** text, **60–99** moderation,
|
||||
**100–199** future (e.g. file transfer = 100–109), **200+** extensions.
|
||||
|
||||
## 4. Connection lifecycle
|
||||
|
||||
```
|
||||
Client Server
|
||||
│ TCP connect ───────────────────────────────────▶│
|
||||
│ ◀──────────────── TLS 1.3 handshake ────────────▶│ server cert (TOFU/PKI, see security.md)
|
||||
│ │
|
||||
│ ClientHello (proto_version, features[], info) ──▶│
|
||||
│ ◀── ServerHello (proto_version, features[], │ feature intersection negotiated here
|
||||
│ server_info, auth_methods, udp_port) │
|
||||
│ │
|
||||
│ AuthRequest (guest{nick} | user{name,pass}) ────▶│ password verified w/ Argon2id
|
||||
│ ◀── AuthResult (ok, session_id, self, perms, │
|
||||
│ udp_token) │
|
||||
│ │
|
||||
│ ◀── ServerStateSnapshot (channel tree, users) ───│ initial sync
|
||||
│ │
|
||||
│ UdpBinding(udp_token) [TCP/TLS] ─────────────────▶│ confirms token, no-ops if mismatched
|
||||
│ ◀── UdpBinding(ack=true) [TCP/TLS] ───────────────│
|
||||
│ │
|
||||
│ ===== UDP side (parallel) ===================== │
|
||||
│ (media keys derived from TLS exporter — no 2nd │
|
||||
│ handshake; see security.md §2) │
|
||||
│ UDP_BINDING frame(udp_token) [plaintext] ───────▶│ binds 5-tuple → session_id
|
||||
│ ── voice frames (AEAD, exported keys) ──────────▶│
|
||||
│ │
|
||||
│ JoinChannelRequest(id, password?) ──────────────▶│
|
||||
│ ◀── JoinChannelResult(ok, members, audio_cfg) ───│
|
||||
│ StreamAnnounce(kind=mic, opus_params) ──────────▶│
|
||||
│ ◀── StreamAnnounceResult(ok, ssrc) │
|
||||
│ ── voice frames flow over UDP ──────────────────▶│
|
||||
│ │
|
||||
│ Ping / Pong (TCP keepalive) ◀──────────────────▶│
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- **Version negotiation.** Each side sends `proto_version` (integer) and a `features`
|
||||
string list. The effective version is `min(client, server)`; the effective feature set
|
||||
is the intersection. A client that doesn't understand a feature simply never uses it.
|
||||
The **current `proto_version` is 2**. v2 widened the UDP voice frame `seq` field from
|
||||
u16 to u64 (voice.md §2) — a wire-format change with no backward compatibility on the
|
||||
media path, so the server rejects any peer not on v2 rather than min-negotiating down.
|
||||
- **Auth over TLS.** Passwords cross the wire only inside TLS 1.3 and are verified against
|
||||
an Argon2id hash at rest (see security.md). `auth_methods` in `ServerHello` advertises
|
||||
whether `guest` is enabled.
|
||||
- **UDP token.** `AuthResult.udp_token` is a short-lived opaque token. The client confirms it
|
||||
over TCP/TLS (`UdpBinding` request/ack) and also sends it as the payload of a plaintext
|
||||
`UDP_BINDING`-type media frame so the server can bind the UDP 5-tuple to the authenticated
|
||||
session without trusting the source address. This bootstrap frame is the only UDP message
|
||||
that carries identity material in the clear; everything after (voice frames) is AEAD-sealed
|
||||
and routed purely by the bound tuple + media-AEAD session.
|
||||
- **Snapshot then deltas.** After auth the server pushes a `ServerStateSnapshot` (full
|
||||
channel tree + visible users), then streams incremental `ChannelEvent`/`UserEvent`
|
||||
deltas. Clients reconcile by id.
|
||||
|
||||
## 5. Message catalog (selected definitions)
|
||||
|
||||
Representative messages; the full `.proto` is the source of truth in `core/proto/`.
|
||||
|
||||
```proto
|
||||
message ClientHello {
|
||||
uint32 proto_version = 1;
|
||||
repeated string features = 2; // e.g. "opus", "fec", "screen-audio"
|
||||
string client_name = 3; // "VoiceCat-macOS"
|
||||
string client_version = 4;
|
||||
string preferred_locale = 5;
|
||||
}
|
||||
|
||||
message ServerHello {
|
||||
uint32 proto_version = 1;
|
||||
repeated string features = 2;
|
||||
string server_name = 3;
|
||||
string server_version = 4;
|
||||
repeated string auth_methods = 5; // "guest", "password"
|
||||
uint32 udp_port = 6;
|
||||
bytes server_identity_fingerprint = 7; // Ed25519 key fp for TOFU display
|
||||
}
|
||||
|
||||
message AuthRequest {
|
||||
oneof method {
|
||||
GuestAuth guest = 1; // { string nickname; }
|
||||
PasswordAuth password = 2; // { string username; string password; }
|
||||
}
|
||||
}
|
||||
|
||||
message AuthResult {
|
||||
bool ok = 1;
|
||||
string error = 2;
|
||||
uint64 session_id = 3;
|
||||
User self = 4;
|
||||
Permissions permissions = 5;
|
||||
bytes udp_token = 6; // bind UDP 5-tuple with this
|
||||
}
|
||||
|
||||
message Channel {
|
||||
uint32 id = 1;
|
||||
uint32 parent_id = 2; // 0 = root
|
||||
string name = 3;
|
||||
string topic = 4;
|
||||
bool password_protected = 5;
|
||||
uint32 max_users = 6;
|
||||
ChannelType type = 7; // PERMANENT / TEMPORARY
|
||||
AudioConfig audio = 8; // per-channel Opus settings (see voice.md)
|
||||
int32 order = 9;
|
||||
}
|
||||
|
||||
message User {
|
||||
uint32 id = 1;
|
||||
string nickname = 2;
|
||||
bool is_guest = 3;
|
||||
uint32 channel_id = 4;
|
||||
bool self_mic_muted = 5;
|
||||
bool self_deafened = 6;
|
||||
bool server_muted = 7;
|
||||
repeated StreamInfo streams = 8; // active media streams this user publishes
|
||||
bool server_deafened = 9; // M5: server-imposed deafen
|
||||
}
|
||||
|
||||
message StreamInfo {
|
||||
uint32 stream_id = 1; // unique within the user
|
||||
uint32 ssrc = 2; // media-plane id assigned by server
|
||||
StreamKind kind = 3; // MIC / SCREEN_AUDIO / AUX_DEVICE
|
||||
AudioConfig audio = 4;
|
||||
string label = 5; // "Microphone", "Desktop audio"
|
||||
}
|
||||
|
||||
message StreamAnnounce { // client → server: "I'm about to publish media"
|
||||
StreamKind kind = 1;
|
||||
AudioConfig requested_audio = 2; // server may clamp to channel policy
|
||||
string label = 3;
|
||||
}
|
||||
message StreamAnnounceResult {
|
||||
bool ok = 1; string error = 2;
|
||||
uint32 stream_id = 3; uint32 ssrc = 4;
|
||||
AudioConfig effective_audio = 5; // authoritative params to encode with
|
||||
}
|
||||
|
||||
message TextMessage {
|
||||
TextScope scope = 1; // CHANNEL / PRIVATE / SERVER
|
||||
uint32 target_id = 2; // channel_id or user_id depending on scope
|
||||
uint32 sender_id = 3; // set by server on relay
|
||||
string body = 4; // UTF-8, server-bounded length
|
||||
uint64 sent_at_unix_ms = 5; // server timestamp on relay
|
||||
string client_msg_id = 6; // client-chosen, echoed in ack (dedup)
|
||||
}
|
||||
```
|
||||
|
||||
> **Text is ephemeral (v1).** The server relays messages live to currently-connected,
|
||||
> subscribed recipients and **does not persist history** — there is no store and no backfill
|
||||
> on join. Clients may keep their own local scrollback for the session. Server-side history
|
||||
> is a deliberate non-feature for now (it can be added later behind a capability flag without
|
||||
> changing `TextMessage`).
|
||||
|
||||
## 6. Request / response & errors
|
||||
|
||||
- Any message a client expects a direct answer to sets a nonzero **`request_id`**; the
|
||||
server echoes it in the response (`*Result` or `GenericResult`). Unsolicited
|
||||
server→client events use `request_id = 0`.
|
||||
- **`GenericResult { bool ok; uint32 code; string message; }`** is the default
|
||||
acknowledgement for operations without a richer reply (create/edit/delete channel, move
|
||||
user, kick, ban, server-mute, set-permission, create/reset/delete account). Error `code`s
|
||||
are an enumerated, stable list.
|
||||
- Fatal conditions send **`Disconnect { code; reason }`** then close the TLS connection.
|
||||
`code ≥ 1` is server-sent (1 = protocol error, 2 = kicked). `code = 0` is client-sent
|
||||
graceful disconnect (§7): the server broadcasts `UserEvent::LEFT` and closes immediately.
|
||||
- **The response is for the request; the broadcast is for the state.** A `*Result` only
|
||||
acknowledges the actor's request (correlation via `request_id`, error text, and any
|
||||
actor-private payload — e.g. the channel `AudioConfig` in `JoinChannelResult`). The
|
||||
resulting *state change* is delivered to **every** connected client **including the actor**
|
||||
via the normal `UserEvent` / `ChannelEvent` / relayed `TextMessage` path. Clients apply
|
||||
those events to their local model and never re-derive their own state from a `*Result`
|
||||
(doing so drifts: the actor would miss its own change and a later event for another user
|
||||
would surface the stale value).
|
||||
|
||||
## 7. Keepalive & timeouts
|
||||
|
||||
- **TCP:** `Ping`/`Pong` every ~15 s; missing 3 consecutive pongs (45 s) → the server's
|
||||
reaper drops the session. `Pong` echoes the `Ping` nonce so RTT is measurable. The
|
||||
client sends `Ping` automatically from its io thread; the server answers with `Pong`
|
||||
in any state.
|
||||
- **`last_seen` reaper.** Every `ConnSession` tracks `last_seen` — bumped on *any*
|
||||
inbound TCP frame (not just `Ping`) and on any inbound UDP voice/keepalive frame. A
|
||||
periodic sweep (`asio::steady_timer`, every 15 s) drops sessions whose `last_seen` is
|
||||
older than 45 s. Each drop calls `close()`, which broadcasts `UserEvent::LEFT` to
|
||||
remaining clients — so half-open connections (NAT timeout, wifi loss without RST,
|
||||
laptop sleep) that never produce a TCP EOF are cleaned up, and peers' audio engines
|
||||
`remove_stream` and stop PLC. The timeout and sweep interval are configurable via
|
||||
`server::Config::reaper_timeout_ms` / `reaper_sweep_ms` (set to 0 to disable).
|
||||
The managed server uses `VoiceServerOptions.IdleTimeout` / `ReaperInterval` with the
|
||||
same 45-second / 15-second defaults (zero idle timeout disables reaping). It refreshes
|
||||
activity on parsed control envelopes, authenticated owned-stream voice, and exact
|
||||
bound-endpoint keepalives; rejected media does not refresh activity. Timing is monotonic.
|
||||
- **UDP:** a separate lightweight keepalive on the media channel (voice.md §6) keeps NAT
|
||||
bindings alive and detects media-path failure independently of the control channel.
|
||||
- **Graceful disconnect.** A client ending its session sends `Disconnect { code = 0;
|
||||
reason }` before closing the socket. The server calls `close()` on receipt —
|
||||
broadcasting `UserEvent::LEFT` immediately, without waiting for TCP EOF or the reaper.
|
||||
The client's `vc_disconnect()` queues this message and waits for the io thread to flush
|
||||
it before closing the socket. `code = 0` is reserved for client-initiated graceful
|
||||
disconnect; server-sent fatal `Disconnect` uses `code ≥ 1` (1 = protocol error,
|
||||
2 = kicked).
|
||||
- **Client reconnection is local policy.** The core reports transport loss but does not reconnect.
|
||||
The iOS client snapshots the server, channel, voice subscription, mute, and deafen state, then
|
||||
reconnects with exponential backoff capped at 30 seconds. `NWPathMonitor` proactively replaces
|
||||
a live session when the active interface changes or becomes unavailable, avoiding the TCP
|
||||
keepalive delay; same-interface refreshes are ignored. A user-initiated disconnect cancels the
|
||||
retry task and path monitor. After authentication, the client rejoins the prior channel before
|
||||
restoring voice and local mute/deafen state.
|
||||
|
||||
## 8. Client-local features (no protocol changes)
|
||||
|
||||
Some features are entirely client-side and involve no changes to the wire format:
|
||||
|
||||
- **External PCM feed (`vc_stream_feed_pcm`)** — the caller supplies interleaved int16 PCM
|
||||
that the core frames, encodes, and sends over the existing UDP media path. From the server
|
||||
and peers' perspective the stream is indistinguishable from a hardware-captured stream. No
|
||||
new messages, fields, or tags are needed.
|
||||
- **PCM tap (`vc_set_pcm_sink`)** — receives decoded per-stream audio before hardware mixing.
|
||||
Entirely local to the listener; no protocol traffic of any kind.
|
||||
|
||||
These are noted here to prevent future contributors from looking for corresponding protocol
|
||||
changes: there are none.
|
||||
|
||||
## 9. Extensibility checklist
|
||||
|
||||
When adding a feature later (e.g. **file transfer**), the rules are:
|
||||
|
||||
1. Add new `oneof` arms in the reserved tag range (file transfer = 100–109) — never reuse
|
||||
or renumber existing tags.
|
||||
2. Advertise a feature string in `ClientHello`/`ServerHello`; only use the feature if both
|
||||
peers list it.
|
||||
3. Prefer extending an existing message with new fields (additive) over inventing a new
|
||||
message where it fits.
|
||||
4. For experimental/out-of-tree features, ride inside `Extension { ns; payload }` until it
|
||||
is promoted to a first-class `oneof` arm.
|
||||
|
||||
This guarantees a v1 client and a v3 server interoperate at the negotiated lowest common
|
||||
denominator.
|
||||
@@ -1,33 +0,0 @@
|
||||
# Roadmap
|
||||
|
||||
The managed rewrite is functionally complete. Work is now release validation, retirement of
|
||||
the old implementation, and focused product development.
|
||||
|
||||
## Release gates
|
||||
|
||||
- Windows: published-client smoke test, NVDA navigation, and sustained real calls.
|
||||
- macOS: VoiceOver matrix, multi-person call, Developer ID signing, notarization, Gatekeeper.
|
||||
- iOS: physical-device VoiceOver, background/lock, interruption and route recovery, Bluetooth,
|
||||
ReplayKit on iOS 18–26, and ScreenCaptureKit audio on iOS 27+.
|
||||
- Server: build and run the container, validate graceful shutdown, and complete a 30-minute or
|
||||
longer concurrent text/voice soak.
|
||||
|
||||
## Cleanup
|
||||
|
||||
- Remove the retired C++ core, server, CLI, tests, CMake presets, and vcpkg tree.
|
||||
- Remove the old Swift macOS/iOS applications after retained assets are detached.
|
||||
- Keep `native/media`, `native/rnnoise`, and `native/apple/broadcast`.
|
||||
- Remove the old Windows P/Invoke project after shared model types are moved into the managed
|
||||
compatibility facade.
|
||||
- Continue reducing historical documentation to current contracts and operating instructions.
|
||||
|
||||
## Deferred product features
|
||||
|
||||
- file transfer
|
||||
- end-to-end media encryption beyond the server-terminated transport encryption
|
||||
- persistent server-side text history
|
||||
- key-based user identities
|
||||
- multi-node/federated servers
|
||||
- PushKit/CallKit incoming-call behavior
|
||||
|
||||
These are not blockers for the current release.
|
||||
@@ -1,195 +0,0 @@
|
||||
# Security Model
|
||||
|
||||
Two encrypted transports: **TLS 1.3** on the TCP control channel, and an encrypted **UDP**
|
||||
media channel. Plus server identity, authentication, accounts at rest, and anti-replay.
|
||||
|
||||
> **Encryption is mandatory — there is no unencrypted mode.** The server has no plaintext
|
||||
> listener, the client has no "insecure" option, and there is no config flag to turn either
|
||||
> off. A connection is encrypted or it does not exist. This is a hard product rule, not a
|
||||
> default. It is also *zero-config* (see §1): the server generates its own key/cert on first
|
||||
> run, so "secured by default" never costs the operator a setup step.
|
||||
|
||||
## 1. Control channel — TLS 1.3 (settled)
|
||||
|
||||
- TCP control is wrapped in **TLS 1.3** (TLS 1.2 disabled). AEAD cipher suites only
|
||||
(AES-128-GCM, AES-256-GCM, ChaCha20-Poly1305). X25519 key exchange.
|
||||
- Library: **mbedTLS 3.6 LTS** — Apache-2.0 (permissive, fine for an eventual closed-source
|
||||
distribution), TLS 1.3 client+server, and `mbedtls_ssl_export_keying_material()` for the
|
||||
media path (§2). It also **static-links cleanly into a single self-host binary**, which is
|
||||
a deliberate choice in service of the easy-deploy goal. (OpenSSL 3.x, also permissive
|
||||
Apache-2.0, is a drop-in alternative behind the same internal interface.)
|
||||
- **Zero-config TLS:** on first launch the server auto-generates a self-signed certificate
|
||||
bound to a freshly generated **Ed25519 identity key** and persists both. The operator does
|
||||
nothing. Clients pin the identity on first connect (TOFU, §1.1). A server *with* a domain
|
||||
can drop in a CA cert later, but it is never required to be encrypted.
|
||||
- All authentication and account material crosses the wire only inside this tunnel.
|
||||
|
||||
### 1.1 Server identity — two modes
|
||||
|
||||
Self-hosting means most servers won't have a CA-signed cert for a hostname. We support
|
||||
both, advertised in `ServerHello`:
|
||||
|
||||
1. **TOFU (Trust On First Use)** — default for hobby servers. On first connect the client
|
||||
shows an identity dialog and, if accepted, pins the value locally. Subsequent connects
|
||||
verify the pin silently; a changed value warns loudly (`MISMATCH`).
|
||||
2. **PKI** — a server with a domain can use a normal CA-signed cert (e.g. Let's Encrypt);
|
||||
clients validate the chain conventionally. TOFU pinning still applies on top.
|
||||
|
||||
**What is actually pinned (M4 implementation):** the **TLS leaf certificate's SHA-256
|
||||
fingerprint** — verifiable directly from the TLS handshake before any application data is
|
||||
trusted. The server also declares an Ed25519 identity fingerprint in `ServerHello`, but this
|
||||
value is **display-only** and is *not* the value that is pinned or verified. Reason: the TLS
|
||||
cert and the Ed25519 identity key are generated independently with no cryptographic binding
|
||||
between them, so pinning the self-declared Ed25519 value (sent *inside* the channel being
|
||||
trust-decided) would be circular — an attacker who impersonates the server at the TLS level
|
||||
would supply whatever Ed25519 value they like. Pinning the TLS cert fingerprint is the only
|
||||
value that is genuinely verifiable at the moment of trust decision.
|
||||
|
||||
This is a known limitation of the current design. Closing it properly requires binding the
|
||||
Ed25519 key into the TLS cert (e.g. as a SubjectAltName or extension), which is a planned
|
||||
future improvement. Until then, clients display both values but gate on the cert fingerprint.
|
||||
|
||||
**Managed rewrite checkpoint:** `dotnet/` uses nonblocking BouncyCastle TLS 1.3 and
|
||||
captures directional exporters during handshake completion. Its client requires an
|
||||
explicit leaf-fingerprint acceptance callback; PKI validation remains unimplemented.
|
||||
New managed server certificates include the Ed25519 public key in SAN URI
|
||||
`urn:voicecat:identity:ed25519:<lowercase-public-key-hex>`. Existing pre-rewrite credentials
|
||||
are imported unchanged. Verifying that URI against the declared ServerHello identity
|
||||
is still deferred to the managed session layer; leaf-certificate TOFU remains the
|
||||
trust gate. Missing members of a persisted credential set cause startup rejection
|
||||
rather than automatic identity rotation. See [api-dotnet.md](api-dotnet.md).
|
||||
|
||||
Client certificates are reserved for a future "key-based identity" option (see roadmap) but
|
||||
are not required in v1.
|
||||
|
||||
## 2. Media channel — UDP encryption (settled: exported-keys + AEAD)
|
||||
|
||||
The UDP media path uses **TLS-exported keys + per-packet AEAD** (an SRTP-style design),
|
||||
mandatory from the first build. This was chosen over DTLS after weighing two findings:
|
||||
|
||||
> **Finding 1 — DTLS 1.3 (RFC 9147) is not in stable OpenSSL or mbedTLS.** It ships
|
||||
> production-ready only in **wolfSSL**, which is **GPLv2-or-commercial** — disqualified,
|
||||
> because the code will eventually be distributed in closed-source form (no GPL/LGPL deps).
|
||||
>
|
||||
> **Finding 2 — mbedTLS 3.6 LTS already exposes `mbedtls_ssl_export_keying_material()`**
|
||||
> (RFC 5705 / RFC 8446 §7.5 exporter). So we can derive media keys from the existing TLS 1.3
|
||||
> control session with *zero* extra handshake and *zero* extra dependency.
|
||||
|
||||
### How it works
|
||||
|
||||
1. After the TLS 1.3 control handshake, both sides call the keying-material exporter with
|
||||
label `"voicecat media v1"` and a one-byte context: `0x00` for client→server,
|
||||
`0x01` for server→client. Each export yields a 32-byte directional media key.
|
||||
No second handshake, no certificates on the UDP path — the UDP channel inherits the
|
||||
authenticated, MITM-resistant TLS session's trust.
|
||||
2. Each UDP voice frame is sealed with **ChaCha20-Poly1305** (managed platform crypto;
|
||||
platform cryptography with a BouncyCastle fallback in .NET).
|
||||
3. The full 20-byte header is AEAD **associated data**. The server authenticates/decrypts
|
||||
inbound media and reseals for each recipient, replacing the sequence with that
|
||||
recipient's next send counter. It forwards the encoded Opus bytes without decoding audio.
|
||||
|
||||
This keeps the entire crypto surface on two permissive libraries (mbedTLS + libsodium), adds
|
||||
no handshake latency to voice startup, and is small enough to audit fully. It is abstracted
|
||||
behind a `MediaCrypto { seal(frame)->bytes; open(bytes)->frame }` interface, so a future
|
||||
DTLS 1.3 backend could slot in later if a permissive implementation matures — but nothing in
|
||||
the design depends on that.
|
||||
|
||||
### Per-frame protections
|
||||
|
||||
- **AEAD** (ChaCha20-Poly1305) over each voice frame — confidentiality + integrity.
|
||||
- **Associated data:** all 20 header bytes remain visible and authenticated; the Opus
|
||||
payload is encrypted and followed by a 16-byte tag.
|
||||
- **Nonce discipline:** `nonce = four_zero_bytes ‖ counter_u64_big_endian`. Counters are
|
||||
per directional session key, shared across its streams. Direction separation comes
|
||||
from exporter contexts, not nonce bits. Automatic epoch rekeying is not implemented;
|
||||
the .NET encryptor refuses counter exhaustion and requires a new session.
|
||||
- **Anti-replay:** a 64-bit sliding-window replay filter keyed on the packet counter (à la
|
||||
IPsec). The window is **advanced only after the AEAD tag verifies** (RFC 3711 §3.3 order:
|
||||
replay-check → authenticate → update). The counter is read from the unauthenticated
|
||||
header, so advancing the high-water mark *before* authentication would let a single
|
||||
corrupted or forged packet jump it far ahead, after which every legitimate packet is
|
||||
rejected as "too old" — a permanent denial of the whole stream. Failed-auth packets leave
|
||||
the window untouched. Replays and out-of-window packets are dropped before decode.
|
||||
|
||||
## 3. UDP session binding
|
||||
|
||||
UDP packets are not individually authenticated to a *user* beyond the transport session.
|
||||
Binding works as:
|
||||
|
||||
1. `AuthResult.udp_token` (issued over TLS) is a random 16-byte token tied to the
|
||||
authenticated session. The client confirms it with `UdpBinding` over TLS.
|
||||
2. Protocol v2 bootstraps UDP with a **plaintext** `UDP_BINDING` packet: the 20-byte
|
||||
binary header followed by the token. This is not a protobuf or an AEAD voice frame.
|
||||
3. Server validates the token and binds the **5-tuple → session_id**. The managed server
|
||||
accepts the first endpoint only; further bootstrap packets cannot replace it.
|
||||
Endpoint changes require a new authenticated session. The token remains available
|
||||
for TLS confirmation but cannot establish a second binding. Session removal retires
|
||||
its endpoint, token and directional keys. Unsupported old releases permitted rebinding
|
||||
with the same token; this differs in policy, not in the packet format.
|
||||
4. Thereafter, frames are accepted only on that bound tuple; ssrcs are checked against the
|
||||
streams the session announced. Source-address spoofing can't hijack a session because the
|
||||
attacker lacks the media key. The bootstrap token is visible on UDP, so it is not
|
||||
a substitute for AEAD authentication and SSRC ownership checks. Header-only keepalives
|
||||
are echoed only for bound endpoints; they provide liveness, not authenticated content.
|
||||
|
||||
## 4. Authentication & accounts (settled: guests + local accounts)
|
||||
|
||||
- **Guests:** toggled by server config (`allow_guests`). A guest picks a nickname and joins;
|
||||
no persistent identity. Nicknames are non-reserved and may be uniquified by the server.
|
||||
- **Local accounts:** username + password, **admin-provisioned** (no self-serve registration
|
||||
in v1). An admin creates/resets/deletes accounts either via the `voicecat-admin` CLI or the
|
||||
in-app admin interface, which sends the privileged `CreateAccount`/`ResetPassword`/
|
||||
`DeleteAccount` messages (protocol.md §3, permission-gated). Stored in **SQLite**; passwords
|
||||
hashed with **Argon2id** (via libsodium `crypto_pwhash`) using per-install-tuned memory/time
|
||||
parameters; never stored or logged in plaintext. Verification runs on the worker pool (it's
|
||||
deliberately slow) to avoid stalling the net thread.
|
||||
- **Channel passwords:** hashed at rest with **BLAKE2b** (libsodium `crypto_generichash`) plus a per-channel salt. BLAKE2b is used instead of Argon2id here because channel-password checks happen on the net thread during `JoinChannelRequest`; a slow hash would block real-time message processing. The password itself still crosses the wire only inside TLS 1.3.
|
||||
- **Brute-force defense:** per-IP and per-account rate limiting on auth attempts with
|
||||
exponential backoff; configurable lockout. Generic `auth_request` failures return a
|
||||
non-enumerating error ("invalid credentials") to avoid username probing.
|
||||
|
||||
```
|
||||
accounts( id INTEGER PK, username TEXT UNIQUE,
|
||||
pw_argon2id TEXT, -- encoded hash incl. params + salt
|
||||
created_at, last_login, flags )
|
||||
bans( id, subject_type, subject, reason, expires_at, created_at )
|
||||
```
|
||||
|
||||
## 5. Permissions (scaffold for v1, enforced server-side)
|
||||
|
||||
A `Permissions` set is attached to each session at auth time and is the *only* authority —
|
||||
clients never self-grant. v1 needs a minimal set (join channel, send text, create temporary
|
||||
channel, kick/move if moderator); the model is a role/flag bitset that the moderation
|
||||
milestone expands. All privileged operations (`CreateChannel`, `Kick`, `Ban`,
|
||||
`MoveUser`, server-mute) are checked against it server-side regardless of client UI.
|
||||
|
||||
## 6. Threat model & non-goals
|
||||
|
||||
**In scope:**
|
||||
- Passive eavesdropping on either transport → defeated by TLS 1.3 (control) and the
|
||||
exported-key AEAD (media).
|
||||
- Active MITM on first connect → mitigated by TOFU pin + Ed25519 identity (user must verify
|
||||
fingerprint out-of-band for the strongest guarantee).
|
||||
- UDP source spoofing / session hijack → defeated by token binding + media-key secrecy +
|
||||
anti-replay.
|
||||
- Password theft at rest → mitigated by Argon2id; in transit → only inside TLS.
|
||||
|
||||
**Explicit non-goals (v1):**
|
||||
- **End-to-end encryption between users.** The server relays Opus and can see who talks to
|
||||
whom; with the SFU relay it does *not* decode audio, but the media key is per
|
||||
client↔server, not per pair. True E2EE (server can't read media) is a possible future
|
||||
feature, not v1.
|
||||
- Anonymity / metadata hiding. The server, by design, knows the channel graph.
|
||||
- DoS resilience at scale beyond basic rate limiting and the bounded-frame guards.
|
||||
|
||||
## 7. Crypto dependency summary
|
||||
|
||||
Two libraries, both permissive (no GPL/LGPL), so a future closed-source distribution stays
|
||||
clean:
|
||||
|
||||
- **TLS 1.3:** **mbedTLS 3.6 LTS** (Apache-2.0). Control-channel TLS + the keying-material
|
||||
exporter that seeds the media path. Static-links into a single binary. (OpenSSL 3.x,
|
||||
Apache-2.0, is an interchangeable alternative.)
|
||||
- **Primitives & password hashing:** **libsodium** (ISC) — Argon2id (account passwords),
|
||||
ChaCha20-Poly1305 (the media AEAD), Ed25519 (server identity), X25519, secure RNG. All
|
||||
non-TLS crypto goes through libsodium so we never hand-roll a primitive.
|
||||
+1
-1
@@ -14,7 +14,7 @@ leaf platform projects.
|
||||
| Persistence | Microsoft.Data.Sqlite | Server-local SQLite with explicit migrations |
|
||||
| Tests | xUnit | Unit, socket integration, allocation, CLI, and production-package checks |
|
||||
|
||||
NuGet packages and their lock files are committed. `dotnet/check-licenses.ps1` enforces the
|
||||
NuGet packages and their lock files are committed. `scripts/check-licenses.ps1` enforces the
|
||||
permissive-license policy.
|
||||
|
||||
## Native media
|
||||
|
||||
-468
@@ -1,468 +0,0 @@
|
||||
# Voice & Media
|
||||
|
||||
Real-time audio runs over **UDP**, secured per [security.md](security.md). The control
|
||||
channel (TCP/TLS) handles *signaling* — announcing streams, channel membership, talk state
|
||||
— while UDP carries only the encoded audio frames. This split keeps media latency low and
|
||||
independent of TCP head-of-line blocking.
|
||||
|
||||
## 1. The multi-stream model
|
||||
|
||||
A **user** publishes one or more **streams**. Each stream is an independent audio source
|
||||
with its own encoder, its own `stream_id` (unique per user) and `ssrc` (media-plane id
|
||||
assigned by the server), and is independently mutable/mutable at the receiver.
|
||||
|
||||
```
|
||||
User "Alex" Receiver "Sam"
|
||||
┌────────────────────┐ ┌──────────────────────────┐
|
||||
│ mic → enc ───┼──ssrc 1001──▶ │ jitter(1001)→dec→┐ │
|
||||
│ desktop → enc ───┼──ssrc 1002──▶ │ jitter(1002)→dec→┤ │
|
||||
│ 2nd mic → enc ───┼──ssrc 1003──▶ │ jitter(1003)→dec→┴─mix──▶ out
|
||||
└────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
Stream **kinds** (v1): `MIC`, `SCREEN_AUDIO` (system/desktop audio for listening together),
|
||||
`AUX_DEVICE` (a second capture device). Receivers can set, per incoming stream: **gain**,
|
||||
**mute**, and **noise reduction** (see §10) — so Sam can turn down Alex's desktop audio
|
||||
while keeping the mic, *and* independently apply noise suppression to a third user who has a
|
||||
loud fan. The mixer sums all active streams from all users in the channel into the local
|
||||
playback device. All of these receiver-side controls are **local to the listener** and carry
|
||||
no protocol traffic.
|
||||
|
||||
`SCREEN_AUDIO` capture is platform-specific and covered in §9 — it is supported on Windows,
|
||||
macOS, **and iOS** (via a ReplayKit broadcast extension).
|
||||
|
||||
## 2. Voice frame format (UDP payload, inside the media AEAD)
|
||||
|
||||
A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.
|
||||
|
||||
The header is **20 bytes** (protocol v2; v1 was 14 bytes with a u16 seq — see note below).
|
||||
|
||||
```
|
||||
0 1 2 3 4 5 6 7 8 ............ 15
|
||||
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
|
||||
│ type │flags │ codec │ ssrc (u32) │ seq (u64) ──▶ │
|
||||
├──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴───────────────┤
|
||||
│ ◀── seq (u64) ──┤ timestamp (u32 @48k) │ payload ... │
|
||||
└──────────────────┴────────────────────────────────────┴───────────────┘
|
||||
bytes [8..15] = seq (u64) [16..19] = timestamp (u32)
|
||||
|
||||
type u8 1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
|
||||
flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present
|
||||
bit2 DTX/comfort-noise · bit3 last-frame-before-stop
|
||||
codec u16 0 = OPUS (room for future codecs)
|
||||
ssrc u32 media-plane stream id. Client sends its own ssrc; the server
|
||||
validates it against the bound session and relays unchanged.
|
||||
seq u64 full monotonic send counter. This IS the AEAD nonce counter, so the
|
||||
receiver derives the nonce directly from it — no rollover guessing.
|
||||
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
|
||||
payload one Opus packet (the encoder's output for one frame)
|
||||
```
|
||||
|
||||
> **Why u64 (protocol v2).** v1 carried only the low 16 bits of the counter and the
|
||||
> receiver zero-extended them to rebuild the AEAD nonce. After 65,536 frames the seq
|
||||
> wrapped, the reconstructed nonce diverged from the sealing nonce, and **every frame
|
||||
> failed authentication permanently** (no rollover counter). v2 puts the full 64-bit
|
||||
> counter on the wire so the nonce is always exact. A v2 server and a v1 client cannot
|
||||
> interoperate; the `Hello` handshake rejects on `proto_version` mismatch.
|
||||
|
||||
This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's
|
||||
full machinery. The server authenticates/decrypts each incoming packet and reseals its
|
||||
encoded Opus bytes for each recipient using that recipient's directional key and send
|
||||
counter. SSRC, timestamp, flags, and codec pass through; sequence and ciphertext/tag change.
|
||||
There is no server-side audio decoding or transcoding.
|
||||
|
||||
### Why client-sends-ssrc is safe
|
||||
|
||||
The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks
|
||||
that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are
|
||||
dropped. So identity is anchored by the session binding + transport encryption, not by
|
||||
trusting the header.
|
||||
|
||||
## 3. Per-channel audio configuration
|
||||
|
||||
Opus is configured **per channel** and pushed to clients in `JoinChannelResult.audio` /
|
||||
`StreamAnnounceResult.effective_audio`. All members of a channel encode with mutually
|
||||
decodable parameters.
|
||||
|
||||
```proto
|
||||
message AudioConfig {
|
||||
uint32 codec = 1; // 0 = OPUS
|
||||
ChannelMode mode = 2; // MONO / STEREO
|
||||
uint32 sample_rate = 3; // 8000/12000/16000/24000/48000 (48000 recommended)
|
||||
uint32 bitrate_bps = 4; // e.g. 24000 (speech) … 128000 (music/stereo)
|
||||
uint32 frame_ms = 5; // 2.5/5/10/20/40/60 (20 default)
|
||||
OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY
|
||||
bool fec = 7; // in-band forward error correction
|
||||
uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness
|
||||
bool dtx = 9; // discontinuous transmission (silence suppression)
|
||||
uint32 complexity = 10; // 0..10 encoder complexity
|
||||
}
|
||||
```
|
||||
|
||||
Guidance baked into defaults / docs:
|
||||
|
||||
- **Sample rate: always run Opus at 48 kHz internally.** Opus resamples internally anyway;
|
||||
48 kHz avoids surprises, and the whole audio stack (capture, `vc_stream_feed_pcm`, mixing,
|
||||
playback) runs at 48 kHz. The per-channel `sample_rate` field is **channel-authoritative**
|
||||
(not a client request) and does *not* change the codec/PCM clock — it caps the encoder's
|
||||
audio bandwidth via `OPUS_SET_MAX_BANDWIDTH` (8000 → narrowband ~4 kHz, 16000 → wideband
|
||||
~8 kHz, 24000 → super-wideband ~12 kHz, 48000 → full ~20 kHz). This lets a low-bitrate room
|
||||
shed out-of-band content while every endpoint keeps a single 48 kHz clock. Default **48000**
|
||||
(full band). See `OpusEncoder::init` and `vc_client::opus_params_from_audio_config`.
|
||||
- **Frame size: 20 ms default.** Smaller (10 ms) lowers latency at the cost of more
|
||||
per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at
|
||||
the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms.
|
||||
The capture engine runs on a fixed 48 kHz / 20 ms clock (960-sample frames), so the send
|
||||
path **reframes** each captured/fed block to the channel's `frame_ms` before encoding
|
||||
(accumulating two 960-frames for a 40 ms channel, splitting each into two 480-frames for a
|
||||
10 ms channel, etc.). This keeps the hardware/`vc_stream_feed_pcm` contract a single 48 kHz
|
||||
clock regardless of the channel's window — see `vc_client::on_capture_frame`.
|
||||
The device/mixer quantum is not the Opus packet duration: managed send streams reframe it
|
||||
into the channel's 5/10/20/40/60 ms packets, and receive streams decode at that duration
|
||||
before slicing decoded PCM back into 20 ms mixer blocks.
|
||||
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 24–32 kbps, DTX on, FEC on.
|
||||
Music/screen-audio channels → `STEREO`, `AUDIO`, 96–128 kbps, DTX off, FEC optional.
|
||||
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
|
||||
monitoring use cases.
|
||||
|
||||
## 4. Packet-loss resilience (Opus 1.6)
|
||||
|
||||
Layered, all configurable per channel:
|
||||
|
||||
1. **In-band FEC** — the encoder embeds a low-bitrate copy of the current frame in the
|
||||
*next* packet (`OPUS_SET_INBAND_FEC`, redundancy scaled by `expected_packet_loss`). On a
|
||||
loss, the receiver decodes that copy out of the next already-buffered packet with
|
||||
`opus_decode(..., decode_fec=1)` — costing one frame of latency on recovery. Gated on the
|
||||
per-stream `fec` flag; if the next packet carries no redundancy libopus yields PLC output,
|
||||
so it is at worst a no-op relative to (2).
|
||||
2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an
|
||||
unrecovered loss; always on, free. The terminal fallback when neither DRED nor FEC applies.
|
||||
3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise
|
||||
updates; cuts bandwidth and is bandwidth-friendly on busy channels.
|
||||
4. **DRED (Deep REDundancy, per-channel toggle)** — Opus 1.6's ML redundancy: the encoder
|
||||
embeds 20 ms of acoustic features in every packet (`bool dred` in `AudioConfig`, off by
|
||||
default). When a packet is lost, the receiver peeks at the next already-buffered packet,
|
||||
parses its DRED extension (`opus_dred_parse`), and reconstructs the lost frame with
|
||||
`opus_decoder_dred_decode` — producing significantly better audio than PLC comfort noise
|
||||
for single-frame gaps. Heavier CPU on the encoder (~5–10 % at 24 kbps); minimal overhead
|
||||
on the decoder (parse is a fast header check on non-DRED packets).
|
||||
|
||||
When a frame is lost, `AudioEngine::on_playback` tries these recovery paths in quality order,
|
||||
falling through on failure: **DRED → in-band FEC → PLC**. DRED and FEC both need the next
|
||||
packet already buffered (one frame of look-ahead); when it has not arrived yet, recovery falls
|
||||
straight through to PLC.
|
||||
|
||||
## 5. Jitter buffer
|
||||
|
||||
Each receiver keeps an **adaptive jitter buffer per ssrc** with **bounded-depth playout**
|
||||
(`dotnet/src/VoiceCat.Audio/ReceiveStream.cs`).
|
||||
|
||||
- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
|
||||
- **The playout clock is always bounded against the stream's *leading edge* (newest buffered
|
||||
frame), never re-synced to the oldest.** The managed playout clock is isolated from hardware
|
||||
drift by the adaptive PCM ring,
|
||||
while the sender omits VAD/PTT/DTX silence from its timestamps, so the two diverge across gaps
|
||||
and late joins. Two corrections keep latency bounded:
|
||||
- **(Re)seed to the leading edge** on first frame, on a talkspurt `marker`, or when the clock
|
||||
has run past the newest frame (starved after silence). Playout retains the adaptive target;
|
||||
when DRED/FEC is enabled that includes one codec frame of recovery look-ahead.
|
||||
- **Frame-skip catch-up:** when the backlog grows past `target + hysteresis` (clock drift,
|
||||
bursty arrival, reordering), fast-forward the clock to leave `target` buffered and drop the
|
||||
now-stale frames. This is the downward force that prevents latency from ratcheting upward.
|
||||
- `target` is an EWMA of arrival-gap variation measured in 48 kHz sample time. DRED or FEC
|
||||
reserves one complete channel Opus frame of look-ahead, variation can raise the target to
|
||||
120 ms, and packet history is bounded to 500 ms. Limits are durations rather than packet
|
||||
counts, so 5 ms and 60 ms channels receive the same policy.
|
||||
- Late frames past the playout point are dropped; gaps are filled by DRED (if the next frame
|
||||
arrived) or PLC.
|
||||
- The `marker` flag (start of talkspurt) — set by the sender on the first frame after a
|
||||
transmission gap — lets the buffer reseed cleanly after silence/DTX without accumulating drift.
|
||||
- One late device callback does not end a talkspurt. Capture resets only after 200 ms of
|
||||
continuous starvation, avoiding a marker/rebuffer cascade from an isolated scheduling miss.
|
||||
- Diagnostics per stream: `packets_lost`, `duplicates`, `underruns`, `target_depth_ms`.
|
||||
|
||||
```
|
||||
incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer
|
||||
▲
|
||||
jitter estimate feeds depth
|
||||
```
|
||||
|
||||
Capture and playback use an allocation-free adaptive PCM ring between the managed 20 ms clock
|
||||
and the hardware clock. Linear-interpolation correction, limited to ±0.5%, holds the ring near
|
||||
its target instead of periodically dropping a block or rendering silence as device clocks drift.
|
||||
The local **Audio buffering** preset is 20, 40 (default), or 60 ms and is persisted per client.
|
||||
|
||||
## 6. UDP keepalive & NAT
|
||||
|
||||
- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to
|
||||
hold NAT bindings and measure media-path RTT/loss independent of TCP. The frame is
|
||||
plaintext (20-byte header, no payload, no AEAD) — the server identifies the sender by
|
||||
its already-verified UDP endpoint (established during the `UdpBinding` handshake). On
|
||||
receipt the server bumps the sender's `last_seen` (so media activity defers the TCP
|
||||
reaper independently of control-channel traffic) and echoes the frame back so the
|
||||
client can measure media-path RTT.
|
||||
- If the media path dies but TCP is alive, the client surfaces a "voice disconnected"
|
||||
state and attempts UDP re-binding (re-derive media keys + fresh `UdpBinding`) without dropping
|
||||
the control session.
|
||||
- No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the **server** is reachable
|
||||
(public IP or port-forward); **clients** sit behind NAT and initiate, so their bindings
|
||||
are created by their outbound first packet.
|
||||
|
||||
## 7. Talk-state signaling
|
||||
|
||||
"Who is talking" can be derived two ways; we use both:
|
||||
|
||||
- **Implicit:** presence of recent voice frames for an ssrc → that stream is "active". The
|
||||
receiver drives talk indicators from the jitter buffer, so they're accurate and need no
|
||||
extra messages.
|
||||
- **Explicit (optional):** `StreamStateUpdate` on TCP for coarse UI state (muted, hold) and
|
||||
for users not currently subscribed to the media. Server-side mute/deafen is authoritative
|
||||
and always signaled on TCP.
|
||||
|
||||
## 8. Capture/playback pipeline (inside the core)
|
||||
|
||||
```
|
||||
mic device ─(miniaudio capture, 48k, mono)→ resample? → send-side VAD/PTT gate
|
||||
→ Opus encode → frame header → AEAD → UDP send
|
||||
|
||||
screen audio ─(WASAPI loopback, 48k, mono or stereo per channel mode)→ Opus encode
|
||||
→ frame header → AEAD → UDP send
|
||||
|
||||
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
|
||||
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
|
||||
→ mixer (sum all ssrc, stereo; mono streams upmixed L=R) → (miniaudio playback,
|
||||
48k, stereo) → device
|
||||
```
|
||||
|
||||
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
|
||||
Playback is genuinely stereo end-to-end. **Mic capture** is mono by default; **stereo mic
|
||||
capture** is supported via `vc_set_capture_channels(stream_id, 2)` — when enabled, the
|
||||
capture device opens in stereo (interleaved L/R). All native clients expose this as a
|
||||
per-user toggle (iOS in Settings; the Windows and macOS desktop clients via a "Stereo
|
||||
microphone" checkbox in Audio settings — a live toggle there restarts the capture device via
|
||||
`vc_audio_restart` so it takes effect immediately). Whether stereo actually reaches the wire
|
||||
depends on the **channel's** Opus mode, which decides the encoder's channel count — the mic's
|
||||
channel count and the channel's mode are independent knobs:
|
||||
- **stereo mic + stereo channel** → real interleaved L/R is encoded directly (no upmix).
|
||||
- **mono mic + stereo channel** → the mono frame is upmixed L=R so the Opus bitstream is
|
||||
still spec-correct stereo.
|
||||
- **stereo mic + mono channel** → the interleaved L/R is folded to mono before the mono
|
||||
encoder. (Handing interleaved pairs straight to a mono `opus_encode` would make it read 2×
|
||||
the samples it should — wrong pitch / garbage — so the fold keeps the toggle safe on any
|
||||
channel.)
|
||||
**Screen-audio (`SCREEN_AUDIO`) loopback** captures
|
||||
in the channel's mode — stereo when the channel is stereo (real interleaved L/R, no
|
||||
downmix), mono when the channel is mono — so a stereo music/screen-share channel gets
|
||||
genuine stereo end-to-end. See §9 for the platform-specific loopback mechanism.
|
||||
- **iOS audio — one path, always external.** On iOS the core **never opens a miniaudio device**:
|
||||
a single `AVAudioEngine` (`IOSAudioEngine`) drives *both* directions, and the core runs fully
|
||||
external for the whole connection. This is the single most important property of the iOS audio
|
||||
stack — there is no second (miniaudio) path to switch to, so a preset/route change cannot leave
|
||||
one direction dropped. The single ordering rule is: `vc_set_external_playback(1)` is set **once
|
||||
at connect** (before the session is activated or any remote stream arrives), and every MIC
|
||||
stream is started with `vc_stream_desc.external_feed=1`.
|
||||
- **core → speaker:** the core's mixer-timer thread decodes+mixes on a ~20 ms cadence and
|
||||
delivers the FINAL mixed PCM via `vc_set_mixed_output_sink`; an `AVAudioSourceNode` pulls it
|
||||
from a lock-free ring and renders it. This runs the whole time we are connected, so remote
|
||||
audio plays even before the user joins voice (kills the "can't hear anyone" race).
|
||||
- **mic → core:** when the mic is active a tap on the engine's input node converts to 48 kHz
|
||||
int16 (`vc_set_capture_channels` decides mono/stereo) and calls `vc_stream_feed_pcm`.
|
||||
- **iOS routing** is still driven from Swift via `AVAudioSession` by the `IOSAudioRouter`
|
||||
singleton — miniaudio never touches `AVAudioSession` on iOS. Input port selection
|
||||
(`availableInputs`), built-in mic orientation (`setPreferredDataSource`: front/back/top/bottom),
|
||||
polar patterns (`setPreferredPolarPattern`: omni/cardioid/subcardioid/bidirectional), mic
|
||||
processing mode (Standard vs `.measurement` Raw), Bluetooth mode (`.allowBluetoothHFP` HFP voice
|
||||
vs `.allowBluetoothA2DP` stereo output vs neither), and stereo capture (`.stereo` polar pattern
|
||||
+ `setPreferredInput` + `setInputDataSource` → `vc_set_capture_channels`) are all set from
|
||||
Swift. Any preset / route / interruption change funnels through one deterministic, Swift-only
|
||||
rebuild: `IOSAudioEngine` stops, `IOSAudioRouter.applyConfiguration()` re-applies the
|
||||
`AVAudioSession`, the graph is rebuilt against the new route, and the engine restarts. No
|
||||
`vc_audio_restart`/`vc_audio_suspend` dance is needed for routing (the core has no hardware
|
||||
devices to reopen) — this is the spirit of TeamTalk5's "close then re-init sound devices", but
|
||||
entirely inside the Swift engine.
|
||||
- **iOS voice processing (AEC/NS/AGC) — native VPIO.** Real iOS echo cancellation, noise
|
||||
suppression and AGC come ONLY from Apple's **Voice-Processing I/O audio unit (VPIO)**, which
|
||||
`inputNode.setVoiceProcessingEnabled(true)` enables; for it to cancel echo it must own BOTH the
|
||||
mic capture and the playback — which the unified engine already does. VPIO forces **mono**, so
|
||||
it is engaged only when the active config wants it (`IOSAudioRouter.currentConfigUsesVoiceProcessing`:
|
||||
mono + standard + non-A2DP + the user's master toggle). iOS exposes no per-stage VPIO control,
|
||||
so the Advanced UI offers exactly two switches: a master **Voice Processing** (AEC + NS bundled)
|
||||
and **AGC** (`isVoiceProcessingAGCEnabled`).
|
||||
- **iOS presets** (`IOSAudioRouter.AudioPreset`): **Voice Chat** (VPIO mono, system output incl.
|
||||
HFP/wired), **Stereo Mic** (internal stereo built-in mic regardless of output, A2DP-capable, no
|
||||
VPIO), **Mono Mic** (internal mono built-in mic regardless of output, A2DP-capable, no VPIO),
|
||||
and **Advanced** (every knob manual). A2DP output requires an internal-mic preset (the Bluetooth
|
||||
device is output-only); the Stereo/Mono Mic presets fall back to the built-in speaker when no
|
||||
external output is connected (`applyA2dpSpeakerFallback`).
|
||||
- **iOS implementation invariants:**
|
||||
- External playback is enabled before connecting because authentication can start the audio
|
||||
engine before the UI receives another turn.
|
||||
- AVAudioEngine callbacks may contain multiple codec frames. The mic path writes them to an
|
||||
SPSC ring and releases complete 20 ms frames at a steady cadence; it never consumes a partial
|
||||
frame, and the pacer is recreated when mono/stereo capture changes.
|
||||
- `AVAudioSession` setters can synchronously emit route-change notifications. Configuration is
|
||||
re-entrancy guarded, and recovery ignores `.categoryChange`, `.routeConfigurationChange`, and
|
||||
`.override` because those reasons are generated by the app's own routing calls. External route
|
||||
changes and engine-configuration notifications still rebuild the graph.
|
||||
- Stereo capture anchors the built-in mic's stereo data source. It does not call
|
||||
`setPreferredInputNumberOfChannels(2)`, which can disrupt A2DP output; the core receives the
|
||||
channel count through `vc_set_capture_channels`.
|
||||
- The A2DP speaker fallback caches its last output override. Reapplying the same override would
|
||||
emit another `.override` notification and recursively trigger recovery.
|
||||
- **DSP engine: see §11.** The original plan was `webrtc-audio-processing` (AEC + NS + AGC +
|
||||
VAD in one tuned module, BSD-licensed) — but it has no working Windows/MSVC build upstream
|
||||
(confirmed via its own issue tracker: GCC-only Meson build, MinGW support unfinished, hard
|
||||
`abseil-cpp` dependency, Linux-tested only —
|
||||
[gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing#1](https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing/-/issues/1)).
|
||||
v1 ships a lightweight, dependency-free energy/RMS VAD instead (§11); there is **no AEC, NS,
|
||||
or AGC implementation at all yet** — not just a deferred VAD, the whole APM is unbuilt. Real
|
||||
`webrtc-audio-processing` stays a tracked future swap, behind the same `ApmProcessor`
|
||||
interface (`core/src/audio/apm_processor.h`), revisit if/when a Linux build target exists or
|
||||
upstream Windows support matures.
|
||||
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
|
||||
|
||||
## 10. Noise reduction — two-sided
|
||||
|
||||
Noise reduction can be applied **at the sender, at the listener, or both** — they are
|
||||
independent.
|
||||
|
||||
- **Sender-side** (the talker's choice): the publishing client runs noise suppression on its
|
||||
mic before the input gain and the VAD/PTT gate, controlled by that user's own settings
|
||||
(`vc_set_input_noise_reduction`). This cleans the signal for *everyone* in one pass and helps
|
||||
bitrate/VAD. MIC stream only.
|
||||
- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding
|
||||
each stream and *before* mixing, the listener can enable an **additional** NS pass on a
|
||||
**specific** sender's stream (`vc_set_remote_stream(..., noise_reduction)`). So even if Alex
|
||||
chose not to denoise his mic, Sam can locally suppress Alex's background noise without
|
||||
affecting how anyone else hears Alex.
|
||||
|
||||
**Backend: RNNoise** (vendored in [`native/rnnoise/`](../native/rnnoise), BSD-3 + CC0).
|
||||
The original plan was WebRTC's APM, but `webrtc-audio-processing` has no working Windows/MSVC
|
||||
build (see §8). RNNoise is a small, dependency-free C library — a hybrid DSP/RNN speech denoiser
|
||||
that runs ~60× faster than real time. Both NR paths share one `ApmProcessor` implementation
|
||||
(`RnnoiseProcessor`, `core/src/audio/apm_processor.cpp`), selected by `ApmProcessor::create()`
|
||||
when the core is built with `VOICECAT_HAS_NS` (a no-op `ApmPassthrough` otherwise). Allocation
|
||||
happens at construction; `process_capture()` runs lock-free on the RT thread (architecture.md §3).
|
||||
|
||||
RNNoise is a **mono, 48 kHz, 480-sample (10 ms)** denoiser. Our engine clock is fixed at 48 kHz
|
||||
and every Opus frame size (480/960/1920/2880) is a multiple of 480, so frames are processed as
|
||||
whole 480-sample chunks with no resampling. Because it's mono-only:
|
||||
- **Send-side:** a stereo mic is downmixed to mono **only when NR is enabled** — with NR off a
|
||||
stereo mic keeps full stereo (we never collapse mic quality unless asked).
|
||||
- **Receive-side:** NR applies to **voice (MIC) streams only**, gated on the stream *kind* — not
|
||||
on its channel count, since a stereo mic with send-side NR off now arrives as stereo voice.
|
||||
When enabled on such a stream the decoded stereo frame is folded to mono, denoised, and
|
||||
duplicated back across both channels (symmetric with the send-side downmix), so that stream
|
||||
plays as mono while NR is on. A **screen-audio share is never voice and is left untouched** —
|
||||
denoising music/video with a speech denoiser would mangle it.
|
||||
|
||||
Implementation: a per-`ssrc` NS instance (`RemoteStream::recv_ns`) on the receive path,
|
||||
instantiated lazily only for streams the listener has flagged; the send-side instance
|
||||
(`vc_client::mic_ns_`) is built once with the MIC stream and gated by an atomic flag so toggling
|
||||
never allocates on the capture callback. State lives entirely on the local machine; toggling
|
||||
either is a local UI action with **no protocol message** and no effect on other users. Because
|
||||
each receive stream is decoded independently before the mixer (voice.md §1), per-user receive
|
||||
NS is a clean drop-in on that per-stream stage.
|
||||
|
||||
All three receive-side controls (gain, mute, NR) are queryable via `vc_get_remote_stream` —
|
||||
the counterpart to `vc_set_remote_stream` — so a UI can reopen its per-stream mix controls at
|
||||
the listener's actual current settings (defaults: gain 1.0, unmuted, NR off). Like the setter,
|
||||
it carries no protocol traffic.
|
||||
|
||||
## 11. Input activation — VAD and PTT (client-configurable)
|
||||
|
||||
Whether the mic transmits is decided locally by the **input gate**, and the client supports
|
||||
**both** modes, switchable per client (`vc_set_input_mode`):
|
||||
|
||||
- **Voice activation (VAD):** v1 implements this as a lightweight, dependency-free
|
||||
energy/RMS-threshold VAD (`EnergyVadProcessor`, `core/src/audio/apm_processor.cpp`) — no
|
||||
external DSP dependency, since real `webrtc-audio-processing` has no working Windows/MSVC
|
||||
build (see §8). It opens the gate when a frame's RMS exceeds a configurable threshold
|
||||
(default ~0.025, normalized to int16 range), with a configurable hang-time (default 300 ms,
|
||||
matching the talk-indicator hangover so "talking" and "gate open" agree) to avoid clipping
|
||||
word tails. DTX naturally complements this — when the gate is closed nothing (or only
|
||||
comfort noise) is sent. This implementation has **no AEC** — a real limitation versus the
|
||||
originally-planned APM, not just a deferred VAD.
|
||||
- **Push-to-talk (PTT):** `vc_set_push_to_talk(active)` opens/closes the gate directly. The UI
|
||||
exposes a configurable keybind; the core just receives gate open/close.
|
||||
|
||||
Gating applies to the **MIC stream only** — `SCREEN_AUDIO`/`AUX_DEVICE` always bypass it
|
||||
(gating a desktop-audio share on the user's own voice activity would silently drop shared
|
||||
music/video audio whenever the user isn't talking, which defeats the feature).
|
||||
|
||||
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
|
||||
needs **no protocol support**; remote talk indicators are still derived from the presence of
|
||||
received frames (§7), so they work identically under VAD or PTT.
|
||||
|
||||
## 9. System / screen audio capture (`SCREEN_AUDIO`)
|
||||
|
||||
"Listen together" needs to capture the audio another app is playing. The capture mechanism
|
||||
differs per OS, but it always feeds the **same** Opus-encode → media-AEAD → UDP path as a
|
||||
normal stream; only the *source* is platform-specific.
|
||||
|
||||
| Platform | Mechanism | Notes |
|
||||
|----------|-----------|-------|
|
||||
| **Windows** | **WASAPI loopback** (whole-device, via miniaudio) **or WASAPI process loopback** (`AUDIOCLIENT_ACTIVATION_PARAMS`, Win10 2004+) for per-app / self-exclude | **Implemented.** Default *entire desktop* uses miniaudio's whole-device loopback in the channel's mode — stereo (interleaved L/R) when the channel is stereo, mono when mono — so a stereo channel gets genuine stereo end-to-end (no downmix). It inherently captures this app's own incoming voice mix (self-echo). The **per-app modes and the "exclude VoiceCat's own audio" option** instead drive `ProcessLoopbackCapture` (process-specific INCLUDE/EXCLUDE) through the external-feed mixer (`vc_stream_feed_pcm`, `external_feed=1`), which avoids self-echo and supports true "everything except". See below. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+) | **Implemented** (`clients/apple/macOS/VoiceCatMac/Audio/ScreenAudioCapture.swift`). OS requires screen-recording permission; capture happens in the main app. An `SCStream` with `capturesAudio` + `excludesCurrentProcessAudio` delivers audio `CMSampleBuffer`s; Swift converts Float32 → int16 (in the channel's mono/stereo mode) and calls `vc_stream_feed_pcm` — no miniaudio loopback device involved (`VOICECAT_HAS_LOOPBACK` is Windows-only). **Supports per-app audio selection** — see below. |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | **Implemented.** See below — separate process, App Group, ~50 MB cap (fine for audio-only). ReplayKit only ever delivers the *mixed* system stream as `.audioApp`, so **per-app filtering / VoiceOver exclusion is not possible on iOS** (it has no per-app granularity, unlike ScreenCaptureKit). |
|
||||
|
||||
### macOS detail — per-app audio selection
|
||||
|
||||
ScreenCaptureKit filters audio at the **application** level, so before sharing starts the user
|
||||
picks a scope in `ScreenSharePickerSheet` (`clients/apple/macOS/VoiceCatMac/Sheets/`):
|
||||
|
||||
- **Everything** — whole display, the original behaviour (`SCContentFilter(display:excludingWindows:)`).
|
||||
- **Only selected apps** — capture just the ticked apps (`init(display:including:exceptingWindows:)`).
|
||||
- **All except selected apps** — capture everything but the ticked apps
|
||||
(`init(display:excludingApplications:exceptingWindows:)`).
|
||||
|
||||
A dedicated **"Exclude screen reader (VoiceOver) audio"** toggle merges the screen-reader
|
||||
process(es) into the exclude set (`ScreenAudioCapture.screenReaderBundleIDs` — VoiceOver plus
|
||||
the speech-synthesis daemon that actually renders the spoken audio). The chosen
|
||||
`ScreenAudioSelection` is passed into `ScreenAudioCapture`, which builds the matching
|
||||
`SCContentFilter`. iOS/ReplayKit has no equivalent control (see the table note above).
|
||||
|
||||
### Windows detail — per-app audio selection and self-echo
|
||||
|
||||
`AppAudioPickerDialog` (`clients/windows/VoiceCat.App/Forms/`) offers the same shape as macOS:
|
||||
|
||||
- **Entire desktop** — whole-device miniaudio loopback handled by the core (default path).
|
||||
- **Only selected apps** — one `ProcessLoopbackCapture` in **INCLUDE** mode per ticked app,
|
||||
mixed by `ProcessAudioMixer` and fed via `vc_stream_feed_pcm`.
|
||||
- **All apps except selected** — a **single** `ProcessLoopbackCapture` in **EXCLUDE** mode of
|
||||
the chosen process tree. WASAPI's `AUDIOCLIENT_PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE`
|
||||
captures the whole render mix minus that tree *dynamically* (apps launched after sharing
|
||||
starts are included automatically). The activation params take a **single** target PID, so
|
||||
exclude is restricted to **one** app — the picker enforces single-selection in this mode.
|
||||
|
||||
An **"Exclude VoiceCat's own audio (prevents echo)"** checkbox (default on, enabled for
|
||||
*entire desktop*) routes the desktop capture through the same EXCLUDE path targeting
|
||||
VoiceCat's **own** process id (`Environment.ProcessId`) — i.e. "entire desktop except this
|
||||
app" — which removes the self-echo loop the whole-device path otherwise has. The per-app
|
||||
INCLUDE modes already never capture this app's tree, so they have no self-echo to remove.
|
||||
|
||||
### iOS detail
|
||||
|
||||
The extension **captures**, the host app **sends**. Unlike a self-connecting extension, this
|
||||
keeps a **single session** — the screen-audio share appears as a second stream of the *same*
|
||||
user (exactly like macOS/Windows), and no credentials are ever persisted to disk.
|
||||
|
||||
- The user starts a broadcast from Control Center's screen-record button; we surface it via
|
||||
`RPSystemBroadcastPickerView` from inside the app (`VoiceControlsView`) for one-tap start.
|
||||
- The **Broadcast Upload Extension** (`native/apple/broadcast/SampleHandler.swift`)
|
||||
receives `RPSampleBufferType.audioApp` (system/app audio), `.audioMic`, and `.video`. We
|
||||
consume **`.audioApp`** only and drop video + mic — video is what blows the **~50 MB**
|
||||
extension memory budget, so an audio-only consumer stays comfortably inside it. The extension
|
||||
does **not** link the managed host or native media shim.
|
||||
- The extension converts each chunk to the core's canonical format (48 kHz int16 stereo, via
|
||||
`AVAudioConverter`) and writes it into a lock-free single-producer/single-consumer ring in a
|
||||
shared **App Group** mmap'd file (`native/apple/broadcast/BroadcastAudioRing.swift`). It
|
||||
posts Darwin notifications on start/stop so the host reacts promptly.
|
||||
- The **host app** owns the stream: its `BroadcastAudioPump` announces the `SCREEN_AUDIO`
|
||||
stream over the control channel (`StreamAnnounce`), drains the ring, and calls
|
||||
`vc_stream_feed_pcm` (the external PCM feed API — see architecture.md §4) to drive the Opus
|
||||
encode + AEAD + send path. It downmixes to mono when the channel's effective config is mono.
|
||||
- Mic + voice also run in the host app. When the broadcast stops (`broadcastFinished`), the
|
||||
extension clears the ring's active flag (and posts a Darwin notification); the host stops
|
||||
feeding and emits `StreamStop`. The host must be alive to relay — always true while in a
|
||||
call (the app declares the `audio` background mode).
|
||||
Reference in New Issue
Block a user