From 268d511f795d331a7c0ac673a99e91dc7c31f209 Mon Sep 17 00:00:00 2001 From: Talon Date: Mon, 15 Jun 2026 20:47:09 +0200 Subject: [PATCH] docs: initial design baseline for VoiceCat voice/text chat Establish the design spec in docs/ before implementation: - README: overview, locked decisions, principles, glossary - architecture: shared C++ core + C ABI, native UIs (Swift/C#), threading model, server design (SFU relay) - protocol: TCP/TLS control plane, protobuf Envelope + message catalog, connection lifecycle, extensibility rules - voice: UDP media frame format, per-channel Opus config, multi-stream model, two-sided noise reduction, VAD/PTT, jitter buffer, iOS ReplayKit screen-audio - security: mandatory encryption (TLS 1.3 + exported-key AEAD), TOFU server identity, admin-provisioned accounts, anti-replay - tech-stack: permissive-only deps (mbedTLS, libsodium, opus, miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL - deployment: zero-config self-host (Docker / binary / source) - roadmap: M0-M5 milestones, resolved decisions Co-Authored-By: Claude Opus 4.8 --- docs/README.md | 68 ++++++++++ docs/architecture.md | 215 ++++++++++++++++++++++++++++++++ docs/deployment.md | 141 +++++++++++++++++++++ docs/protocol.md | 291 +++++++++++++++++++++++++++++++++++++++++++ docs/roadmap.md | 99 +++++++++++++++ docs/security.md | 157 +++++++++++++++++++++++ docs/tech-stack.md | 91 ++++++++++++++ docs/voice.md | 240 +++++++++++++++++++++++++++++++++++ 8 files changed, 1302 insertions(+) create mode 100644 docs/README.md create mode 100644 docs/architecture.md create mode 100644 docs/deployment.md create mode 100644 docs/protocol.md create mode 100644 docs/roadmap.md create mode 100644 docs/security.md create mode 100644 docs/tech-stack.md create mode 100644 docs/voice.md diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..db9ecf2 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,68 @@ +# VoiceCat — Design Documentation + +VoiceCat is a self-hosted, server-based voice and text chat system in the spirit of +classic TeamSpeak / Mumble: channel-based voice, channel and private text chat, and a +single server you own and run. It deliberately avoids WebRTC. The media path is plain +**UDP**, the control path is plain **TCP**, and both are encrypted. + +This folder is the design spec. No code yet — these documents define the architecture, +the wire protocol, the audio pipeline, the security model, and the dependency list, so +that implementation can start from a shared, agreed plan. + +## Decisions locked so far + +| Area | Decision | +|------|----------| +| Code architecture | **Shared C++ core** (`libvoicecat`) consumed by native UIs over a **C ABI**. Server reuses the same core. | +| Native clients | macOS/iOS in **Swift** (SwiftUI; Swift↔C++ interop), Windows in **C#** (`LibraryImport` P/Invoke). | +| Control transport | **TCP + TLS 1.3** (mbedTLS) | +| Media transport | **UDP** secured by **TLS-exported keys + ChaCha20-Poly1305 AEAD** — mandatory, no plaintext mode (see [security.md](security.md)) | +| Crypto libraries | **mbedTLS** (TLS 1.3) + **libsodium** (AEAD, Argon2id, Ed25519) — both permissive, **no GPL/LGPL anywhere** | +| Voice codec | **Opus** (libopus 1.6), per-channel configurable mono/stereo, bitrate, frame size, FEC/DTX | +| Audio DSP | **webrtc-audio-processing (APM)** — AEC/NS/AGC/VAD. NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally. Input gate supports **VAD and PTT**, client-configurable. | +| Identity | **Guests + admin-provisioned local accounts** (Argon2id, SQLite). No self-serve registration; guests toggleable per server. | +| Text | **Ephemeral** — live relay, no server-side history in v1. | +| Serialization | **Protocol Buffers** for the control plane; **custom binary** for voice frames | +| Deployment | **One `docker run`, one static binary, or `cmake --build`** — zero-config, secured by default (see [deployment.md](deployment.md)) | + +## Document index + +1. [architecture.md](architecture.md) — System layers, the shared core, threading model, the C ABI, server design. +2. [protocol.md](protocol.md) — The control protocol: framing, connection lifecycle, the full message catalog, encoding, versioning/extensibility. +3. [voice.md](voice.md) — The UDP media protocol: voice frame format, Opus configuration, multi-stream model, jitter buffer, packet-loss handling. +4. [security.md](security.md) — Mandatory encryption (TLS 1.3 + exported-key media AEAD), server identity (TOFU), authentication, accounts, anti-replay, threat model. +5. [tech-stack.md](tech-stack.md) — Concrete libraries with versions and rationale, the permissive-license rule, build tooling, per-platform notes. +6. [deployment.md](deployment.md) — The "set it up in a few minutes" story: Docker, single binary, source build, zero-config defaults. +7. [roadmap.md](roadmap.md) — Milestones, what ships when, and the list of open questions still to resolve. + +## Design principles + +- **One core, many faces.** Protocol, crypto, Opus, networking, jitter buffering, and + mixing live once in C++. UIs are thin. This keeps behavior identical across platforms + and the security-sensitive code reviewed in a single place. +- **Boringly simple transport.** TCP for control, UDP for media. No ICE, no SDP, no + TURN. A user opens a port (or port-forwards) and runs a server. +- **Extensible from day one.** Every message rides in a versioned envelope; capabilities + are negotiated at connect time; unknown fields are ignored. File transfer, screen-audio, + and moderation slot in without breaking older clients. +- **Real-time correctness.** The audio thread never blocks, never allocates, never takes a + lock. Network and audio communicate through lock-free ring buffers. +- **Encrypted, always.** There is no unencrypted mode to misconfigure. The server has no + plaintext listener; encryption is on because it can't be turned off. And it's free to the + operator — the server self-provisions its key/cert on first run. +- **Stupid-easy to self-host.** The target reaction is "oh, I (or my agent) can stand this up + in a few minutes." One `docker run`, or one static binary, or a plain `cmake --build` — no + certificate wrangling, no external database, sane defaults out of the box. Permissive + licenses only (no GPL/LGPL) so it can be redistributed freely, including closed-source. + +## Glossary + +- **Core** — `libvoicecat`, the shared C++ library. +- **Control channel** — the TCP/TLS connection carrying protobuf messages. +- **Media channel** — the UDP connection carrying voice frames. +- **Stream** — one audio source from one user (e.g. mic, screen audio, second device). A + user may publish several streams at once; each is independently controllable. +- **Channel** — a room in the channel tree. Voice is scoped to a channel. +- **Session** — an authenticated connection; ties a TCP control channel to a UDP 5-tuple. +- **SFU relay** — the server forwards Opus packets between channel members without decoding + them (selective forwarding, no transcoding). diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..2ad12ee --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,215 @@ +# Architecture + +## 1. The shared-core model + +All non-UI logic lives in one C++ library, **`libvoicecat`**. The same library is linked +into every client and into the server. Platform UIs are thin and call the core through a +stable **C ABI** (`voicecat.h`). + +``` + ┌───────────────────────────────────────────┐ + macOS / iOS (Swift) │ │ Windows (C#) + ┌──────────────────┐ │ libvoicecat (C++) │ ┌──────────────────┐ + │ SwiftUI views │ │ ┌─────────────────────────────────────┐ │ │ WinUI/Avalonia │ + │ AVAudioSession │──┼─▶│ C ABI (voicecat.h) │◀─┼──│ LibraryImport │ + │ Swift↔C++ interop│ │ ├─────────────────────────────────────┤ │ │ P/Invoke │ + └──────────────────┘ │ │ Session / Protocol state machine │ │ └──────────────────┘ + │ │ Text + voice signaling │ │ + Linux/macOS/Windows │ │ Audio engine: capture→encode→send, │ │ + server │ │ recv→jitter→decode→mix→playback │ │ + ┌──────────────────┐ │ │ Codec layer (Opus 1.6) │ │ + │ voicecat-server │──┼─▶│ Crypto + transport (TLS 1.3 + AEAD) │ │ + │ (reuses core) │ │ │ Net I/O (Asio: TCP + UDP + timers) │ │ + └──────────────────┘ │ └─────────────────────────────────────┘ │ + └───────────────────────────────────────────┘ +``` + +Why this shape: + +- **Swift** (5.9+) can import C++ directly, but we still ship a C ABI because it is the + lowest-friction, most stable boundary and it is what **C#** needs (`LibraryImport`/ + P/Invoke). One ABI serves both. +- The **server** is not a separate codebase. It links the same protocol, crypto, and Opus + code as the client, so framing/encryption can never drift between the two ends. + +## 2. Layered design inside the core + +From the OS up: + +| Layer | Responsibility | Key deps | +|-------|----------------|----------| +| **Platform I/O** | Sockets, timers; audio device capture/playback | Asio, miniaudio | +| **Transport** | TLS 1.3 (TCP), exported-key ChaCha20-Poly1305 AEAD (UDP), framing, anti-replay | mbedTLS, libsodium | +| **Codec & DSP** | Opus encode/decode; APM (AEC/NS/AGC/VAD) send-side + per-user NR receive-side; resample; jitter buffer; mixer | libopus, webrtc-audio-processing, speexdsp | +| **Protocol** | Message (de)serialization, request/response correlation, state machine | protobuf | +| **Session/domain** | Channels, users, streams, permissions, text routing | — | +| **C ABI façade** | Handle-based API + event callbacks exposed to UIs | — | + +A UI never sees a socket, an Opus packet, or a protobuf message. It sees: "connect", +"join channel", "start a stream from this device", "send this text", and a stream of +events ("user joined", "user is talking", "message received", "level meter = 0.4"). + +## 3. Threading model + +Three classes of thread, with strict rules. + +``` + ┌──────────────┐ lock-free ┌──────────────┐ lock-free ┌──────────────┐ + │ Audio capture│ ──ring buffer─▶│ Net thread │ ──ring buffer─▶│Audio playback│ + │ (RT, miniaudio│ │ (Asio loop) │ │ (RT, miniaudio│ + │ callback) │◀───ring buffer─│ │◀───ring buffer─│ callback) │ + │ capture→Opus │ │ TLS + AEAD, │ │ jitter→Opus │ + │ encode │ │ route, relay │ │ decode→mix │ + └──────────────┘ └──────────────┘ └──────────────┘ + │ + ┌─────▼──────┐ + │ Worker pool│ DB, Argon2id, file I/O, + │ (blocking) │ TLS handshakes, codec setup + └────────────┘ +``` + +Rules: + +- **Audio (real-time) threads** are driven by the OS audio callback. They must not + allocate, lock, log, or do syscalls beyond the ring-buffer hand-off. Opus encode/decode + runs here (it is allocation-free after init). +- **Net thread(s)** run the Asio event loop: TLS records, AEAD seal/open, protobuf parse, channel + routing, jitter-buffer feed. On the server, this is where the SFU relay copies packets + to subscribers. +- **Worker pool** absorbs anything that can block: SQLite, Argon2id verification, DNS, + TLS handshake CPU, codec (re)configuration. +- **Communication** between audio and net is single-producer/single-consumer lock-free + ring buffers (one per direction per stream). Control-plane events to the UI go through a + thread-safe queue drained on the UI's terms. + +## 4. The C ABI (`voicecat.h`) — shape + +Handle-based, opaque pointers, C-linkage. Illustrative (final names in implementation): + +```c +typedef struct vc_client vc_client; + +typedef struct { + void (*on_event)(void* user, const vc_event* ev); // state changes, messages + void (*on_level)(void* user, uint32_t stream_id, float rms); // meters (throttled) + void* user; +} vc_callbacks; + +vc_client* vc_client_create(const vc_config* cfg, vc_callbacks cb); +void vc_client_destroy(vc_client*); + +int vc_connect(vc_client*, const char* host, uint16_t port); // async; result via event +int vc_authenticate_guest(vc_client*, const char* nickname); +int vc_authenticate_user(vc_client*, const char* user, const char* password); + +int vc_join_channel(vc_client*, uint32_t channel_id, const char* password /*nullable*/); +int vc_leave_channel(vc_client*); + +// Streams (mic / screen audio / aux device) +int vc_stream_start(vc_client*, const vc_stream_desc* desc, uint32_t* out_stream_id); +int vc_stream_stop(vc_client*, uint32_t stream_id); +int vc_set_input_device(vc_client*, uint32_t stream_id, const char* device_id); +int vc_set_self_mute(vc_client*, bool mic_muted, bool deafened); + +// Text +int vc_send_text(vc_client*, vc_text_scope scope, uint32_t target_id, const char* utf8); + +// Enumeration helpers for UI device pickers +int vc_list_devices(vc_client*, vc_device_kind kind, vc_device_list* out); +``` + +Design notes: + +- **Async, event-driven.** Calls return immediately; results and state changes arrive via + `on_event`. This maps cleanly onto SwiftUI/`async` and C# `event`/`Task` patterns. +- **The core owns audio.** Capture, encode, decode, mixing, and playback happen inside the + core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders + meters/state*. This keeps the real-time path identical on every OS. (iOS is the one + exception that needs UI-side cooperation — see below.) +- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are + caller-allocated or returned with a paired `vc_free`. + +### Per-platform binding notes + +- **Swift / Apple.** Import the C ABI via a module map; SwiftUI on top. On **iOS** the app + must still own `AVAudioSession` (category `.playAndRecord`, `.voiceChat` mode), request + mic permission, and handle interruptions/route changes — the core exposes hooks + (`vc_audio_suspend`/`vc_audio_resume`) the Swift layer calls from `AVAudioSession` + notifications. Background voice and VoIP push (CallKit/PushKit) are a later milestone. +- **iOS screen / system-audio sharing** is supported via a **ReplayKit Broadcast Upload + Extension** (the same mechanism Discord uses; triggered from Control Center's screen-record + button via `RPSystemBroadcastPickerView`). The extension receives + `RPSampleBufferType.audioApp` (system/app audio) and `.audioMic`. We capture **`.audioApp`** + for the `SCREEN_AUDIO` "listen together" stream and ignore video. The extension runs in a + **separate process with a ~50 MB memory cap** — that cap is a problem only for video + frames, so audio-only stays well within budget. It links a *minimal* slice of the core + (Opus encode + media send), shares the session/credentials with the host app through an + **App Group**, and re-derives its own media keys. This is detailed in [voice.md](voice.md) §9. +- **C# / Windows.** `LibraryImport` (source-generated P/Invoke, .NET 7+) over the C ABI. + Marshal the `on_event` callback as a `[UnmanagedCallersOnly]`/function-pointer to avoid + delegate lifetime pitfalls. UI in **WinUI 3** (most native) or **Avalonia** (if we later + want a single C# UI across desktop OSes). + +## 5. Server architecture + +`voicecat-server` is a headless process linking the core. + +``` + TCP/TLS 1.3 UDP + media AEAD + │ │ + ┌────────▼─────────┐ ┌─────────▼──────────┐ + │ Connection mgr │ │ UDP demux │ + │ (accept, TLS, │ │ 5-tuple → session │ + │ per-conn state) │ │ anti-replay window │ + └────────┬─────────┘ └─────────┬──────────┘ + │ │ + ┌────────▼───────────────────────────────────▼──────────┐ + │ Session registry (session_id ↔ TCP conn ↔ UDP tuple) │ + └────────┬───────────────────────────────┬───────────────┘ + │ │ + ┌────────▼─────────┐ ┌──────────────┐ ┌▼─────────────────┐ + │ Channel manager │ │ Text router │ │ Voice router/SFU │ + │ tree, configs, │ │ channel + PM │ │ relay Opus to │ + │ membership, perms│ │ │ │ channel members │ + └────────┬─────────┘ └──────────────┘ └──────────────────┘ + │ + ┌────────▼─────────┐ + │ Persistence │ accounts (Argon2id), channels, bans, config + │ SQLite │ + └──────────────────┘ +``` + +- **Voice router is a relay, not a mixer.** For each incoming voice frame it looks up the + sender's channel and forwards the *unmodified Opus payload* (restamped with the sender's + user id) to every other subscribed member. No server-side decode/transcode → low CPU, + low latency, and end-to-content is just Opus. Per-channel Opus params are enforced so all + members are mutually decodable. +- **Subscriptions.** Clients implicitly subscribe to their current channel's voice; text + and presence can be subscribed more broadly. This keeps fan-out bounded on big servers. +- **Stateless-ish media.** UDP carries no auth per packet beyond the media-AEAD session; + the 5-tuple→session binding is established once via a token (see protocol.md §4). +- **Single process, scalable later.** v1 is one process, one machine. The session registry + and router are written behind interfaces so a future build can sit them behind a shared + bus for multi-node, but that is explicitly out of scope for now. + +## 6. Repository layout (proposed) + +``` +voice-cat/ +├── docs/ # this folder +├── core/ # libvoicecat (C++) +│ ├── include/voicecat.h # the C ABI +│ ├── src/{net,crypto,codec,protocol,session,audio}/ +│ └── proto/ # .proto definitions (shared source of truth) +├── server/ # voicecat-server (C++, links core) +├── clients/ +│ ├── apple/ # Swift package + Xcode project (macOS + iOS) +│ └── windows/ # .NET solution (C#) +├── tools/ +│ └── vccli/ # headless test client (C++), for protocol bring-up +├── third_party/ # vendored / vcpkg manifest +└── CMakeLists.txt +``` + +Build is **CMake** with **vcpkg** (manifest mode) for C/C++ deps; the Apple and Windows UI +projects consume the built core as a binary + headers. See [tech-stack.md](tech-stack.md). diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..9dff336 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,141 @@ +# Deployment & Self-Hosting + +The product goal: someone looks at this and thinks *"oh, I (or my agent) can stand this up +in a few minutes."* Everything below is in service of that. Three install paths, all +**zero-config and encrypted by default**. + +## 1. The three paths + +### A. Docker (recommended) + +```bash +docker run -d --name voicecat \ + -p 8384:8384/tcp \ # control (TLS 1.3) + -p 8384:8384/udp \ # media (encrypted) + -v voicecat-data:/data \ + ghcr.io//voicecat:latest +``` + +That's the whole thing. On first start it generates its Ed25519 identity + self-signed +cert, creates the SQLite database under `/data`, prints the **server fingerprint** (for +clients to verify), and listens. Control and media share one port number on TCP+UDP to keep +firewall rules trivial. + +A `docker-compose.yml` is provided for people who prefer it, but it isn't required. + +### B. Single static binary + +```bash +# download for your OS, then: +./voicecat-server # uses ./voicecat-data/ , prints fingerprint, runs +``` + +The server is a **single statically linked executable** (mbedTLS, libsodium, opus, sqlite, +etc. linked in — all permissive licenses). No runtime, no shared libraries to install, no +package manager. Linux (primary), macOS, and Windows builds. + +### C. From source + +```bash +git clone … && cd voice-cat +cmake --preset server-release # vcpkg fetches & pins all deps +cmake --build --preset server-release +./build/voicecat-server +``` + +One `cmake` invocation; vcpkg (manifest mode) resolves the dependency graph reproducibly. +No system packages to chase. + +## 2. Zero-config defaults + +The server runs with **no config file at all**. Sensible defaults: + +| Setting | Default | +|---------|---------| +| Encryption | On, always (not configurable off) | +| TLS cert / identity | Auto-generated on first run, persisted to the data dir | +| Database | Embedded SQLite in the data dir (no external DB) | +| Guests | Enabled (so the very first connect "just works"); easily disabled | +| Ports | `8384` TCP + UDP | +| A default channel | One "Lobby" voice/text channel created on first run | +| Opus policy | 48 kHz, 20 ms frames, mono/VOIP defaults; per-channel overrides allowed | +| Argon2id cost | Auto-tuned to the host on first run | + +Override only what you care about, via env vars or an optional `server.toml`: + +```toml +# server.toml — every key is optional +server_name = "Cats United" +allow_guests = false +bind_port = 8384 +data_dir = "/data" + +[tls] # only if you want a real CA cert; otherwise self-signed +cert_file = "/data/fullchain.pem" +key_file = "/data/privkey.pem" + +[opus.defaults] # default Opus policy for new channels +mode = "mono" +bitrate_bps = 24000 +frame_ms = 20 +fec = true +dtx = true + +[opus.limits] # server-enforced ceilings (bound bandwidth) +max_bitrate_bps = 128000 # channels can't be configured above this +``` + +Every key also has an `VOICECAT_*` env var form, which is what the Docker path uses. + +## 3. Connecting (client side) + +- **Pure direct-connect.** Enter `host:port` (and a nickname or account). There is no central + directory or server browser — you connect to a server you know. +- **Saved server list.** The client keeps a local list of saved servers (host:port, pinned + fingerprint, nickname/credentials per server) so you can store several and pick one to + join. This lives entirely in the client. +- On first connect the client shows the server's **fingerprint** and pins it (TOFU). No + accounts or certs needed to try it; if the operator disabled guests, the client prompts for + the username/password an admin gave you. + +That's the entire flow: run the server, share `host:port` + fingerprint, friends save it and +connect. + +## 3a. Provisioning accounts (admin) + +Accounts are **admin-provisioned** — there is no self-serve registration. Two equivalent ways, +both writing the same SQLite store: + +```bash +# CLI against the server's data dir or a running server +voicecat-admin account add # prompts for / generates a password +voicecat-admin account reset +voicecat-admin account del +voicecat-admin account list +``` + +…or from the **in-app admin interface** (a user with the admin permission), which sends the +privileged `CreateAccount`/`ResetPassword`/`DeleteAccount` control messages over TLS +(protocol.md §3). Guests need no provisioning; they just pick a nickname (if guests are +enabled). + +## 4. Why it stays this easy (design constraints that protect the goal) + +- **No external services.** No separate database, no Redis, no TURN/STUN server, no reverse + proxy required. SQLite is embedded; media is plain UDP. +- **No certificate chore.** Self-signed + Ed25519 TOFU means encryption needs zero operator + action. A domain owner *can* drop in a Let's Encrypt cert, but never *has* to. +- **One port pair.** TCP+UDP on the same number; one firewall/port-forward rule. +- **Static linking + permissive licenses.** The binary has no install-time dependencies and + can be redistributed (including closed-source) without copyleft obligations. +- **Agent-friendly.** The run command is a single line with no interactive prompts, the + server logs its fingerprint and listen address in machine-readable form, and `--help` / + `--print-config` expose everything an automation needs. Health endpoint for liveness checks. + +## 5. Operational niceties (planned, not blocking v1) + +- `voicecat-server --print-fingerprint` and a `/healthz` TCP check. +- Graceful reload of `server.toml` on `SIGHUP`. +- `voicecat-admin` (see §3a) also handles bans and channel admin, talking to the same SQLite + file or a running server. +- Prebuilt images for `linux/amd64` + `linux/arm64` (Raspberry Pi / cheap VPS friendly). diff --git a/docs/protocol.md b/docs/protocol.md new file mode 100644 index 0000000..89f6019 --- /dev/null +++ b/docs/protocol.md @@ -0,0 +1,291 @@ +# Control Protocol + +The control plane runs over **TCP, wrapped in TLS 1.3**. It carries everything that is not +real-time media: handshake, authentication, channel/user/presence state, text chat, and +voice *signaling* (announcing that a media stream is starting/stopping). Real-time voice +travels separately over UDP — see [voice.md](voice.md). + +## 1. Framing + +Inside the TLS stream, messages are length-prefixed: + +``` +┌──────────────┬───────────────────────────────────────────────┐ +│ u32 length │ protobuf-encoded Envelope (length bytes) │ +│ (big-endian)│ │ +└──────────────┴───────────────────────────────────────────────┘ +``` + +- `length` is the byte count of the payload that follows (not including the 4 length + bytes). Hard cap (e.g. 16 MiB) to bound memory; oversized frame → protocol error + + disconnect. +- The payload is a single **`Envelope`** protobuf message. We do **not** add our own type + byte; the type is the `oneof` discriminator inside the Envelope, which keeps the framing + trivial and lets protobuf own all forward/backward compatibility. + +TLS already provides record framing, integrity, and ordering; we only add message +boundaries on top of the TLS byte stream. + +## 2. Why Protocol Buffers for control + +- Schema-driven codegen for **C++, C#, and Swift** (all first-class) → no hand-rolled + parsers, no drift between client and server. +- **Forward/backward compatible by construction**: unknown fields are preserved/ignored, + new fields and new `oneof` arms are additive. This is exactly the "extensible protocol" + requirement. +- Compact enough for a control plane (text/state, not media). We use **`oneof`** envelopes + rather than `Any` so the wire stays tight and the switch is exhaustive. + +> Media frames do **not** use protobuf — they use a fixed binary header (see voice.md), +> because per-packet protobuf overhead and allocation are unacceptable on the RT path. + +## 3. The Envelope + +```proto +syntax = "proto3"; +package voicecat.v1; + +message Envelope { + // Monotonic per-connection id set by the sender of a request; echoed in the + // matching response so async callers can correlate. 0 for unsolicited events. + uint64 request_id = 1; + + oneof body { + // ── Session / handshake ─────────────────────────────── + ClientHello client_hello = 10; + ServerHello server_hello = 11; + AuthRequest auth_request = 12; + AuthResult auth_result = 13; + Disconnect disconnect = 14; + Ping ping = 15; + Pong pong = 16; + + // ── State sync ──────────────────────────────────────── + ServerStateSnapshot server_state = 20; + ChannelEvent channel_event = 21; // created/updated/deleted + UserEvent user_event = 22; // joined/left/updated + SubscribeRequest subscribe = 23; + + // ── Channel operations ──────────────────────────────── + JoinChannelRequest join_channel = 30; + JoinChannelResult join_channel_result= 31; + LeaveChannelRequest leave_channel = 32; + CreateChannelRequest create_channel = 33; + EditChannelRequest edit_channel = 34; + DeleteChannelRequest delete_channel = 35; + MoveUserRequest move_user = 36; + GenericResult generic_result = 37; // ack/err for the above + + // ── Voice signaling (media is on UDP) ───────────────── + StreamAnnounce stream_announce = 40; + StreamAnnounceResult stream_announce_result = 41; + StreamStop stream_stop = 42; + StreamStateUpdate stream_state = 43; // talking/muted indicator + UdpBinding udp_binding = 44; // token to bind the UDP 5-tuple + + // ── Text ────────────────────────────────────────────── + TextMessage text_message = 50; + TextMessageAck text_message_ack = 51; + TypingIndicator typing = 52; + + // ── Moderation / permissions ────────────────────────── + KickRequest kick = 60; + BanRequest ban = 61; + SetPermissionRequest set_permission = 62; + + // ── Admin account management (privileged) ───────────── + // Accounts are admin-provisioned (no self-serve registration in v1). + // These ride the same TLS control channel and require an admin permission. + CreateAccountRequest create_account = 70; + ResetPasswordRequest reset_password = 71; + DeleteAccountRequest delete_account = 72; + ListAccountsRequest list_accounts = 73; + + // ── Extension escape hatch ──────────────────────────── + Extension extension = 200; // {string ns; bytes payload;} + } +} +``` + +Reserved tag ranges keep future families from colliding: **10–19** session, **20–29** state, +**30–39** channels, **40–49** voice signaling, **50–59** text, **60–99** moderation, +**100–199** future (e.g. file transfer = 100–109), **200+** extensions. + +## 4. Connection lifecycle + +``` +Client Server + │ TCP connect ───────────────────────────────────▶│ + │ ◀──────────────── TLS 1.3 handshake ────────────▶│ server cert (TOFU/PKI, see security.md) + │ │ + │ ClientHello (proto_version, features[], info) ──▶│ + │ ◀── ServerHello (proto_version, features[], │ feature intersection negotiated here + │ server_info, auth_methods, udp_port) │ + │ │ + │ AuthRequest (guest{nick} | user{name,pass}) ────▶│ password verified w/ Argon2id + │ ◀── AuthResult (ok, session_id, self, perms, │ + │ udp_token) │ + │ │ + │ ◀── ServerStateSnapshot (channel tree, users) ───│ initial sync + │ │ + │ ===== UDP side (parallel) ===================== │ + │ (media keys derived from TLS exporter — no 2nd │ + │ handshake; see security.md §2) │ + │ UdpBinding(udp_token) [AEAD, exported keys] ────▶│ binds 5-tuple → session_id + │ ◀── UdpBinding ack [AEAD] ───────────────────────│ + │ │ + │ JoinChannelRequest(id, password?) ──────────────▶│ + │ ◀── JoinChannelResult(ok, members, audio_cfg) ───│ + │ StreamAnnounce(kind=mic, opus_params) ──────────▶│ + │ ◀── StreamAnnounceResult(ok, ssrc) │ + │ ── voice frames flow over UDP ──────────────────▶│ + │ │ + │ Ping / Pong (TCP keepalive) ◀──────────────────▶│ +``` + +Notes: + +- **Version negotiation.** Each side sends `proto_version` (integer) and a `features` + string list. The effective version is `min(client, server)`; the effective feature set + is the intersection. A client that doesn't understand a feature simply never uses it. +- **Auth over TLS.** Passwords cross the wire only inside TLS 1.3 and are verified against + an Argon2id hash at rest (see security.md). `auth_methods` in `ServerHello` advertises + whether `guest` is enabled. +- **UDP token.** `AuthResult.udp_token` is a short-lived opaque token. The client sends it + in the first UDP message (`UdpBinding`) so the server can bind the UDP 5-tuple to the + authenticated session without trusting the source address. This is the only UDP message + that carries identity material; everything after is implicit via the bound tuple + + media-AEAD session. +- **Snapshot then deltas.** After auth the server pushes a `ServerStateSnapshot` (full + channel tree + visible users), then streams incremental `ChannelEvent`/`UserEvent` + deltas. Clients reconcile by id. + +## 5. Message catalog (selected definitions) + +Representative messages; the full `.proto` is the source of truth in `core/proto/`. + +```proto +message ClientHello { + uint32 proto_version = 1; + repeated string features = 2; // e.g. "opus", "fec", "screen-audio" + string client_name = 3; // "VoiceCat-macOS" + string client_version = 4; + string preferred_locale = 5; +} + +message ServerHello { + uint32 proto_version = 1; + repeated string features = 2; + string server_name = 3; + string server_version = 4; + repeated string auth_methods = 5; // "guest", "password" + uint32 udp_port = 6; + bytes server_identity_fingerprint = 7; // Ed25519 key fp for TOFU display +} + +message AuthRequest { + oneof method { + GuestAuth guest = 1; // { string nickname; } + PasswordAuth password = 2; // { string username; string password; } + } +} + +message AuthResult { + bool ok = 1; + string error = 2; + uint64 session_id = 3; + User self = 4; + Permissions permissions = 5; + bytes udp_token = 6; // bind UDP 5-tuple with this +} + +message Channel { + uint32 id = 1; + uint32 parent_id = 2; // 0 = root + string name = 3; + string topic = 4; + bool password_protected = 5; + uint32 max_users = 6; + ChannelType type = 7; // PERMANENT / TEMPORARY + AudioConfig audio = 8; // per-channel Opus settings (see voice.md) + int32 order = 9; +} + +message User { + uint32 id = 1; + string nickname = 2; + bool is_guest = 3; + uint32 channel_id = 4; + bool self_mic_muted = 5; + bool self_deafened = 6; + bool server_muted = 7; + repeated StreamInfo streams = 8; // active media streams this user publishes +} + +message StreamInfo { + uint32 stream_id = 1; // unique within the user + uint32 ssrc = 2; // media-plane id assigned by server + StreamKind kind = 3; // MIC / SCREEN_AUDIO / AUX_DEVICE + AudioConfig audio = 4; + string label = 5; // "Microphone", "Desktop audio" +} + +message StreamAnnounce { // client → server: "I'm about to publish media" + StreamKind kind = 1; + AudioConfig requested_audio = 2; // server may clamp to channel policy + string label = 3; +} +message StreamAnnounceResult { + bool ok = 1; string error = 2; + uint32 stream_id = 3; uint32 ssrc = 4; + AudioConfig effective_audio = 5; // authoritative params to encode with +} + +message TextMessage { + TextScope scope = 1; // CHANNEL / PRIVATE / SERVER + uint32 target_id = 2; // channel_id or user_id depending on scope + uint32 sender_id = 3; // set by server on relay + string body = 4; // UTF-8, server-bounded length + uint64 sent_at_unix_ms = 5; // server timestamp on relay + string client_msg_id = 6; // client-chosen, echoed in ack (dedup) +} +``` + +> **Text is ephemeral (v1).** The server relays messages live to currently-connected, +> subscribed recipients and **does not persist history** — there is no store and no backfill +> on join. Clients may keep their own local scrollback for the session. Server-side history +> is a deliberate non-feature for now (it can be added later behind a capability flag without +> changing `TextMessage`). + +## 6. Request / response & errors + +- Any message a client expects a direct answer to sets a nonzero **`request_id`**; the + server echoes it in the response (`*Result` or `GenericResult`). Unsolicited + server→client events use `request_id = 0`. +- **`GenericResult { bool ok; uint32 code; string message; }`** is the default + acknowledgement for operations without a richer reply (create/edit/delete channel, move + user, etc.). Error `code`s are an enumerated, stable list. +- Fatal conditions send **`Disconnect { code; reason }`** then close the TLS connection. + +## 7. Keepalive & timeouts + +- **TCP:** `Ping`/`Pong` every ~15 s; missing N consecutive pongs → drop. `Pong` echoes the + `Ping` nonce so RTT is measurable. +- **UDP:** a separate lightweight keepalive on the media channel (voice.md §6) keeps NAT + bindings alive and detects media-path failure independently of the control channel. + +## 8. Extensibility checklist + +When adding a feature later (e.g. **file transfer**), the rules are: + +1. Add new `oneof` arms in the reserved tag range (file transfer = 100–109) — never reuse + or renumber existing tags. +2. Advertise a feature string in `ClientHello`/`ServerHello`; only use the feature if both + peers list it. +3. Prefer extending an existing message with new fields (additive) over inventing a new + message where it fits. +4. For experimental/out-of-tree features, ride inside `Extension { ns; payload }` until it + is promoted to a first-class `oneof` arm. + +This guarantees a v1 client and a v3 server interoperate at the negotiated lowest common +denominator. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..9959a81 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,99 @@ +# Roadmap & Open Questions + +## 1. Milestones + +Each milestone is shippable/testable on its own. The headless C++ test client (`vccli`) +exists from M1 so the protocol can be exercised long before any GUI. + +### M0 — Scaffolding +- Repo layout (see architecture.md §6), CMake + vcpkg manifest, CI matrix. +- `core/proto/` skeleton; `protoc` codegen wired for C++ (C#/Swift later). +- Empty `libvoicecat` with the C ABI header and stub implementations. +- **Exit:** core + server + `vccli` compile and link on Linux/macOS/Windows. + +### M1 — Control plane (TCP/TLS, no audio yet) +- TLS 1.3 transport; framing; Envelope; ClientHello/ServerHello negotiation. +- Auth: **guest + admin-provisioned local account** (Argon2id, SQLite); server identity + (TOFU/Ed25519); `voicecat-admin` account add/reset/del/list. +- Channel tree: snapshot + deltas; join/leave; create/edit/delete (perm-checked). +- **Text chat (ephemeral):** channel + private messages, acks; live relay, no history store. +- `vccli` can connect, auth, browse channels, and chat. +- **Exit:** two `vccli` instances chat through a real server over TLS. + +### M2 — Voice, single stream +- UDP transport with exported-key + ChaCha20-Poly1305 AEAD (mandatory, no plaintext path); + UDP token binding; anti-replay. +- miniaudio capture/playback; libopus encode/decode; one `MIC` stream per user. +- **Send-side webrtc APM** (AEC + NS + AGC + VAD) and a **VAD/PTT input gate** (both modes, + client-configurable) — AEC is in from the start, not deferred. +- Per-ssrc adaptive jitter buffer; mixer; FEC/PLC/DTX. +- Per-channel `AudioConfig` enforced (incl. server `max_bitrate_bps` ceiling); SFU relay. +- **Exit:** talk between two `vccli`/early-GUI clients in a channel; loss resilience visible. + +### M3 — Multi-stream & per-channel tuning +- Multiple concurrent streams per user (`MIC`, `SCREEN_AUDIO`, `AUX_DEVICE`). +- Per-stream receiver gain/mute; **listener-side per-user noise reduction** (APM NS on the + receive path, per ssrc, local-only); talk indicators. +- Full per-channel Opus configurability (mono/stereo, bitrate, frame size, FEC/DTX). +- **Exit:** a user shares mic + desktop audio; listeners control each independently. + +### M4 — Native clients +- **Windows (C#/WinUI):** connect, saved-server list, channel tree, voice, text, device + pickers, meters, VAD/PTT + per-user NR controls. +- **macOS (Swift/SwiftUI):** same. +- **iOS (Swift):** AVAudioSession integration, mic permission, foreground voice; ReplayKit + broadcast extension for `SCREEN_AUDIO`. +- In-app **admin interface** (account provisioning, bans) for admin users. +- **Exit:** non-technical user installs a client, saves a server, and joins. + +### M5 — Moderation, polish, and beyond +- Permissions/roles, kick/ban/server-mute, channel passwords UI. +- DRED toggle, audio-quality polish. (AEC and VAD/PTT already shipped in M2.) +- **Then (post-v1, protocol already reserves space):** file transfer, E2EE option, + CallKit/PushKit background voice, key-based identity, server-side text history, + multi-node server. + +## 2. Resolved decisions + +Settled and reflected throughout the docs: + +- **Media crypto:** exported-key + ChaCha20-Poly1305 AEAD from day one, **mandatory** — no + DTLS, no plaintext mode. (security.md §2) +- **TLS library / licensing:** **mbedTLS** (Apache-2.0) + **libsodium** (ISC). **No GPL/LGPL** + anywhere; code is redistributable closed-source. wolfSSL is rejected. (tech-stack.md §5) +- **iOS screen/system audio:** supported via a ReplayKit Broadcast Upload Extension + (`.audioApp`); audio-only stays within the extension memory cap. (voice.md §9) +- **Self-host UX:** zero-config, encrypted-by-default; Docker / single binary / source build. + (deployment.md) +- **DSP engine:** **webrtc-audio-processing (APM)** — AEC in from the start, plus NS/AGC/VAD. + (voice.md §8) +- **Input activation:** **VAD *and* PTT**, both modes client-configurable. (voice.md §11) +- **Noise reduction is two-sided:** sender can denoise its mic, *and* each listener can apply + NS to a **specific** other user, locally, with no protocol traffic. (voice.md §10) +- **Accounts:** **admin-provisioned** (no self-serve registration) via `voicecat-admin` or the + in-app admin interface. (security.md §4, protocol.md §3, deployment.md §3a) +- **Text:** **ephemeral** — live relay, no server-side history in v1. (protocol.md §5) +- **Connect UX:** pure direct-connect with a **client-side saved-server list** (no central + directory). (deployment.md §3) +- **Bitrate ceiling:** server-config `opus.limits.max_bitrate_bps`. (deployment.md §2) +- **Name:** "VoiceCat" stays as the internal placeholder. + +## 3. Open questions + +All initial open questions are resolved (§2). Two **second-order considerations** to keep in +mind during implementation — not blockers: + +- **APM in constrained contexts.** webrtc-audio-processing is a heavier build; confirm it + static-links cleanly for the single-binary goal, and note the iOS **broadcast extension only + does Opus encode + send** (no APM), so it stays under the ~50 MB cap. Listener-side per-user + NS runs only in the full host app. +- **Receive-side NR cost at scale.** A per-ssrc APM NS instance per flagged user adds CPU on + busy channels; instantiate lazily (only for flagged streams) and cap concurrent instances. + +## 4. What's intentionally deferred + +To keep v1 focused (voice + text), these are designed-for but not built: file transfer, +end-to-end encryption, key-based identities, multi-node/federated servers, mobile +background VoIP push, and any server-side audio mixing/transcoding. The protocol's +versioning + feature negotiation + reserved tag ranges (protocol.md §8) ensure each can be +added without breaking deployed clients. diff --git a/docs/security.md b/docs/security.md new file mode 100644 index 0000000..888bdd6 --- /dev/null +++ b/docs/security.md @@ -0,0 +1,157 @@ +# Security Model + +Two encrypted transports: **TLS 1.3** on the TCP control channel, and an encrypted **UDP** +media channel. Plus server identity, authentication, accounts at rest, and anti-replay. + +> **Encryption is mandatory — there is no unencrypted mode.** The server has no plaintext +> listener, the client has no "insecure" option, and there is no config flag to turn either +> off. A connection is encrypted or it does not exist. This is a hard product rule, not a +> default. It is also *zero-config* (see §1): the server generates its own key/cert on first +> run, so "secured by default" never costs the operator a setup step. + +## 1. Control channel — TLS 1.3 (settled) + +- TCP control is wrapped in **TLS 1.3** (TLS 1.2 disabled). AEAD cipher suites only + (AES-128-GCM, AES-256-GCM, ChaCha20-Poly1305). X25519 key exchange. +- Library: **mbedTLS 3.6 LTS** — Apache-2.0 (permissive, fine for an eventual closed-source + distribution), TLS 1.3 client+server, and `mbedtls_ssl_export_keying_material()` for the + media path (§2). It also **static-links cleanly into a single self-host binary**, which is + a deliberate choice in service of the easy-deploy goal. (OpenSSL 3.x, also permissive + Apache-2.0, is a drop-in alternative behind the same internal interface.) +- **Zero-config TLS:** on first launch the server auto-generates a self-signed certificate + bound to a freshly generated **Ed25519 identity key** and persists both. The operator does + nothing. Clients pin the identity on first connect (TOFU, §1.1). A server *with* a domain + can drop in a CA cert later, but it is never required to be encrypted. +- All authentication and account material crosses the wire only inside this tunnel. + +### 1.1 Server identity — two modes + +Self-hosting means most servers won't have a CA-signed cert for a hostname. We support +both, advertised in `ServerHello`: + +1. **TOFU (Trust On First Use)** — default for hobby servers. The server has a long-lived + **Ed25519 identity key**; its fingerprint is shown to the user on first connect (like + SSH host keys / TeamSpeak server keys) and pinned locally. Subsequent connects verify + the pin; a changed key warns loudly. The TLS cert is self-signed and bound to this + identity key. +2. **PKI** — a server with a domain can use a normal CA-signed cert (e.g. Let's Encrypt); + clients validate the chain conventionally. TOFU pinning still applies on top. + +Client certificates are reserved for a future "key-based identity" option (see roadmap) but +are not required in v1. + +## 2. Media channel — UDP encryption (settled: exported-keys + AEAD) + +The UDP media path uses **TLS-exported keys + per-packet AEAD** (an SRTP-style design), +mandatory from the first build. This was chosen over DTLS after weighing two findings: + +> **Finding 1 — DTLS 1.3 (RFC 9147) is not in stable OpenSSL or mbedTLS.** It ships +> production-ready only in **wolfSSL**, which is **GPLv2-or-commercial** — disqualified, +> because the code will eventually be distributed in closed-source form (no GPL/LGPL deps). +> +> **Finding 2 — mbedTLS 3.6 LTS already exposes `mbedtls_ssl_export_keying_material()`** +> (RFC 5705 / RFC 8446 §7.5 exporter). So we can derive media keys from the existing TLS 1.3 +> control session with *zero* extra handshake and *zero* extra dependency. + +### How it works + +1. During the TLS 1.3 control handshake, both sides call the keying-material exporter with a + fixed label (`"voicecat media v1"`) to derive independent **send/recv media keys** and a + salt. No second handshake, no certificates on the UDP path — the UDP channel inherits the + authenticated, MITM-resistant TLS session's trust. +2. Each UDP voice frame is sealed with **ChaCha20-Poly1305** (libsodium, ISC license). +3. The readable routing field (`ssrc`) is passed as AEAD **associated data** so the relay can + route without decrypting and an attacker cannot tamper with it undetected. + +This keeps the entire crypto surface on two permissive libraries (mbedTLS + libsodium), adds +no handshake latency to voice startup, and is small enough to audit fully. It is abstracted +behind a `MediaCrypto { seal(frame)->bytes; open(bytes)->frame }` interface, so a future +DTLS 1.3 backend could slot in later if a permissive implementation matures — but nothing in +the design depends on that. + +### Per-frame protections + +- **AEAD** (ChaCha20-Poly1305) over each voice frame — confidentiality + integrity. +- **Associated data:** the `ssrc` (and version/flags) are authenticated-but-visible so the + relay routes without decrypting; everything else is encrypted. +- **Nonce discipline:** `nonce = direction_bit ‖ ssrc ‖ monotonic_packet_counter`. The + counter never repeats under one key; the session **rekeys** (re-derives via the exporter + with a bumped epoch) well before counter exhaustion or on a time/byte budget. +- **Anti-replay:** a sliding-window replay filter per ssrc (à la IPsec) keyed on the packet + counter. Replays and out-of-window packets are dropped before decode. + +## 3. UDP session binding + +UDP packets are not individually authenticated to a *user* beyond the transport session. +Binding works as: + +1. `AuthResult.udp_token` (issued over TLS) is a short-lived, single-use, random token tied + to `session_id`. +2. Client's first UDP message is `UdpBinding{udp_token}`, sent as the first AEAD media frame + using the keys exported from the TLS session. +3. Server validates the token, binds the **5-tuple → session_id**, and discards the token. +4. Thereafter, frames are accepted only on that bound tuple; ssrcs are checked against the + streams the session announced. Source-address spoofing can't hijack a session because the + attacker lacks the media key and the token. + +## 4. Authentication & accounts (settled: guests + local accounts) + +- **Guests:** toggled by server config (`allow_guests`). A guest picks a nickname and joins; + no persistent identity. Nicknames are non-reserved and may be uniquified by the server. +- **Local accounts:** username + password, **admin-provisioned** (no self-serve registration + in v1). An admin creates/resets/deletes accounts either via the `voicecat-admin` CLI or the + in-app admin interface, which sends the privileged `CreateAccount`/`ResetPassword`/ + `DeleteAccount` messages (protocol.md §3, permission-gated). Stored in **SQLite**; passwords + hashed with **Argon2id** (via libsodium `crypto_pwhash`) using per-install-tuned memory/time + parameters; never stored or logged in plaintext. Verification runs on the worker pool (it's + deliberately slow) to avoid stalling the net thread. +- **Channel passwords:** hashed at rest too; join attempts compare server-side. +- **Brute-force defense:** per-IP and per-account rate limiting on auth attempts with + exponential backoff; configurable lockout. Generic `auth_request` failures return a + non-enumerating error ("invalid credentials") to avoid username probing. + +``` +accounts( id INTEGER PK, username TEXT UNIQUE, + pw_argon2id TEXT, -- encoded hash incl. params + salt + created_at, last_login, flags ) +bans( id, subject_type, subject, reason, expires_at ) +``` + +## 5. Permissions (scaffold for v1, enforced server-side) + +A `Permissions` set is attached to each session at auth time and is the *only* authority — +clients never self-grant. v1 needs a minimal set (join channel, send text, create temporary +channel, kick/move if moderator); the model is a role/flag bitset that the moderation +milestone expands. All privileged operations (`CreateChannel`, `Kick`, `Ban`, +`MoveUser`, server-mute) are checked against it server-side regardless of client UI. + +## 6. Threat model & non-goals + +**In scope:** +- Passive eavesdropping on either transport → defeated by TLS 1.3 (control) and the + exported-key AEAD (media). +- Active MITM on first connect → mitigated by TOFU pin + Ed25519 identity (user must verify + fingerprint out-of-band for the strongest guarantee). +- UDP source spoofing / session hijack → defeated by token binding + media-key secrecy + + anti-replay. +- Password theft at rest → mitigated by Argon2id; in transit → only inside TLS. + +**Explicit non-goals (v1):** +- **End-to-end encryption between users.** The server relays Opus and can see who talks to + whom; with the SFU relay it does *not* decode audio, but the media key is per + client↔server, not per pair. True E2EE (server can't read media) is a possible future + feature, not v1. +- Anonymity / metadata hiding. The server, by design, knows the channel graph. +- DoS resilience at scale beyond basic rate limiting and the bounded-frame guards. + +## 7. Crypto dependency summary + +Two libraries, both permissive (no GPL/LGPL), so a future closed-source distribution stays +clean: + +- **TLS 1.3:** **mbedTLS 3.6 LTS** (Apache-2.0). Control-channel TLS + the keying-material + exporter that seeds the media path. Static-links into a single binary. (OpenSSL 3.x, + Apache-2.0, is an interchangeable alternative.) +- **Primitives & password hashing:** **libsodium** (ISC) — Argon2id (account passwords), + ChaCha20-Poly1305 (the media AEAD), Ed25519 (server identity), X25519, secure RNG. All + non-TLS crypto goes through libsodium so we never hand-roll a primitive. diff --git a/docs/tech-stack.md b/docs/tech-stack.md new file mode 100644 index 0000000..4716358 --- /dev/null +++ b/docs/tech-stack.md @@ -0,0 +1,91 @@ +# Tech Stack & Dependencies + +Concrete library choices with versions and rationale. Everything in the **core** is C++ +(C++20). UIs are Swift and C#. Build is CMake + vcpkg. + +## 1. Core library (`libvoicecat`, C++20) + +| Concern | Choice | Version (as of 2026-06) | Why / notes | +|---------|--------|-------------------------|-------------| +| Sockets, timers, async | **Standalone Asio** | 1.30.x | Header-only, no Boost dependency, cross-platform TCP+UDP+timers, one reactor for client and server. (Boost.Asio is interchangeable if we already pull Boost.) | +| TLS 1.3 (control) | **mbedTLS 3.6 LTS** | 3.6.x (LTS ≥ Mar 2027) | **Apache-2.0** (permissive — clean for eventual closed-source distribution). TLS 1.3 client+server, plus `mbedtls_ssl_export_keying_material()` to seed the media AEAD. **Static-links cleanly → single self-host binary.** OpenSSL 3.x (Apache-2.0) is an interchangeable alternative. No DTLS/wolfSSL (GPL) — see [security.md](security.md) §2. | +| Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. | +| Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. | +| Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. | +| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) | 1.x (standalone APM) | **BSD-3.** The primary DSP engine: high-quality acoustic echo cancellation, noise suppression, AGC, and VAD in one tuned module. Used **send-side** (clean the mic) and **receive-side per user** (listener-chosen NS on a specific stream — voice.md §10). AEC is in from the start, not deferred. | +| Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) | +| Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. | +| Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. | +| Logging | **spdlog** | 1.14.x | Fast, async-capable; off the RT path. | + +Resampling note: Opus runs internally at 48 kHz; miniaudio can deliver 48 kHz directly, so +explicit resampling (speexdsp/libsamplerate) is only needed when a device can't do 48 kHz. + +## 2. Clients + +### macOS / iOS — Swift + +| Concern | Choice | Notes | +|---------|--------|-------| +| Language | **Swift 5.9+** | Direct **Swift↔C++ interop** available, but we bind through the C ABI for parity with Windows. | +| UI | **SwiftUI** | Single UI codebase for macOS + iOS where practical; AppKit/UIKit shims as needed. | +| Audio session (iOS) | **AVAudioSession** | App owns category `.playAndRecord` + `.voiceChat` mode, mic permission, interruption/route-change handling; calls `vc_audio_suspend/resume` on the core. macOS uses CoreAudio via the core directly. | +| Packaging | Swift Package + Xcode project | Core shipped as an XCFramework (device + simulator + macOS slices). | +| Future | CallKit / PushKit | For background VoIP + incoming-call UX on iOS. Post-v1. | + +### Windows — C# + +| Concern | Choice | Notes | +|---------|--------|-------| +| Runtime | **.NET 8+** | LTS. | +| Interop | **`LibraryImport`** (source-gen P/Invoke) over the C ABI | Marshal the event callback as a function pointer (`[UnmanagedCallersOnly]`) to avoid delegate-lifetime bugs; keep the interface "chunky" not "chatty" to minimize managed↔native transitions. | +| UI | **WinUI 3** (most native) or **Avalonia** | WinUI for a first-class Windows look; Avalonia if we later want one C# UI across desktop OSes. | +| Audio | handled by the core (miniaudio/WASAPI) | C# only drives device selection + meters. | + +## 3. Server (`voicecat-server`) + +- Pure C++ linking the core; **no GUI**. Runs on **Linux** (primary), **macOS**, **Windows**. +- Config via a `server.toml` (`allow_guests`, ports, channel defaults, Opus policy, TLS cert + paths or auto-self-signed + Ed25519 identity, Argon2id cost params, rate limits). +- SQLite for state. Single process for v1; interfaces drawn so a multi-node build is + *possible* later but explicitly out of scope. +- Packaging: static-ish binary per OS; systemd unit + Docker image for Linux. + +## 4. Build & tooling + +| Tool | Use | +|------|-----| +| **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. | +| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, webrtc-audio-processing, speexdsp, asio, miniaudio). Reproducible across OSes. | +| **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). | +| **clang-format / clang-tidy** | Style + static analysis on the core. | +| **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). | +| **GitHub Actions** (or similar) | Matrix CI: Linux/macOS/Windows core+server; Xcode build for Apple; `dotnet` build for Windows. | + +## 5. Licensing — permissive only (hard rule) + +The code will eventually be distributed in **closed-source** form, so **no GPL/LGPL +dependencies are permitted.** Every dependency below is BSD / MIT / ISC / Apache-2.0 / +public-domain: + +- **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ · + **miniaudio** — public domain / MIT-0 ✅ · **speexdsp** — BSD ✅ · **protobuf** — BSD ✅ · + **SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅ · + **webrtc-audio-processing** — BSD-3 ✅ (heavier build, but core to the DSP path). +- **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag + in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for + one entirely. +- CI runs a license scanner over the resolved vcpkg graph and **fails the build on any + GPL/LGPL transitive dependency**, so this rule can't silently regress. + +## 6. Why not the obvious alternatives + +- **WebRTC** — explicitly rejected: ICE/SDP/TURN complexity, huge dependency, opaque. We + want plain TCP+UDP we fully control. +- **QUIC** — capable (reliable streams + datagrams + TLS 1.3 in one), but heavier and drifts + toward the complexity we're avoiding. Revisit only if NAT traversal/multiplexing pain + appears. +- **gRPC** for control — pulls HTTP/2 and a lot of surface for what is a simple framed + message stream over TLS. Plain protobuf-over-framed-TLS is enough. +- **A Rust core** — viable and memory-safe, but the user prefers C++ and the Swift/C# + binding story is marginally simpler from C++ (Swift can even consume C++ directly). diff --git a/docs/voice.md b/docs/voice.md new file mode 100644 index 0000000..a2ffa93 --- /dev/null +++ b/docs/voice.md @@ -0,0 +1,240 @@ +# Voice & Media + +Real-time audio runs over **UDP**, secured per [security.md](security.md). The control +channel (TCP/TLS) handles *signaling* — announcing streams, channel membership, talk state +— while UDP carries only the encoded audio frames. This split keeps media latency low and +independent of TCP head-of-line blocking. + +## 1. The multi-stream model + +A **user** publishes one or more **streams**. Each stream is an independent audio source +with its own encoder, its own `stream_id` (unique per user) and `ssrc` (media-plane id +assigned by the server), and is independently mutable/mutable at the receiver. + +``` + User "Alex" Receiver "Sam" + ┌────────────────────┐ ┌──────────────────────────┐ + │ mic → enc ───┼──ssrc 1001──▶ │ jitter(1001)→dec→┐ │ + │ desktop → enc ───┼──ssrc 1002──▶ │ jitter(1002)→dec→┤ │ + │ 2nd mic → enc ───┼──ssrc 1003──▶ │ jitter(1003)→dec→┴─mix──▶ out + └────────────────────┘ └──────────────────────────┘ +``` + +Stream **kinds** (v1): `MIC`, `SCREEN_AUDIO` (system/desktop audio for listening together), +`AUX_DEVICE` (a second capture device). Receivers can set, per incoming stream: **gain**, +**mute**, and **noise reduction** (see §10) — so Sam can turn down Alex's desktop audio +while keeping the mic, *and* independently apply noise suppression to a third user who has a +loud fan. The mixer sums all active streams from all users in the channel into the local +playback device. All of these receiver-side controls are **local to the listener** and carry +no protocol traffic. + +`SCREEN_AUDIO` capture is platform-specific and covered in §9 — it is supported on Windows, +macOS, **and iOS** (via a ReplayKit broadcast extension). + +## 2. Voice frame format (UDP payload, inside the media AEAD) + +A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian. + +``` + 0 1 2 3 4 5 6 7 8 ... +┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐ +│ type │flags │ codec │ ssrc (u32) │ +├──────┴──────┴──────┴──────┼──────┬──────┬──────┬──────┬──────────────┤ +│ seq (u16) │ timestamp (u32, in samples @48k) │ payload ... │ +└─────────────┴──────────────────────────────────────────┴──────────────┘ + +type u8 1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake) +flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present + bit2 DTX/comfort-noise · bit3 last-frame-before-stop +codec u16 0 = OPUS (room for future codecs) +ssrc u32 media-plane stream id. Client sends its own ssrc; the server + validates it against the bound session and relays unchanged. +seq u16 per-ssrc sequence number, wraps; drives loss detection + reorder +timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer +payload one Opus packet (the encoder's output for one frame) +``` + +This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's +full machinery. The **server relays the payload unmodified** — it only reads the header to +route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once +assigned at `StreamAnnounce`). No server-side decode. + +### Why client-sends-ssrc is safe + +The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks +that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are +dropped. So identity is anchored by the session binding + transport encryption, not by +trusting the header. + +## 3. Per-channel audio configuration + +Opus is configured **per channel** and pushed to clients in `JoinChannelResult.audio` / +`StreamAnnounceResult.effective_audio`. All members of a channel encode with mutually +decodable parameters. + +```proto +message AudioConfig { + uint32 codec = 1; // 0 = OPUS + ChannelMode mode = 2; // MONO / STEREO + uint32 sample_rate = 3; // 8000/12000/16000/24000/48000 (48000 recommended) + uint32 bitrate_bps = 4; // e.g. 24000 (speech) … 128000 (music/stereo) + uint32 frame_ms = 5; // 2.5/5/10/20/40/60 (20 default) + OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY + bool fec = 7; // in-band forward error correction + uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness + bool dtx = 9; // discontinuous transmission (silence suppression) + uint32 complexity = 10; // 0..10 encoder complexity +} +``` + +Guidance baked into defaults / docs: + +- **Sample rate: always run Opus at 48 kHz internally.** Opus resamples internally anyway; + 48 kHz avoids surprises. The `sample_rate` field mainly constrains capture/narrowband + modes for very low bitrate channels. Default **48000**. +- **Frame size: 20 ms default.** Smaller (10 ms) lowers latency at the cost of more + per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at + the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms. +- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 24–32 kbps, DTX on, FEC on. + Music/screen-audio channels → `STEREO`, `AUDIO`, 96–128 kbps, DTX off, FEC optional. +- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for + monitoring use cases. + +## 4. Packet-loss resilience (Opus 1.6) + +Layered, all configurable per channel: + +1. **In-band FEC** — Opus embeds a low-bitrate copy of the previous frame; the decoder + recovers a lost packet from the *next* one (costs one frame of latency on recovery). + Tuned by `expected_packet_loss`. +2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an + unrecovered loss; always on, free. +3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise + updates; cuts bandwidth and is bandwidth-friendly on busy channels. +4. **DRED (Deep REDundancy, optional/feature-gated)** — Opus 1.6's ML redundancy carries + acoustic features so the decoder can reconstruct longer loss bursts. Heavier CPU; gate + behind a `features` flag and per-channel toggle, off by default. + +## 5. Jitter buffer + +Each receiver keeps an **adaptive jitter buffer per ssrc**. + +- Frames are inserted by `timestamp`; playback reads in order at the device callback rate. +- Target depth adapts to observed network jitter between a configurable **min/max latency** + (channel-level "stability vs latency" knob). A "low-latency" channel runs a shallow + buffer; a "stable" channel runs deeper. +- Late frames past the playout point are dropped; gaps are filled by FEC (if the next frame + arrived) or PLC. +- The `marker` flag (start of talkspurt) lets the buffer resynchronize cleanly after + silence/DTX without accumulating drift. + +``` + incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer + ▲ + jitter estimate feeds depth +``` + +## 6. UDP keepalive & NAT + +- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to + hold NAT bindings and measure media-path RTT/loss independent of TCP. +- If the media path dies but TCP is alive, the client surfaces a "voice disconnected" + state and attempts UDP re-binding (re-derive media keys + fresh `UdpBinding`) without dropping + the control session. +- No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the **server** is reachable + (public IP or port-forward); **clients** sit behind NAT and initiate, so their bindings + are created by their outbound first packet. + +## 7. Talk-state signaling + +"Who is talking" can be derived two ways; we use both: + +- **Implicit:** presence of recent voice frames for an ssrc → that stream is "active". The + receiver drives talk indicators from the jitter buffer, so they're accurate and need no + extra messages. +- **Explicit (optional):** `StreamStateUpdate` on TCP for coarse UI state (muted, hold) and + for users not currently subscribed to the media. Server-side mute/deafen is authoritative + and always signaled on TCP. + +## 8. Capture/playback pipeline (inside the core) + +``` + device ─(miniaudio capture, 48k)→ resample? → send-side APM + (AEC + NS + AGC + VAD/PTT gate) → Opus encode → frame header → AEAD → UDP send + + UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode + → per-stream recv-side NS (optional, per user) → per-stream gain/mute + → mixer (sum all ssrc) → (miniaudio playback, 48k) → device +``` + +- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA). +- **DSP engine: webrtc-audio-processing (APM)** — the "better one". It provides + high-quality **AEC** (acoustic echo cancellation, essential for speaker users), **noise + suppression**, **AGC**, and a **VAD** in one tuned module, BSD-licensed. speexdsp is kept + only for resampling and as a lightweight jitter-buffer reference. AEC is **in from the + start**, not deferred. +- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus. + +## 10. Noise reduction — two-sided + +Noise reduction can be applied **at the sender, at the listener, or both** — they are +independent. + +- **Sender-side** (the talker's choice): the publishing client runs APM noise suppression on + its mic before encoding, controlled by that user's own settings. This cleans the signal for + *everyone* and saves bitrate. +- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding + each stream and *before* mixing, the listener can enable an **additional** NS pass on a + **specific** sender's stream. So even if Alex chose not to denoise his mic, Sam can locally + suppress Alex's background noise without affecting how anyone else hears Alex. + +Implementation: a per-`ssrc` APM NS instance on the receive path, instantiated lazily only +for streams the listener has flagged. State lives entirely on the listener's machine; toggling +it is a local UI action with **no protocol message** and no effect on other listeners. Because +each receive stream is decoded independently before the mixer (voice.md §1), per-user receive +NS is a clean drop-in on that per-stream stage. + +## 11. Input activation — VAD and PTT (client-configurable) + +Whether the mic transmits is decided locally by the **input gate**, and the client supports +**both** modes, switchable per client (and ideally per input device): + +- **Voice activation (VAD):** the APM VAD opens the gate when speech is detected, with a + configurable threshold and hang-time to avoid clipping word tails. DTX naturally + complements this — when the gate is closed nothing (or only comfort noise) is sent. +- **Push-to-talk (PTT):** a held key/button opens the gate. The UI exposes a configurable + keybind; the core just receives gate open/close. + +This is purely a send-side, client-local concern — it gates what gets encoded and sent. It +needs **no protocol support**; remote talk indicators are still derived from the presence of +received frames (§7), so they work identically under VAD or PTT. + +## 9. System / screen audio capture (`SCREEN_AUDIO`) + +"Listen together" needs to capture the audio another app is playing. The capture mechanism +differs per OS, but it always feeds the **same** Opus-encode → media-AEAD → UDP path as a +normal stream; only the *source* is platform-specific. + +| Platform | Mechanism | Notes | +|----------|-----------|-------| +| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | Cleanest case; no extra process. Can capture system mix or a specific endpoint. | +| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. | +| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). | + +### iOS detail + +- The user starts a broadcast from Control Center's screen-record button; we surface it via + `RPSystemBroadcastPickerView` from inside the app for one-tap start. +- The **Broadcast Upload Extension** receives `RPSampleBufferType.audioApp` (system/app + audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video + buffers entirely — video is what blows the **~50 MB** extension memory budget, so an + audio-only consumer stays comfortably inside it. +- The extension is a *separate process*. It links a **minimal slice of the core** (Opus + encode + media send only — not the full client), reads the active session token and + server endpoint from a shared **App Group** container that the host app wrote at join + time, derives its own media keys, and publishes the `SCREEN_AUDIO` stream directly. The + host app announces the stream over its control channel (`StreamAnnounce`) so the server and + peers learn about it. +- Mic + voice continue to run in the **host app**; only the system-audio share lives in the + extension. When the broadcast stops (`broadcastFinished`), the extension sends a final + frame with the `last-frame-before-stop` flag and the host app emits `StreamStop`.