docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation: - README: overview, locked decisions, principles, glossary - architecture: shared C++ core + C ABI, native UIs (Swift/C#), threading model, server design (SFU relay) - protocol: TCP/TLS control plane, protobuf Envelope + message catalog, connection lifecycle, extensibility rules - voice: UDP media frame format, per-channel Opus config, multi-stream model, two-sided noise reduction, VAD/PTT, jitter buffer, iOS ReplayKit screen-audio - security: mandatory encryption (TLS 1.3 + exported-key AEAD), TOFU server identity, admin-provisioned accounts, anti-replay - tech-stack: permissive-only deps (mbedTLS, libsodium, opus, miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL - deployment: zero-config self-host (Docker / binary / source) - roadmap: M0-M5 milestones, resolved decisions Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
68
docs/README.md
Normal file
68
docs/README.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# VoiceCat — Design Documentation
|
||||
|
||||
VoiceCat is a self-hosted, server-based voice and text chat system in the spirit of
|
||||
classic TeamSpeak / Mumble: channel-based voice, channel and private text chat, and a
|
||||
single server you own and run. It deliberately avoids WebRTC. The media path is plain
|
||||
**UDP**, the control path is plain **TCP**, and both are encrypted.
|
||||
|
||||
This folder is the design spec. No code yet — these documents define the architecture,
|
||||
the wire protocol, the audio pipeline, the security model, and the dependency list, so
|
||||
that implementation can start from a shared, agreed plan.
|
||||
|
||||
## Decisions locked so far
|
||||
|
||||
| Area | Decision |
|
||||
|------|----------|
|
||||
| Code architecture | **Shared C++ core** (`libvoicecat`) consumed by native UIs over a **C ABI**. Server reuses the same core. |
|
||||
| Native clients | macOS/iOS in **Swift** (SwiftUI; Swift↔C++ interop), Windows in **C#** (`LibraryImport` P/Invoke). |
|
||||
| Control transport | **TCP + TLS 1.3** (mbedTLS) |
|
||||
| Media transport | **UDP** secured by **TLS-exported keys + ChaCha20-Poly1305 AEAD** — mandatory, no plaintext mode (see [security.md](security.md)) |
|
||||
| Crypto libraries | **mbedTLS** (TLS 1.3) + **libsodium** (AEAD, Argon2id, Ed25519) — both permissive, **no GPL/LGPL anywhere** |
|
||||
| Voice codec | **Opus** (libopus 1.6), per-channel configurable mono/stereo, bitrate, frame size, FEC/DTX |
|
||||
| Audio DSP | **webrtc-audio-processing (APM)** — AEC/NS/AGC/VAD. NR is **two-sided**: sender can denoise, and each listener can denoise a *specific* other user locally. Input gate supports **VAD and PTT**, client-configurable. |
|
||||
| Identity | **Guests + admin-provisioned local accounts** (Argon2id, SQLite). No self-serve registration; guests toggleable per server. |
|
||||
| Text | **Ephemeral** — live relay, no server-side history in v1. |
|
||||
| Serialization | **Protocol Buffers** for the control plane; **custom binary** for voice frames |
|
||||
| Deployment | **One `docker run`, one static binary, or `cmake --build`** — zero-config, secured by default (see [deployment.md](deployment.md)) |
|
||||
|
||||
## Document index
|
||||
|
||||
1. [architecture.md](architecture.md) — System layers, the shared core, threading model, the C ABI, server design.
|
||||
2. [protocol.md](protocol.md) — The control protocol: framing, connection lifecycle, the full message catalog, encoding, versioning/extensibility.
|
||||
3. [voice.md](voice.md) — The UDP media protocol: voice frame format, Opus configuration, multi-stream model, jitter buffer, packet-loss handling.
|
||||
4. [security.md](security.md) — Mandatory encryption (TLS 1.3 + exported-key media AEAD), server identity (TOFU), authentication, accounts, anti-replay, threat model.
|
||||
5. [tech-stack.md](tech-stack.md) — Concrete libraries with versions and rationale, the permissive-license rule, build tooling, per-platform notes.
|
||||
6. [deployment.md](deployment.md) — The "set it up in a few minutes" story: Docker, single binary, source build, zero-config defaults.
|
||||
7. [roadmap.md](roadmap.md) — Milestones, what ships when, and the list of open questions still to resolve.
|
||||
|
||||
## Design principles
|
||||
|
||||
- **One core, many faces.** Protocol, crypto, Opus, networking, jitter buffering, and
|
||||
mixing live once in C++. UIs are thin. This keeps behavior identical across platforms
|
||||
and the security-sensitive code reviewed in a single place.
|
||||
- **Boringly simple transport.** TCP for control, UDP for media. No ICE, no SDP, no
|
||||
TURN. A user opens a port (or port-forwards) and runs a server.
|
||||
- **Extensible from day one.** Every message rides in a versioned envelope; capabilities
|
||||
are negotiated at connect time; unknown fields are ignored. File transfer, screen-audio,
|
||||
and moderation slot in without breaking older clients.
|
||||
- **Real-time correctness.** The audio thread never blocks, never allocates, never takes a
|
||||
lock. Network and audio communicate through lock-free ring buffers.
|
||||
- **Encrypted, always.** There is no unencrypted mode to misconfigure. The server has no
|
||||
plaintext listener; encryption is on because it can't be turned off. And it's free to the
|
||||
operator — the server self-provisions its key/cert on first run.
|
||||
- **Stupid-easy to self-host.** The target reaction is "oh, I (or my agent) can stand this up
|
||||
in a few minutes." One `docker run`, or one static binary, or a plain `cmake --build` — no
|
||||
certificate wrangling, no external database, sane defaults out of the box. Permissive
|
||||
licenses only (no GPL/LGPL) so it can be redistributed freely, including closed-source.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Core** — `libvoicecat`, the shared C++ library.
|
||||
- **Control channel** — the TCP/TLS connection carrying protobuf messages.
|
||||
- **Media channel** — the UDP connection carrying voice frames.
|
||||
- **Stream** — one audio source from one user (e.g. mic, screen audio, second device). A
|
||||
user may publish several streams at once; each is independently controllable.
|
||||
- **Channel** — a room in the channel tree. Voice is scoped to a channel.
|
||||
- **Session** — an authenticated connection; ties a TCP control channel to a UDP 5-tuple.
|
||||
- **SFU relay** — the server forwards Opus packets between channel members without decoding
|
||||
them (selective forwarding, no transcoding).
|
||||
215
docs/architecture.md
Normal file
215
docs/architecture.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# Architecture
|
||||
|
||||
## 1. The shared-core model
|
||||
|
||||
All non-UI logic lives in one C++ library, **`libvoicecat`**. The same library is linked
|
||||
into every client and into the server. Platform UIs are thin and call the core through a
|
||||
stable **C ABI** (`voicecat.h`).
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────┐
|
||||
macOS / iOS (Swift) │ │ Windows (C#)
|
||||
┌──────────────────┐ │ libvoicecat (C++) │ ┌──────────────────┐
|
||||
│ SwiftUI views │ │ ┌─────────────────────────────────────┐ │ │ WinUI/Avalonia │
|
||||
│ AVAudioSession │──┼─▶│ C ABI (voicecat.h) │◀─┼──│ LibraryImport │
|
||||
│ Swift↔C++ interop│ │ ├─────────────────────────────────────┤ │ │ P/Invoke │
|
||||
└──────────────────┘ │ │ Session / Protocol state machine │ │ └──────────────────┘
|
||||
│ │ Text + voice signaling │ │
|
||||
Linux/macOS/Windows │ │ Audio engine: capture→encode→send, │ │
|
||||
server │ │ recv→jitter→decode→mix→playback │ │
|
||||
┌──────────────────┐ │ │ Codec layer (Opus 1.6) │ │
|
||||
│ voicecat-server │──┼─▶│ Crypto + transport (TLS 1.3 + AEAD) │ │
|
||||
│ (reuses core) │ │ │ Net I/O (Asio: TCP + UDP + timers) │ │
|
||||
└──────────────────┘ │ └─────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Why this shape:
|
||||
|
||||
- **Swift** (5.9+) can import C++ directly, but we still ship a C ABI because it is the
|
||||
lowest-friction, most stable boundary and it is what **C#** needs (`LibraryImport`/
|
||||
P/Invoke). One ABI serves both.
|
||||
- The **server** is not a separate codebase. It links the same protocol, crypto, and Opus
|
||||
code as the client, so framing/encryption can never drift between the two ends.
|
||||
|
||||
## 2. Layered design inside the core
|
||||
|
||||
From the OS up:
|
||||
|
||||
| Layer | Responsibility | Key deps |
|
||||
|-------|----------------|----------|
|
||||
| **Platform I/O** | Sockets, timers; audio device capture/playback | Asio, miniaudio |
|
||||
| **Transport** | TLS 1.3 (TCP), exported-key ChaCha20-Poly1305 AEAD (UDP), framing, anti-replay | mbedTLS, libsodium |
|
||||
| **Codec & DSP** | Opus encode/decode; APM (AEC/NS/AGC/VAD) send-side + per-user NR receive-side; resample; jitter buffer; mixer | libopus, webrtc-audio-processing, speexdsp |
|
||||
| **Protocol** | Message (de)serialization, request/response correlation, state machine | protobuf |
|
||||
| **Session/domain** | Channels, users, streams, permissions, text routing | — |
|
||||
| **C ABI façade** | Handle-based API + event callbacks exposed to UIs | — |
|
||||
|
||||
A UI never sees a socket, an Opus packet, or a protobuf message. It sees: "connect",
|
||||
"join channel", "start a stream from this device", "send this text", and a stream of
|
||||
events ("user joined", "user is talking", "message received", "level meter = 0.4").
|
||||
|
||||
## 3. Threading model
|
||||
|
||||
Three classes of thread, with strict rules.
|
||||
|
||||
```
|
||||
┌──────────────┐ lock-free ┌──────────────┐ lock-free ┌──────────────┐
|
||||
│ Audio capture│ ──ring buffer─▶│ Net thread │ ──ring buffer─▶│Audio playback│
|
||||
│ (RT, miniaudio│ │ (Asio loop) │ │ (RT, miniaudio│
|
||||
│ callback) │◀───ring buffer─│ │◀───ring buffer─│ callback) │
|
||||
│ capture→Opus │ │ TLS + AEAD, │ │ jitter→Opus │
|
||||
│ encode │ │ route, relay │ │ decode→mix │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘
|
||||
│
|
||||
┌─────▼──────┐
|
||||
│ Worker pool│ DB, Argon2id, file I/O,
|
||||
│ (blocking) │ TLS handshakes, codec setup
|
||||
└────────────┘
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- **Audio (real-time) threads** are driven by the OS audio callback. They must not
|
||||
allocate, lock, log, or do syscalls beyond the ring-buffer hand-off. Opus encode/decode
|
||||
runs here (it is allocation-free after init).
|
||||
- **Net thread(s)** run the Asio event loop: TLS records, AEAD seal/open, protobuf parse, channel
|
||||
routing, jitter-buffer feed. On the server, this is where the SFU relay copies packets
|
||||
to subscribers.
|
||||
- **Worker pool** absorbs anything that can block: SQLite, Argon2id verification, DNS,
|
||||
TLS handshake CPU, codec (re)configuration.
|
||||
- **Communication** between audio and net is single-producer/single-consumer lock-free
|
||||
ring buffers (one per direction per stream). Control-plane events to the UI go through a
|
||||
thread-safe queue drained on the UI's terms.
|
||||
|
||||
## 4. The C ABI (`voicecat.h`) — shape
|
||||
|
||||
Handle-based, opaque pointers, C-linkage. Illustrative (final names in implementation):
|
||||
|
||||
```c
|
||||
typedef struct vc_client vc_client;
|
||||
|
||||
typedef struct {
|
||||
void (*on_event)(void* user, const vc_event* ev); // state changes, messages
|
||||
void (*on_level)(void* user, uint32_t stream_id, float rms); // meters (throttled)
|
||||
void* user;
|
||||
} vc_callbacks;
|
||||
|
||||
vc_client* vc_client_create(const vc_config* cfg, vc_callbacks cb);
|
||||
void vc_client_destroy(vc_client*);
|
||||
|
||||
int vc_connect(vc_client*, const char* host, uint16_t port); // async; result via event
|
||||
int vc_authenticate_guest(vc_client*, const char* nickname);
|
||||
int vc_authenticate_user(vc_client*, const char* user, const char* password);
|
||||
|
||||
int vc_join_channel(vc_client*, uint32_t channel_id, const char* password /*nullable*/);
|
||||
int vc_leave_channel(vc_client*);
|
||||
|
||||
// Streams (mic / screen audio / aux device)
|
||||
int vc_stream_start(vc_client*, const vc_stream_desc* desc, uint32_t* out_stream_id);
|
||||
int vc_stream_stop(vc_client*, uint32_t stream_id);
|
||||
int vc_set_input_device(vc_client*, uint32_t stream_id, const char* device_id);
|
||||
int vc_set_self_mute(vc_client*, bool mic_muted, bool deafened);
|
||||
|
||||
// Text
|
||||
int vc_send_text(vc_client*, vc_text_scope scope, uint32_t target_id, const char* utf8);
|
||||
|
||||
// Enumeration helpers for UI device pickers
|
||||
int vc_list_devices(vc_client*, vc_device_kind kind, vc_device_list* out);
|
||||
```
|
||||
|
||||
Design notes:
|
||||
|
||||
- **Async, event-driven.** Calls return immediately; results and state changes arrive via
|
||||
`on_event`. This maps cleanly onto SwiftUI/`async` and C# `event`/`Task` patterns.
|
||||
- **The core owns audio.** Capture, encode, decode, mixing, and playback happen inside the
|
||||
core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
|
||||
meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
|
||||
exception that needs UI-side cooperation — see below.)
|
||||
- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
|
||||
caller-allocated or returned with a paired `vc_free`.
|
||||
|
||||
### Per-platform binding notes
|
||||
|
||||
- **Swift / Apple.** Import the C ABI via a module map; SwiftUI on top. On **iOS** the app
|
||||
must still own `AVAudioSession` (category `.playAndRecord`, `.voiceChat` mode), request
|
||||
mic permission, and handle interruptions/route changes — the core exposes hooks
|
||||
(`vc_audio_suspend`/`vc_audio_resume`) the Swift layer calls from `AVAudioSession`
|
||||
notifications. Background voice and VoIP push (CallKit/PushKit) are a later milestone.
|
||||
- **iOS screen / system-audio sharing** is supported via a **ReplayKit Broadcast Upload
|
||||
Extension** (the same mechanism Discord uses; triggered from Control Center's screen-record
|
||||
button via `RPSystemBroadcastPickerView`). The extension receives
|
||||
`RPSampleBufferType.audioApp` (system/app audio) and `.audioMic`. We capture **`.audioApp`**
|
||||
for the `SCREEN_AUDIO` "listen together" stream and ignore video. The extension runs in a
|
||||
**separate process with a ~50 MB memory cap** — that cap is a problem only for video
|
||||
frames, so audio-only stays well within budget. It links a *minimal* slice of the core
|
||||
(Opus encode + media send), shares the session/credentials with the host app through an
|
||||
**App Group**, and re-derives its own media keys. This is detailed in [voice.md](voice.md) §9.
|
||||
- **C# / Windows.** `LibraryImport` (source-generated P/Invoke, .NET 7+) over the C ABI.
|
||||
Marshal the `on_event` callback as a `[UnmanagedCallersOnly]`/function-pointer to avoid
|
||||
delegate lifetime pitfalls. UI in **WinUI 3** (most native) or **Avalonia** (if we later
|
||||
want a single C# UI across desktop OSes).
|
||||
|
||||
## 5. Server architecture
|
||||
|
||||
`voicecat-server` is a headless process linking the core.
|
||||
|
||||
```
|
||||
TCP/TLS 1.3 UDP + media AEAD
|
||||
│ │
|
||||
┌────────▼─────────┐ ┌─────────▼──────────┐
|
||||
│ Connection mgr │ │ UDP demux │
|
||||
│ (accept, TLS, │ │ 5-tuple → session │
|
||||
│ per-conn state) │ │ anti-replay window │
|
||||
└────────┬─────────┘ └─────────┬──────────┘
|
||||
│ │
|
||||
┌────────▼───────────────────────────────────▼──────────┐
|
||||
│ Session registry (session_id ↔ TCP conn ↔ UDP tuple) │
|
||||
└────────┬───────────────────────────────┬───────────────┘
|
||||
│ │
|
||||
┌────────▼─────────┐ ┌──────────────┐ ┌▼─────────────────┐
|
||||
│ Channel manager │ │ Text router │ │ Voice router/SFU │
|
||||
│ tree, configs, │ │ channel + PM │ │ relay Opus to │
|
||||
│ membership, perms│ │ │ │ channel members │
|
||||
└────────┬─────────┘ └──────────────┘ └──────────────────┘
|
||||
│
|
||||
┌────────▼─────────┐
|
||||
│ Persistence │ accounts (Argon2id), channels, bans, config
|
||||
│ SQLite │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
- **Voice router is a relay, not a mixer.** For each incoming voice frame it looks up the
|
||||
sender's channel and forwards the *unmodified Opus payload* (restamped with the sender's
|
||||
user id) to every other subscribed member. No server-side decode/transcode → low CPU,
|
||||
low latency, and end-to-content is just Opus. Per-channel Opus params are enforced so all
|
||||
members are mutually decodable.
|
||||
- **Subscriptions.** Clients implicitly subscribe to their current channel's voice; text
|
||||
and presence can be subscribed more broadly. This keeps fan-out bounded on big servers.
|
||||
- **Stateless-ish media.** UDP carries no auth per packet beyond the media-AEAD session;
|
||||
the 5-tuple→session binding is established once via a token (see protocol.md §4).
|
||||
- **Single process, scalable later.** v1 is one process, one machine. The session registry
|
||||
and router are written behind interfaces so a future build can sit them behind a shared
|
||||
bus for multi-node, but that is explicitly out of scope for now.
|
||||
|
||||
## 6. Repository layout (proposed)
|
||||
|
||||
```
|
||||
voice-cat/
|
||||
├── docs/ # this folder
|
||||
├── core/ # libvoicecat (C++)
|
||||
│ ├── include/voicecat.h # the C ABI
|
||||
│ ├── src/{net,crypto,codec,protocol,session,audio}/
|
||||
│ └── proto/ # .proto definitions (shared source of truth)
|
||||
├── server/ # voicecat-server (C++, links core)
|
||||
├── clients/
|
||||
│ ├── apple/ # Swift package + Xcode project (macOS + iOS)
|
||||
│ └── windows/ # .NET solution (C#)
|
||||
├── tools/
|
||||
│ └── vccli/ # headless test client (C++), for protocol bring-up
|
||||
├── third_party/ # vendored / vcpkg manifest
|
||||
└── CMakeLists.txt
|
||||
```
|
||||
|
||||
Build is **CMake** with **vcpkg** (manifest mode) for C/C++ deps; the Apple and Windows UI
|
||||
projects consume the built core as a binary + headers. See [tech-stack.md](tech-stack.md).
|
||||
141
docs/deployment.md
Normal file
141
docs/deployment.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# Deployment & Self-Hosting
|
||||
|
||||
The product goal: someone looks at this and thinks *"oh, I (or my agent) can stand this up
|
||||
in a few minutes."* Everything below is in service of that. Three install paths, all
|
||||
**zero-config and encrypted by default**.
|
||||
|
||||
## 1. The three paths
|
||||
|
||||
### A. Docker (recommended)
|
||||
|
||||
```bash
|
||||
docker run -d --name voicecat \
|
||||
-p 8384:8384/tcp \ # control (TLS 1.3)
|
||||
-p 8384:8384/udp \ # media (encrypted)
|
||||
-v voicecat-data:/data \
|
||||
ghcr.io/<org>/voicecat:latest
|
||||
```
|
||||
|
||||
That's the whole thing. On first start it generates its Ed25519 identity + self-signed
|
||||
cert, creates the SQLite database under `/data`, prints the **server fingerprint** (for
|
||||
clients to verify), and listens. Control and media share one port number on TCP+UDP to keep
|
||||
firewall rules trivial.
|
||||
|
||||
A `docker-compose.yml` is provided for people who prefer it, but it isn't required.
|
||||
|
||||
### B. Single static binary
|
||||
|
||||
```bash
|
||||
# download for your OS, then:
|
||||
./voicecat-server # uses ./voicecat-data/ , prints fingerprint, runs
|
||||
```
|
||||
|
||||
The server is a **single statically linked executable** (mbedTLS, libsodium, opus, sqlite,
|
||||
etc. linked in — all permissive licenses). No runtime, no shared libraries to install, no
|
||||
package manager. Linux (primary), macOS, and Windows builds.
|
||||
|
||||
### C. From source
|
||||
|
||||
```bash
|
||||
git clone … && cd voice-cat
|
||||
cmake --preset server-release # vcpkg fetches & pins all deps
|
||||
cmake --build --preset server-release
|
||||
./build/voicecat-server
|
||||
```
|
||||
|
||||
One `cmake` invocation; vcpkg (manifest mode) resolves the dependency graph reproducibly.
|
||||
No system packages to chase.
|
||||
|
||||
## 2. Zero-config defaults
|
||||
|
||||
The server runs with **no config file at all**. Sensible defaults:
|
||||
|
||||
| Setting | Default |
|
||||
|---------|---------|
|
||||
| Encryption | On, always (not configurable off) |
|
||||
| TLS cert / identity | Auto-generated on first run, persisted to the data dir |
|
||||
| Database | Embedded SQLite in the data dir (no external DB) |
|
||||
| Guests | Enabled (so the very first connect "just works"); easily disabled |
|
||||
| Ports | `8384` TCP + UDP |
|
||||
| A default channel | One "Lobby" voice/text channel created on first run |
|
||||
| Opus policy | 48 kHz, 20 ms frames, mono/VOIP defaults; per-channel overrides allowed |
|
||||
| Argon2id cost | Auto-tuned to the host on first run |
|
||||
|
||||
Override only what you care about, via env vars or an optional `server.toml`:
|
||||
|
||||
```toml
|
||||
# server.toml — every key is optional
|
||||
server_name = "Cats United"
|
||||
allow_guests = false
|
||||
bind_port = 8384
|
||||
data_dir = "/data"
|
||||
|
||||
[tls] # only if you want a real CA cert; otherwise self-signed
|
||||
cert_file = "/data/fullchain.pem"
|
||||
key_file = "/data/privkey.pem"
|
||||
|
||||
[opus.defaults] # default Opus policy for new channels
|
||||
mode = "mono"
|
||||
bitrate_bps = 24000
|
||||
frame_ms = 20
|
||||
fec = true
|
||||
dtx = true
|
||||
|
||||
[opus.limits] # server-enforced ceilings (bound bandwidth)
|
||||
max_bitrate_bps = 128000 # channels can't be configured above this
|
||||
```
|
||||
|
||||
Every key also has an `VOICECAT_*` env var form, which is what the Docker path uses.
|
||||
|
||||
## 3. Connecting (client side)
|
||||
|
||||
- **Pure direct-connect.** Enter `host:port` (and a nickname or account). There is no central
|
||||
directory or server browser — you connect to a server you know.
|
||||
- **Saved server list.** The client keeps a local list of saved servers (host:port, pinned
|
||||
fingerprint, nickname/credentials per server) so you can store several and pick one to
|
||||
join. This lives entirely in the client.
|
||||
- On first connect the client shows the server's **fingerprint** and pins it (TOFU). No
|
||||
accounts or certs needed to try it; if the operator disabled guests, the client prompts for
|
||||
the username/password an admin gave you.
|
||||
|
||||
That's the entire flow: run the server, share `host:port` + fingerprint, friends save it and
|
||||
connect.
|
||||
|
||||
## 3a. Provisioning accounts (admin)
|
||||
|
||||
Accounts are **admin-provisioned** — there is no self-serve registration. Two equivalent ways,
|
||||
both writing the same SQLite store:
|
||||
|
||||
```bash
|
||||
# CLI against the server's data dir or a running server
|
||||
voicecat-admin account add <username> # prompts for / generates a password
|
||||
voicecat-admin account reset <username>
|
||||
voicecat-admin account del <username>
|
||||
voicecat-admin account list
|
||||
```
|
||||
|
||||
…or from the **in-app admin interface** (a user with the admin permission), which sends the
|
||||
privileged `CreateAccount`/`ResetPassword`/`DeleteAccount` control messages over TLS
|
||||
(protocol.md §3). Guests need no provisioning; they just pick a nickname (if guests are
|
||||
enabled).
|
||||
|
||||
## 4. Why it stays this easy (design constraints that protect the goal)
|
||||
|
||||
- **No external services.** No separate database, no Redis, no TURN/STUN server, no reverse
|
||||
proxy required. SQLite is embedded; media is plain UDP.
|
||||
- **No certificate chore.** Self-signed + Ed25519 TOFU means encryption needs zero operator
|
||||
action. A domain owner *can* drop in a Let's Encrypt cert, but never *has* to.
|
||||
- **One port pair.** TCP+UDP on the same number; one firewall/port-forward rule.
|
||||
- **Static linking + permissive licenses.** The binary has no install-time dependencies and
|
||||
can be redistributed (including closed-source) without copyleft obligations.
|
||||
- **Agent-friendly.** The run command is a single line with no interactive prompts, the
|
||||
server logs its fingerprint and listen address in machine-readable form, and `--help` /
|
||||
`--print-config` expose everything an automation needs. Health endpoint for liveness checks.
|
||||
|
||||
## 5. Operational niceties (planned, not blocking v1)
|
||||
|
||||
- `voicecat-server --print-fingerprint` and a `/healthz` TCP check.
|
||||
- Graceful reload of `server.toml` on `SIGHUP`.
|
||||
- `voicecat-admin` (see §3a) also handles bans and channel admin, talking to the same SQLite
|
||||
file or a running server.
|
||||
- Prebuilt images for `linux/amd64` + `linux/arm64` (Raspberry Pi / cheap VPS friendly).
|
||||
291
docs/protocol.md
Normal file
291
docs/protocol.md
Normal file
@@ -0,0 +1,291 @@
|
||||
# Control Protocol
|
||||
|
||||
The control plane runs over **TCP, wrapped in TLS 1.3**. It carries everything that is not
|
||||
real-time media: handshake, authentication, channel/user/presence state, text chat, and
|
||||
voice *signaling* (announcing that a media stream is starting/stopping). Real-time voice
|
||||
travels separately over UDP — see [voice.md](voice.md).
|
||||
|
||||
## 1. Framing
|
||||
|
||||
Inside the TLS stream, messages are length-prefixed:
|
||||
|
||||
```
|
||||
┌──────────────┬───────────────────────────────────────────────┐
|
||||
│ u32 length │ protobuf-encoded Envelope (length bytes) │
|
||||
│ (big-endian)│ │
|
||||
└──────────────┴───────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- `length` is the byte count of the payload that follows (not including the 4 length
|
||||
bytes). Hard cap (e.g. 16 MiB) to bound memory; oversized frame → protocol error +
|
||||
disconnect.
|
||||
- The payload is a single **`Envelope`** protobuf message. We do **not** add our own type
|
||||
byte; the type is the `oneof` discriminator inside the Envelope, which keeps the framing
|
||||
trivial and lets protobuf own all forward/backward compatibility.
|
||||
|
||||
TLS already provides record framing, integrity, and ordering; we only add message
|
||||
boundaries on top of the TLS byte stream.
|
||||
|
||||
## 2. Why Protocol Buffers for control
|
||||
|
||||
- Schema-driven codegen for **C++, C#, and Swift** (all first-class) → no hand-rolled
|
||||
parsers, no drift between client and server.
|
||||
- **Forward/backward compatible by construction**: unknown fields are preserved/ignored,
|
||||
new fields and new `oneof` arms are additive. This is exactly the "extensible protocol"
|
||||
requirement.
|
||||
- Compact enough for a control plane (text/state, not media). We use **`oneof`** envelopes
|
||||
rather than `Any` so the wire stays tight and the switch is exhaustive.
|
||||
|
||||
> Media frames do **not** use protobuf — they use a fixed binary header (see voice.md),
|
||||
> because per-packet protobuf overhead and allocation are unacceptable on the RT path.
|
||||
|
||||
## 3. The Envelope
|
||||
|
||||
```proto
|
||||
syntax = "proto3";
|
||||
package voicecat.v1;
|
||||
|
||||
message Envelope {
|
||||
// Monotonic per-connection id set by the sender of a request; echoed in the
|
||||
// matching response so async callers can correlate. 0 for unsolicited events.
|
||||
uint64 request_id = 1;
|
||||
|
||||
oneof body {
|
||||
// ── Session / handshake ───────────────────────────────
|
||||
ClientHello client_hello = 10;
|
||||
ServerHello server_hello = 11;
|
||||
AuthRequest auth_request = 12;
|
||||
AuthResult auth_result = 13;
|
||||
Disconnect disconnect = 14;
|
||||
Ping ping = 15;
|
||||
Pong pong = 16;
|
||||
|
||||
// ── State sync ────────────────────────────────────────
|
||||
ServerStateSnapshot server_state = 20;
|
||||
ChannelEvent channel_event = 21; // created/updated/deleted
|
||||
UserEvent user_event = 22; // joined/left/updated
|
||||
SubscribeRequest subscribe = 23;
|
||||
|
||||
// ── Channel operations ────────────────────────────────
|
||||
JoinChannelRequest join_channel = 30;
|
||||
JoinChannelResult join_channel_result= 31;
|
||||
LeaveChannelRequest leave_channel = 32;
|
||||
CreateChannelRequest create_channel = 33;
|
||||
EditChannelRequest edit_channel = 34;
|
||||
DeleteChannelRequest delete_channel = 35;
|
||||
MoveUserRequest move_user = 36;
|
||||
GenericResult generic_result = 37; // ack/err for the above
|
||||
|
||||
// ── Voice signaling (media is on UDP) ─────────────────
|
||||
StreamAnnounce stream_announce = 40;
|
||||
StreamAnnounceResult stream_announce_result = 41;
|
||||
StreamStop stream_stop = 42;
|
||||
StreamStateUpdate stream_state = 43; // talking/muted indicator
|
||||
UdpBinding udp_binding = 44; // token to bind the UDP 5-tuple
|
||||
|
||||
// ── Text ──────────────────────────────────────────────
|
||||
TextMessage text_message = 50;
|
||||
TextMessageAck text_message_ack = 51;
|
||||
TypingIndicator typing = 52;
|
||||
|
||||
// ── Moderation / permissions ──────────────────────────
|
||||
KickRequest kick = 60;
|
||||
BanRequest ban = 61;
|
||||
SetPermissionRequest set_permission = 62;
|
||||
|
||||
// ── Admin account management (privileged) ─────────────
|
||||
// Accounts are admin-provisioned (no self-serve registration in v1).
|
||||
// These ride the same TLS control channel and require an admin permission.
|
||||
CreateAccountRequest create_account = 70;
|
||||
ResetPasswordRequest reset_password = 71;
|
||||
DeleteAccountRequest delete_account = 72;
|
||||
ListAccountsRequest list_accounts = 73;
|
||||
|
||||
// ── Extension escape hatch ────────────────────────────
|
||||
Extension extension = 200; // {string ns; bytes payload;}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Reserved tag ranges keep future families from colliding: **10–19** session, **20–29** state,
|
||||
**30–39** channels, **40–49** voice signaling, **50–59** text, **60–99** moderation,
|
||||
**100–199** future (e.g. file transfer = 100–109), **200+** extensions.
|
||||
|
||||
## 4. Connection lifecycle
|
||||
|
||||
```
|
||||
Client Server
|
||||
│ TCP connect ───────────────────────────────────▶│
|
||||
│ ◀──────────────── TLS 1.3 handshake ────────────▶│ server cert (TOFU/PKI, see security.md)
|
||||
│ │
|
||||
│ ClientHello (proto_version, features[], info) ──▶│
|
||||
│ ◀── ServerHello (proto_version, features[], │ feature intersection negotiated here
|
||||
│ server_info, auth_methods, udp_port) │
|
||||
│ │
|
||||
│ AuthRequest (guest{nick} | user{name,pass}) ────▶│ password verified w/ Argon2id
|
||||
│ ◀── AuthResult (ok, session_id, self, perms, │
|
||||
│ udp_token) │
|
||||
│ │
|
||||
│ ◀── ServerStateSnapshot (channel tree, users) ───│ initial sync
|
||||
│ │
|
||||
│ ===== UDP side (parallel) ===================== │
|
||||
│ (media keys derived from TLS exporter — no 2nd │
|
||||
│ handshake; see security.md §2) │
|
||||
│ UdpBinding(udp_token) [AEAD, exported keys] ────▶│ binds 5-tuple → session_id
|
||||
│ ◀── UdpBinding ack [AEAD] ───────────────────────│
|
||||
│ │
|
||||
│ JoinChannelRequest(id, password?) ──────────────▶│
|
||||
│ ◀── JoinChannelResult(ok, members, audio_cfg) ───│
|
||||
│ StreamAnnounce(kind=mic, opus_params) ──────────▶│
|
||||
│ ◀── StreamAnnounceResult(ok, ssrc) │
|
||||
│ ── voice frames flow over UDP ──────────────────▶│
|
||||
│ │
|
||||
│ Ping / Pong (TCP keepalive) ◀──────────────────▶│
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- **Version negotiation.** Each side sends `proto_version` (integer) and a `features`
|
||||
string list. The effective version is `min(client, server)`; the effective feature set
|
||||
is the intersection. A client that doesn't understand a feature simply never uses it.
|
||||
- **Auth over TLS.** Passwords cross the wire only inside TLS 1.3 and are verified against
|
||||
an Argon2id hash at rest (see security.md). `auth_methods` in `ServerHello` advertises
|
||||
whether `guest` is enabled.
|
||||
- **UDP token.** `AuthResult.udp_token` is a short-lived opaque token. The client sends it
|
||||
in the first UDP message (`UdpBinding`) so the server can bind the UDP 5-tuple to the
|
||||
authenticated session without trusting the source address. This is the only UDP message
|
||||
that carries identity material; everything after is implicit via the bound tuple +
|
||||
media-AEAD session.
|
||||
- **Snapshot then deltas.** After auth the server pushes a `ServerStateSnapshot` (full
|
||||
channel tree + visible users), then streams incremental `ChannelEvent`/`UserEvent`
|
||||
deltas. Clients reconcile by id.
|
||||
|
||||
## 5. Message catalog (selected definitions)
|
||||
|
||||
Representative messages; the full `.proto` is the source of truth in `core/proto/`.
|
||||
|
||||
```proto
|
||||
message ClientHello {
|
||||
uint32 proto_version = 1;
|
||||
repeated string features = 2; // e.g. "opus", "fec", "screen-audio"
|
||||
string client_name = 3; // "VoiceCat-macOS"
|
||||
string client_version = 4;
|
||||
string preferred_locale = 5;
|
||||
}
|
||||
|
||||
message ServerHello {
|
||||
uint32 proto_version = 1;
|
||||
repeated string features = 2;
|
||||
string server_name = 3;
|
||||
string server_version = 4;
|
||||
repeated string auth_methods = 5; // "guest", "password"
|
||||
uint32 udp_port = 6;
|
||||
bytes server_identity_fingerprint = 7; // Ed25519 key fp for TOFU display
|
||||
}
|
||||
|
||||
message AuthRequest {
|
||||
oneof method {
|
||||
GuestAuth guest = 1; // { string nickname; }
|
||||
PasswordAuth password = 2; // { string username; string password; }
|
||||
}
|
||||
}
|
||||
|
||||
message AuthResult {
|
||||
bool ok = 1;
|
||||
string error = 2;
|
||||
uint64 session_id = 3;
|
||||
User self = 4;
|
||||
Permissions permissions = 5;
|
||||
bytes udp_token = 6; // bind UDP 5-tuple with this
|
||||
}
|
||||
|
||||
message Channel {
|
||||
uint32 id = 1;
|
||||
uint32 parent_id = 2; // 0 = root
|
||||
string name = 3;
|
||||
string topic = 4;
|
||||
bool password_protected = 5;
|
||||
uint32 max_users = 6;
|
||||
ChannelType type = 7; // PERMANENT / TEMPORARY
|
||||
AudioConfig audio = 8; // per-channel Opus settings (see voice.md)
|
||||
int32 order = 9;
|
||||
}
|
||||
|
||||
message User {
|
||||
uint32 id = 1;
|
||||
string nickname = 2;
|
||||
bool is_guest = 3;
|
||||
uint32 channel_id = 4;
|
||||
bool self_mic_muted = 5;
|
||||
bool self_deafened = 6;
|
||||
bool server_muted = 7;
|
||||
repeated StreamInfo streams = 8; // active media streams this user publishes
|
||||
}
|
||||
|
||||
message StreamInfo {
|
||||
uint32 stream_id = 1; // unique within the user
|
||||
uint32 ssrc = 2; // media-plane id assigned by server
|
||||
StreamKind kind = 3; // MIC / SCREEN_AUDIO / AUX_DEVICE
|
||||
AudioConfig audio = 4;
|
||||
string label = 5; // "Microphone", "Desktop audio"
|
||||
}
|
||||
|
||||
message StreamAnnounce { // client → server: "I'm about to publish media"
|
||||
StreamKind kind = 1;
|
||||
AudioConfig requested_audio = 2; // server may clamp to channel policy
|
||||
string label = 3;
|
||||
}
|
||||
message StreamAnnounceResult {
|
||||
bool ok = 1; string error = 2;
|
||||
uint32 stream_id = 3; uint32 ssrc = 4;
|
||||
AudioConfig effective_audio = 5; // authoritative params to encode with
|
||||
}
|
||||
|
||||
message TextMessage {
|
||||
TextScope scope = 1; // CHANNEL / PRIVATE / SERVER
|
||||
uint32 target_id = 2; // channel_id or user_id depending on scope
|
||||
uint32 sender_id = 3; // set by server on relay
|
||||
string body = 4; // UTF-8, server-bounded length
|
||||
uint64 sent_at_unix_ms = 5; // server timestamp on relay
|
||||
string client_msg_id = 6; // client-chosen, echoed in ack (dedup)
|
||||
}
|
||||
```
|
||||
|
||||
> **Text is ephemeral (v1).** The server relays messages live to currently-connected,
|
||||
> subscribed recipients and **does not persist history** — there is no store and no backfill
|
||||
> on join. Clients may keep their own local scrollback for the session. Server-side history
|
||||
> is a deliberate non-feature for now (it can be added later behind a capability flag without
|
||||
> changing `TextMessage`).
|
||||
|
||||
## 6. Request / response & errors
|
||||
|
||||
- Any message a client expects a direct answer to sets a nonzero **`request_id`**; the
|
||||
server echoes it in the response (`*Result` or `GenericResult`). Unsolicited
|
||||
server→client events use `request_id = 0`.
|
||||
- **`GenericResult { bool ok; uint32 code; string message; }`** is the default
|
||||
acknowledgement for operations without a richer reply (create/edit/delete channel, move
|
||||
user, etc.). Error `code`s are an enumerated, stable list.
|
||||
- Fatal conditions send **`Disconnect { code; reason }`** then close the TLS connection.
|
||||
|
||||
## 7. Keepalive & timeouts
|
||||
|
||||
- **TCP:** `Ping`/`Pong` every ~15 s; missing N consecutive pongs → drop. `Pong` echoes the
|
||||
`Ping` nonce so RTT is measurable.
|
||||
- **UDP:** a separate lightweight keepalive on the media channel (voice.md §6) keeps NAT
|
||||
bindings alive and detects media-path failure independently of the control channel.
|
||||
|
||||
## 8. Extensibility checklist
|
||||
|
||||
When adding a feature later (e.g. **file transfer**), the rules are:
|
||||
|
||||
1. Add new `oneof` arms in the reserved tag range (file transfer = 100–109) — never reuse
|
||||
or renumber existing tags.
|
||||
2. Advertise a feature string in `ClientHello`/`ServerHello`; only use the feature if both
|
||||
peers list it.
|
||||
3. Prefer extending an existing message with new fields (additive) over inventing a new
|
||||
message where it fits.
|
||||
4. For experimental/out-of-tree features, ride inside `Extension { ns; payload }` until it
|
||||
is promoted to a first-class `oneof` arm.
|
||||
|
||||
This guarantees a v1 client and a v3 server interoperate at the negotiated lowest common
|
||||
denominator.
|
||||
99
docs/roadmap.md
Normal file
99
docs/roadmap.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Roadmap & Open Questions
|
||||
|
||||
## 1. Milestones
|
||||
|
||||
Each milestone is shippable/testable on its own. The headless C++ test client (`vccli`)
|
||||
exists from M1 so the protocol can be exercised long before any GUI.
|
||||
|
||||
### M0 — Scaffolding
|
||||
- Repo layout (see architecture.md §6), CMake + vcpkg manifest, CI matrix.
|
||||
- `core/proto/` skeleton; `protoc` codegen wired for C++ (C#/Swift later).
|
||||
- Empty `libvoicecat` with the C ABI header and stub implementations.
|
||||
- **Exit:** core + server + `vccli` compile and link on Linux/macOS/Windows.
|
||||
|
||||
### M1 — Control plane (TCP/TLS, no audio yet)
|
||||
- TLS 1.3 transport; framing; Envelope; ClientHello/ServerHello negotiation.
|
||||
- Auth: **guest + admin-provisioned local account** (Argon2id, SQLite); server identity
|
||||
(TOFU/Ed25519); `voicecat-admin` account add/reset/del/list.
|
||||
- Channel tree: snapshot + deltas; join/leave; create/edit/delete (perm-checked).
|
||||
- **Text chat (ephemeral):** channel + private messages, acks; live relay, no history store.
|
||||
- `vccli` can connect, auth, browse channels, and chat.
|
||||
- **Exit:** two `vccli` instances chat through a real server over TLS.
|
||||
|
||||
### M2 — Voice, single stream
|
||||
- UDP transport with exported-key + ChaCha20-Poly1305 AEAD (mandatory, no plaintext path);
|
||||
UDP token binding; anti-replay.
|
||||
- miniaudio capture/playback; libopus encode/decode; one `MIC` stream per user.
|
||||
- **Send-side webrtc APM** (AEC + NS + AGC + VAD) and a **VAD/PTT input gate** (both modes,
|
||||
client-configurable) — AEC is in from the start, not deferred.
|
||||
- Per-ssrc adaptive jitter buffer; mixer; FEC/PLC/DTX.
|
||||
- Per-channel `AudioConfig` enforced (incl. server `max_bitrate_bps` ceiling); SFU relay.
|
||||
- **Exit:** talk between two `vccli`/early-GUI clients in a channel; loss resilience visible.
|
||||
|
||||
### M3 — Multi-stream & per-channel tuning
|
||||
- Multiple concurrent streams per user (`MIC`, `SCREEN_AUDIO`, `AUX_DEVICE`).
|
||||
- Per-stream receiver gain/mute; **listener-side per-user noise reduction** (APM NS on the
|
||||
receive path, per ssrc, local-only); talk indicators.
|
||||
- Full per-channel Opus configurability (mono/stereo, bitrate, frame size, FEC/DTX).
|
||||
- **Exit:** a user shares mic + desktop audio; listeners control each independently.
|
||||
|
||||
### M4 — Native clients
|
||||
- **Windows (C#/WinUI):** connect, saved-server list, channel tree, voice, text, device
|
||||
pickers, meters, VAD/PTT + per-user NR controls.
|
||||
- **macOS (Swift/SwiftUI):** same.
|
||||
- **iOS (Swift):** AVAudioSession integration, mic permission, foreground voice; ReplayKit
|
||||
broadcast extension for `SCREEN_AUDIO`.
|
||||
- In-app **admin interface** (account provisioning, bans) for admin users.
|
||||
- **Exit:** non-technical user installs a client, saves a server, and joins.
|
||||
|
||||
### M5 — Moderation, polish, and beyond
|
||||
- Permissions/roles, kick/ban/server-mute, channel passwords UI.
|
||||
- DRED toggle, audio-quality polish. (AEC and VAD/PTT already shipped in M2.)
|
||||
- **Then (post-v1, protocol already reserves space):** file transfer, E2EE option,
|
||||
CallKit/PushKit background voice, key-based identity, server-side text history,
|
||||
multi-node server.
|
||||
|
||||
## 2. Resolved decisions
|
||||
|
||||
Settled and reflected throughout the docs:
|
||||
|
||||
- **Media crypto:** exported-key + ChaCha20-Poly1305 AEAD from day one, **mandatory** — no
|
||||
DTLS, no plaintext mode. (security.md §2)
|
||||
- **TLS library / licensing:** **mbedTLS** (Apache-2.0) + **libsodium** (ISC). **No GPL/LGPL**
|
||||
anywhere; code is redistributable closed-source. wolfSSL is rejected. (tech-stack.md §5)
|
||||
- **iOS screen/system audio:** supported via a ReplayKit Broadcast Upload Extension
|
||||
(`.audioApp`); audio-only stays within the extension memory cap. (voice.md §9)
|
||||
- **Self-host UX:** zero-config, encrypted-by-default; Docker / single binary / source build.
|
||||
(deployment.md)
|
||||
- **DSP engine:** **webrtc-audio-processing (APM)** — AEC in from the start, plus NS/AGC/VAD.
|
||||
(voice.md §8)
|
||||
- **Input activation:** **VAD *and* PTT**, both modes client-configurable. (voice.md §11)
|
||||
- **Noise reduction is two-sided:** sender can denoise its mic, *and* each listener can apply
|
||||
NS to a **specific** other user, locally, with no protocol traffic. (voice.md §10)
|
||||
- **Accounts:** **admin-provisioned** (no self-serve registration) via `voicecat-admin` or the
|
||||
in-app admin interface. (security.md §4, protocol.md §3, deployment.md §3a)
|
||||
- **Text:** **ephemeral** — live relay, no server-side history in v1. (protocol.md §5)
|
||||
- **Connect UX:** pure direct-connect with a **client-side saved-server list** (no central
|
||||
directory). (deployment.md §3)
|
||||
- **Bitrate ceiling:** server-config `opus.limits.max_bitrate_bps`. (deployment.md §2)
|
||||
- **Name:** "VoiceCat" stays as the internal placeholder.
|
||||
|
||||
## 3. Open questions
|
||||
|
||||
All initial open questions are resolved (§2). Two **second-order considerations** to keep in
|
||||
mind during implementation — not blockers:
|
||||
|
||||
- **APM in constrained contexts.** webrtc-audio-processing is a heavier build; confirm it
|
||||
static-links cleanly for the single-binary goal, and note the iOS **broadcast extension only
|
||||
does Opus encode + send** (no APM), so it stays under the ~50 MB cap. Listener-side per-user
|
||||
NS runs only in the full host app.
|
||||
- **Receive-side NR cost at scale.** A per-ssrc APM NS instance per flagged user adds CPU on
|
||||
busy channels; instantiate lazily (only for flagged streams) and cap concurrent instances.
|
||||
|
||||
## 4. What's intentionally deferred
|
||||
|
||||
To keep v1 focused (voice + text), these are designed-for but not built: file transfer,
|
||||
end-to-end encryption, key-based identities, multi-node/federated servers, mobile
|
||||
background VoIP push, and any server-side audio mixing/transcoding. The protocol's
|
||||
versioning + feature negotiation + reserved tag ranges (protocol.md §8) ensure each can be
|
||||
added without breaking deployed clients.
|
||||
157
docs/security.md
Normal file
157
docs/security.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# Security Model
|
||||
|
||||
Two encrypted transports: **TLS 1.3** on the TCP control channel, and an encrypted **UDP**
|
||||
media channel. Plus server identity, authentication, accounts at rest, and anti-replay.
|
||||
|
||||
> **Encryption is mandatory — there is no unencrypted mode.** The server has no plaintext
|
||||
> listener, the client has no "insecure" option, and there is no config flag to turn either
|
||||
> off. A connection is encrypted or it does not exist. This is a hard product rule, not a
|
||||
> default. It is also *zero-config* (see §1): the server generates its own key/cert on first
|
||||
> run, so "secured by default" never costs the operator a setup step.
|
||||
|
||||
## 1. Control channel — TLS 1.3 (settled)
|
||||
|
||||
- TCP control is wrapped in **TLS 1.3** (TLS 1.2 disabled). AEAD cipher suites only
|
||||
(AES-128-GCM, AES-256-GCM, ChaCha20-Poly1305). X25519 key exchange.
|
||||
- Library: **mbedTLS 3.6 LTS** — Apache-2.0 (permissive, fine for an eventual closed-source
|
||||
distribution), TLS 1.3 client+server, and `mbedtls_ssl_export_keying_material()` for the
|
||||
media path (§2). It also **static-links cleanly into a single self-host binary**, which is
|
||||
a deliberate choice in service of the easy-deploy goal. (OpenSSL 3.x, also permissive
|
||||
Apache-2.0, is a drop-in alternative behind the same internal interface.)
|
||||
- **Zero-config TLS:** on first launch the server auto-generates a self-signed certificate
|
||||
bound to a freshly generated **Ed25519 identity key** and persists both. The operator does
|
||||
nothing. Clients pin the identity on first connect (TOFU, §1.1). A server *with* a domain
|
||||
can drop in a CA cert later, but it is never required to be encrypted.
|
||||
- All authentication and account material crosses the wire only inside this tunnel.
|
||||
|
||||
### 1.1 Server identity — two modes
|
||||
|
||||
Self-hosting means most servers won't have a CA-signed cert for a hostname. We support
|
||||
both, advertised in `ServerHello`:
|
||||
|
||||
1. **TOFU (Trust On First Use)** — default for hobby servers. The server has a long-lived
|
||||
**Ed25519 identity key**; its fingerprint is shown to the user on first connect (like
|
||||
SSH host keys / TeamSpeak server keys) and pinned locally. Subsequent connects verify
|
||||
the pin; a changed key warns loudly. The TLS cert is self-signed and bound to this
|
||||
identity key.
|
||||
2. **PKI** — a server with a domain can use a normal CA-signed cert (e.g. Let's Encrypt);
|
||||
clients validate the chain conventionally. TOFU pinning still applies on top.
|
||||
|
||||
Client certificates are reserved for a future "key-based identity" option (see roadmap) but
|
||||
are not required in v1.
|
||||
|
||||
## 2. Media channel — UDP encryption (settled: exported-keys + AEAD)
|
||||
|
||||
The UDP media path uses **TLS-exported keys + per-packet AEAD** (an SRTP-style design),
|
||||
mandatory from the first build. This was chosen over DTLS after weighing two findings:
|
||||
|
||||
> **Finding 1 — DTLS 1.3 (RFC 9147) is not in stable OpenSSL or mbedTLS.** It ships
|
||||
> production-ready only in **wolfSSL**, which is **GPLv2-or-commercial** — disqualified,
|
||||
> because the code will eventually be distributed in closed-source form (no GPL/LGPL deps).
|
||||
>
|
||||
> **Finding 2 — mbedTLS 3.6 LTS already exposes `mbedtls_ssl_export_keying_material()`**
|
||||
> (RFC 5705 / RFC 8446 §7.5 exporter). So we can derive media keys from the existing TLS 1.3
|
||||
> control session with *zero* extra handshake and *zero* extra dependency.
|
||||
|
||||
### How it works
|
||||
|
||||
1. During the TLS 1.3 control handshake, both sides call the keying-material exporter with a
|
||||
fixed label (`"voicecat media v1"`) to derive independent **send/recv media keys** and a
|
||||
salt. No second handshake, no certificates on the UDP path — the UDP channel inherits the
|
||||
authenticated, MITM-resistant TLS session's trust.
|
||||
2. Each UDP voice frame is sealed with **ChaCha20-Poly1305** (libsodium, ISC license).
|
||||
3. The readable routing field (`ssrc`) is passed as AEAD **associated data** so the relay can
|
||||
route without decrypting and an attacker cannot tamper with it undetected.
|
||||
|
||||
This keeps the entire crypto surface on two permissive libraries (mbedTLS + libsodium), adds
|
||||
no handshake latency to voice startup, and is small enough to audit fully. It is abstracted
|
||||
behind a `MediaCrypto { seal(frame)->bytes; open(bytes)->frame }` interface, so a future
|
||||
DTLS 1.3 backend could slot in later if a permissive implementation matures — but nothing in
|
||||
the design depends on that.
|
||||
|
||||
### Per-frame protections
|
||||
|
||||
- **AEAD** (ChaCha20-Poly1305) over each voice frame — confidentiality + integrity.
|
||||
- **Associated data:** the `ssrc` (and version/flags) are authenticated-but-visible so the
|
||||
relay routes without decrypting; everything else is encrypted.
|
||||
- **Nonce discipline:** `nonce = direction_bit ‖ ssrc ‖ monotonic_packet_counter`. The
|
||||
counter never repeats under one key; the session **rekeys** (re-derives via the exporter
|
||||
with a bumped epoch) well before counter exhaustion or on a time/byte budget.
|
||||
- **Anti-replay:** a sliding-window replay filter per ssrc (à la IPsec) keyed on the packet
|
||||
counter. Replays and out-of-window packets are dropped before decode.
|
||||
|
||||
## 3. UDP session binding
|
||||
|
||||
UDP packets are not individually authenticated to a *user* beyond the transport session.
|
||||
Binding works as:
|
||||
|
||||
1. `AuthResult.udp_token` (issued over TLS) is a short-lived, single-use, random token tied
|
||||
to `session_id`.
|
||||
2. Client's first UDP message is `UdpBinding{udp_token}`, sent as the first AEAD media frame
|
||||
using the keys exported from the TLS session.
|
||||
3. Server validates the token, binds the **5-tuple → session_id**, and discards the token.
|
||||
4. Thereafter, frames are accepted only on that bound tuple; ssrcs are checked against the
|
||||
streams the session announced. Source-address spoofing can't hijack a session because the
|
||||
attacker lacks the media key and the token.
|
||||
|
||||
## 4. Authentication & accounts (settled: guests + local accounts)
|
||||
|
||||
- **Guests:** toggled by server config (`allow_guests`). A guest picks a nickname and joins;
|
||||
no persistent identity. Nicknames are non-reserved and may be uniquified by the server.
|
||||
- **Local accounts:** username + password, **admin-provisioned** (no self-serve registration
|
||||
in v1). An admin creates/resets/deletes accounts either via the `voicecat-admin` CLI or the
|
||||
in-app admin interface, which sends the privileged `CreateAccount`/`ResetPassword`/
|
||||
`DeleteAccount` messages (protocol.md §3, permission-gated). Stored in **SQLite**; passwords
|
||||
hashed with **Argon2id** (via libsodium `crypto_pwhash`) using per-install-tuned memory/time
|
||||
parameters; never stored or logged in plaintext. Verification runs on the worker pool (it's
|
||||
deliberately slow) to avoid stalling the net thread.
|
||||
- **Channel passwords:** hashed at rest too; join attempts compare server-side.
|
||||
- **Brute-force defense:** per-IP and per-account rate limiting on auth attempts with
|
||||
exponential backoff; configurable lockout. Generic `auth_request` failures return a
|
||||
non-enumerating error ("invalid credentials") to avoid username probing.
|
||||
|
||||
```
|
||||
accounts( id INTEGER PK, username TEXT UNIQUE,
|
||||
pw_argon2id TEXT, -- encoded hash incl. params + salt
|
||||
created_at, last_login, flags )
|
||||
bans( id, subject_type, subject, reason, expires_at )
|
||||
```
|
||||
|
||||
## 5. Permissions (scaffold for v1, enforced server-side)
|
||||
|
||||
A `Permissions` set is attached to each session at auth time and is the *only* authority —
|
||||
clients never self-grant. v1 needs a minimal set (join channel, send text, create temporary
|
||||
channel, kick/move if moderator); the model is a role/flag bitset that the moderation
|
||||
milestone expands. All privileged operations (`CreateChannel`, `Kick`, `Ban`,
|
||||
`MoveUser`, server-mute) are checked against it server-side regardless of client UI.
|
||||
|
||||
## 6. Threat model & non-goals
|
||||
|
||||
**In scope:**
|
||||
- Passive eavesdropping on either transport → defeated by TLS 1.3 (control) and the
|
||||
exported-key AEAD (media).
|
||||
- Active MITM on first connect → mitigated by TOFU pin + Ed25519 identity (user must verify
|
||||
fingerprint out-of-band for the strongest guarantee).
|
||||
- UDP source spoofing / session hijack → defeated by token binding + media-key secrecy +
|
||||
anti-replay.
|
||||
- Password theft at rest → mitigated by Argon2id; in transit → only inside TLS.
|
||||
|
||||
**Explicit non-goals (v1):**
|
||||
- **End-to-end encryption between users.** The server relays Opus and can see who talks to
|
||||
whom; with the SFU relay it does *not* decode audio, but the media key is per
|
||||
client↔server, not per pair. True E2EE (server can't read media) is a possible future
|
||||
feature, not v1.
|
||||
- Anonymity / metadata hiding. The server, by design, knows the channel graph.
|
||||
- DoS resilience at scale beyond basic rate limiting and the bounded-frame guards.
|
||||
|
||||
## 7. Crypto dependency summary
|
||||
|
||||
Two libraries, both permissive (no GPL/LGPL), so a future closed-source distribution stays
|
||||
clean:
|
||||
|
||||
- **TLS 1.3:** **mbedTLS 3.6 LTS** (Apache-2.0). Control-channel TLS + the keying-material
|
||||
exporter that seeds the media path. Static-links into a single binary. (OpenSSL 3.x,
|
||||
Apache-2.0, is an interchangeable alternative.)
|
||||
- **Primitives & password hashing:** **libsodium** (ISC) — Argon2id (account passwords),
|
||||
ChaCha20-Poly1305 (the media AEAD), Ed25519 (server identity), X25519, secure RNG. All
|
||||
non-TLS crypto goes through libsodium so we never hand-roll a primitive.
|
||||
91
docs/tech-stack.md
Normal file
91
docs/tech-stack.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Tech Stack & Dependencies
|
||||
|
||||
Concrete library choices with versions and rationale. Everything in the **core** is C++
|
||||
(C++20). UIs are Swift and C#. Build is CMake + vcpkg.
|
||||
|
||||
## 1. Core library (`libvoicecat`, C++20)
|
||||
|
||||
| Concern | Choice | Version (as of 2026-06) | Why / notes |
|
||||
|---------|--------|-------------------------|-------------|
|
||||
| Sockets, timers, async | **Standalone Asio** | 1.30.x | Header-only, no Boost dependency, cross-platform TCP+UDP+timers, one reactor for client and server. (Boost.Asio is interchangeable if we already pull Boost.) |
|
||||
| TLS 1.3 (control) | **mbedTLS 3.6 LTS** | 3.6.x (LTS ≥ Mar 2027) | **Apache-2.0** (permissive — clean for eventual closed-source distribution). TLS 1.3 client+server, plus `mbedtls_ssl_export_keying_material()` to seed the media AEAD. **Static-links cleanly → single self-host binary.** OpenSSL 3.x (Apache-2.0) is an interchangeable alternative. No DTLS/wolfSSL (GPL) — see [security.md](security.md) §2. |
|
||||
| Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash`), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. |
|
||||
| Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. |
|
||||
| Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio**. One real-time abstraction across all desktop targets; keeps the RT path identical. |
|
||||
| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) | 1.x (standalone APM) | **BSD-3.** The primary DSP engine: high-quality acoustic echo cancellation, noise suppression, AGC, and VAD in one tuned module. Used **send-side** (clean the mic) and **receive-side per user** (listener-chosen NS on a specific stream — voice.md §10). AEC is in from the start, not deferred. |
|
||||
| Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) |
|
||||
| Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. |
|
||||
| Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. |
|
||||
| Logging | **spdlog** | 1.14.x | Fast, async-capable; off the RT path. |
|
||||
|
||||
Resampling note: Opus runs internally at 48 kHz; miniaudio can deliver 48 kHz directly, so
|
||||
explicit resampling (speexdsp/libsamplerate) is only needed when a device can't do 48 kHz.
|
||||
|
||||
## 2. Clients
|
||||
|
||||
### macOS / iOS — Swift
|
||||
|
||||
| Concern | Choice | Notes |
|
||||
|---------|--------|-------|
|
||||
| Language | **Swift 5.9+** | Direct **Swift↔C++ interop** available, but we bind through the C ABI for parity with Windows. |
|
||||
| UI | **SwiftUI** | Single UI codebase for macOS + iOS where practical; AppKit/UIKit shims as needed. |
|
||||
| Audio session (iOS) | **AVAudioSession** | App owns category `.playAndRecord` + `.voiceChat` mode, mic permission, interruption/route-change handling; calls `vc_audio_suspend/resume` on the core. macOS uses CoreAudio via the core directly. |
|
||||
| Packaging | Swift Package + Xcode project | Core shipped as an XCFramework (device + simulator + macOS slices). |
|
||||
| Future | CallKit / PushKit | For background VoIP + incoming-call UX on iOS. Post-v1. |
|
||||
|
||||
### Windows — C#
|
||||
|
||||
| Concern | Choice | Notes |
|
||||
|---------|--------|-------|
|
||||
| Runtime | **.NET 8+** | LTS. |
|
||||
| Interop | **`LibraryImport`** (source-gen P/Invoke) over the C ABI | Marshal the event callback as a function pointer (`[UnmanagedCallersOnly]`) to avoid delegate-lifetime bugs; keep the interface "chunky" not "chatty" to minimize managed↔native transitions. |
|
||||
| UI | **WinUI 3** (most native) or **Avalonia** | WinUI for a first-class Windows look; Avalonia if we later want one C# UI across desktop OSes. |
|
||||
| Audio | handled by the core (miniaudio/WASAPI) | C# only drives device selection + meters. |
|
||||
|
||||
## 3. Server (`voicecat-server`)
|
||||
|
||||
- Pure C++ linking the core; **no GUI**. Runs on **Linux** (primary), **macOS**, **Windows**.
|
||||
- Config via a `server.toml` (`allow_guests`, ports, channel defaults, Opus policy, TLS cert
|
||||
paths or auto-self-signed + Ed25519 identity, Argon2id cost params, rate limits).
|
||||
- SQLite for state. Single process for v1; interfaces drawn so a multi-node build is
|
||||
*possible* later but explicitly out of scope.
|
||||
- Packaging: static-ish binary per OS; systemd unit + Docker image for Linux.
|
||||
|
||||
## 4. Build & tooling
|
||||
|
||||
| Tool | Use |
|
||||
|------|-----|
|
||||
| **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. |
|
||||
| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, webrtc-audio-processing, speexdsp, asio, miniaudio). Reproducible across OSes. |
|
||||
| **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). |
|
||||
| **clang-format / clang-tidy** | Style + static analysis on the core. |
|
||||
| **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). |
|
||||
| **GitHub Actions** (or similar) | Matrix CI: Linux/macOS/Windows core+server; Xcode build for Apple; `dotnet` build for Windows. |
|
||||
|
||||
## 5. Licensing — permissive only (hard rule)
|
||||
|
||||
The code will eventually be distributed in **closed-source** form, so **no GPL/LGPL
|
||||
dependencies are permitted.** Every dependency below is BSD / MIT / ISC / Apache-2.0 /
|
||||
public-domain:
|
||||
|
||||
- **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ ·
|
||||
**miniaudio** — public domain / MIT-0 ✅ · **speexdsp** — BSD ✅ · **protobuf** — BSD ✅ ·
|
||||
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅ ·
|
||||
**webrtc-audio-processing** — BSD-3 ✅ (heavier build, but core to the DSP path).
|
||||
- **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag
|
||||
in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for
|
||||
one entirely.
|
||||
- CI runs a license scanner over the resolved vcpkg graph and **fails the build on any
|
||||
GPL/LGPL transitive dependency**, so this rule can't silently regress.
|
||||
|
||||
## 6. Why not the obvious alternatives
|
||||
|
||||
- **WebRTC** — explicitly rejected: ICE/SDP/TURN complexity, huge dependency, opaque. We
|
||||
want plain TCP+UDP we fully control.
|
||||
- **QUIC** — capable (reliable streams + datagrams + TLS 1.3 in one), but heavier and drifts
|
||||
toward the complexity we're avoiding. Revisit only if NAT traversal/multiplexing pain
|
||||
appears.
|
||||
- **gRPC** for control — pulls HTTP/2 and a lot of surface for what is a simple framed
|
||||
message stream over TLS. Plain protobuf-over-framed-TLS is enough.
|
||||
- **A Rust core** — viable and memory-safe, but the user prefers C++ and the Swift/C#
|
||||
binding story is marginally simpler from C++ (Swift can even consume C++ directly).
|
||||
240
docs/voice.md
Normal file
240
docs/voice.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# Voice & Media
|
||||
|
||||
Real-time audio runs over **UDP**, secured per [security.md](security.md). The control
|
||||
channel (TCP/TLS) handles *signaling* — announcing streams, channel membership, talk state
|
||||
— while UDP carries only the encoded audio frames. This split keeps media latency low and
|
||||
independent of TCP head-of-line blocking.
|
||||
|
||||
## 1. The multi-stream model
|
||||
|
||||
A **user** publishes one or more **streams**. Each stream is an independent audio source
|
||||
with its own encoder, its own `stream_id` (unique per user) and `ssrc` (media-plane id
|
||||
assigned by the server), and is independently mutable/mutable at the receiver.
|
||||
|
||||
```
|
||||
User "Alex" Receiver "Sam"
|
||||
┌────────────────────┐ ┌──────────────────────────┐
|
||||
│ mic → enc ───┼──ssrc 1001──▶ │ jitter(1001)→dec→┐ │
|
||||
│ desktop → enc ───┼──ssrc 1002──▶ │ jitter(1002)→dec→┤ │
|
||||
│ 2nd mic → enc ───┼──ssrc 1003──▶ │ jitter(1003)→dec→┴─mix──▶ out
|
||||
└────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
Stream **kinds** (v1): `MIC`, `SCREEN_AUDIO` (system/desktop audio for listening together),
|
||||
`AUX_DEVICE` (a second capture device). Receivers can set, per incoming stream: **gain**,
|
||||
**mute**, and **noise reduction** (see §10) — so Sam can turn down Alex's desktop audio
|
||||
while keeping the mic, *and* independently apply noise suppression to a third user who has a
|
||||
loud fan. The mixer sums all active streams from all users in the channel into the local
|
||||
playback device. All of these receiver-side controls are **local to the listener** and carry
|
||||
no protocol traffic.
|
||||
|
||||
`SCREEN_AUDIO` capture is platform-specific and covered in §9 — it is supported on Windows,
|
||||
macOS, **and iOS** (via a ReplayKit broadcast extension).
|
||||
|
||||
## 2. Voice frame format (UDP payload, inside the media AEAD)
|
||||
|
||||
A fixed binary header — no protobuf on the RT path. Multi-byte fields are big-endian.
|
||||
|
||||
```
|
||||
0 1 2 3 4 5 6 7 8 ...
|
||||
┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────────────┐
|
||||
│ type │flags │ codec │ ssrc (u32) │
|
||||
├──────┴──────┴──────┴──────┼──────┬──────┬──────┬──────┬──────────────┤
|
||||
│ seq (u16) │ timestamp (u32, in samples @48k) │ payload ... │
|
||||
└─────────────┴──────────────────────────────────────────┴──────────────┘
|
||||
|
||||
type u8 1 = VOICE, 2 = KEEPALIVE, 3 = UDP_BINDING (handshake)
|
||||
flags u8 bit0 marker (start of talkspurt) · bit1 FEC-present
|
||||
bit2 DTX/comfort-noise · bit3 last-frame-before-stop
|
||||
codec u16 0 = OPUS (room for future codecs)
|
||||
ssrc u32 media-plane stream id. Client sends its own ssrc; the server
|
||||
validates it against the bound session and relays unchanged.
|
||||
seq u16 per-ssrc sequence number, wraps; drives loss detection + reorder
|
||||
timestamp u32 RTP-style sample clock @48 kHz; drives the jitter buffer
|
||||
payload one Opus packet (the encoder's output for one frame)
|
||||
```
|
||||
|
||||
This is intentionally RTP-shaped (familiar semantics: ssrc/seq/timestamp) without RTP's
|
||||
full machinery. The **server relays the payload unmodified** — it only reads the header to
|
||||
route by ssrc→channel and may restamp nothing (the client's ssrc is globally unique once
|
||||
assigned at `StreamAnnounce`). No server-side decode.
|
||||
|
||||
### Why client-sends-ssrc is safe
|
||||
|
||||
The UDP 5-tuple is bound to an authenticated session (protocol.md §4). The server checks
|
||||
that the ssrc in each frame belongs to a stream that session announced; spoofed ssrcs are
|
||||
dropped. So identity is anchored by the session binding + transport encryption, not by
|
||||
trusting the header.
|
||||
|
||||
## 3. Per-channel audio configuration
|
||||
|
||||
Opus is configured **per channel** and pushed to clients in `JoinChannelResult.audio` /
|
||||
`StreamAnnounceResult.effective_audio`. All members of a channel encode with mutually
|
||||
decodable parameters.
|
||||
|
||||
```proto
|
||||
message AudioConfig {
|
||||
uint32 codec = 1; // 0 = OPUS
|
||||
ChannelMode mode = 2; // MONO / STEREO
|
||||
uint32 sample_rate = 3; // 8000/12000/16000/24000/48000 (48000 recommended)
|
||||
uint32 bitrate_bps = 4; // e.g. 24000 (speech) … 128000 (music/stereo)
|
||||
uint32 frame_ms = 5; // 2.5/5/10/20/40/60 (20 default)
|
||||
OpusApplication application = 6;// VOIP / AUDIO / LOWDELAY
|
||||
bool fec = 7; // in-band forward error correction
|
||||
uint32 expected_packet_loss = 8;// %, tunes FEC aggressiveness
|
||||
bool dtx = 9; // discontinuous transmission (silence suppression)
|
||||
uint32 complexity = 10; // 0..10 encoder complexity
|
||||
}
|
||||
```
|
||||
|
||||
Guidance baked into defaults / docs:
|
||||
|
||||
- **Sample rate: always run Opus at 48 kHz internally.** Opus resamples internally anyway;
|
||||
48 kHz avoids surprises. The `sample_rate` field mainly constrains capture/narrowband
|
||||
modes for very low bitrate channels. Default **48000**.
|
||||
- **Frame size: 20 ms default.** Smaller (10 ms) lowers latency at the cost of more
|
||||
per-packet overhead and CPU; larger (40/60 ms) improves efficiency and loss resilience at
|
||||
the cost of latency. Expose it per channel for "low-latency talk" vs "stable music" rooms.
|
||||
- **Mode/bitrate:** speech channels → `MONO`, `VOIP`, 24–32 kbps, DTX on, FEC on.
|
||||
Music/screen-audio channels → `STEREO`, `AUDIO`, 96–128 kbps, DTX off, FEC optional.
|
||||
- **`application`:** `VOIP` for talk, `AUDIO` for music/screen-share, `LOWDELAY` for
|
||||
monitoring use cases.
|
||||
|
||||
## 4. Packet-loss resilience (Opus 1.6)
|
||||
|
||||
Layered, all configurable per channel:
|
||||
|
||||
1. **In-band FEC** — Opus embeds a low-bitrate copy of the previous frame; the decoder
|
||||
recovers a lost packet from the *next* one (costs one frame of latency on recovery).
|
||||
Tuned by `expected_packet_loss`.
|
||||
2. **PLC (packet loss concealment)** — decoder synthesizes a plausible frame for an
|
||||
unrecovered loss; always on, free.
|
||||
3. **DTX** — sender stops transmitting during silence and sends sparse comfort-noise
|
||||
updates; cuts bandwidth and is bandwidth-friendly on busy channels.
|
||||
4. **DRED (Deep REDundancy, optional/feature-gated)** — Opus 1.6's ML redundancy carries
|
||||
acoustic features so the decoder can reconstruct longer loss bursts. Heavier CPU; gate
|
||||
behind a `features` flag and per-channel toggle, off by default.
|
||||
|
||||
## 5. Jitter buffer
|
||||
|
||||
Each receiver keeps an **adaptive jitter buffer per ssrc**.
|
||||
|
||||
- Frames are inserted by `timestamp`; playback reads in order at the device callback rate.
|
||||
- Target depth adapts to observed network jitter between a configurable **min/max latency**
|
||||
(channel-level "stability vs latency" knob). A "low-latency" channel runs a shallow
|
||||
buffer; a "stable" channel runs deeper.
|
||||
- Late frames past the playout point are dropped; gaps are filled by FEC (if the next frame
|
||||
arrived) or PLC.
|
||||
- The `marker` flag (start of talkspurt) lets the buffer resynchronize cleanly after
|
||||
silence/DTX without accumulating drift.
|
||||
|
||||
```
|
||||
incoming (out of order) ──▶ [ reorder by ts | adaptive depth ] ──▶ Opus decode ──▶ mixer
|
||||
▲
|
||||
jitter estimate feeds depth
|
||||
```
|
||||
|
||||
## 6. UDP keepalive & NAT
|
||||
|
||||
- A `KEEPALIVE` (type 2) frame flows both directions on the media channel every ~5 s to
|
||||
hold NAT bindings and measure media-path RTT/loss independent of TCP.
|
||||
- If the media path dies but TCP is alive, the client surfaces a "voice disconnected"
|
||||
state and attempts UDP re-binding (re-derive media keys + fresh `UdpBinding`) without dropping
|
||||
the control session.
|
||||
- No ICE/STUN/TURN. The expectation matches TeamSpeak/Mumble: the **server** is reachable
|
||||
(public IP or port-forward); **clients** sit behind NAT and initiate, so their bindings
|
||||
are created by their outbound first packet.
|
||||
|
||||
## 7. Talk-state signaling
|
||||
|
||||
"Who is talking" can be derived two ways; we use both:
|
||||
|
||||
- **Implicit:** presence of recent voice frames for an ssrc → that stream is "active". The
|
||||
receiver drives talk indicators from the jitter buffer, so they're accurate and need no
|
||||
extra messages.
|
||||
- **Explicit (optional):** `StreamStateUpdate` on TCP for coarse UI state (muted, hold) and
|
||||
for users not currently subscribed to the media. Server-side mute/deafen is authoritative
|
||||
and always signaled on TCP.
|
||||
|
||||
## 8. Capture/playback pipeline (inside the core)
|
||||
|
||||
```
|
||||
device ─(miniaudio capture, 48k)→ resample? → send-side APM
|
||||
(AEC + NS + AGC + VAD/PTT gate) → Opus encode → frame header → AEAD → UDP send
|
||||
|
||||
UDP recv → AEAD open → parse header → jitter(ssrc) → Opus decode
|
||||
→ per-stream recv-side NS (optional, per user) → per-stream gain/mute
|
||||
→ mixer (sum all ssrc) → (miniaudio playback, 48k) → device
|
||||
```
|
||||
|
||||
- Capture and playback run on miniaudio's real-time callbacks (WASAPI / CoreAudio / ALSA).
|
||||
- **DSP engine: webrtc-audio-processing (APM)** — the "better one". It provides
|
||||
high-quality **AEC** (acoustic echo cancellation, essential for speaker users), **noise
|
||||
suppression**, **AGC**, and a **VAD** in one tuned module, BSD-licensed. speexdsp is kept
|
||||
only for resampling and as a lightweight jitter-buffer reference. AEC is **in from the
|
||||
start**, not deferred.
|
||||
- The mixer sums decoded streams; clipping is handled by soft limiting on the master bus.
|
||||
|
||||
## 10. Noise reduction — two-sided
|
||||
|
||||
Noise reduction can be applied **at the sender, at the listener, or both** — they are
|
||||
independent.
|
||||
|
||||
- **Sender-side** (the talker's choice): the publishing client runs APM noise suppression on
|
||||
its mic before encoding, controlled by that user's own settings. This cleans the signal for
|
||||
*everyone* and saves bitrate.
|
||||
- **Listener-side, per user** (the listener's choice): on the receive path, *after* decoding
|
||||
each stream and *before* mixing, the listener can enable an **additional** NS pass on a
|
||||
**specific** sender's stream. So even if Alex chose not to denoise his mic, Sam can locally
|
||||
suppress Alex's background noise without affecting how anyone else hears Alex.
|
||||
|
||||
Implementation: a per-`ssrc` APM NS instance on the receive path, instantiated lazily only
|
||||
for streams the listener has flagged. State lives entirely on the listener's machine; toggling
|
||||
it is a local UI action with **no protocol message** and no effect on other listeners. Because
|
||||
each receive stream is decoded independently before the mixer (voice.md §1), per-user receive
|
||||
NS is a clean drop-in on that per-stream stage.
|
||||
|
||||
## 11. Input activation — VAD and PTT (client-configurable)
|
||||
|
||||
Whether the mic transmits is decided locally by the **input gate**, and the client supports
|
||||
**both** modes, switchable per client (and ideally per input device):
|
||||
|
||||
- **Voice activation (VAD):** the APM VAD opens the gate when speech is detected, with a
|
||||
configurable threshold and hang-time to avoid clipping word tails. DTX naturally
|
||||
complements this — when the gate is closed nothing (or only comfort noise) is sent.
|
||||
- **Push-to-talk (PTT):** a held key/button opens the gate. The UI exposes a configurable
|
||||
keybind; the core just receives gate open/close.
|
||||
|
||||
This is purely a send-side, client-local concern — it gates what gets encoded and sent. It
|
||||
needs **no protocol support**; remote talk indicators are still derived from the presence of
|
||||
received frames (§7), so they work identically under VAD or PTT.
|
||||
|
||||
## 9. System / screen audio capture (`SCREEN_AUDIO`)
|
||||
|
||||
"Listen together" needs to capture the audio another app is playing. The capture mechanism
|
||||
differs per OS, but it always feeds the **same** Opus-encode → media-AEAD → UDP path as a
|
||||
normal stream; only the *source* is platform-specific.
|
||||
|
||||
| Platform | Mechanism | Notes |
|
||||
|----------|-----------|-------|
|
||||
| **Windows** | **WASAPI loopback** capture of the default render endpoint (via miniaudio's loopback mode) | Cleanest case; no extra process. Can capture system mix or a specific endpoint. |
|
||||
| **macOS** | **ScreenCaptureKit** system-audio capture (macOS 13+), or a virtual audio device fallback on older OSes | OS requires screen-recording permission; capture happens in the main app. |
|
||||
| **iOS** | **ReplayKit Broadcast Upload Extension** (the Discord mechanism) | See below — separate process, App Group, ~50 MB cap (fine for audio-only). |
|
||||
|
||||
### iOS detail
|
||||
|
||||
- The user starts a broadcast from Control Center's screen-record button; we surface it via
|
||||
`RPSystemBroadcastPickerView` from inside the app for one-tap start.
|
||||
- The **Broadcast Upload Extension** receives `RPSampleBufferType.audioApp` (system/app
|
||||
audio) and `.audioMic`. We consume **`.audioApp`** for `SCREEN_AUDIO` and drop the video
|
||||
buffers entirely — video is what blows the **~50 MB** extension memory budget, so an
|
||||
audio-only consumer stays comfortably inside it.
|
||||
- The extension is a *separate process*. It links a **minimal slice of the core** (Opus
|
||||
encode + media send only — not the full client), reads the active session token and
|
||||
server endpoint from a shared **App Group** container that the host app wrote at join
|
||||
time, derives its own media keys, and publishes the `SCREEN_AUDIO` stream directly. The
|
||||
host app announces the stream over its control channel (`StreamAnnounce`) so the server and
|
||||
peers learn about it.
|
||||
- Mic + voice continue to run in the **host app**; only the system-audio share lives in the
|
||||
extension. When the broadcast stops (`broadcastFinished`), the extension sends a final
|
||||
frame with the `last-frame-before-stop` flag and the host app emits `StreamStop`.
|
||||
Reference in New Issue
Block a user