voice-cat/docs/architecture.md

# Architecture

## 1. The shared-core model

All non-UI logic lives in one C++ library, **`libvoicecat`**. The same library is linked
into every client and into the server. Platform UIs are thin and call the core through a
stable **C ABI** (`voicecat.h`).

```
                         ┌───────────────────────────────────────────┐
   macOS / iOS (Swift)   │                                           │   Windows (C#)
   ┌──────────────────┐  │            libvoicecat (C++)              │  ┌──────────────────┐
   │ SwiftUI views    │  │  ┌─────────────────────────────────────┐ │  │ WinUI/Avalonia   │
   │ AVAudioSession   │──┼─▶│ C ABI  (voicecat.h)                 │◀─┼──│ LibraryImport    │
   │ Swift↔C++ interop│  │  ├─────────────────────────────────────┤ │  │ P/Invoke         │
   └──────────────────┘  │  │ Session / Protocol state machine    │ │  └──────────────────┘
                         │  │ Text + voice signaling              │ │
   Linux/macOS/Windows   │  │ Audio engine: capture→encode→send,  │ │
   server                │  │   recv→jitter→decode→mix→playback   │ │
   ┌──────────────────┐  │  │ Codec layer (Opus 1.6)              │ │
   │ voicecat-server  │──┼─▶│ Crypto + transport (TLS 1.3 + AEAD) │ │
   │ (reuses core)    │  │  │ Net I/O (Asio: TCP + UDP + timers)  │ │
   └──────────────────┘  │  └─────────────────────────────────────┘ │
                         └───────────────────────────────────────────┘
```

Why this shape:

- **Swift** (5.9+) can import C++ directly, but we still ship a C ABI because it is the
  lowest-friction, most stable boundary and it is what **C#** needs (`LibraryImport`/
  P/Invoke). One ABI serves both.
- The **server** is not a separate codebase. It links the same protocol, crypto, and Opus
  code as the client, so framing/encryption can never drift between the two ends.

## 2. Layered design inside the core

From the OS up:

| Layer | Responsibility | Key deps |
|-------|----------------|----------|
| **Platform I/O** | Sockets, timers; audio device capture/playback | Asio, miniaudio |
| **Transport** | TLS 1.3 (TCP), exported-key ChaCha20-Poly1305 AEAD (UDP), framing, anti-replay | mbedTLS, libsodium |
| **Codec & DSP** | Opus encode/decode; APM (AEC/NS/AGC/VAD) send-side + per-user NR receive-side; resample; jitter buffer; mixer | libopus, webrtc-audio-processing, speexdsp |
| **Protocol** | Message (de)serialization, request/response correlation, state machine | protobuf |
| **Session/domain** | Channels, users, streams, permissions, text routing | — |
| **C ABI façade** | Handle-based API + event callbacks exposed to UIs | — |

A UI never sees a socket, an Opus packet, or a protobuf message. It sees: "connect",
"join channel", "start a stream from this device", "send this text", and a stream of
events ("user joined", "user is talking", "message received", "level meter = 0.4").

## 3. Threading model

Three classes of thread, with strict rules.

```
 ┌──────────────┐   lock-free    ┌──────────────┐   lock-free    ┌──────────────┐
 │ Audio capture│ ──ring buffer─▶│  Net thread  │ ──ring buffer─▶│Audio playback│
 │ (RT, miniaudio│                │  (Asio loop) │                │ (RT, miniaudio│
 │  callback)   │◀───ring buffer─│              │◀───ring buffer─│  callback)   │
 │ capture→Opus │                │ TLS + AEAD,  │                │ jitter→Opus  │
 │  encode      │                │ route, relay │                │ decode→mix   │
 └──────────────┘                └──────────────┘                └──────────────┘
                                        │
                                  ┌─────▼──────┐
                                  │ Worker pool│  DB, Argon2id, file I/O,
                                  │ (blocking) │  TLS handshakes, codec setup
                                  └────────────┘
```

Rules:

- **Audio (real-time) threads** are driven by the OS audio callback. They must not
  allocate, lock, log, or do syscalls beyond the ring-buffer hand-off. Opus encode/decode
  runs here (it is allocation-free after init).
- **Net thread(s)** run the Asio event loop: TLS records, AEAD seal/open, protobuf parse, channel
  routing, jitter-buffer feed. On the server, this is where the SFU relay copies packets
  to subscribers.
- **Worker pool** absorbs anything that can block: SQLite, Argon2id verification, DNS,
  TLS handshake CPU, codec (re)configuration.
- **Communication** between audio and net is single-producer/single-consumer lock-free
  ring buffers (one per direction per stream). Control-plane events to the UI go through a
  thread-safe queue drained on the UI's terms.

## 4. The C ABI (`voicecat.h`) — shape

Handle-based, opaque pointers, C-linkage. Illustrative (final names in implementation):

```c
typedef struct vc_client vc_client;

typedef struct {
    void (*on_event)(void* user, const vc_event* ev);   // state changes, messages
    void (*on_level)(void* user, uint32_t stream_id, float rms);  // meters (throttled)
    void* user;
} vc_callbacks;

vc_client*  vc_client_create(const vc_config* cfg, vc_callbacks cb);
void        vc_client_destroy(vc_client*);

int  vc_connect(vc_client*, const char* host, uint16_t port);     // async; result via event
int  vc_authenticate_guest(vc_client*, const char* nickname);
int  vc_authenticate_user(vc_client*, const char* user, const char* password);

int  vc_join_channel(vc_client*, uint32_t channel_id, const char* password /*nullable*/);
int  vc_leave_channel(vc_client*);

// Streams (mic / screen audio / aux device)
int  vc_stream_start(vc_client*, const vc_stream_desc* desc, uint32_t* out_stream_id);
int  vc_stream_stop(vc_client*, uint32_t stream_id);
int  vc_set_input_device(vc_client*, uint32_t stream_id, const char* device_id);
int  vc_set_self_mute(vc_client*, bool mic_muted, bool deafened);

// Text
int  vc_send_text(vc_client*, vc_text_scope scope, uint32_t target_id, const char* utf8);

// Enumeration helpers for UI device pickers
int  vc_list_devices(vc_client*, vc_device_kind kind, vc_device_list* out);
```

Design notes:

- **Async, event-driven.** Calls return immediately; results and state changes arrive via
  `on_event`. This maps cleanly onto SwiftUI/`async` and C# `event`/`Task` patterns.
- **The core owns audio.** Capture, encode, decode, mixing, and playback happen inside the
  core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
  meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
  exception that needs UI-side cooperation — see below.)
- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
  caller-allocated or returned with a paired `vc_free`.

### Per-platform binding notes

- **Swift / Apple.** Import the C ABI via a module map; SwiftUI on top. On **iOS** the app
  must still own `AVAudioSession` (category `.playAndRecord`, `.voiceChat` mode), request
  mic permission, and handle interruptions/route changes — the core exposes hooks
  (`vc_audio_suspend`/`vc_audio_resume`) the Swift layer calls from `AVAudioSession`
  notifications. Background voice and VoIP push (CallKit/PushKit) are a later milestone.
- **iOS screen / system-audio sharing** is supported via a **ReplayKit Broadcast Upload
  Extension** (the same mechanism Discord uses; triggered from Control Center's screen-record
  button via `RPSystemBroadcastPickerView`). The extension receives
  `RPSampleBufferType.audioApp` (system/app audio) and `.audioMic`. We capture **`.audioApp`**
  for the `SCREEN_AUDIO` "listen together" stream and ignore video. The extension runs in a
  **separate process with a ~50 MB memory cap** — that cap is a problem only for video
  frames, so audio-only stays well within budget. It links a *minimal* slice of the core
  (Opus encode + media send), shares the session/credentials with the host app through an
  **App Group**, and re-derives its own media keys. This is detailed in [voice.md](voice.md) §9.
- **C# / Windows.** `LibraryImport` (source-generated P/Invoke, .NET 7+) over the C ABI.
  Marshal the `on_event` callback as a `[UnmanagedCallersOnly]`/function-pointer to avoid
  delegate lifetime pitfalls. UI in **WinUI 3** (most native) or **Avalonia** (if we later
  want a single C# UI across desktop OSes).

## 5. Server architecture

`voicecat-server` is a headless process linking the core.

```
        TCP/TLS 1.3                       UDP + media AEAD
            │                                   │
   ┌────────▼─────────┐               ┌─────────▼──────────┐
   │ Connection mgr   │               │ UDP demux          │
   │ (accept, TLS,    │               │ 5-tuple → session  │
   │  per-conn state) │               │ anti-replay window │
   └────────┬─────────┘               └─────────┬──────────┘
            │                                   │
   ┌────────▼───────────────────────────────────▼──────────┐
   │ Session registry  (session_id ↔ TCP conn ↔ UDP tuple)  │
   └────────┬───────────────────────────────┬───────────────┘
            │                                │
   ┌────────▼─────────┐   ┌──────────────┐  ┌▼─────────────────┐
   │ Channel manager  │   │ Text router  │  │ Voice router/SFU │
   │ tree, configs,   │   │ channel + PM │  │ relay Opus to    │
   │ membership, perms│   │              │  │ channel members  │
   └────────┬─────────┘   └──────────────┘  └──────────────────┘
            │
   ┌────────▼─────────┐
   │ Persistence      │  accounts (Argon2id), channels, bans, config
   │ SQLite           │
   └──────────────────┘
```

- **Voice router is a relay, not a mixer.** For each incoming voice frame it looks up the
  sender's channel and forwards the *unmodified Opus payload* (restamped with the sender's
  user id) to every other subscribed member. No server-side decode/transcode → low CPU,
  low latency, and end-to-content is just Opus. Per-channel Opus params are enforced so all
  members are mutually decodable.
- **Subscriptions.** Clients implicitly subscribe to their current channel's voice; text
  and presence can be subscribed more broadly. This keeps fan-out bounded on big servers.
- **Stateless-ish media.** UDP carries no auth per packet beyond the media-AEAD session;
  the 5-tuple→session binding is established once via a token (see protocol.md §4).
- **Single process, scalable later.** v1 is one process, one machine. The session registry
  and router are written behind interfaces so a future build can sit them behind a shared
  bus for multi-node, but that is explicitly out of scope for now.

## 6. Repository layout (proposed)

```
voice-cat/
├── docs/                  # this folder
├── core/                  # libvoicecat (C++)
│   ├── include/voicecat.h # the C ABI
│   ├── src/{net,crypto,codec,protocol,session,audio}/
│   └── proto/             # .proto definitions (shared source of truth)
├── server/                # voicecat-server (C++, links core)
├── clients/
│   ├── apple/             # Swift package + Xcode project (macOS + iOS)
│   └── windows/           # .NET solution (C#)
├── tools/
│   └── vccli/             # headless test client (C++), for protocol bring-up
├── third_party/           # vendored / vcpkg manifest
└── CMakeLists.txt
```

Build is **CMake** with **vcpkg** (manifest mode) for C/C++ deps; the Apple and Windows UI
projects consume the built core as a binary + headers. See [tech-stack.md](tech-stack.md).