voice-cat/docs/architecture.md

# Architecture

## 1. The shared-core model

All non-UI logic lives in one C++ library, **`libvoicecat`**. The same library is linked
into every client and into the server. Platform UIs are thin and call the core through a
stable **C ABI** (`voicecat.h`).

```
                         ┌───────────────────────────────────────────┐
   macOS / iOS (Swift)   │                                           │   Windows (C#)
   ┌──────────────────┐  │            libvoicecat (C++)              │  ┌──────────────────┐
   │ SwiftUI views    │  │  ┌─────────────────────────────────────┐ │  │ WinForms (.NET 10│
   │ AVAudioSession   │──┼─▶│ C ABI  (voicecat.h)                 │◀─┼──│ LibraryImport    │
   │ Swift↔C++ interop│  │  ├─────────────────────────────────────┤ │  │ P/Invoke         │
   └──────────────────┘  │  │ Session / Protocol state machine    │ │  └──────────────────┘
                         │  │ Text + voice signaling              │ │
   Linux/macOS/Windows   │  │ Audio engine: capture→encode→send,  │ │
   server                │  │   recv→jitter→decode→mix→playback   │ │
   ┌──────────────────┐  │  │ Codec layer (Opus 1.6)              │ │
   │ voicecat-server  │──┼─▶│ Crypto + transport (TLS 1.3 + AEAD) │ │
   │ (reuses core)    │  │  │ Net I/O (Asio: TCP + UDP + timers)  │ │
   └──────────────────┘  │  └─────────────────────────────────────┘ │
                         └───────────────────────────────────────────┘
```

Why this shape:

- **Swift** (5.9+) can import C++ directly, but we still ship a C ABI because it is the
  lowest-friction, most stable boundary and it is what **C#** needs (`LibraryImport`/
  P/Invoke). One ABI serves both.
- The **server** is not a separate codebase. It links the same protocol, crypto, and Opus
  code as the client, so framing/encryption can never drift between the two ends.

## 2. Layered design inside the core

From the OS up:

| Layer | Responsibility | Key deps |
|-------|----------------|----------|
| **Platform I/O** | Sockets, timers; audio device capture/playback | Asio, miniaudio |
| **Transport** | TLS 1.3 (TCP), exported-key ChaCha20-Poly1305 AEAD (UDP), framing, anti-replay | mbedTLS, libsodium |
| **Codec & DSP** | Opus encode/decode; APM (AEC/NS/AGC/VAD) send-side + per-user NR receive-side; resample; jitter buffer; mixer | libopus, webrtc-audio-processing, speexdsp |
| **Protocol** | Message (de)serialization, request/response correlation, state machine | protobuf |
| **Session/domain** | Channels, users, streams, permissions, text routing | — |
| **C ABI façade** | Handle-based API + event callbacks exposed to UIs | — |

A UI never sees a socket, an Opus packet, or a protobuf message. It sees: "connect",
"join channel", "start a stream from this device", "send this text", and a stream of
events ("user joined", "user is talking", "message received", "level meter = 0.4").

## 3. Threading model

Three classes of thread, with strict rules.

```
 ┌──────────────┐   lock-free    ┌──────────────┐   lock-free    ┌──────────────┐
 │ Audio capture│ ──ring buffer─▶│  Net thread  │ ──ring buffer─▶│Audio playback│
 │ (RT, miniaudio│                │  (Asio loop) │                │ (RT, miniaudio│
 │  callback)   │◀───ring buffer─│              │◀───ring buffer─│  callback)   │
 │ capture→Opus │                │ TLS + AEAD,  │                │ jitter→Opus  │
 │  encode      │                │ route, relay │                │ decode→mix   │
 └──────────────┘                └──────────────┘                └──────────────┘
                                        │
                                  ┌─────▼──────┐
                                  │ Worker pool│  DB, Argon2id, file I/O,
                                  │ (blocking) │  TLS handshakes, codec setup
                                  └────────────┘
```

Rules:

- **Audio (real-time) threads** are driven by the OS audio callback. They must not
  allocate, lock, log, or do syscalls beyond the ring-buffer hand-off. Opus encode/decode
  runs here (it is allocation-free after init).
- **Net thread(s)** run the Asio event loop: TLS records, AEAD seal/open, protobuf parse, channel
  routing, jitter-buffer feed. On the server, this is where the SFU relay copies packets
  to subscribers.
- **Worker pool** absorbs anything that can block: SQLite, Argon2id verification, DNS,
  TLS handshake CPU, codec (re)configuration.
- **Communication** between audio and net is single-producer/single-consumer lock-free
  ring buffers (one per direction per stream). Control-plane events to the UI go through a
  thread-safe queue drained on the UI's terms.

## 4. The C ABI (`voicecat.h`) — shape

Handle-based, opaque pointers, C-linkage. Illustrative (final names in implementation):

```c
typedef struct vc_client vc_client;

typedef struct {
    void (*on_event)(void* user, const vc_event* ev);   // state changes, messages
    void (*on_level)(void* user, uint32_t stream_id, float rms);  // meters (throttled)
    void* user;
} vc_callbacks;

vc_client*  vc_client_create(const vc_config* cfg, vc_callbacks cb);
void        vc_client_destroy(vc_client*);

int  vc_connect(vc_client*, const char* host, uint16_t port);     // async; result via event
int  vc_authenticate_guest(vc_client*, const char* nickname);
int  vc_authenticate_user(vc_client*, const char* user, const char* password);

int  vc_join_channel(vc_client*, uint32_t channel_id, const char* password /*nullable*/);
int  vc_leave_channel(vc_client*);

// Streams (mic / screen audio / aux device)
int  vc_stream_start(vc_client*, const vc_stream_desc* desc, uint32_t* out_stream_id);
int  vc_stream_stop(vc_client*, uint32_t stream_id);
int  vc_set_input_device(vc_client*, uint32_t stream_id, const char* device_id);
int  vc_set_self_mute(vc_client*, bool mic_muted, bool deafened);

// Text
int  vc_send_text(vc_client*, vc_text_scope scope, uint32_t target_id, const char* utf8);

// Enumeration helpers for UI device pickers
int  vc_list_devices(vc_client*, vc_device_kind kind, vc_device_list* out);
```

Design notes:

- **Async, event-driven.** Calls return immediately; results and state changes arrive via
  `on_event`. This maps cleanly onto SwiftUI/`async` and C# `event`/`Task` patterns.
- **The core owns audio.** Capture, encode, decode, mixing, and playback happen inside the
  core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
  meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
  exception that needs UI-side cooperation — see below.)
- **Device enumeration works pre-connect.** `vc_list_devices` needs no live session — device
  pickers can populate before `vc_connect`. `vc_device.id` is an opaque, internally-encoded
  handle (currently a hex-encoded `ma_device_id`) — always round-trip an id that came from
  `vc_list_devices`/`vc_get_stream_audio_config`; never construct one by hand. Tolerate an
  empty list (a machine can legitimately have zero input or output devices).
- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
  caller-allocated or returned with a paired `vc_free`.

### Per-platform binding notes

- **Swift / Apple.** Import the C ABI via a **module map** (`module VoiceCatC { header "voicecat.h" }`) staged into the XCFramework headers by `clients/apple/scripts/build-xcframework.sh` — Swift gets a clean `import VoiceCatC` with all C enums/structs/functions available directly (no manual redeclaration, unlike the C# P/Invoke layer). A **Swift wrapper** (`VoiceCatCore` package at `clients/apple/`) provides Swift-idiomatic types (`VoiceCatResult`, `VoiceCatEvent`, `Channel`, `User`, etc.) on top, mirroring the C# `VoiceCat.Interop` layer. Callbacks use `@convention(c)` closures (plain C function pointers, not ARC-managed closures) + `Unmanaged.passUnretained(self)` as the `user` context (the Swift analog of C#'s `[UnmanagedCallersOnly]` + `GCHandle`). Events are delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (one async block scheduled at a time) — the Swift analog of C#'s `Channel<VoiceCatEvent>` + 30ms WinForms Timer pump. `deinit` calls `vc_client_destroy` (joins all threads) then frees native CString config storage (the core stores raw pointers, doesn't copy). **macOS UI: AppKit** (chosen over SwiftUI for the most mature VoiceOver accessibility story — same rationale as the Windows client's WinForms choice); **iOS UI: SwiftUI** (narrower control surface, sufficient VoiceOver support). On **iOS** the app owns `AVAudioSession` (category `.playAndRecord`), requests mic permission, and handles interruptions/route changes — the core exposes hooks (`vc_audio_suspend`/`vc_audio_resume`/`vc_audio_restart`, implemented) the Swift layer calls from `AVAudioSession` notifications and `IOSAudioRouter` setting changes. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) is driven from the Swift `IOSAudioRouter` singleton via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The core is told the capture channel count via `vc_set_capture_channels` (append-only ABI). `vc_audio_restart` does a full stop + re-init (unlike `suspend`/`resume` which only stop/start) so devices reopen against a new route after `AVAudioSession` reconfiguration. iOS 18.0 deployment target. Background voice and VoIP push (CallKit/PushKit) are a later milestone. The XCFramework carries a **fat static library** (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps so the Swift Package links a single self-contained `.a` per slice.
- **iOS screen / system-audio sharing** is supported via a **ReplayKit Broadcast Upload
  Extension** (the same mechanism Discord uses; triggered from Control Center's screen-record
  button via `RPSystemBroadcastPickerView`). The extension receives
  `RPSampleBufferType.audioApp` (system/app audio) and `.audioMic`. We capture **`.audioApp`**
  for the `SCREEN_AUDIO` "listen together" stream and ignore video. The extension runs in a
  **separate process with a ~50 MB memory cap** — that cap is a problem only for video
  frames, so audio-only stays well within budget. It links a *minimal* slice of the core
  (Opus encode + media send), shares the session/credentials with the host app through an
  **App Group**, and re-derives its own media keys. This is detailed in [voice.md](voice.md) §9.
- **C# / Windows.** `[LibraryImport]` (source-generated P/Invoke, .NET 7+) over the C ABI.
  `[UnmanagedCallersOnly]` static methods for `on_event`/`on_level` to avoid delegate-lifetime
  pitfalls. UI in **WinForms (.NET 10)** — chosen over WinUI 3/Avalonia for its mature,
  predictable screen-reader (NVDA/JAWS/Narrator) UIA support (see roadmap.md §2).
  Events are delivered via `System.Threading.Channels.Channel<VoiceCatEvent>`, drained by a
  30ms `System.Windows.Forms.Timer` on the UI thread — simpler than a message-only HWND +
  `PostMessage` with no meaningful latency cost. `VoiceCatClientHandle : SafeHandle` wraps
  the `vc_client*` and guarantees `vc_client_destroy` runs on GC/Dispose.

### External PCM feed/tap

Two API functions let callers bypass miniaudio entirely for a stream:

| Function | Direction | Contract |
|----------|-----------|----------|
| `vc_stream_feed_pcm(c, stream_id, pcm, samples_per_channel, channels)` | **Send** — caller → network | Caller supplies interleaved int16 at the stream's sample rate (`channels` = 1 mono, 2 stereo). The core frames, Opus-encodes, AEAD-seals, and sends over UDP — identical wire path to hardware capture. The stream must already be started with `vc_stream_start`. Thread-safe; may be called from any thread (audio callback, ReplayKit delegate, SCStream callback). |
| `vc_set_pcm_sink(c, cb, user)` | **Receive** — network → caller | `cb` is called on the audio (playback) thread once per decoded Opus frame per remote stream, with `(user_id, stream_id, pcm, samples_per_channel, channels, sample_rate)`. PCM is delivered to the sink **and** the hardware device — dual output; the hardware mix is unaffected. Pass `cb=NULL` to disable (default). **Must not block** — copy what you need and return. |

`vc_test_inject_capture` (the old TEST-ONLY mono-only predecessor) is a deprecated alias
for `vc_stream_feed_pcm(..., channels=1)` — kept for source compatibility.

**Use cases:** ReplayKit Broadcast Extension (iOS `SCREEN_AUDIO`), ScreenCaptureKit (macOS
`SCREEN_AUDIO`), music/TTS/relay bots, soundboards, transcription clients. The extension or
bot links Opus + the feed entry point — no `ma_device`, no hardware, headless.

**Threading:** the feed path is thread-safe (ring buffer, no lock on the RT path). The sink
callback runs on the miniaudio playback thread — observe the same rules as the capture
callback: no allocations, no blocking calls.

## 5. Server architecture

`voicecat-server` is a headless process linking the core.

```
        TCP/TLS 1.3                       UDP + media AEAD
            │                                   │
   ┌────────▼─────────┐               ┌─────────▼──────────┐
   │ Connection mgr   │               │ UDP demux          │
   │ (accept, TLS,    │               │ 5-tuple → session  │
   │  per-conn state) │               │ anti-replay window │
   └────────┬─────────┘               └─────────┬──────────┘
            │                                   │
   ┌────────▼───────────────────────────────────▼──────────┐
   │ Session registry  (session_id ↔ TCP conn ↔ UDP tuple)  │
   └────────┬───────────────────────────────┬───────────────┘
            │                                │
   ┌────────▼─────────┐   ┌──────────────┐  ┌▼─────────────────┐
   │ Channel manager  │   │ Text router  │  │ Voice router/SFU │
   │ tree, configs,   │   │ channel + PM │  │ relay Opus to    │
   │ membership, perms│   │              │  │ channel members  │
   └────────┬─────────┘   └──────────────┘  └──────────────────┘
            │
   ┌────────▼─────────┐
   │ Persistence      │  accounts (Argon2id), channels, bans, config
   │ SQLite           │
   └──────────────────┘
```

- **Voice router is a relay, not a mixer.** For each incoming voice frame it looks up the
  sender's channel and forwards the *unmodified Opus payload* (restamped with the sender's
  user id) to every other subscribed member. No server-side decode/transcode → low CPU,
  low latency, and end-to-content is just Opus. Per-channel Opus params are enforced so all
  members are mutually decodable.
- **Subscriptions.** Clients implicitly subscribe to their current channel's voice; text
  and presence can be subscribed more broadly. This keeps fan-out bounded on big servers.
- **Stateless-ish media.** UDP carries no auth per packet beyond the media-AEAD session;
  the 5-tuple→session binding is established once via a token (see protocol.md §4).
- **Keepalive reaper.** An `asio::steady_timer` sweeps every 15 s and drops any session
  whose `last_seen` (bumped on every inbound TCP or UDP frame) is older than 45 s. Each
  drop broadcasts `UserEvent::LEFT` so peers clean up immediately. This catches half-open
  connections that never produce a TCP EOF. Configurable via `server::Config`.
- **Single process, scalable later.** v1 is one process, one machine. The session registry
  and router are written behind interfaces so a future build can sit them behind a shared
  bus for multi-node, but that is explicitly out of scope for now.

## 6. Repository layout (proposed)

```
voice-cat/
├── docs/                  # this folder
├── core/                  # libvoicecat (C++)
│   ├── include/voicecat.h # the C ABI
│   ├── src/{net,crypto,codec,protocol,session,audio}/
│   └── proto/             # .proto definitions (shared source of truth)
├── server/                # voicecat-server (C++, links core)
├── clients/
│   ├── apple/             # Swift package + Xcode project (macOS + iOS)
│   └── windows/           # .NET solution (C#)
├── tools/
│   └── vccli/             # headless test client (C++), for protocol bring-up
├── third_party/           # vendored / vcpkg manifest
└── CMakeLists.txt
```

Build is **CMake** with **vcpkg** (manifest mode) for C/C++ deps; the Apple and Windows UI
projects consume the built core as a binary + headers. See [tech-stack.md](tech-stack.md).