Retire legacy sources and verify managed iOS deployment
This commit is contained in:
+57
-232
@@ -1,251 +1,76 @@
|
||||
# Architecture
|
||||
|
||||
The parallel .NET rewrite under `dotnet/` currently implements shared protocol framing,
|
||||
voice headers, and media crypto. Existing server/client/audio behavior remains in C++.
|
||||
See `docs/api-dotnet.md` for the initial managed contract and
|
||||
`docs/porting-to-dotnet.md` for subsequent migration phases.
|
||||
VoiceCat is a .NET 10 client/server application with a deliberately narrow native media
|
||||
boundary. The managed implementation is authoritative.
|
||||
|
||||
## 1. The shared-core model
|
||||
## Runtime components
|
||||
|
||||
All non-UI logic lives in one C++ library, **`libvoicecat`**. The same library is linked
|
||||
into every client and into the server. Platform UIs are thin and call the core through a
|
||||
stable **C ABI** (`voicecat.h`).
|
||||
```text
|
||||
Windows WinForms ─┐
|
||||
macOS AppKit ─────┼─ VoiceCat.Core ─ VoiceCat.Audio ─ native media shim
|
||||
iOS UIKit ────────┘ │
|
||||
├─ VoiceCat.Protocol
|
||||
└─ VoiceCat.Crypto
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────┐
|
||||
macOS / iOS (Swift) │ │ Windows (C#)
|
||||
┌──────────────────┐ │ libvoicecat (C++) │ ┌──────────────────┐
|
||||
│ SwiftUI views │ │ ┌─────────────────────────────────────┐ │ │ WinForms (.NET 10│
|
||||
│ AVAudioSession │──┼─▶│ C ABI (voicecat.h) │◀─┼──│ LibraryImport │
|
||||
│ Swift↔C++ interop│ │ ├─────────────────────────────────────┤ │ │ P/Invoke │
|
||||
└──────────────────┘ │ │ Session / Protocol state machine │ │ └──────────────────┘
|
||||
│ │ Text + voice signaling │ │
|
||||
Linux/macOS/Windows │ │ Audio engine: capture→encode→send, │ │
|
||||
server │ │ recv→jitter→decode→mix→playback │ │
|
||||
┌──────────────────┐ │ │ Codec layer (Opus 1.6) │ │
|
||||
│ voicecat-server │──┼─▶│ Crypto + transport (TLS 1.3 + AEAD) │ │
|
||||
│ (reuses core) │ │ │ Net I/O (Asio: TCP + UDP + timers) │ │
|
||||
└──────────────────┘ │ └─────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────┘
|
||||
VoiceCat.Server ──────────── VoiceCat.Protocol + VoiceCat.Crypto + SQLite
|
||||
```
|
||||
|
||||
Why this shape:
|
||||
The server terminates TLS control sessions and relays authenticated encrypted Opus packets.
|
||||
It does not decode, mix, or transcode media. Clients own capture, encoding, jitter/loss
|
||||
recovery, decoding, mixing, and playback.
|
||||
|
||||
- **Swift** (5.9+) can import C++ directly, but we still ship a C ABI because it is the
|
||||
lowest-friction, most stable boundary and it is what **C#** needs (`LibraryImport`/
|
||||
P/Invoke). One ABI serves both.
|
||||
- The **server** is not a separate codebase. It links the same protocol, crypto, and Opus
|
||||
code as the client, so framing/encryption can never drift between the two ends.
|
||||
## Managed projects
|
||||
|
||||
## 2. Layered design inside the core
|
||||
- `VoiceCat.Protocol`: generated protobuf types and bounded length-prefixed framing.
|
||||
- `VoiceCat.Crypto`: BouncyCastle TLS 1.3, exporter-derived media secrets, certificate TOFU,
|
||||
Ed25519 server identity, ChaCha20-Poly1305, replay windows, and Argon2id.
|
||||
- `VoiceCat.Codec`: safe ownership around fixed Opus shim handles.
|
||||
- `VoiceCat.Dsp`: RNNoise and energy-VAD ownership.
|
||||
- `VoiceCat.Audio`: streams, packet-loss recovery, jitter buffers, mixing, PCM rings, and
|
||||
activation policy.
|
||||
- `VoiceCat.Core`: connection lifecycle, authentication, state snapshots/events, requests,
|
||||
stream negotiation, encrypted UDP, reconnect, and administration helpers.
|
||||
- `VoiceCat.Server`: listener/session ownership, SQLite state, permissions, moderation,
|
||||
channel management, encrypted UDP routing, and process administration commands.
|
||||
- `VoiceCat.Cli`: interactive client and deterministic text/voice behavior driver.
|
||||
|
||||
From the OS up:
|
||||
Leaf UI projects reference the managed core; platform audio objects translate between native
|
||||
device buffers and bounded PCM rings owned by `VoiceCat.Audio`.
|
||||
|
||||
| Layer | Responsibility | Key deps |
|
||||
|-------|----------------|----------|
|
||||
| **Platform I/O** | Sockets, timers; audio device capture/playback | Asio, miniaudio |
|
||||
| **Transport** | TLS 1.3 (TCP), exported-key ChaCha20-Poly1305 AEAD (UDP), framing, anti-replay | mbedTLS, libsodium |
|
||||
| **Codec & DSP** | Opus encode/decode; APM (AEC/NS/AGC/VAD) send-side + per-user NR receive-side; resample; jitter buffer; mixer | libopus, webrtc-audio-processing, speexdsp |
|
||||
| **Protocol** | Message (de)serialization, request/response correlation, state machine | protobuf |
|
||||
| **Session/domain** | Channels, users, streams, permissions, text routing | — |
|
||||
| **C ABI façade** | Handle-based API + event callbacks exposed to UIs | — |
|
||||
## Native boundary
|
||||
|
||||
A UI never sees a socket, an Opus packet, or a protobuf message. It sees: "connect",
|
||||
"join channel", "start a stream from this device", "send this text", and a stream of
|
||||
events ("user joined", "user is talking", "message received", "level meter = 0.4").
|
||||
`native/media` builds `voicecat_media` for desktop and static Apple targets. It contains only
|
||||
fixed C entry points for Opus/DRED and RNNoise. Vendored RNNoise is under `native/rnnoise`.
|
||||
Networking, TLS, media encryption, session state, jitter, and mixing do not live in native code.
|
||||
|
||||
## 3. Threading model
|
||||
`native/apple/broadcast` is a Swift ReplayKit upload extension. It captures application audio,
|
||||
converts it to 48 kHz stereo int16 PCM, and writes the frozen App Group ring documented in
|
||||
`broadcast-ring-format.md`. The managed iOS host drains that ring and performs encoding and
|
||||
networking. The extension deliberately has no managed runtime or VoiceCat protocol stack.
|
||||
|
||||
Three classes of thread, with strict rules.
|
||||
The iOS host also has small Objective-C bridges in its managed project for platform APIs that
|
||||
need direct native entry points. These are platform adapters, not a second core.
|
||||
|
||||
```
|
||||
┌──────────────┐ lock-free ┌──────────────┐ lock-free ┌──────────────┐
|
||||
│ Audio capture│ ──ring buffer─▶│ Net thread │ ──ring buffer─▶│Audio playback│
|
||||
│ (RT, miniaudio│ │ (Asio loop) │ │ (RT, miniaudio│
|
||||
│ callback) │◀───ring buffer─│ │◀───ring buffer─│ callback) │
|
||||
│ capture→Opus │ │ TLS + AEAD, │ │ jitter→Opus │
|
||||
│ encode │ │ route, relay │ │ decode→mix │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘
|
||||
│
|
||||
┌─────▼──────┐
|
||||
│ Worker pool│ DB, Argon2id, file I/O,
|
||||
│ (blocking) │ TLS handshakes, codec setup
|
||||
└────────────┘
|
||||
```
|
||||
## Ownership and concurrency
|
||||
|
||||
Rules:
|
||||
- A `VoiceCatClient` owns one TLS control connection, one media session, and its audio engine.
|
||||
- Control/TLS operations have one serialized owner. UI code submits requests and observes
|
||||
events; it does not call TLS concurrently.
|
||||
- The server publishes immutable routing state to its UDP loop. The UDP loop owns endpoint
|
||||
binding, packet authentication, replay checks, and recipient resealing.
|
||||
- Audio callbacks consume or produce preallocated ring-buffer memory. They never allocate,
|
||||
lock, block, log, or access sockets.
|
||||
- Teardown establishes a quiescence barrier before callback-owned state is released.
|
||||
- Blocking file, database, TLS, and device lifecycle work stays off real-time callbacks.
|
||||
|
||||
- **Audio (real-time) threads** are driven by the OS audio callback. They must not
|
||||
allocate, lock, log, or do syscalls beyond the ring-buffer hand-off. Opus encode/decode
|
||||
runs here (it is allocation-free after init).
|
||||
- **Net thread(s)** run the Asio event loop: TLS records, AEAD seal/open, protobuf parse, channel
|
||||
routing, jitter-buffer feed. On the server, this is where the SFU relay copies packets
|
||||
to subscribers.
|
||||
- **Worker pool** absorbs anything that can block: SQLite, Argon2id verification, DNS,
|
||||
TLS handshake CPU, codec (re)configuration.
|
||||
- **Communication** between audio and net is single-producer/single-consumer lock-free
|
||||
ring buffers (one per direction per stream). Control-plane events to the UI go through a
|
||||
thread-safe queue drained on the UI's terms.
|
||||
## Data and contracts
|
||||
|
||||
## 4. The C ABI (`voicecat.h`) — shape
|
||||
- `proto/voicecat.proto` is the control-plane schema.
|
||||
- Media uses the fixed header and AEAD construction described in `protocol.md` and
|
||||
`security.md`.
|
||||
- SQLite is the server's persistent store; schema changes require explicit migrations.
|
||||
- Client profiles and TOFU pins are local platform data.
|
||||
- The ReplayKit ring layout is separately versioned and frozen.
|
||||
|
||||
Handle-based, opaque pointers, C-linkage. Illustrative (final names in implementation):
|
||||
|
||||
```c
|
||||
typedef struct vc_client vc_client;
|
||||
|
||||
typedef struct {
|
||||
void (*on_event)(void* user, const vc_event* ev); // state changes, messages
|
||||
void (*on_level)(void* user, uint32_t stream_id, float rms); // meters (throttled)
|
||||
void* user;
|
||||
} vc_callbacks;
|
||||
|
||||
vc_client* vc_client_create(const vc_config* cfg, vc_callbacks cb);
|
||||
void vc_client_destroy(vc_client*);
|
||||
|
||||
int vc_connect(vc_client*, const char* host, uint16_t port); // async; result via event
|
||||
int vc_authenticate_guest(vc_client*, const char* nickname);
|
||||
int vc_authenticate_user(vc_client*, const char* user, const char* password);
|
||||
|
||||
int vc_join_channel(vc_client*, uint32_t channel_id, const char* password /*nullable*/);
|
||||
int vc_leave_channel(vc_client*);
|
||||
|
||||
// Streams (mic / screen audio / aux device)
|
||||
int vc_stream_start(vc_client*, const vc_stream_desc* desc, uint32_t* out_stream_id);
|
||||
int vc_stream_stop(vc_client*, uint32_t stream_id);
|
||||
int vc_set_input_device(vc_client*, uint32_t stream_id, const char* device_id);
|
||||
int vc_set_self_mute(vc_client*, bool mic_muted, bool deafened);
|
||||
|
||||
// Text
|
||||
int vc_send_text(vc_client*, vc_text_scope scope, uint32_t target_id, const char* utf8);
|
||||
|
||||
// Enumeration helpers for UI device pickers
|
||||
int vc_list_devices(vc_client*, vc_device_kind kind, vc_device_list* out);
|
||||
```
|
||||
|
||||
Design notes:
|
||||
|
||||
- **Async, event-driven.** Calls return immediately; results and state changes arrive via
|
||||
`on_event`. This maps cleanly onto SwiftUI/`async` and C# `event`/`Task` patterns.
|
||||
- **The core owns audio.** Capture, encode, decode, mixing, and playback happen inside the
|
||||
core via miniaudio. The UI only *selects devices*, *starts/stops streams*, and *renders
|
||||
meters/state*. This keeps the real-time path identical on every OS. (iOS is the one
|
||||
exception that needs UI-side cooperation — see below.)
|
||||
- **Device enumeration works pre-connect.** `vc_list_devices` needs no live session — device
|
||||
pickers can populate before `vc_connect`. `vc_device.id` is an opaque, internally-encoded
|
||||
handle (currently a hex-encoded `ma_device_id`) — always round-trip an id that came from
|
||||
`vc_list_devices`/`vc_get_stream_audio_config`; never construct one by hand. Tolerate an
|
||||
empty list (a machine can legitimately have zero input or output devices).
|
||||
- **Strings are UTF-8 `const char*`; ownership is explicit.** Output buffers are
|
||||
caller-allocated or returned with a paired `vc_free`.
|
||||
|
||||
### Per-platform binding notes
|
||||
|
||||
- **Swift / Apple.** Import the C ABI via a **module map** (`module VoiceCatC { header "voicecat.h" }`) staged into the XCFramework headers by `clients/apple/scripts/build-xcframework.sh` — Swift gets a clean `import VoiceCatC` with all C enums/structs/functions available directly (no manual redeclaration, unlike the C# P/Invoke layer). A **Swift wrapper** (`VoiceCatCore` package at `clients/apple/`) provides Swift-idiomatic types (`VoiceCatResult`, `VoiceCatEvent`, `Channel`, `User`, etc.) on top, mirroring the C# `VoiceCat.Interop` layer. Callbacks use `@convention(c)` closures (plain C function pointers, not ARC-managed closures) + `Unmanaged.passUnretained(self)` as the `user` context (the Swift analog of C#'s `[UnmanagedCallersOnly]` + `GCHandle`). Events are delivered on `@MainActor` via a coalesced `DispatchQueue.main` drain (one async block scheduled at a time) — the Swift analog of C#'s `Channel<VoiceCatEvent>` + 30ms WinForms Timer pump. `deinit` calls `vc_client_destroy` (joins all threads) then frees native CString config storage (the core stores raw pointers, doesn't copy). **macOS UI: AppKit** (chosen over SwiftUI for the most mature VoiceOver accessibility story — same rationale as the Windows client's WinForms choice); **iOS UI: SwiftUI** (narrower control surface, sufficient VoiceOver support). On **iOS** the app owns `AVAudioSession` (category `.playAndRecord`), requests mic permission, and handles interruptions/route changes — the core exposes hooks (`vc_audio_suspend`/`vc_audio_resume`/`vc_audio_restart`, implemented) the Swift layer calls from `AVAudioSession` notifications and `IOSAudioRouter` setting changes. All iOS audio routing (input port selection, mic orientation/polar patterns, HFP vs A2DP, measurement/raw mode, stereo capture) is driven from the Swift `IOSAudioRouter` singleton via `AVAudioSession` *before* the core (miniaudio) opens its device — miniaudio does NOT touch `AVAudioSession` on iOS. The core is told the capture channel count via `vc_set_capture_channels` (append-only ABI). `vc_audio_restart` does a full stop + re-init (unlike `suspend`/`resume` which only stop/start) so devices reopen against a new route after `AVAudioSession` reconfiguration. iOS 18.0 deployment target. Background voice and VoIP push (CallKit/PushKit) are a later milestone. The XCFramework carries a **fat static library** (`libvoicecat-fat.a`) bundling `libvoicecat.a` + all vcpkg static deps so the Swift Package links a single self-contained `.a` per slice.
|
||||
- **iOS screen / system-audio sharing** is supported via a **ReplayKit Broadcast Upload
|
||||
Extension** (the same mechanism Discord uses; triggered from Control Center's screen-record
|
||||
button via `RPSystemBroadcastPickerView`). The extension receives
|
||||
`RPSampleBufferType.audioApp` (system/app audio) and `.audioMic`. We capture **`.audioApp`**
|
||||
for the `SCREEN_AUDIO` "listen together" stream and ignore video. The extension runs in a
|
||||
**separate process with a ~50 MB memory cap** — that cap is a problem only for video
|
||||
frames, so audio-only stays well within budget. It links a *minimal* slice of the core
|
||||
(Opus encode + media send), shares the session/credentials with the host app through an
|
||||
**App Group**, and re-derives its own media keys. This is detailed in [voice.md](voice.md) §9.
|
||||
- **C# / Windows.** `[LibraryImport]` (source-generated P/Invoke, .NET 7+) over the C ABI.
|
||||
`[UnmanagedCallersOnly]` static methods for `on_event`/`on_level` to avoid delegate-lifetime
|
||||
pitfalls. UI in **WinForms (.NET 10)** — chosen over WinUI 3/Avalonia for its mature,
|
||||
predictable screen-reader (NVDA/JAWS/Narrator) UIA support (see roadmap.md §2).
|
||||
Events are delivered via `System.Threading.Channels.Channel<VoiceCatEvent>`, drained by a
|
||||
30ms `System.Windows.Forms.Timer` on the UI thread — simpler than a message-only HWND +
|
||||
`PostMessage` with no meaningful latency cost. `VoiceCatClientHandle : SafeHandle` wraps
|
||||
the `vc_client*` and guarantees `vc_client_destroy` runs on GC/Dispose.
|
||||
|
||||
### External PCM feed/tap
|
||||
|
||||
Two API functions let callers bypass miniaudio entirely for a stream:
|
||||
|
||||
| Function | Direction | Contract |
|
||||
|----------|-----------|----------|
|
||||
| `vc_stream_feed_pcm(c, stream_id, pcm, samples_per_channel, channels)` | **Send** — caller → network | Caller supplies interleaved int16 at the stream's sample rate (`channels` = 1 mono, 2 stereo). The core frames, Opus-encodes, AEAD-seals, and sends over UDP — identical wire path to hardware capture. The stream must already be started with `vc_stream_start`. Thread-safe; may be called from any thread (audio callback, ReplayKit delegate, SCStream callback). |
|
||||
| `vc_set_pcm_sink(c, cb, user)` | **Receive** — network → caller | `cb` is called on the audio (playback) thread once per decoded Opus frame per remote stream, with `(user_id, stream_id, pcm, samples_per_channel, channels, sample_rate)`. PCM is delivered to the sink **and** the hardware device — dual output; the hardware mix is unaffected. Pass `cb=NULL` to disable (default); once that call returns, no callback using the old `user` pointer remains in flight. **Must not block** — copy what you need and return. |
|
||||
|
||||
`vc_test_inject_capture` (the old TEST-ONLY mono-only predecessor) is a deprecated alias
|
||||
for `vc_stream_feed_pcm(..., channels=1)` — kept for source compatibility.
|
||||
|
||||
**Use cases:** ReplayKit Broadcast Extension (iOS `SCREEN_AUDIO`), ScreenCaptureKit (macOS
|
||||
`SCREEN_AUDIO`), music/TTS/relay bots, soundboards, transcription clients. The extension or
|
||||
bot links Opus + the feed entry point — no `ma_device`, no hardware, headless.
|
||||
|
||||
**Threading:** the feed path is thread-safe (ring buffer, no lock on the RT path). The sink
|
||||
callback runs on the miniaudio playback thread — observe the same rules as the capture
|
||||
callback: no allocations, no blocking calls.
|
||||
|
||||
## 5. Server architecture
|
||||
|
||||
`voicecat-server` is a headless process linking the core.
|
||||
|
||||
```
|
||||
TCP/TLS 1.3 UDP + media AEAD
|
||||
│ │
|
||||
┌────────▼─────────┐ ┌─────────▼──────────┐
|
||||
│ Connection mgr │ │ UDP demux │
|
||||
│ (accept, TLS, │ │ 5-tuple → session │
|
||||
│ per-conn state) │ │ anti-replay window │
|
||||
└────────┬─────────┘ └─────────┬──────────┘
|
||||
│ │
|
||||
┌────────▼───────────────────────────────────▼──────────┐
|
||||
│ Session registry (session_id ↔ TCP conn ↔ UDP tuple) │
|
||||
└────────┬───────────────────────────────┬───────────────┘
|
||||
│ │
|
||||
┌────────▼─────────┐ ┌──────────────┐ ┌▼─────────────────┐
|
||||
│ Channel manager │ │ Text router │ │ Voice router/SFU │
|
||||
│ tree, configs, │ │ channel + PM │ │ relay Opus to │
|
||||
│ membership, perms│ │ │ │ channel members │
|
||||
└────────┬─────────┘ └──────────────┘ └──────────────────┘
|
||||
│
|
||||
┌────────▼─────────┐
|
||||
│ Persistence │ accounts (Argon2id), channels, bans, config
|
||||
│ SQLite │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
- **Voice router is a relay, not a mixer.** For each incoming voice frame it looks up the
|
||||
sender's channel and forwards the *unmodified encoded Opus bytes* to other members.
|
||||
It authenticates/decrypts incoming media, then reseals with each recipient's directional
|
||||
key and counter. SSRC/timestamp/flags/codec pass through; sequence and ciphertext/tag change.
|
||||
No server-side decode/transcode → low CPU,
|
||||
low latency, and end-to-content is just Opus. Per-channel Opus params are enforced so all
|
||||
members are mutually decodable.
|
||||
- **Subscriptions.** Clients implicitly subscribe to their current channel's voice; text
|
||||
and presence can be subscribed more broadly. This keeps fan-out bounded on big servers.
|
||||
- **Stateless-ish media.** UDP carries no auth per packet beyond the media-AEAD session;
|
||||
the 5-tuple→session binding is established once via a token (see protocol.md §4).
|
||||
- **Keepalive reaper.** An `asio::steady_timer` sweeps every 15 s and drops any session
|
||||
whose `last_seen` (bumped on every inbound TCP or UDP frame) is older than 45 s. Each
|
||||
drop broadcasts `UserEvent::LEFT` so peers clean up immediately. This catches half-open
|
||||
connections that never produce a TCP EOF. Configurable via `server::Config`.
|
||||
- **Single process, scalable later.** v1 is one process, one machine. The session registry
|
||||
and router are written behind interfaces so a future build can sit them behind a shared
|
||||
bus for multi-node, but that is explicitly out of scope for now.
|
||||
|
||||
## 6. Repository layout (proposed)
|
||||
|
||||
```
|
||||
voice-cat/
|
||||
├── docs/ # this folder
|
||||
├── core/ # libvoicecat (C++)
|
||||
│ ├── include/voicecat.h # the C ABI
|
||||
│ ├── src/{net,crypto,codec,protocol,session,audio}/
|
||||
│ └── proto/ # .proto definitions (shared source of truth)
|
||||
├── server/ # voicecat-server (C++, links core)
|
||||
├── clients/
|
||||
│ ├── apple/ # Swift package + Xcode project (macOS + iOS)
|
||||
│ └── windows/ # .NET solution (C#)
|
||||
├── tools/
|
||||
│ └── vccli/ # headless test client (C++), for protocol bring-up
|
||||
├── third_party/ # vendored / vcpkg manifest
|
||||
└── CMakeLists.txt
|
||||
```
|
||||
|
||||
Build is **CMake** with **vcpkg** (manifest mode) for C/C++ deps; the Apple and Windows UI
|
||||
projects consume the built core as a binary + headers. See [tech-stack.md](tech-stack.md).
|
||||
Unsupported C++ and Swift applications may remain temporarily during repository cleanup, but
|
||||
they are not dependencies, compatibility targets, or design authorities.
|
||||
|
||||
Reference in New Issue
Block a user