docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation:
- README: overview, locked decisions, principles, glossary
- architecture: shared C++ core + C ABI, native UIs (Swift/C#),
threading model, server design (SFU relay)
- protocol: TCP/TLS control plane, protobuf Envelope + message
catalog, connection lifecycle, extensibility rules
- voice: UDP media frame format, per-channel Opus config,
multi-stream model, two-sided noise reduction, VAD/PTT,
jitter buffer, iOS ReplayKit screen-audio
- security: mandatory encryption (TLS 1.3 + exported-key AEAD),
TOFU server identity, admin-provisioned accounts, anti-replay
- tech-stack: permissive-only deps (mbedTLS, libsodium, opus,
miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL
- deployment: zero-config self-host (Docker / binary / source)
- roadmap: M0-M5 milestones, resolved decisions
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:47:09 +02:00
# Tech Stack & Dependencies
Concrete library choices with versions and rationale. Everything in the **core** is C++
(C++20). UIs are Swift and C#. Build is CMake + vcpkg.
## 1. Core library (`libvoicecat`, C++20)
| Concern | Choice | Version (as of 2026-06) | Why / notes |
|---------|--------|-------------------------|-------------|
| Sockets, timers, async | **Standalone Asio** | 1.30.x | Header-only, no Boost dependency, cross-platform TCP+UDP+timers, one reactor for client and server. (Boost.Asio is interchangeable if we already pull Boost.) |
| TLS 1.3 (control) | **mbedTLS 3.6 LTS** | 3.6.x (LTS ≥ Mar 2027) | **Apache-2.0** (permissive — clean for eventual closed-source distribution). TLS 1.3 client+server, plus `mbedtls_ssl_export_keying_material()` to seed the media AEAD. **Static-links cleanly → single self-host binary.** OpenSSL 3.x (Apache-2.0) is an interchangeable alternative. No DTLS/wolfSSL (GPL) — see [security.md ](security.md ) §2. |
| Crypto primitives + password hashing + media AEAD | **libsodium** | 1.0.20 | **ISC.** Argon2id (`crypto_pwhash` ), ChaCha20-Poly1305 (per-frame media encryption), Ed25519 server identity, X25519, CSPRNG. Audited, hard to misuse. |
| Audio codec | **libopus** | **1.6** (2025-12) | Per-channel mono/stereo, bitrate, frame size; in-band FEC, DTX, PLC, and optional **DRED** deep redundancy; Opus HD/96 kHz available. The whole reason the design is codec-flexible. |
| Audio capture/playback | **miniaudio** | 0.11.x | Single-header, public-domain, backends for **WASAPI / CoreAudio / ALSA / PulseAudio** . One real-time abstraction across all desktop targets; keeps the RT path identical. |
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
| Audio DSP — AEC/NS/AGC/VAD | **webrtc-audio-processing** (APM) — **planned, not built** | 1.x (standalone APM) | **BSD-3** , but has no working Windows/MSVC build upstream (GCC-only Meson, MinGW support unfinished, hard `abseil-cpp` dep — see roadmap.md §2). v1 ships a lightweight, dependency-free energy/RMS VAD instead (`core/src/audio/apm_processor.cpp` ); there is **no AEC, NS, or AGC implementation at all** yet. Real APM stays a tracked future swap behind the same `ApmProcessor` interface. |
docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation:
- README: overview, locked decisions, principles, glossary
- architecture: shared C++ core + C ABI, native UIs (Swift/C#),
threading model, server design (SFU relay)
- protocol: TCP/TLS control plane, protobuf Envelope + message
catalog, connection lifecycle, extensibility rules
- voice: UDP media frame format, per-channel Opus config,
multi-stream model, two-sided noise reduction, VAD/PTT,
jitter buffer, iOS ReplayKit screen-audio
- security: mandatory encryption (TLS 1.3 + exported-key AEAD),
TOFU server identity, admin-provisioned accounts, anti-replay
- tech-stack: permissive-only deps (mbedTLS, libsodium, opus,
miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL
- deployment: zero-config self-host (Docker / binary / source)
- roadmap: M0-M5 milestones, resolved decisions
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:47:09 +02:00
| Resampling + jitter ref | **speexdsp** | 1.2.x | BSD. Resampler for non-48 kHz devices; lightweight jitter-buffer reference. (No longer the NS/AGC/VAD source — APM replaces it.) |
| Control serialization | **Protocol Buffers** (protobuf-lite) | 5.x (proto3) | Codegen for C++/C#/Swift ; additive, forward/backward compatible; `oneof` envelopes. `nanopb` is a fallback if footprint matters. |
| Server persistence | **SQLite** | 3.4x | Accounts, channels, bans, config. Zero-admin, single file, ships everywhere. |
| Logging | **spdlog** | 1.14.x | Fast, async-capable; off the RT path. |
Resampling note: Opus runs internally at 48 kHz; miniaudio can deliver 48 kHz directly, so
explicit resampling (speexdsp/libsamplerate) is only needed when a device can't do 48 kHz.
## 2. Clients
### macOS / iOS — Swift
| Concern | Choice | Notes |
|---------|--------|-------|
| Language | **Swift 5.9+** | Direct **Swift↔C++ interop** available, but we bind through the C ABI for parity with Windows. |
| UI | **SwiftUI** | Single UI codebase for macOS + iOS where practical; AppKit/UIKit shims as needed. |
| Audio session (iOS) | **AVAudioSession** | App owns category `.playAndRecord` + `.voiceChat` mode, mic permission, interruption/route-change handling; calls `vc_audio_suspend/resume` on the core. macOS uses CoreAudio via the core directly. |
| Packaging | Swift Package + Xcode project | Core shipped as an XCFramework (device + simulator + macOS slices). |
| Future | CallKit / PushKit | For background VoIP + incoming-call UX on iOS. Post-v1. |
2026-06-17 00:52:02 +02:00
### Windows — C# (shipped in M4, 2026-06-17)
docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation:
- README: overview, locked decisions, principles, glossary
- architecture: shared C++ core + C ABI, native UIs (Swift/C#),
threading model, server design (SFU relay)
- protocol: TCP/TLS control plane, protobuf Envelope + message
catalog, connection lifecycle, extensibility rules
- voice: UDP media frame format, per-channel Opus config,
multi-stream model, two-sided noise reduction, VAD/PTT,
jitter buffer, iOS ReplayKit screen-audio
- security: mandatory encryption (TLS 1.3 + exported-key AEAD),
TOFU server identity, admin-provisioned accounts, anti-replay
- tech-stack: permissive-only deps (mbedTLS, libsodium, opus,
miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL
- deployment: zero-config self-host (Docker / binary / source)
- roadmap: M0-M5 milestones, resolved decisions
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:47:09 +02:00
| Concern | Choice | Notes |
|---------|--------|-------|
2026-06-17 00:52:02 +02:00
| Runtime | ** .NET 10 LTS** (`net10.0-windows` ) | In-service until 2028. |
| Interop | ** `[LibraryImport]` ** (source-gen P/Invoke) over the C ABI | `[UnmanagedCallersOnly]` static methods for `on_event` /`on_level` ; `VoiceCatClientHandle : SafeHandle` owns the `vc_client*` lifetime. |
| Event delivery | ** `System.Threading.Channels.Channel<VoiceCatEvent>` ** | Single-writer/reader, unbounded; drained by a 30ms `System.Windows.Forms.Timer` on the UI thread. Simpler than a message-only HWND with no meaningful latency cost. |
| UI | **WinForms** | Chosen over WinUI 3 / Avalonia for mature, predictable NVDA/JAWS/Narrator UIA support. Win32 HWND controls have the most complete accessibility story on .NET 10 today. See roadmap.md §2. |
| Persistence | ** `System.Text.Json` ** (`servers.json` ), ** `ProtectedData` ** (DPAPI) | Saved-server list in `%AppData%\VoiceCat\` ; passwords DPAPI-encrypted at rest, opt-in, `CurrentUser` scope. |
| Audio | Handled by the core (miniaudio/WASAPI) | C# only drives device selection + meters. |
docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation:
- README: overview, locked decisions, principles, glossary
- architecture: shared C++ core + C ABI, native UIs (Swift/C#),
threading model, server design (SFU relay)
- protocol: TCP/TLS control plane, protobuf Envelope + message
catalog, connection lifecycle, extensibility rules
- voice: UDP media frame format, per-channel Opus config,
multi-stream model, two-sided noise reduction, VAD/PTT,
jitter buffer, iOS ReplayKit screen-audio
- security: mandatory encryption (TLS 1.3 + exported-key AEAD),
TOFU server identity, admin-provisioned accounts, anti-replay
- tech-stack: permissive-only deps (mbedTLS, libsodium, opus,
miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL
- deployment: zero-config self-host (Docker / binary / source)
- roadmap: M0-M5 milestones, resolved decisions
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:47:09 +02:00
## 3. Server (`voicecat-server`)
- Pure C++ linking the core; **no GUI** . Runs on **Linux** (primary), **macOS** , **Windows** .
- Config via a `server.toml` (`allow_guests` , ports, channel defaults, Opus policy, TLS cert
paths or auto-self-signed + Ed25519 identity, Argon2id cost params, rate limits).
- SQLite for state. Single process for v1; interfaces drawn so a multi-node build is
*possible* later but explicitly out of scope.
- Packaging: static-ish binary per OS; systemd unit + Docker image for Linux.
## 4. Build & tooling
| Tool | Use |
|------|-----|
| **CMake** (3.25+) | One build graph for core + server + test CLI; UI projects consume the built core. |
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
| **vcpkg** (manifest mode) | Pin C/C++ deps (opus, libsodium, mbedtls, protobuf, sqlite3, spdlog, asio, miniaudio — see `vcpkg.json` ). `webrtc-audio-processing` /`speexdsp` are **not** in the manifest: no working vcpkg port / no working Windows/MSVC build exists upstream for the former; the latter was never actually wired up (the lightweight VAD needs no resampler). Reproducible across OSes. |
docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation:
- README: overview, locked decisions, principles, glossary
- architecture: shared C++ core + C ABI, native UIs (Swift/C#),
threading model, server design (SFU relay)
- protocol: TCP/TLS control plane, protobuf Envelope + message
catalog, connection lifecycle, extensibility rules
- voice: UDP media frame format, per-channel Opus config,
multi-stream model, two-sided noise reduction, VAD/PTT,
jitter buffer, iOS ReplayKit screen-audio
- security: mandatory encryption (TLS 1.3 + exported-key AEAD),
TOFU server identity, admin-provisioned accounts, anti-replay
- tech-stack: permissive-only deps (mbedTLS, libsodium, opus,
miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL
- deployment: zero-config self-host (Docker / binary / source)
- roadmap: M0-M5 milestones, resolved decisions
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:47:09 +02:00
| **protoc** | Generate C++/C#/Swift from `core/proto/*.proto` (single source of truth). |
| **clang-format / clang-tidy** | Style + static analysis on the core. |
| **CTest + a fuzz target** | Unit/integration tests; fuzz the frame parser and protobuf boundary (security-sensitive). |
| **GitHub Actions** (or similar) | Matrix CI: Linux/macOS/Windows core+server; Xcode build for Apple; `dotnet` build for Windows. |
## 5. Licensing — permissive only (hard rule)
The code will eventually be distributed in **closed-source** form, so **no GPL/LGPL
dependencies are permitted.** Every dependency below is BSD / MIT / ISC / Apache-2.0 /
public-domain:
- **mbedTLS** — Apache-2.0 ✅ · **libsodium** — ISC ✅ · **libopus** — BSD ✅ ·
feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 16:11:52 +02:00
**miniaudio** — public domain / MIT-0 ✅ · **protobuf** — BSD ✅ ·
**SQLite** — public domain ✅ · **Asio** (standalone) — Boost ✅ · **spdlog** — MIT ✅.
**webrtc-audio-processing** would be BSD-3 ✅ if/when it's actually built in (see §1) —
not a live dependency today, so not part of the resolved vcpkg graph the license scanner
below checks.
docs: initial design baseline for VoiceCat voice/text chat
Establish the design spec in docs/ before implementation:
- README: overview, locked decisions, principles, glossary
- architecture: shared C++ core + C ABI, native UIs (Swift/C#),
threading model, server design (SFU relay)
- protocol: TCP/TLS control plane, protobuf Envelope + message
catalog, connection lifecycle, extensibility rules
- voice: UDP media frame format, per-channel Opus config,
multi-stream model, two-sided noise reduction, VAD/PTT,
jitter buffer, iOS ReplayKit screen-audio
- security: mandatory encryption (TLS 1.3 + exported-key AEAD),
TOFU server identity, admin-provisioned accounts, anti-replay
- tech-stack: permissive-only deps (mbedTLS, libsodium, opus,
miniaudio, webrtc-apm, ...), build tooling, no GPL/LGPL
- deployment: zero-config self-host (Docker / binary / source)
- roadmap: M0-M5 milestones, resolved decisions
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:47:09 +02:00
- **Explicitly rejected:** **wolfSSL** (GPLv2/commercial) and any DTLS stack that would drag
in copyleft. The exported-keys + AEAD media design (security.md §2) removes the need for
one entirely.
- CI runs a license scanner over the resolved vcpkg graph and **fails the build on any
GPL/LGPL transitive dependency**, so this rule can't silently regress.
## 6. Why not the obvious alternatives
- **WebRTC** — explicitly rejected: ICE/SDP/TURN complexity, huge dependency, opaque. We
want plain TCP+UDP we fully control.
- **QUIC** — capable (reliable streams + datagrams + TLS 1.3 in one), but heavier and drifts
toward the complexity we're avoiding. Revisit only if NAT traversal/multiplexing pain
appears.
- **gRPC** for control — pulls HTTP/2 and a lot of surface for what is a simple framed
message stream over TLS. Plain protobuf-over-framed-TLS is enough.
- **A Rust core** — viable and memory-safe, but the user prefers C++ and the Swift/C#
binding story is marginally simpler from C++ (Swift can even consume C++ directly).