Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
694494a5beecfe8d4986bc01a38d7a82a4fb201c
voice-cat/PROGRESS.md
Talon 694494a5be feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer 
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.

Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 01:31:14 +02:00

6.8 KiB
Raw Blame History

PROGRESS — VoiceCat

Living status. Update this file in the same commit as your work so the next agent picks up instantly. Newest status at the top.

  • Date convention: ISO (YYYY-MM-DD).
  • Statuses: [ ] not started · [~] in progress · [x] done.

▶ Where we left off / next action

  • Done: M2 — voice, single stream ✓ complete (2026-06-16). ctest --preset m1-dev — all 9/9 tests green including the new M2 exit criterion (test_m2_voice): two headless clients encrypt Opus frames via ChaCha20-Poly1305, bind UDP sockets, and the server SFU relay re-encrypts + forwards frames.
  • Next: M3 — multi-stream & per-channel tuning (screen audio, listener-side per-user NR, jitter buffer stats API). See docs/roadmap.md §M3.

Milestones (see docs/roadmap.md for full detail)

  • M0 — Scaffolding ✓ complete
  • M1 — Control plane ✓ complete (2026-06-15)
  • M2 — Voice, single stream ✓ complete (2026-06-16)
  • M3 — Multi-stream & per-channel tuning ← next
  • M3 — Multi-stream & per-channel tuning (screen audio, listener-side per-user NR)
  • M4 — Native clients (Windows C#, macOS/iOS Swift)
  • M5 — Moderation, polish, beyond (perms, bans, DRED; then file transfer, E2EE, …)

M0 — Scaffolding ✓ (completed)

  • Repo layout (core/ server/ tools/ clients/ tests/), CMake + presets, vcpkg manifest.
  • C ABI header core/include/voicecat.h (full surface, stubbed).
  • Protocol source-of-truth core/proto/voicecat.proto (matches docs/protocol.md).
  • Core stubs for all six subsystems (net/crypto/codec/protocol/session/audio) + vc_client.
  • voicecat-server (arg parsing, config, stub run) and vccli (drives the C ABI).
  • CTest smoke test asserting the C ABI contract (not just "it compiles").
  • .gitattributes (LF), .gitignore, .clang-format, onboarding docs.
  • Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.

M1 — Control plane ✓ (completed 2026-06-15)

Exit criterion: ✓ test_m1_integration — two clients authenticate over TLS 1.3 (guest

  • Argon2id password), exchange channel and private text messages. Passes in ~1 s.
  • vcpkg baseline + m1-dev preset; find_package for protobuf/mbedTLS/libsodium/asio/sqlite3.
  • FrameCodec feed + emit; encode_envelope / decode_envelope.
  • Asio TCP acceptor + TcpServerConn (TLS path: blocking handshake thread + tls_read_loop).
  • TlsContext (mbedTLS 1.3, server cert/identity, ECDSA-P256 self-signed, TOFU on client).
  • WorkerPool (3 threads, used for Argon2id).
  • Database — SQLite, Argon2id via libsodium, create_account / authenticate / bootstrap admin.
  • voicecat-admin — account add/reset/del/list against live DB file.
  • ServerIdentityManager — generate/persist Ed25519 key + cert; fingerprint display.
  • ConnSession — WaitingHello → WaitingAuth → Authenticated state machine; full protocol relay.
  • SessionRegistry — channel tree, user map, broadcast, text routing.
  • vc_client (client.cpp) — full M1 C ABI: connect/TLS/ClientHello/AuthRequest/text/disconnect.
  • Server::run() — io_context, acceptor, worker pool, signal handling, on_ready callback.
  • test_m1_integration — M1 exit criterion. Verified green 2026-06-15.

Key bug fixed: double-framing in ConnSession::send_envelope — encode_envelope was pre-framing the protobuf, then TcpServerConn::send_frame re-framed it. Fixed by serializing raw protobuf bytes directly and letting send_frame add the single [4-byte len] prefix.

M2 — Voice, single stream ✓ (completed 2026-06-16)

Exit criterion: ✓ test_m2_voice — two headless clients authenticate over TLS, bind UDP, announce a MIC stream, send 50 encrypted Opus frames; server SFU relay re-encrypts + forwards to the second client; B receives ≥ 25 frames and all decrypt correctly. Passes in ~4 s.

  • m2-dev preset (inherits vcpkg-base, binaryDir build/m2-dev); m1-dev also builds all M2 code.
  • core/CMakeLists.txt — find_package(Opus), find_path(MINIAUDIO_INCLUDE_DIR).
  • core/src/net/voice_frame.h — 14-byte UDP header (type/flags/codec/ssrc/seq/ts), serialize/parse, make_udp_binding_packet.
  • SodiumMediaCrypto — ChaCha20-Poly1305 AEAD; counter-nonce; 64-bit sliding-window anti-replay; derive_send/recv from TLS RFC 5705 exporter.
  • OpusEncoder / OpusDecoder — libopus 1.6, FEC, DTX, PLC (free; nullptr → decoder extrapolates).
  • UdpMediaChannel — async UDP socket (asio); thread-safe send_to; async recv loop.
  • JitterBuffer — per-ssrc, EWMA jitter estimation, adaptive depth 20–200 ms, late-drop at 500 ms.
  • AudioEngine — miniaudio capture+playback; inject_capture() bypass for headless tests; per-ssrc RemoteStream with OpusDecoder + JitterBuffer.
  • ApmProcessor — ApmPassthrough stub (VAD always open); WebRTC APM deferred until M3.
  • on_tls_ready callback in TcpChannelCallbacks — server derives and stores media AEAD keys immediately after TLS handshake.
  • ConnSession M2 — udp_token generated at construction; included in AuthResult; handle_udp_binding (verifies token, TCP ack); handle_stream_announce (assigns SSRC via registry); udp_media_port in ServerHello.
  • SessionRegistry M2 — register_udp_token, find_by_udp_token, register_udp_endpoint, find_by_udp_endpoint, assign_ssrc, find_channel_sessions, user_channel.
  • MediaRelay — SFU UDP relay; kFrameUdpBinding → endpoint binding; kFrameVoice → decrypt/re-encrypt/forward to channel members.
  • Server::run() — creates and binds MediaRelay; passes media port to ConnSession; wires on_tls_ready to derive per-connection media AEAD keys.
  • test_voice_frame — header round-trip, big-endian layout, binding packet format.
  • test_media_aead — seal/open round-trip, anti-replay, tamper detection, multi-packet sequence.
  • test_opus_codec — encode/decode round-trip energy check (within 3 dB), PLC, frame-samples helper.
  • test_m2_voice — M2 exit criterion. Verified green 2026-06-16.

Decisions log

All architecture/scope decisions are settled and recorded in docs/roadmap.md §2 "Resolved decisions" and reflected across docs/. If you make a new decision, record it there and link it here.

How to update this file

  1. Check off tasks as you complete them; flip a milestone to [x] only when its exit criterion test passes.
  2. Keep the "Where we left off / next action" block at the top accurate — it's the first thing the next agent reads.
  3. When you start a milestone, copy its task list from docs/roadmap.md into a section here.