feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer

Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.

Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-16 01:31:14 +02:00
parent 63f457fc54
commit 694494a5be
29 changed files with 2548 additions and 86 deletions

View File

@@ -10,12 +10,12 @@ up instantly. Newest status at the top.
## ▶ Where we left off / next action
- **Done:** **M1control plane** ✓ complete (2026-06-15).
`ctest --preset m1-dev` — all 5 tests green (smoke, frame_codec, envelope, tls_loopback,
m1_integration). Two clients authenticate over TLS 1.3 and exchange channel + private text.
- **Next:** **M2 — voice, single stream**. First task: Opus encode/decode stub → real
implementation; UDP socket + ChaCha20-Poly1305 AEAD media frame; jitter buffer.
See `docs/voice.md` and `docs/roadmap.md §M2`.
- **Done:** **M2voice, single stream** ✓ complete (2026-06-16).
`ctest --preset m1-dev` — all **9/9 tests** green including the new M2 exit criterion
(`test_m2_voice`): two headless clients encrypt Opus frames via ChaCha20-Poly1305, bind
UDP sockets, and the server SFU relay re-encrypts + forwards frames.
- **Next:** **M3 — multi-stream & per-channel tuning** (screen audio, listener-side per-user
NR, jitter buffer stats API). See `docs/roadmap.md §M3`.
---
@@ -23,7 +23,8 @@ up instantly. Newest status at the top.
- [x] **M0 — Scaffolding** ✓ complete
- [x] **M1 — Control plane** ✓ complete (2026-06-15)
- [~] **M2 — Voice, single stream** (UDP, Opus, jitter buffer, APM send-side, VAD/PTT) ← current
- [x] **M2 — Voice, single stream** ✓ complete (2026-06-16)
- [ ] **M3 — Multi-stream & per-channel tuning** ← next
- [ ] **M3 — Multi-stream & per-channel tuning** (screen audio, listener-side per-user NR)
- [ ] **M4 — Native clients** (Windows C#, macOS/iOS Swift)
- [ ] **M5 — Moderation, polish, beyond** (perms, bans, DRED; then file transfer, E2EE, …)
@@ -68,6 +69,35 @@ raw protobuf bytes directly and letting `send_frame` add the single `[4-byte len
---
---
## M2 — Voice, single stream ✓ (completed 2026-06-16)
**Exit criterion:**`test_m2_voice` — two headless clients authenticate over TLS, bind UDP,
announce a MIC stream, send 50 encrypted Opus frames; server SFU relay re-encrypts + forwards
to the second client; B receives ≥ 25 frames and all decrypt correctly. Passes in ~4 s.
- [x] `m2-dev` preset (inherits `vcpkg-base`, binaryDir `build/m2-dev`); `m1-dev` also builds all M2 code.
- [x] `core/CMakeLists.txt``find_package(Opus)`, `find_path(MINIAUDIO_INCLUDE_DIR)`.
- [x] `core/src/net/voice_frame.h` — 14-byte UDP header (type/flags/codec/ssrc/seq/ts), serialize/parse, `make_udp_binding_packet`.
- [x] `SodiumMediaCrypto` — ChaCha20-Poly1305 AEAD; counter-nonce; 64-bit sliding-window anti-replay; `derive_send/recv` from TLS RFC 5705 exporter.
- [x] `OpusEncoder` / `OpusDecoder` — libopus 1.6, FEC, DTX, PLC (free; nullptr → decoder extrapolates).
- [x] `UdpMediaChannel` — async UDP socket (asio); thread-safe `send_to`; async recv loop.
- [x] `JitterBuffer` — per-ssrc, EWMA jitter estimation, adaptive depth 20200 ms, late-drop at 500 ms.
- [x] `AudioEngine` — miniaudio capture+playback; `inject_capture()` bypass for headless tests; per-ssrc RemoteStream with OpusDecoder + JitterBuffer.
- [x] `ApmProcessor``ApmPassthrough` stub (VAD always open); WebRTC APM deferred until M3.
- [x] `on_tls_ready` callback in `TcpChannelCallbacks` — server derives and stores media AEAD keys immediately after TLS handshake.
- [x] `ConnSession` M2 — `udp_token` generated at construction; included in `AuthResult`; `handle_udp_binding` (verifies token, TCP ack); `handle_stream_announce` (assigns SSRC via registry); `udp_media_port` in `ServerHello`.
- [x] `SessionRegistry` M2 — `register_udp_token`, `find_by_udp_token`, `register_udp_endpoint`, `find_by_udp_endpoint`, `assign_ssrc`, `find_channel_sessions`, `user_channel`.
- [x] `MediaRelay` — SFU UDP relay; `kFrameUdpBinding` → endpoint binding; `kFrameVoice` → decrypt/re-encrypt/forward to channel members.
- [x] `Server::run()` — creates and binds `MediaRelay`; passes media port to `ConnSession`; wires `on_tls_ready` to derive per-connection media AEAD keys.
- [x] `test_voice_frame` — header round-trip, big-endian layout, binding packet format.
- [x] `test_media_aead` — seal/open round-trip, anti-replay, tamper detection, multi-packet sequence.
- [x] `test_opus_codec` — encode/decode round-trip energy check (within 3 dB), PLC, frame-samples helper.
- [x] `test_m2_voice` — M2 exit criterion. Verified green 2026-06-16.
---
## Decisions log
All architecture/scope decisions are settled and recorded in