Talon
19c2fb6ec9
The iOS mic was unusable — a consistent ~40-60ms flutter + volume fade
('slow fan') on every preset. The core sends each captured frame
synchronously (no send pacer), so packet cadence == capture cadence, and
the receiver's playout keeps near-zero buffering by design (its jitter
estimate keys off the regular sender timestamp, so it's blind to arrival
jitter). That's smooth only for a steady sender (desktop miniaudio =
steady 20ms); the iOS AVAudioEngine tap delivers ~2 frames per ~40ms
callback -> bursty -> receiver underruns -> PLC fade.
Fix (iOS-only): the mic tap writes converted 48kHz int16 to an SPSC ring;
a 20ms feed pump drains it and calls feedPcm at a steady cadence. The pump
primes a small prebuffer cushion (3 frames ~60ms, self-healing up to
~120ms on underrun) before releasing, so the tap's bursts can't drain it
to empty. Never reads a partial frame (read consumes what it returns ->
partials were the crackle), and rebuilds with the current channel count
each rebuild() (a frozen count fed mono-as-stereo = octave-up on a
Stereo->Voice Chat switch).
Trade-off: ~60-120ms added mic-send latency, unavoidable when de-bursting
for a near-zero-buffer receiver. PROGRESS.md notes the proper follow-up:
make the jitter buffer measure real RFC-3550 arrival jitter so the
receiver absorbs bursts itself.
Verified: xcodebuild Debug BUILD SUCCEEDED (iOS Simulator, arm64).
966 lines
73 KiB
Markdown
966 lines
73 KiB
Markdown
# PROGRESS — VoiceCat
|
||
|
||
Living status. **Update this file in the same commit as your work** so the next agent picks
|
||
up instantly. Newest status at the top.
|
||
|
||
- **Date convention:** ISO (YYYY-MM-DD).
|
||
- Statuses: `[ ]` not started · `[~]` in progress · `[x]` done.
|
||
|
||
---
|
||
|
||
## ▶ Where we left off / next action
|
||
|
||
- **[ ] Soon — jitter buffer should measure REAL arrival jitter (RFC 3550), not sender
|
||
timestamps.** `JitterBuffer::push` (`core/src/audio/audio_engine.cpp:84-108`) estimates
|
||
jitter from `gap = ts - last_push_ts_`, where `ts` is the **sender's timestamp** — which is
|
||
perfectly regular (`ls.timestamp += samples` every frame, independent of when the packet is
|
||
actually sent). So `diff` is always ~0, `jitter_est_` stays 0, and `target_depth_ms_` is
|
||
pinned at its ~20 ms floor. The buffer is therefore **blind to real network/arrival jitter
|
||
and to bursty senders** — it never deepens. Combined with the playout deliberately seeding
|
||
to near-zero depth (`on_playback`, ~line 715), the receiver tolerates only a *steady*
|
||
sender. This is exactly why the iOS mic needed a send-side pacing cushion (below) and why
|
||
genuine network jitter would also cause underruns. **Fix:** measure inter-arrival jitter
|
||
the RFC 3550 way — `D = (arrival_j - arrival_i) - (ts_j - ts_i)` using a wall-clock arrival
|
||
stamp captured in `push()` — and drive `target_depth_ms_` off that EWMA (keep the existing
|
||
marker/silence-gap outlier rejection). Then the receiver absorbs bursts itself and the iOS
|
||
send cushion could be reduced or removed. Shared-core change → add a test and re-verify
|
||
desktop↔desktop stays low-latency (steady sender ⇒ ~0 arrival jitter ⇒ no regression).
|
||
|
||
- **Done (2026-06-23):** **Fixed iOS mic flutter / crackle / octave-up.** The iOS mic was
|
||
unusable: a consistent ~40–60 ms flutter with volume fade ("talking through a slow fan") on
|
||
every preset. Root cause: the core sends each captured frame **synchronously**
|
||
(`on_capture_frame` → `encode_and_send_frame`, no send pacer), so packet cadence == capture
|
||
cadence; and the receiver's playout keeps **near-zero buffering** by design and its jitter
|
||
estimate is blind to arrival timing (see RFC-3550 item above). That's smooth only for a
|
||
*steady* sender (desktop miniaudio = steady 20 ms), but the iOS `AVAudioEngine` input tap
|
||
delivers ~2 frames per ~40 ms callback (more under VPIO) → bursty → receiver underruns → PLC
|
||
fade.
|
||
- **Fix (iOS-only, `clients/apple/iOS/VoiceCatiOS/IOSVoiceProcessingEngine.swift`):** the
|
||
mic tap converts to 48 kHz int16 and writes a lock-free SPSC ring; a 20 ms feed pump
|
||
drains it and calls `feedPcm` at a **steady** cadence so packets leave the core every
|
||
20 ms (what the receiver expects). The pump **primes a small prebuffer cushion**
|
||
(`PumpState.targetFrames`, 3 frames ≈ 60 ms, self-healing up to ~120 ms on underrun)
|
||
before releasing, so the tap's bursts can't drain it to empty. Two correctness rules
|
||
(each had bit us): never read a partial frame (`read` consumes what it returns →
|
||
discarding partials caused crackle), and rebuild the pump with the current channel count
|
||
every `rebuild()` (a frozen channel count fed mono-as-stereo = octave-up on a Stereo→Voice
|
||
Chat switch). Trade-off: ~60–120 ms added mic-send latency — unavoidable when de-bursting
|
||
for a near-zero-buffer receiver; the RFC-3550 fix above would let us shrink it.
|
||
- **Verify:** `xcodebuild` Debug **BUILD SUCCEEDED** (iOS Simulator, arm64). Audible test
|
||
requires a real device (simulator has no real mic route): mic should be smooth on Voice
|
||
Chat / Mono Mic / Stereo Mic, including switching presets while live (no octave).
|
||
|
||
- **Done (2026-06-23):** **Fixed Apple client link failure (stale xcframework missing
|
||
RNNoise).** Both `VoiceCatMac` and `VoiceCatiOS` failed to link with `Undefined symbols for
|
||
architecture arm64: _rnnoise_create / _rnnoise_destroy / _rnnoise_process_frame`. Root
|
||
cause: `clients/apple/scripts/build-xcframework.sh` merged vcpkg deps into the fat static
|
||
lib but NOT the locally-built vendored `librnnoise.a` (a CMake target from
|
||
`third_party/rnnoise/`, linked privately into `voicecat` via `VOICECAT_HAS_NS` — not a
|
||
vcpkg dep). The xcframework had been rebuilt at 14:17 after the RNNoise commit but still
|
||
omitted the symbols, so every slice's `libvoicecat-fat.a` referenced `_rnnoise_*` with no
|
||
defining object. The iOS slices were also stale (pre-rnnoise) and absent from the
|
||
xcframework entirely.
|
||
- **Fix:** `build-xcframework.sh` now collects `.a` files from `build/<preset>/lib/`
|
||
(excluding `libvoicecat*`) in addition to `vcpkg_installed/<triplet>/lib/`, so vendored
|
||
CMake-target static libs like `librnnoise.a` are merged into the fat lib. Future-proof:
|
||
any new vendored static-lib target landing in `build/<preset>/lib/` is picked up
|
||
automatically. README "Fat static library" section updated.
|
||
- **Verify:** rebuilt `VoiceCatCore.xcframework --all` → all 3 slices (macos-arm64,
|
||
ios-arm64, ios-arm64-simulator) now carry 10 `_rnnoise_*` symbols each; fat lib
|
||
~30 MB → ~33 MB. `xcodebuild` Debug **BUILD SUCCEEDED** for `VoiceCatMac`,
|
||
`VoiceCatiOS` (iphonesimulator arm64), and `VoiceCatiOS` (iphoneos arm64,
|
||
`CODE_SIGNING_ALLOWED=NO`). No core/ABI/proto changes — xcframework artifact only.
|
||
|
||
- **Done (2026-06-23):** **Remote-stream noise suppression — real backend (RNNoise).** The
|
||
two-sided NR plumbing (`RemoteStream::recv_ns` + `vc_set_remote_stream(... noise_reduction)`)
|
||
was wired but **inert** — `ApmProcessor::create()` returned a no-op passthrough, because the
|
||
originally-planned `webrtc-audio-processing` won't build on Windows/macOS. Replaced with
|
||
**RNNoise** (BSD-3 + CC0), vendored at `third_party/rnnoise/` (the vcpkg port is `!windows
|
||
!arm`), built as a standalone C static lib + `VOICECAT_HAS_NS`. One `RnnoiseProcessor`
|
||
(`core/src/audio/apm_processor.cpp`) now backs **both** NR paths:
|
||
- **Receive-side** (per-listener, per-`ssrc`): lit up automatically via the factory; gated to
|
||
mono streams (`audio_engine.cpp` ~L791).
|
||
- **Send-side** (mic, new): `vc_set_input_noise_reduction(client, enable)` ABI +
|
||
`vc_client::mic_ns_`, run before input gain/VAD in `on_capture_frame`. A stereo mic is
|
||
downmixed to mono **only when NR is on**; with NR off a stereo mic keeps full stereo.
|
||
- RNNoise is mono/48 kHz/480-sample; our clock is fixed 48 kHz and Opus frame sizes are all
|
||
multiples of 480, so no resampling. RT-safe: alloc at construction, lock-free in the callback.
|
||
- **Verify status:** `ctest --preset dev` green — **28/28** (new `noise_suppression` test:
|
||
feeds white noise through `ApmProcessor::create()`, measures **99.9%** RMS reduction). Build
|
||
clean on the `dev` MinGW preset. **Next (manual):** add the on/off toggles to the client UIs
|
||
(Windows Audio Settings dialog, macOS/iOS settings) calling the two ABIs; build `windows-client`
|
||
+ `apple-dev` presets to confirm RNNoise compiles under MinGW-DLL and arm64; two-client E2E.
|
||
|
||
- **Done (2026-06-23):** **Aux outgoing stream (mic + a second input device) — Windows + macOS.**
|
||
Users can now transmit a second hardware input device (e.g. line-in / aux) alongside the mic, with
|
||
its own device picker and volume, from Audio Settings. **No core/ABI/proto changes** — the aux is
|
||
a `VC_STREAM_AUX_DEVICE` stream started with `external_feed=1`, captured client-side, and fed via
|
||
`vc_stream_feed_pcm` (the same external-feed pipeline screen-audio uses). Per-kind `local_streams_`
|
||
already allows mic + screen + one aux to coexist; volume is a client-side gain multiply (the core's
|
||
`vc_set_input_gain` is mic-only/global). The aux is always-on (the core never gates `AUX_DEVICE` on
|
||
VAD/PTT) and is tied to the voice session (started on Join Voice when enabled, stopped on Leave).
|
||
- **Windows:** new `Audio/InputDeviceCapture.cs` (WASAPI shared-mode capture from a real input
|
||
endpoint via `IMMDevice.Activate(IAudioClient)`, 48 kHz/s16, 20 ms frames) + `InputDeviceEnumerator`
|
||
(WASAPI capture-endpoint list — separate from the core's miniaudio ids). Aux section in
|
||
`AudioSettingsForm.cs` (enable checkbox, device combo, refresh, volume slider, accessible names,
|
||
live-apply + Cancel revert via callbacks). Lifecycle in `MainForm.cs` (`_auxStreamId` +
|
||
`InputDeviceCapture`). Persisted in `VoiceSettings.cs` (`AuxEnabled/AuxDeviceId/AuxGain`).
|
||
- **macOS:** new `Audio/InputDeviceCapture.swift` (AVAudioEngine input-node tap pinned to the chosen
|
||
Core Audio device via `kAudioOutputUnitProperty_CurrentDevice`; AVAudioConverter → 48 kHz int16;
|
||
20 ms framing modelled on `ScreenAudioCapture`) + `InputDeviceEnumerator` (Core Audio device list
|
||
by stable UID). Aux section in `SettingsWindowController.swift`; lifecycle + UserDefaults
|
||
persistence (`voice.aux*`) in `MainWindowController.swift`. New file added to `project.pbxproj`.
|
||
- **Verify status:** Windows C# solution builds clean (0 warn/0 err); `ctest` core suite unchanged
|
||
(no core edits). **Next (manual):** on a Mac, build `VoiceCatMac.xcodeproj`; then two-client E2E —
|
||
enable aux on a second input device, confirm two distinct streams for the sender and that the aux
|
||
volume slider moves the aux level independently of the mic; confirm persistence across relaunch.
|
||
|
||
- **Done (2026-06-23):** **Input-settings persistence, mic input gain, + two iOS bugs (all 3
|
||
clients).** Four fixes:
|
||
1. **Input settings now persist.** Transmission mode (VAD/PTT/Always-On), VAD threshold, and the
|
||
new mic gain were applied to the core + UI but never saved, so every relaunch reset to VAD
|
||
defaults. Each client now persists them and re-applies on connect: iOS via `UserDefaults`
|
||
(`SessionState.loadAndApplyVoiceSettings` + setter writes, keys `voice.*`); macOS via
|
||
`UserDefaults` (`MainWindowController` `didSet` + `loadPersistedAudioSettings`, also restores
|
||
the VAD slider from the stored threshold); Windows via new
|
||
`VoiceCat.App/Models/VoiceSettings.cs` (JSON at `%AppData%\VoiceCat\voice.json`, mirrors
|
||
`FeedbackSettings`) loaded/applied in `MainForm`.
|
||
2. **Microphone input gain.** New global send-side API `vc_set_input_gain` (voicecat.h →
|
||
`client.cpp::on_capture_frame`, applied to MIC PCM before the VAD gate, clamped to int16) plus
|
||
Swift (`setInputGain`) and C# (`SetInputGain`) bindings. Mic-volume slider (0–300 %, default
|
||
100 %) added to all three clients' input settings, persisted with the rest.
|
||
3. **iOS chat send fixed.** `ChatView` called `sendText(scope:.channel)` with no `targetId` (→ 0),
|
||
so channel messages went nowhere; now passes `session.currentChannelId`.
|
||
4. **iOS per-user tuning reachable via VoiceOver.** The tuning sheet was long-press
|
||
`.contextMenu` only (invisible to VoiceOver); `UserRow` now also exposes the same buttons as
|
||
`.accessibilityActions` (no visual change), so the actions rotor reaches tuning + admin actions.
|
||
- **Verified:** core `cmake --build --preset dev` clean; `ctest --preset dev` = 24/27 (the 3
|
||
failures — `external_pcm`, `frame_ms_reframe`, `channel_samplerate` — are a pre-existing
|
||
teardown crash on this machine, reproduced identically with the changes stashed). xcframework
|
||
rebuilt (`--all`); **VoiceCatMac** and **VoiceCatiOS** (arm64 sim) → BUILD SUCCEEDED;
|
||
`VoiceCat.Interop` (`dotnet build`) succeeded. **Windows App not built** (WinForms
|
||
net10.0-windows can't build on macOS) — changes follow existing patterns; needs a Windows
|
||
build + manual check.
|
||
- **Next (manual):** on each client, set PTT + non-default VAD/mic-gain, relaunch → settings
|
||
restored; boost a quiet mic and confirm others hear it louder; iOS send a channel message;
|
||
iOS VoiceOver → focus a user → actions rotor opens tuning.
|
||
|
||
- **Done (2026-06-22):** **Fixed growing voice latency (jitter-buffer depth ratchet).** Symptom:
|
||
end-to-end latency grew to multiple seconds and "drifted backward," reset only by leaving/
|
||
rejoining voice (DTX/FEC/DRED on, 10% loss). Root cause was **not** the codec settings (10% loss
|
||
is just an `OPUS_SET_PACKET_LOSS_PERC` encoder hint; FEC/DRED add no standing latency) but the
|
||
receiver playout logic in `core/src/audio/audio_engine.cpp`: the playout clock free-ran in real
|
||
time while the sender omitted silence from its timestamps and set **no header flags at all**, and
|
||
the only correction snapped the clock to the *oldest* buffered frame (could only *add* latency) —
|
||
with `target_depth_ms_` computed but never enforced, so latency could only grow or be reset.
|
||
**Fix:** bounded-depth playout — (re)seed to the *leading edge* (newest frame) on start/marker/
|
||
starve, and **frame-skip catch-up** that trims a backlog beyond `target + hysteresis` (the missing
|
||
downward force). Plus hardening: adaptive late-drop window, talkspurt `kFlagMarker`/`kFlagDtx`
|
||
now actually stamped by the sender (`client.cpp` send path) and consumed on recv, EWMA outlier
|
||
rejection (silence gaps/stragglers no longer poison the estimate), duplicate counting, ring-
|
||
underrun diagnostics (`stream_underruns`/`stream_duplicates`). New regression test
|
||
`tests/test_jitter_depth.cpp` asserts depth stays bounded (<200 ms) while arrivals outrun playout
|
||
for ~4 s. `ctest --preset dev` green — **27/27**. Docs: `docs/voice.md` §5 rewritten.
|
||
- **Next (manual E2E):** two clients in a channel, DTX/FEC/DRED on — talk in alternating bursts
|
||
for several minutes and confirm latency stays low/stable (no backward drift, no rejoin needed).
|
||
|
||
- **Windows done / Apple awaiting Mac build (2026-06-22):** **Event sound effects + optional
|
||
text-to-speech for all clients.** Clients now play a cue per session event and can optionally
|
||
speak it (TTS off by default; when on it announces joins/leaves and reads message/PM bodies).
|
||
One canonical event→sound mapping (defined off the shared C ABI `vc_event` stream) is mirrored
|
||
across all three clients; `self` vs others is `user_id == self_user_id`, and outgoing messages
|
||
echo back as events so sent/recv cues need no separate send-path hook. Conservative defaults
|
||
(join/leave, channel/PM sent+recv, login, logout, connection-lost, mic on/off ON; per-utterance
|
||
self voice-activity `va_start/va_stop` and the PTT cue OFF). WAVs ship from `assets/sounds/`.
|
||
- **Windows (built + verified):** new `VoiceCat.App/Notifications/` (`FeedbackSettings` →
|
||
`%AppData%\VoiceCat\feedback.json`, `SoundPlayerPool` via `System.Media.SoundPlayer`,
|
||
`SpeechAnnouncer` via the **Prismatoid** NuGet 0.3.0, `EventFeedback` dispatcher); hooks in
|
||
`Forms/MainForm.cs`; `Forms/NotificationSettingsForm.cs` under a new **Settings ▸ Notifications**
|
||
menu. `.csproj` adds the Prismatoid PackageRef and copies the WAVs into `sounds\`. `dotnet build`
|
||
clean; WAVs + `Prismatoid.dll` confirmed in output. Note: `SoundPlayer` has no gain control, so
|
||
volume is honoured as a mute gate (0 = silent) — swap to NAudio if finer/overlap control is needed.
|
||
- **macOS + iOS (written, NOT yet built — needs a Mac):** shared `Sources/VoiceCatCore/Feedback/`
|
||
(`SoundEvent`, `EventFeedback` = `AVAudioPlayer` pool + native `AVSpeechSynthesizer`,
|
||
`FeedbackSettings` over `UserDefaults`); WAVs copied into `Sources/VoiceCatCore/Sounds/` and
|
||
bundled via `Package.swift` `resources: [.process("Sounds")]` (`Bundle.module`). Hooks: iOS
|
||
`SessionState.handleEvent` (+ split `userJoined`/`userLeft`, added a `.disconnected` cue case),
|
||
`AppState` auth-success login cue, PTT cue in `setPushToTalk`; macOS `MainWindowController`
|
||
handlers + NSEvent PTT monitor. Settings UI: iOS `SettingsView` Notifications section
|
||
(`@AppStorage`), macOS `SettingsWindowController` checkboxes + volume slider. No `.pbxproj`
|
||
edits needed (shared files are SPM-managed; app files already in the projects).
|
||
- **Next:** on a Mac, `clients/apple/scripts/build-xcframework.sh --all` then build
|
||
VoiceCatMac/VoiceCatiOS; fix compile fallout. **Watch the iOS audio session:** cues/TTS play over
|
||
the live VPIO `playAndRecord` session — verify they mix and don't duck/interrupt the call or get
|
||
silenced by the mute switch (most likely bug site). Then run `ctest --preset dev` (unchanged —
|
||
no core/server code touched).
|
||
|
||
- **Done (2026-06-22):** **UDP media now shares the TCP port (self-host port-forward fix).** Symptom: a
|
||
remote self-hosted server (`iamtalon.me:8384`, TCP+UDP 8384 forwarded) accepted TCP connections but
|
||
passed no voice. Root cause: `Config::media_port` defaulted to `0` = OS-assigned, and `main.cpp`'s
|
||
`--port` only set `bind_port` (TCP) — so the UDP relay bound a *random high port*, advertised it to
|
||
clients in HELLO (`udp_port`), and clients sent voice there. With only `8384/udp` forwarded those
|
||
packets were dropped → connect OK, no audio. This contradicted `docs/deployment.md` ("Control and media
|
||
share one port number on TCP+UDP"). **Fix (`server/src/server.cpp`):** media follows bind_port when
|
||
`media_port == 0` — `media_want = cfg_.media_port != 0 ? cfg_.media_port : cfg_.bind_port`. The
|
||
`0 = OS-assigned` escape hatch survives when `bind_port` is also 0, so tests that bind ephemeral ports
|
||
are unaffected (kept the logic in server.cpp rather than hardcoding 8384 as the default, which would
|
||
collide parallel tests on UDP 8384). Banner now reads `TCP :8384 UDP :8384`. Build + `ctest --preset
|
||
dev` green (24/24); live-verified banner with `--port 8390` → `UDP :8390`. **Action for self-hosters:**
|
||
redeploy and confirm the startup banner shows matching TCP/UDP ports; the existing single forward rule
|
||
is now correct. If voice still fails, watch the server's rate-limited `[media] dropped frames —
|
||
unmapped-endpoint=…` line (NAT source-port rewrite would be the next suspect).
|
||
|
||
- **Done (2026-06-23, Swift-only — no core/ABI change; awaiting on-device verification):** **iOS audio
|
||
stack unified — one always-external `AVAudioEngine`, miniaudio dropped on iOS.** The iOS audio path was
|
||
a fragile hybrid: Voice-Chat-class presets ran a native VPIO `AVAudioEngine` (core external) while
|
||
Stereo/Studio/A2DP presets ran the core's miniaudio devices. Nearly every bug lived in the seam
|
||
(lingering miniaudio capture unit fighting VPIO, the `audioRestart` ordering dance, the route-change
|
||
"glitching" loop, stereo↔mono stickiness, "can't hear anyone"), and switching presets/routes mid-call
|
||
routinely dropped input, output, or both. **Fix: drive *all* iOS audio through one `AVAudioEngine` with
|
||
the core fully external at all times** — `vc_set_external_playback(1)` once at connect, every MIC stream
|
||
`external_feed=1`, mic via `vc_stream_feed_pcm`, playback via `vc_set_mixed_output_sink`.
|
||
- `IOSVoiceProcessingEngine.swift` → **`IOSAudioEngine`** (same file): always-on `AVAudioSourceNode`
|
||
playback (runs whenever connected, so remote audio plays before you join voice); conditional mic tap;
|
||
VPIO + AGC toggled per config. One private `rebuild()` (stop → set VPIO → install tap → start) backs
|
||
`startListening`/`stop`/`startMic`/`stopMic`/`reconfigure`/`setCaptureChannels`. Kept the `PCMRing`
|
||
and ring-stats diagnostics.
|
||
- `IOSAudioRouter`: presets cut from seven to **four** — Voice Chat (VPIO mono, system output),
|
||
Stereo Mic / Mono Mic (internal built-in mic regardless of output, A2DP-capable, no VPIO), Advanced
|
||
(manual). New persisted `voiceProcessingEnabled` (master AEC+NS) + `agcEnabled`; setters now call
|
||
`IOSAudioEngine.reconfigure()` instead of `client.audioRestart()` + `reconcileVoicePath`. Kept the
|
||
proven AVAudioSession recipes (category/mode/options, stereo capsule, `applyA2dpSpeakerFallback`).
|
||
- `AudioSessionManager` slimmed (drops `client`/`activeMicStreamId`/`reconcileVoicePath`; adds
|
||
`isActive`); interruption-end & device-change now `reconfigure()` the engine. `SessionState`
|
||
`doStartMicStream`/`stopMicStream` collapsed to start-stream + `startMic`/`stopMic` (no
|
||
`setExternalPlayback`/`audioRestart` toggling); `reconcileVoicePath` deleted. `AppState` sets external
|
||
playback + `startListening` at connect, `stop()` at disconnect. `SettingsView` → four presets +
|
||
Advanced VPIO/AGC toggles.
|
||
- **No core/ABI/test change** — relies on the already-shipped `vc_set_external_playback` /
|
||
`external_feed` / `vc_set_mixed_output_sink` / `vc_stream_feed_pcm` path (`test_external_pcm`,
|
||
`test_external_playback`). `xcodebuild` iOS device Debug **BUILD SUCCEEDED**. **Rebuild the
|
||
xcframework is NOT required** (no new symbols).
|
||
- **Next (user, on device):** two iPhones in a channel — verify BOTH directions survive every
|
||
transition and are never silent unless intended: Voice Chat (no echo, NR), listen-only before joining,
|
||
join↔leave repeatedly, switch Voice Chat↔Stereo↔Mono↔Advanced *while in voice*, A2DP connect/unplug,
|
||
wired connect/unplug, phone-call interruption + resume, screen-audio share.
|
||
|
||
- **Superseded by the 2026-06-23 unification above (2026-06-22):** **iOS real echo cancellation / noise
|
||
suppression via native VPIO.** Root cause of "voice chat doesn't sound like a call" (echo + no NR): real iOS
|
||
AEC/NS/AGC come only from Apple's Voice-Processing I/O unit (VPIO), but the core uses miniaudio's
|
||
plain RemoteIO units — so `.voiceChat` mode alone never engaged AEC. Fix moves both mic capture and
|
||
playback to a native Swift `AVAudioEngine` (`setVoiceProcessingEnabled`) on the AEC presets, with the
|
||
core in external mode.
|
||
- **Core (done, builds + tests green):** new ABI `vc_set_mixed_output_sink` + `vc_set_external_playback`
|
||
(voicecat.h PATCH→2). `AudioEngine` gains a mixer-timer thread that drives `on_playback` (decode+mix)
|
||
on a ~20 ms cadence with NO hardware playback device and ships the final mix to the mixed-output
|
||
sink; `start()` also skips the hardware capture device when the MIC stream is `external_feed`
|
||
(`AudioParams.external_capture`). New white-box test `test_external_playback` (23/24;
|
||
pre-existing `external_pcm` teardown crash on Darwin 25.5 is UNRELATED — original tree crashes too).
|
||
- **Swift (done, builds):** `VoiceCatCore` wrappers (`externalFeed` on `StreamDescriptor`,
|
||
`setMixedOutputSink`, `setExternalPlayback`); new `IOSVoiceProcessingEngine.swift` (VPIO
|
||
`AVAudioEngine`: mic tap→`feedPcm`, mixed-sink lock-free ring→`AVAudioSourceNode`);
|
||
`IOSAudioRouter.currentConfigUsesVoiceProcessing` gates the path per preset; `SessionState`
|
||
join/leave + `reconcileVoicePath()` switch between VPIO and the miniaudio path; Voice Chat defaults
|
||
to speaker; SettingsView shows AEC/NS state. **Rebuild the xcframework** before building the app:
|
||
`clients/apple/scripts/build-xcframework.sh --all` (new ABI symbols). `xcodebuild` iOS sim Debug
|
||
BUILD SUCCEEDED.
|
||
- **Post-verification fixes (2026-06-22, Swift-only — no core/ABI change):** two on-device bugs fixed.
|
||
- **Voice Chat (VPIO) silent playback:** `SessionState.doStartMicStream()` called `audioRestart()`
|
||
BEFORE `startStream`, so when the engine was already running (a remote stream had started it) it
|
||
reopened with `external_capture=false` and opened a hardware miniaudio capture device; the
|
||
announce-result restart then early-returned (engine already running) so that device was never
|
||
dropped and fought the `AVAudioEngine` VPIO unit, silencing playback. Fix: set
|
||
`setExternalPlayback` first, then `startStream` (which stores `external_feed` synchronously), THEN
|
||
`audioRestart()` — the core reopens in full external mode (no hardware devices). Added VPIO
|
||
diagnostics (graph/route formats at start; ring written/read totals at teardown).
|
||
- **Stereo Mic / Studio quiet earpiece:** the `.builtInMicBtA2dp` presets omit `.defaultToSpeaker`
|
||
(it breaks A2DP) and skip `forceSpeaker`, so with no Bluetooth connected output pinned to the quiet
|
||
receiver. New `IOSAudioRouter.applyA2dpSpeakerFallback()` overrides to the built-in speaker when no
|
||
external (A2DP/wired/AirPlay) output is present, clears the override when one is — called after
|
||
activation and on device-change route changes (`AudioSessionManager`).
|
||
- **Next (user, on device):** two iPhones on speaker, Voice Chat preset → confirm (a) no echo, (b)
|
||
background noise suppressed, (c) speaker output by default AND remote audio is now audible; then
|
||
Stereo Mic / Studio with no BT → confirm loud speaker (not earpiece), and A2DP takes over when a BT
|
||
headset connects. Tune the mixer-timer/ring sizing if there's under/overrun.
|
||
|
||
- **Done (2026-06-21):** **Docker + Linux deployment + GitHub Actions cross-build.** Added the complete Linux server
|
||
deployment story (the only missing platform — Windows and macOS already have native
|
||
binaries):
|
||
- `Dockerfile` — multi-stage (builder: `ubuntu:24.04` + vcpkg + `cmake --preset
|
||
server-release`; runtime: `ubuntu:24.04`, non-root `voicecat` user, `/data` volume,
|
||
TCP+UDP 8384). vcpkg is fetched via the GitHub archive tarball at the exact
|
||
`builtin-baseline` commit (`d46283cf…`), avoiding a full git-history clone. BuildKit
|
||
cache mounts on `/vcpkg/downloads`, `/vcpkg/buildtrees`, `/vcpkg/packages` (scoped by
|
||
`TARGETARCH`) keep rebuilds fast. Both `voicecat-server` and `voicecat-admin` are
|
||
copied into the runtime image.
|
||
- `docker-compose.yml` — single-service compose file with `restart: unless-stopped`,
|
||
named volume `voicecat-data`, and port mappings for TCP+UDP 8384. `command:` shows
|
||
how to set `--name`.
|
||
- `.dockerignore` — excludes `.git/`, `build/`, `clients/` (Swift/C# code), `docs/`,
|
||
markdown, editor config; build context is just `core/`, `server/`, `tools/`, `cmake/`,
|
||
and the three root CMake/vcpkg files.
|
||
- `deploy/linux/voicecat.service` — hardened systemd unit (non-root, `ProtectSystem`,
|
||
`NoNewPrivileges`, `AmbientCapabilities=CAP_NET_BIND_SERVICE`) for bare-metal deploys.
|
||
- Multi-arch: `docker buildx build --platform linux/amd64,linux/arm64 .` works without
|
||
any triplet override — `cmake/voicecat-toolchain.cmake` auto-detects from the host
|
||
arch cmake sees inside the buildx container.
|
||
- Quick start: `docker compose up -d` (or `docker run -d -p 8384:8384/tcp -p
|
||
8384:8384/udp -v voicecat-data:/data voicecat`). First run auto-generates identity
|
||
+ cert + DB; check logs for fingerprint + admin password.
|
||
- **GitHub Actions** (`.github/workflows/build-linux.yml`): primary cross-platform
|
||
binary build path — amd64 uses `ubuntu-24.04`, arm64 uses `ubuntu-24.04-arm`
|
||
(native, not QEMU). Triggers on push to main (when C++/cmake files change) and
|
||
manually via `workflow_dispatch`. Downloads land as 90-day artifacts.
|
||
`scripts/build-linux-binaries.sh` is the local Docker fallback (needs ~10–15 GB
|
||
free disk; suits Linux dev machines, not Windows Docker Desktop).
|
||
|
||
- **Done (2026-06-21):** **Fix permanent voice-loss bug + harden the UDP media path (protocol v2).**
|
||
Field report: two iOS users lost all audio mid-call after a bad-network blip and could not
|
||
recover even by restarting the apps. Root causes found in the UDP media path:
|
||
1. **Anti-replay window poisoned by unauthenticated packets (the trigger).**
|
||
`SodiumMediaCrypto::open()` advanced `recv_highest_` from the plaintext header `seq`
|
||
*before* verifying the AEAD tag and never rolled it back on failure. One corrupted/forged
|
||
frame (a bit-flip on flaky wifi) shoved the high-water mark far ahead, after which every
|
||
legitimate frame was rejected as "too old" — permanently. Fixed by reordering to
|
||
replay-check → authenticate → update (RFC 3711 §3.3): the window is now touched only after
|
||
a successful tag check. Regression test in `test_media_aead.cpp`
|
||
(`test_corrupted_seq_does_not_poison_window`) — fails on the old code, passes now.
|
||
2. **16-bit seq wrap with no rollover counter.** The wire header carried only the low 16 bits
|
||
of the nonce counter (zero-extended on receive); after 65,536 frames the reconstructed
|
||
nonce diverged and all frames failed auth. **Wire format widened to a full u64 seq**
|
||
(`voice_frame.h`: header 14 → 20 bytes, `seq` u16 → u64; `crypto.cpp`, `client.cpp`,
|
||
`media_relay.cpp` updated; `JitterBuffer::Frame::seq` widened). This is a **versioned wire
|
||
change → `VOICECAT_PROTOCOL_VERSION` 1 → 2**; the `Hello` handshake rejects on mismatch
|
||
(`conn_session.cpp`). The voice frame is parsed only in `core/`+`server/`+`tests/`, so the
|
||
Swift/C# clients need only a rebuild — no parser changes.
|
||
3. **Server leaked UDP state on disconnect.** `SessionRegistry::unregister_session()` now also
|
||
frees `udp_endpoints_`/`udp_tokens_`/`ssrc_to_session_` (scan-and-erase by session id).
|
||
4. **Diagnostics.** `MediaRelay` now emits rate-limited dropped-frame counters
|
||
(unmapped-endpoint / no-recv-crypto / open-failed) so a wedged media path is observable.
|
||
- **Verified:** `cmake --build --preset dev` clean; `ctest --preset dev -E external_pcm`
|
||
**22/22 pass** (incl. `m2_voice` e2e relay + the two new AEAD regressions). `external_pcm`
|
||
still aborts on the **pre-existing** CoreAudio shutdown mutex race (confirmed identical on a
|
||
clean baseline checkout under the same harness — unrelated to these changes). Docs updated:
|
||
`voice.md` §2 (header), `protocol.md` (v2 + negotiation), `security.md` (authenticate-then-advance).
|
||
|
||
- **Done (2026-06-21):** **Expose all channel codec params + guest nickname in every client.**
|
||
- **DRED everywhere + ABI fix.** `dred` (Opus 1.6 Deep REDundancy) existed in the C ABI
|
||
(`vc_audio_config.dred`) and proto but was absent from *both* client marshaling layers — a
|
||
latent ABI mismatch: Swift `AudioConfig` and the C# `VcAudioConfigNative` blittable struct
|
||
were each one `int` short of the native struct passed to `vc_create_channel`/`vc_edit_channel`.
|
||
Added `dred` through Swift (`Models.swift`, `Marshaling.swift`, `VoiceCatClient.toNative`) and
|
||
C# (`Structs.cs`, `Models.cs`, `Marshaling.cs`, `VoiceCatClient.cs`).
|
||
- **Windows:** added the one missing DRED checkbox to `ChannelEditDialog` (all other params
|
||
were already present).
|
||
- **macOS:** `ChannelEditSheet` now exposes the previously-hidden params — application profile,
|
||
sample rate, expected packet loss, complexity, and DRED (was only stereo/bitrate/frame/FEC/DTX).
|
||
- **iOS:** `ChannelEditView` was name+topic only; rebuilt into a full create **and edit** form
|
||
(General: name/topic/parent/password/max-users/sort-order; Audio: stereo/bitrate/sample-rate/
|
||
frame/application/packet-loss/complexity/FEC/DTX/DRED). Added `SessionState.editChannel` and an
|
||
"Edit" swipe action (admins) in `ChannelTreeView` + `ChannelBrowserView` (iOS previously had no
|
||
edit-channel UI at all). Note: the channel list doesn't carry the current audio config, so on
|
||
edit the audio fields start from codec defaults — same limitation as macOS/Windows.
|
||
- **Guest nickname.** Guests could not set a display name on iOS *or* macOS (the field was
|
||
absent/disabled; only Windows had it). Added a dedicated `nickname` to `SavedServer` on both
|
||
(backward-compatible Codable), a Nickname field shown in Guest mode (`AddServerView` /
|
||
`AddServerSheet`), and wired the guest auth path to use it (`AppState`, `ConnectWindowController`).
|
||
- **Verified:** `xcodebuild` Debug — macOS BUILD SUCCEEDED; iOS (sim, `ARCHS=arm64`) BUILD
|
||
SUCCEEDED. Core `ctest --preset dev` 22/23 (only `external_pcm` aborts on a pre-existing
|
||
shutdown mutex race; no C++ was changed). Windows C# not buildable on macOS — changes reviewed.
|
||
|
||
- **Done (2026-06-21):** **iOS iPhone-layout UX fixes.** (1) Channels are now a **drill-down**
|
||
on iPhone — new `ChannelBrowserView` (root list of top-level channels) → `ChannelDetailView`
|
||
(people in the channel + sub-channels + an explicit "Join Channel" button with password
|
||
prompt). The iPad 3-column `NavigationSplitView` is unchanged. (2) Extracted a self-contained
|
||
`UserRow` (context menu + sheets) from `UserListView` so admin actions are reused in the
|
||
drill-down. (3) Fixed the **off-screen chat compose box**: `MainView` now places
|
||
`VoiceControlsView` via `.safeAreaInset(edge: .bottom)` instead of a floating `.overlay`, so
|
||
it reserves layout space above the tab bar and cooperates with keyboard avoidance. (4)
|
||
**Collapsed Activity into Chat** like macOS/Windows: `ChatView` renders a merged, time-sorted
|
||
timeline of `messages` + `activityLog` (activity rows in gray); the separate Activity tab and
|
||
`ActivityLogView.swift` are removed. `xcodebuild` Debug for `generic/platform=iOS` BUILD
|
||
SUCCEEDED (sim slice still arm64-only → simulator run N/A). Next: on-device check of the
|
||
drill-down + compose box + unified timeline.
|
||
|
||
- **Done (2026-06-22):** **Windows exclude mode is now a real native exclude + self-echo
|
||
removal.** The "All apps except selected" mode previously captured the *complement of a frozen
|
||
app snapshot* in INCLUDE mode (missed late-launched apps, system sounds; wasted captures on
|
||
silent windows). It now opens a **single `ProcessLoopbackCapture` in EXCLUDE mode**
|
||
(`AUDIOCLIENT_PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE`) of the one chosen app — true
|
||
system-mix-minus-one, dynamic. `AppAudioPickerDialog` enforces single-selection in exclude
|
||
mode (the API takes one target PID). Added an **"Exclude VoiceCat's own audio (prevents echo)"**
|
||
checkbox (default on, entire-desktop only) that routes the desktop capture through the same
|
||
EXCLUDE path targeting `Environment.ProcessId`, killing the whole-device self-echo loop.
|
||
Touched `ProcessAudioMixer.cs` (`ResolveCaptures`), `AppAudioPickerDialog.cs`, `MainForm.cs`,
|
||
`AudioSessionEnumerator.cs` (`EntireDesktop(bool ExcludeSelf)`); docs in voice.md §9. No C++ /
|
||
ABI changes. `dotnet build` clean. **Still to verify on-device:** exclude actually silences
|
||
the chosen app while the rest plays, late-launched apps appear without restart, and the
|
||
self-exclude checkbox removes the echo.
|
||
|
||
- **Done (2026-06-21):** **Screen-audio sharing on macOS + iOS.** macOS uses ScreenCaptureKit
|
||
(`ScreenAudioCapture.swift`) → `vc_stream_feed_pcm`; iOS uses a ReplayKit Broadcast Upload
|
||
Extension (`VoiceCatBroadcast`) that forwards captured `.audioApp` PCM through a shared App
|
||
Group SPSC ring (`BroadcastAudioRing.swift`) to the host's `BroadcastAudioPump`, which owns
|
||
the `SCREEN_AUDIO` stream and feeds it — single session, no creds on disk. No C++ changes
|
||
(the core was already ready). macOS `xcodebuild` Debug BUILD SUCCEEDED; iOS app + extension
|
||
build for device (the xcframework sim slice is arm64-only, so x86_64-simulator link is N/A).
|
||
Next: on-device end-to-end verification (two clients hear the shared audio; iOS broadcast
|
||
start/stop). NOTE: `ctest --preset dev` is 22/23 — `external_pcm` passes its assertions but
|
||
aborts at shutdown (`mutex lock failed`), a **pre-existing** teardown crash unrelated to this
|
||
change (no C++ was modified).
|
||
|
||
- **Done (2026-06-21):** **macOS per-app screen-audio selection.** Before sharing, a new
|
||
`ScreenSharePickerSheet` lets the user choose scope — share Everything / Only selected apps /
|
||
All except selected apps — plus a first-class **"Exclude screen reader (VoiceOver) audio"**
|
||
toggle. `ScreenAudioCapture` now takes a `ScreenAudioSelection` and builds the matching
|
||
`SCContentFilter` (`including:` / `excludingApplications:`); app list comes from
|
||
`SCShareableContent`. macOS `xcodebuild` Debug BUILD SUCCEEDED. iOS deliberately untouched —
|
||
ReplayKit only delivers the mixed system stream, so per-app/VoiceOver filtering is impossible
|
||
there (documented in voice.md §9). **Still to verify on-device:** which process actually
|
||
carries VoiceOver speech (VoiceOver app vs. `com.apple.speech.speechsynthesisd`) — the exclude
|
||
set covers both candidates in `ScreenAudioCapture.screenReaderBundleIDs`; confirm exclusion
|
||
actually silences it in a real share.
|
||
|
||
- **Done (2026-06-20):** **macOS client UI overhaul** — mirrors the Windows client's UI
|
||
overhaul (commit 97fa659 + 540ec13), adapted to Mac-native conventions. Also fixed and
|
||
verified the previously-uncompiled Swift changes from the external PCM feed/tap commit
|
||
(615d2a8). The main window is now just toolbar + channels + users + chat; audio device
|
||
settings (input mode, VAD, PTT key, device picker, level meter) moved to a modeless
|
||
Settings window (⌘,). Details in M5 section below. `swift test` 10/10; `xcodebuild` Debug
|
||
+ Release BUILD SUCCEEDED with 0 Swift warnings.
|
||
Next: live manual verification (toolbar toggles, unified log colors, PM windows, channel
|
||
counts, volume slider, settings window); then iOS ReplayKit and macOS ScreenCaptureKit
|
||
consumers of `vc_stream_feed_pcm`.
|
||
|
||
- **Awaiting on-device verification:** **iOS stereo mic kills headphone/A2DP output — REAL
|
||
root cause found & fixed** (2026-06-20, on Windows; verify on Mac). All prior "fixes" (the
|
||
2026-06-19 entries below) targeted the Swift `IOSAudioRouter` on the false premise that
|
||
"miniaudio does NOT touch AVAudioSession on iOS." **It does.** The core opened its miniaudio
|
||
devices with `ma_device_init(nullptr, ...)`; with a NULL context, miniaudio (0.11.25) runs an
|
||
iOS "hack" (`miniaudio.h` ~44057) that picks a session category by device type, then
|
||
`ma_context_init__coreaudio` (~36552) calls `setCategory()` + `setActive()` on **every device
|
||
open** — capture → `AVAudioSessionCategoryRecord` with **zero options**. That wiped the
|
||
`.playAndRecord` category, the mode, and `.allowBluetoothA2DP`/`.mixWithOthers`/`.allowAirPlay`
|
||
that `IOSAudioRouter` had just configured → headphone/A2DP (and even wired) output died. The
|
||
stereo presets broke worst because they depend on the A2DP output route the wipe removed.
|
||
TeamTalk never hits this: its SDK opens RemoteIO/VPIO AudioUnits directly and leaves the
|
||
session entirely to the app (`UtilSound.swift`); miniaudio insists on managing it.
|
||
- **Fix (core, cross-platform safe):** `AudioEngine` now owns a `ma_context` built by
|
||
`make_context_config()` with `coreaudio.sessionCategory = ma_ios_session_category_none` +
|
||
`noAudioSessionActivate`/`noAudioSessionDeactivate = MA_TRUE`, and passes it to **all**
|
||
`ma_device_init` calls (playback, capture, loopback) and to `enumerate_devices`'s context.
|
||
miniaudio now never touches AVAudioSession; the Swift `IOSAudioRouter` is the sole owner
|
||
(session is already activated on connect in `AppState.swift:authResult`, before any device
|
||
opens, so removing miniaudio's self-activation is safe). Context is lazily inited in
|
||
`start()`, reused across restarts, uninited in `~AudioEngine`.
|
||
Files: `core/src/audio/audio_engine.{h,cpp}`.
|
||
- **TEMP diagnostics (remove after verification):** `AudioSessionManager.logSessionState(_:)`
|
||
logs category/mode/options/route; called after `ensureSessionActive`, on every route change,
|
||
and on `.streamStarted` (right after the core opens its devices). On Mac, watch the log when
|
||
joining voice with the Stereo Mic preset: category must stay `…PlayAndRecord` with
|
||
`allowBluetoothA2DP` and the output route must remain the headphones/A2DP device — NOT flip
|
||
to `…Record`. If confirmed, delete the `logSessionState` calls + method and the prior
|
||
band-aid comments in `IOSAudioRouter`/`audio_engine.cpp` can be trimmed.
|
||
- **Verified on Windows:** `cmake --build --preset dev` clean, `ctest --preset dev` 23/23
|
||
(22/22 prior + `test_external_pcm` new binary). iOS build & on-device run still to be done by the user on the Mac.
|
||
|
||
- **Done (2026-06-20):** **External PCM feed/tap API (`vc_stream_feed_pcm` +
|
||
`vc_set_pcm_sink`)** — see detail in M5 section below. `ctest --preset dev` 23/23 (was 22/22 + 1 new test binary with 3 sub-tests).
|
||
Next: iOS ReplayKit and macOS ScreenCaptureKit consumers of this API. A public,
|
||
documented API for driving audio streams with externally-provided PCM instead of (or in
|
||
addition to) miniaudio's hardware device. Motivated by four concrete use cases — all in our
|
||
roadmap — that the current "miniaudio owns the device" model can't serve:
|
||
1. **ReplayKit Broadcast Upload Extension (iOS `SCREEN_AUDIO`)** — the extension is a
|
||
*separate process* with a ~50 MB memory cap and can't link the full `AudioEngine`
|
||
(`ma_device`, capture/playback threads). It needs to feed `CMSampleBuffer` audio (system
|
||
app audio) into the encode path without any audio hardware. The current plan in
|
||
`docs/voice.md §9` says the extension links "a minimal slice of the core (Opus encode +
|
||
media send only)" — a public feed-PCM API *is* that minimal slice. The extension links
|
||
Opus + the feed entry point, no `ma_device` needed.
|
||
2. **ScreenCaptureKit (macOS `SCREEN_AUDIO`)** — `SCStream` delivers `CMSampleBuffer` in a
|
||
callback; convert to int16 and feed. No need to route through miniaudio's device layer.
|
||
**This is how macOS screen-audio actually gets implemented** — today it does NOT work:
|
||
`VOICECAT_HAS_LOOPBACK` is Windows-only (`core/CMakeLists.txt:88-95`), so on macOS
|
||
`AudioEngine::start_loopback_capture()` hits the `#else` stub (`audio_engine.cpp:647-649`)
|
||
and returns `false`. The macOS client's "Share Screen Audio" button
|
||
(`MainWindowController.swift:800-816`) calls `startStream(.screenAudio)` which announces
|
||
the stream to peers but captures **zero audio** — peers hear silence. The button is left
|
||
in place (not touched per user request); it'll work once this API + a ScreenCaptureKit
|
||
tap ship on Mac.
|
||
3. **Bots** — music bot, TTS bot, radio relay, transcription bot. They create a
|
||
`SCREEN_AUDIO`/`AUX_DEVICE` stream and feed synthesized or decoded PCM via the feed API.
|
||
No audio hardware required — runs headless on a server. Today the only way to feed
|
||
external PCM is `vc_test_inject_capture` (TEST-ONLY, name signals "don't ship this") or
|
||
re-implementing Opus encode + AEAD + UDP framing yourself (~500 lines of duplicated
|
||
crypto/codec code per consumer).
|
||
4. **Custom clients / accessibility** — soundboard, DAW integration, TTS of incoming chat,
|
||
recording/transcription of remote audio. Need either feed (send) or tap (receive) or
|
||
both.
|
||
|
||
**What we already have (input half, gated as test-only):** `vc_test_inject_capture
|
||
(stream_id, pcm, samples)` (`voicecat.h`, `client.cpp:1452`) feeds raw int16 PCM into the
|
||
encode pipeline via `AudioEngine::inject_capture(kind, pcm, n)`. It works for any stream
|
||
kind, supports multiple concurrent injection taps (one ring buffer per local kind), and
|
||
goes through the full encode → AEAD → UDP path. The encode path already handles
|
||
`channels == 1 || 2` (proven by the WASAPI stereo loopback work, 2026-06-17 entry below).
|
||
The only problems: it's marked TEST-ONLY in the header, the name signals "don't use this in
|
||
production," and it hardcodes mono (no `channels` parameter).
|
||
|
||
**What's missing (output half):** today decoded remote audio is mixed and pushed to the
|
||
miniaudio playback device (`on_playback`). There's no way for an external consumer to
|
||
intercept the decoded PCM of a specific remote stream — it all goes to the hardware device.
|
||
A bot that wants to record, transcribe, or re-broadcast remote audio has no hook.
|
||
|
||
**Plan (API design — clean, append-only, no struct changes, ABI-stable):**
|
||
- **`vc_stream_feed_pcm`** — promote `vc_test_inject_capture` to a public, documented API
|
||
and add a `channels` parameter:
|
||
```c
|
||
/* External PCM feed — replaces the hardware capture device for this stream. Caller
|
||
provides interleaved int16 PCM at the stream's sample rate. The core frames it,
|
||
encodes (Opus), seals (AEAD), and sends (UDP). Works for any stream kind
|
||
(MIC/SCREEN_AUDIO/AUX_DEVICE). The stream must be started first (vc_stream_start);
|
||
this just replaces the capture source. channels = 1 (mono) or 2 (stereo interleaved).
|
||
Thread-safe; may be called from any thread including audio callbacks. */
|
||
vc_result vc_stream_feed_pcm(vc_client* c, uint32_t stream_id,
|
||
const int16_t* pcm, size_t samples_per_channel,
|
||
uint32_t channels);
|
||
```
|
||
- **`vc_set_pcm_sink`** — symmetric output side: receive decoded remote audio as int16 PCM
|
||
instead of (or in addition to) the hardware playback device:
|
||
```c
|
||
/* External PCM tap — receive decoded, mixed remote audio as int16 PCM. The callback
|
||
fires on the audio thread with the mixed output for a specific remote stream. Pass
|
||
cb=NULL to disable (default: disabled, hardware playback only). When enabled, PCM is
|
||
delivered to the sink AND the hardware device (dual output) so a bot can record
|
||
without disabling local monitoring. user_id+stream_id identify the source stream.
|
||
The callback MUST NOT block — copy what you need and return (same contract as
|
||
vc_callbacks.on_event). */
|
||
typedef void (*vc_pcm_sink_cb)(void* user, uint32_t user_id, uint32_t stream_id,
|
||
const int16_t* pcm, size_t samples_per_channel,
|
||
uint32_t channels, uint32_t sample_rate);
|
||
vc_result vc_set_pcm_sink(vc_client* c, vc_pcm_sink_cb cb, void* user);
|
||
```
|
||
- **Core changes:**
|
||
- `core/include/voicecat.h` — add `vc_pcm_sink_cb` typedef + the two function
|
||
declarations (append-only, after `vc_test_inject_capture`). Full doc comments on both
|
||
(contract, thread-safety, lifetime, use cases).
|
||
- `core/src/voicecat.cpp` — thin C trampolines → `vc_client::stream_feed_pcm` /
|
||
`set_pcm_sink`.
|
||
- `core/src/core/client.{h,cpp}` — `stream_feed_pcm`: validates `stream_id`, looks up
|
||
the `LocalStream`'s kind, calls `audio_engine_.inject_capture(kind, pcm, n)` (existing
|
||
path) with the channel count forwarded. `set_pcm_sink`: stores the callback + user
|
||
pointer; `on_playback` (or a new fan-out in the mixer) invokes it per remote stream
|
||
alongside the existing hardware write. Keep `vc_test_inject_capture` as a deprecated
|
||
alias calling `stream_feed_pcm(..., channels=1)` for source compatibility.
|
||
- `core/src/audio/audio_engine.{h,cpp}` — `inject_capture` already exists per-kind; add
|
||
a `channels` parameter to the ring-buffer write path (or a parallel stereo-aware
|
||
variant). The encode path in `client.cpp::on_capture_frame` already handles
|
||
`channels==2` via the stereo encode branch — just plumb the value through. For the
|
||
sink: add a `pcm_sink_` member (callback + user); in `on_playback` after mixing, if the
|
||
sink is set, copy the mixed PCM for the current stream and invoke the callback. The
|
||
copy must stay off the RT-critical path — document the non-blocking contract.
|
||
- **Skeleton stub path:** update `client.cpp`'s `#else` (no-deps) stub section to add
|
||
`vc_stream_feed_pcm`/`vc_set_pcm_sink` returning `VC_ERR_NOT_IMPLEMENTED` — keeps the
|
||
skeleton preset green.
|
||
- **Swift `VoiceCatCore`:** add `feedPcm(streamId:pcm:samplesPerChannel:channels:)` and
|
||
`setPcmSink(_:user:)` (the Swift wrapper around `vc_pcm_sink_cb` — a
|
||
`@convention(c)` closure + `Unmanaged` context, mirroring `Callbacks.swift`). Wraps both
|
||
new ABI functions.
|
||
- **C# `VoiceCat.Interop`:** add `StreamFeedPcm(streamId, pcm, samples, channels)` (with
|
||
`int16[]` marshaling) and `SetPcmSink` (delegates via `[UnmanagedCallersOnly]` thunk,
|
||
mirroring the event-callback pattern). Wraps both new ABI functions.
|
||
- **Tests:**
|
||
- `tests/test_external_pcm.cpp` (new) — `test_feed_pcm_round_trip`: two clients, A feeds
|
||
a known mono sine wave via `vc_stream_feed_pcm` on a MIC stream, B receives via the
|
||
normal decode path and asserts energy matches. `test_feed_pcm_stereo`: same with
|
||
`channels=2`, assert L≠R end-to-end (mirrors the WASAPI loopback stereo test).
|
||
`test_pcm_sink`: B sets a `vc_pcm_sink_cb`, A feeds PCM, assert the sink callback
|
||
receives the decoded PCM with matching energy. All headless, no audio hardware.
|
||
- `clients/apple/Tests/VoiceCatCoreTests/` — Swift wrapper round-trip for `feedPcm`.
|
||
- `clients/windows/VoiceCat.Interop.Tests/` — C# wrapper round-trip.
|
||
- **Docs:**
|
||
- `docs/architecture.md §4` — new subsection on external PCM feed/tap: the contract
|
||
(caller provides interleaved int16 at the stream's sample rate; core frames/encodes/
|
||
seals/sends for feed; core decodes/mixes/delivers for sink; sink callback must not
|
||
block), the use cases (ReplayKit, ScreenCaptureKit, bots, custom clients), and the
|
||
relationship to `vc_test_inject_capture` (deprecated alias).
|
||
- `docs/voice.md §9` — update the iOS ReplayKit and macOS ScreenCaptureKit rows: both
|
||
now consume `vc_stream_feed_pcm` instead of a "minimal slice of the core." Update the
|
||
iOS detail bullets: the extension links Opus + `vc_stream_feed_pcm` (not a parallel
|
||
media stack). Add a macOS ScreenCaptureKit note: convert `CMSampleBuffer` → int16,
|
||
feed via `vc_stream_feed_pcm` — this is how macOS screen-audio actually ships.
|
||
- `docs/protocol.md` — no protocol changes (the feed/sink are client-local; the wire
|
||
format is identical whether PCM came from miniaudio or an external source). Note this
|
||
explicitly.
|
||
- `docs/roadmap.md` — add a milestone entry; update the iOS ReplayKit and macOS
|
||
ScreenCaptureKit pending items to reference `vc_stream_feed_pcm`.
|
||
- **Implementation order:**
|
||
1. C ABI + core (`voicecat.h`, `voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`) +
|
||
skeleton stub. Verify `ctest --preset dev` green.
|
||
2. `tests/test_external_pcm.cpp` — the three behavior tests. Verify green.
|
||
3. Swift `VoiceCatCore` wrapper + `VoiceCatCoreTests` round-trip.
|
||
4. C# `VoiceCat.Interop` wrapper + `VoiceCatClientSmokeTests` round-trip.
|
||
5. Docs (`architecture.md`, `voice.md`, `protocol.md`, `roadmap.md`, header comments).
|
||
6. **Then** ReplayKit (iOS) and ScreenCaptureKit (macOS) become ~100-line consumers of
|
||
this API instead of parallel media stacks.
|
||
- **Verification:** `ctest --preset dev` green (3 new tests); `swift test` green; `dotnet
|
||
test` green; `xcodebuild` (skeleton) green. The feed/sink tests are fully headless — no
|
||
audio hardware, no simulator, no device — so they run in CI on every platform.
|
||
- **Files to touch:**
|
||
- Core C++: `core/include/voicecat.h`, `core/src/voicecat.cpp`,
|
||
`core/src/core/client.{h,cpp}`, `core/src/audio/audio_engine.{h,cpp}`.
|
||
- Tests: `tests/test_external_pcm.cpp` (new), `tests/CMakeLists.txt`.
|
||
- Swift: `clients/apple/Sources/VoiceCatCore/VoiceCatClient.swift`,
|
||
`clients/apple/Sources/VoiceCatCore/Callbacks.swift`,
|
||
`clients/apple/Tests/VoiceCatCoreTests/ExternalPcmTests.swift` (new).
|
||
- C#: `clients/windows/VoiceCat.Interop/VoiceCatClient.cs`,
|
||
`clients/windows/VoiceCat.Interop/NativeMethods.cs`,
|
||
`clients/windows/VoiceCat.Interop.Tests/ExternalPcmTests.cs` (new).
|
||
- Docs: `docs/architecture.md`, `docs/voice.md`, `docs/protocol.md`, `docs/roadmap.md`.
|
||
- **ABI stability:** append-only — two new functions + one new typedef, no existing
|
||
structs/enums changed. `vc_test_inject_capture` stays as a deprecated alias for source
|
||
compatibility. Treat as a deliberate, versioned ABI event per `docs/protocol.md §8`.
|
||
- **Relationship to the iOS audio routing plan:** orthogonal. The iOS routing layer controls
|
||
*which hardware route* miniaudio opens (AVAudioSession config in Swift). This plan is about
|
||
*bypassing miniaudio's hardware entirely* (external PCM feed/tap). Both ship; they don't
|
||
conflict. ReplayKit/ScreenCaptureKit consume this API; the iOS routing layer controls the
|
||
*mic* path which still uses miniaudio's device.
|
||
|
||
---
|
||
|
||
## Recent completed work
|
||
|
||
All items below are `[x]` done; `ctest --preset dev` 26/26 on Windows after all.
|
||
|
||
- **Per-channel sample_rate as a bandwidth cap** (2026-06-22): the channel `sample_rate` field
|
||
was inert (the codec is pinned to 48 kHz). Made it meaningful without changing the 48 kHz
|
||
clock: it's carried as `OpusParams::max_bandwidth_hz` and applied via `OPUS_SET_MAX_BANDWIDTH`
|
||
in `OpusEncoder::init` (8000→narrowband … 48000→full). Made it **channel-authoritative** on
|
||
the server (`conn_session.cpp` no longer overrides effective `sample_rate` with the client's
|
||
always-48000 request — like `frame_ms`/`mode`). `vc_get_stream_audio_config` now reports the
|
||
channel's configured rate for own streams too. New ctest `channel_samplerate`: a 7 kHz tone is
|
||
attenuated ~1000× on an 8 kHz channel vs a 48 kHz channel. Files: `opus_codec.{h,cpp}`,
|
||
`client.cpp`, `server/src/conn_session.cpp`, `docs/voice.md`, `tests/test_channel_samplerate.cpp`,
|
||
`tests/CMakeLists.txt`. (Future: a true non-48k stack is possible but unnecessary — 48 kHz is
|
||
what nearly all hard/software runs at; the bandwidth cap covers the narrowband use case.)
|
||
|
||
- **Non-20ms channel frame_ms fix** (2026-06-22): the AudioEngine capture clock is fixed at
|
||
48 kHz / 20 ms (960-sample frames), but a channel may set any Opus `frame_ms` (2.5…60 ms,
|
||
docs/voice.md §3) and the server enforces it unclamped. The send path handed the engine's
|
||
960-sample frame straight to an encoder configured for the channel's window — silently
|
||
ignoring `frame_ms > 20` and **breaking `frame_ms < 20` entirely** (receiver sized its decode
|
||
buffer too small → `OPUS_BUFFER_TOO_SMALL` → dead audio). Affected the hardware mic AND
|
||
`vc_stream_feed_pcm`. Fix: `vc_client::on_capture_frame` now reframes each captured/fed block
|
||
to `ls.frame_samples` via a per-`LocalStream` accumulator (pre-sized at announce, no RT-thread
|
||
alloc) before `encode_and_send_frame`; the 20 ms case stays a zero-copy fast path. Also pinned
|
||
the codec to 48 kHz internally in `opus_params_from_audio_config` (was honoring a non-48k
|
||
effective sample_rate against a 48k PCM clock). New ctest `frame_ms_reframe` (40 ms accumulate
|
||
+ 10 ms split round trips). Files: `client.{h,cpp}`, `voicecat.h` (feed doc), `docs/voice.md`,
|
||
`tests/test_frame_ms_reframe.cpp`, `tests/CMakeLists.txt`.
|
||
|
||
- **External PCM feed/tap API** (2026-06-20): `vc_stream_feed_pcm` + `vc_set_pcm_sink` shipped.
|
||
Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public, stereo-capable API.
|
||
Adds symmetric PCM sink on the playback thread. Swift wrapper (`feedPcm`/`setPcmSink` in
|
||
`VoiceCatClient.swift`, 4 XCTest smoke tests). C# wrapper (`StreamFeedPcm`/`SetPcmSink` in
|
||
`VoiceCatClient.cs` + `NativeMethods.cs`, 4 xUnit smoke tests in `ExternalPcmTests.cs`).
|
||
Three new headless C++ ctests. Docs: architecture.md §4 new subsection, voice.md §9 updated,
|
||
protocol.md §8 explicit no-protocol-change note, roadmap.md M5 entry. Files: `voicecat.h`,
|
||
`voicecat.cpp`, `client.{h,cpp}`, `audio_engine.{h,cpp}`, `tests/test_external_pcm.cpp`,
|
||
`tests/CMakeLists.txt`, Swift + C# wrappers.
|
||
|
||
- **iOS A2DP + stereo root cause fix** (2026-06-20): miniaudio's NULL-context `ma_device_init`
|
||
was calling `AVAudioSession setCategory(Record)` on every device open, wiping the session
|
||
config `IOSAudioRouter` had set. Fixed by sharing a `ma_context` with
|
||
`sessionCategory=none` + `noAudioSessionActivate/Deactivate=MA_TRUE` — miniaudio never
|
||
touches `AVAudioSession`; `IOSAudioRouter` is the sole owner. Files: `audio_engine.{h,cpp}`.
|
||
|
||
- **iOS audio routing overhaul** (2026-06-19): Full `IOSAudioRouter` singleton drives all
|
||
`AVAudioSession` config before miniaudio opens devices. Fixed stereo mic polar-pattern setup
|
||
(WWDC20 recipe: `setPreferredInput` + `setInputDataSource` + `.stereo` polar pattern + no
|
||
`setPreferredInputNumberOfChannels`). Added `vc_audio_restart` ABI (full stop+reinit for
|
||
close→reconfigure→reopen ordering). Added `vc_set_capture_channels` ABI (core stereo-mic
|
||
support). AVAudioSession activated proactively on `.authResult`, not lazily on
|
||
`.streamStarted`. Join/Leave Voice button added (parity with macOS). Channel-id sync fixed
|
||
(mic button was permanently dimmed). iOS deployment target raised to 18.0.
|
||
|
||
- **iOS SwiftUI client** (2026-06-19): `VoiceCatiOS.xcodeproj` at `clients/apple/iOS/`.
|
||
Full feature parity with macOS/Windows: saved server list (JSON + Keychain, App Group
|
||
`group.cat.voice.VoiceCat`), TOFU, connect flow, channel tree, user list with context menus,
|
||
chat, admin sheets, voice controls, settings. `xcodebuild` → BUILD SUCCEEDED.
|
||
|
||
- **macOS AppKit client** (2026-06-18): `VoiceCatMac.xcodeproj` at `clients/apple/macOS/`.
|
||
Fixed compile errors (`NSAccessibility` call-site arg order, `StreamSummary.id` vs
|
||
`.streamId`) and linker issues (`OTHER_LDFLAGS = -lc++`, `ONLY_ACTIVE_ARCH = YES` for
|
||
Release). Debug + Release both BUILD SUCCEEDED.
|
||
|
||
- **Swift `VoiceCatCore` package + XCFramework** (2026-06-18): Shared Swift wrapper at
|
||
`clients/apple/`. `build-xcframework.sh` merges `libvoicecat.a` + 107 vcpkg static deps into
|
||
a fat `.a` via `libtool -static`. 6/6 Swift tests green (real server, mirrors C# Interop
|
||
tests). Supports macOS-arm64 + iOS-arm64 + iOS-sim slices.
|
||
|
||
- **macOS port validated** (2026-06-18): 21/21 on macOS. Three cross-platform bugs fixed:
|
||
missing `<netdb.h>` in POSIX test branch; SIGPIPE kills (added `SIG_IGN`); use-after-free of
|
||
Asio kqueue reactor on server shutdown (fixed `TcpAcceptor` shutdown/connection-drain
|
||
sequence).
|
||
|
||
- **CMake preset cleanup** (2026-06-18): `m1-dev`→`dev`, `dev`→`skeleton`, `m2-dev` dropped.
|
||
New `release`, `server-release` (stripped), `apple-dev`/`apple-ios`/`apple-ios-sim`. Cross-
|
||
platform triplet auto-resolved by `cmake/voicecat-toolchain.cmake`.
|
||
|
||
- **Disconnect, keepalive & reaper** (2026-06-18): Client sends `Ping` every 15 s; server
|
||
reaper drops sessions after 45 s; UDP `KEEPALIVE` every 5 s keeps NAT alive. `vc_disconnect`
|
||
sends graceful `Disconnect` proto. Stale-user LEFT broadcast on drop. PLC capped at ~2 s.
|
||
Three new tests: `test_disconnect_left`, `test_plc_cap`, `test_reaper_timeout`.
|
||
|
||
- **Stereo screen-audio loopback** (2026-06-17): WASAPI loopback opens in channel's
|
||
stereo/mono mode (was hardcoded mono). Real stereo flows end-to-end through loopback → encode
|
||
→ decode → mixer. New `test_loopback_stereo_capture`.
|
||
|
||
- **Windows screen-audio UI wired** (2026-06-17): `btnScreenShareToggle` in `MainForm.cs`.
|
||
No core/proto/ABI changes — all the plumbing was already there. `dotnet test` 4/4 green.
|
||
|
||
- **Bug fixes** (2026-06-16 – 2026-06-17):
|
||
- *AEAD nonce desync in SFU relay* — relay forwarded sender's `seq` verbatim; recipient
|
||
nonce reconstruction used the wrong counter. Fixed by rewriting the outgoing `seq` field
|
||
to the recipient's `peek_send_counter()`.
|
||
- *Playout clock free-ran* — `playout_ts` advanced even during VAD/PTT silence gaps,
|
||
eventually dropping all frames as too-late. Fixed with resync in `on_playback` via
|
||
`JitterBuffer::peek_front_ts()`.
|
||
- *Stale users after disconnect* — `ConnSession::close()` didn't broadcast `UserEvent::LEFT`
|
||
before erasing. Fixed; PLC cap added as defense-in-depth.
|
||
- *"Randomly bumped to Lobby"* — server excluded the actor from its own state-change
|
||
broadcasts. Fixed: `UserEvent::UPDATED` now goes to all clients including the actor.
|
||
- *Silent playback after join* — `opus_decode` received hardware callback frame count as
|
||
`max_samples` instead of the Opus frame size. Fixed with a decode ring buffer.
|
||
|
||
---
|
||
|
||
## Milestones (see [docs/roadmap.md](docs/roadmap.md) for full detail)
|
||
|
||
- [x] **M0 — Scaffolding** ✓ complete
|
||
- [x] **M1 — Control plane** ✓ complete (2026-06-15)
|
||
- [x] **M2 — Voice, single stream** ✓ complete (2026-06-16)
|
||
- [x] **M3 — Multi-stream & per-channel tuning** ✓ complete (2026-06-16)
|
||
- [x] **M4 — Native clients** — Windows WinForms ✓ (2026-06-17); macOS AppKit ✓ (2026-06-18); iOS SwiftUI ✓ (2026-06-19)
|
||
- [~] **M5 — Moderation, polish, beyond** (perms, bans, DRED; then file transfer, E2EE, …)
|
||
|
||
---
|
||
|
||
## M0 — Scaffolding ✓
|
||
|
||
Repo layout (`core/ server/ tools/ clients/ tests/`), CMake + vcpkg manifest, C ABI header
|
||
(`voicecat.h`), proto source of truth, core stubs for all six subsystems, `voicecat-server` +
|
||
`vccli` skeletons, smoke CTest, `.clang-format`/`.gitattributes`/`.gitignore`.
|
||
|
||
---
|
||
|
||
## M1 — Control plane ✓ (completed 2026-06-15)
|
||
|
||
**Exit criterion:** `test_m1_integration` — two clients authenticate over TLS 1.3 (guest +
|
||
Argon2id), exchange channel + private text. ~1 s.
|
||
|
||
`FrameCodec`, `TlsContext` (mbedTLS 1.3, ECDSA-P256 self-signed, TOFU pins TLS leaf-cert
|
||
SHA-256), `WorkerPool`, `Database` (SQLite + Argon2id), `ServerIdentityManager`,
|
||
`ConnSession` state machine, `SessionRegistry`, `vc_client` full M1 C ABI, `voicecat-admin`
|
||
CLI, dual-stack `TcpAcceptor`. **Key bug fixed:** `send_frame` double-framing — `encode_envelope`
|
||
was pre-framing the protobuf; fixed by passing raw protobuf bytes.
|
||
|
||
---
|
||
|
||
## M2 — Voice, single stream ✓ (completed 2026-06-16)
|
||
|
||
**Exit criterion:** `test_m2_voice` + `test_voice_client_abi` — two headless clients auth, bind
|
||
UDP, 50 Opus frames relayed + re-encrypted by SFU, B receives ≥25 and decrypts. ~4 s.
|
||
|
||
14-byte UDP voice header, `SodiumMediaCrypto` (ChaCha20-Poly1305 + 64-bit anti-replay),
|
||
`OpusEncoder`/`OpusDecoder` (FEC, PLC), `UdpMediaChannel`, `JitterBuffer`, `AudioEngine`
|
||
(miniaudio), `MediaRelay` SFU. **Key bug fixed:** `on_playback` passed hardware callback frame
|
||
count as `opus_decode` max_samples; fixed with a per-stream decode ring buffer.
|
||
|
||
---
|
||
|
||
## M3 — Multi-stream & per-channel tuning ✓ (completed 2026-06-16)
|
||
|
||
**Exit criterion:** `test_m3_multistream` — client A runs two concurrent streams (MIC +
|
||
SCREEN_AUDIO); B sees both; per-stream gain/mute/NR independent; effective Opus config matches
|
||
channel's server-enforced settings. ~2.4 s.
|
||
|
||
Fixed server `stream_id` counter bug (always wrote `1`). Per-channel `AudioConfig` populated
|
||
(Lobby: mono/24kbps/VOIP + DTX; Music Room: stereo/128kbps/AUDIO). `LocalStream` map,
|
||
`pending_announce_kind_`, `run_talk_timer()`, thread-join race in `teardown_voice()` fixed.
|
||
New C ABI: `vc_get_stream_audio_config`, `vc_test_inject_capture`.
|
||
|
||
---
|
||
|
||
## Post-M3 follow-up ✓ (completed 2026-06-16)
|
||
|
||
- **Device enumeration** — `vc_list_devices`/`vc_set_input_device`; opaque hex device ids;
|
||
`vc_free_device_list` now frees. Works pre-connect.
|
||
- **VAD/PTT gate** — `EnergyVadProcessor` (RMS threshold ~0.025, 300 ms hang-time);
|
||
`vc_set_input_mode`/`vc_set_push_to_talk`; MIC-only (SCREEN_AUDIO/AUX_DEVICE bypass).
|
||
- **True stereo playback** — `playback_channels=2`; stereo decoded L→L R→R in mixer; mono
|
||
upmixed L=R; hardware fallback to mono on failure.
|
||
- **WASAPI loopback** — `loopback_device_` with `ma_device_type_loopback`;
|
||
`VOICECAT_HAS_LOOPBACK` macro (Windows-only). `vccli --share-screen-audio`.
|
||
|
||
**Known deferred (still open):** AEC/NS/AGC (no working Windows/MSVC WebRTC APM build);
|
||
process-specific WASAPI loopback; RT-thread rule violation in `on_capture_frame` (mutex lock
|
||
on audio callback thread — pre-existing, needs lock-free ring-buffer refactor).
|
||
|
||
---
|
||
|
||
## M4 — Native clients ✓ (completed 2026-06-17 – 2026-06-19)
|
||
|
||
**Exit criterion:** `ctest --preset dev` 21/21 green; `dotnet build` 0 warnings; `xcodebuild`
|
||
BUILD SUCCEEDED (macOS + iOS); manually verified: connect, TOFU, channel tree, join, voice,
|
||
text, device pickers, level meter on each platform.
|
||
|
||
**New C ABI (additive):** `vc_list_channels`/`vc_list_users`/`vc_list_user_streams`,
|
||
`vc_join_channel`, `VC_EVENT_SERVER_IDENTITY` + `vc_confirm_server_identity`,
|
||
`vc_config::tofu_store_path`, `VC_INPUT_ALWAYS_ON`, `vc_set_vad_threshold`,
|
||
`vc_audio_suspend`/`vc_audio_resume`, `vc_audio_restart`, `vc_set_capture_channels`.
|
||
|
||
**Windows** (`clients/windows/`): `VoiceCat.Interop` (P/Invoke, `[UnmanagedCallersOnly]`),
|
||
`VoiceCat.App` (ConnectDialog, ServerIdentityDialog, MainForm with full M5 moderation UI,
|
||
PerUserTuningDialog, PttKeyCaptureDialog), `VoiceCat.Interop.Tests`. PTT is focus-scoped.
|
||
|
||
**macOS** (`clients/apple/macOS/VoiceCatMac.xcodeproj`): NSOutlineView channel tree,
|
||
NSTableView user list, NSTextView chat, voice controls, full VoiceOver accessibility, admin
|
||
menu, 17 Swift source files. `build-xcframework.sh` produces `VoiceCatCore.xcframework`.
|
||
|
||
**iOS** (`clients/apple/iOS/VoiceCatiOS.xcodeproj`): SwiftUI, `NavigationSplitView`/`TabView`,
|
||
`OutlineGroup` channel tree, `IOSAudioRouter` AVAudioSession driver, 24 Swift source files,
|
||
iOS 18.0 deployment target. App Group `group.cat.voice.VoiceCat` for Keychain sharing.
|
||
|
||
---
|
||
|
||
## M5 — Moderation, polish, and beyond [~] (in progress 2026-06-17)
|
||
|
||
**Exit criterion:** four ABI-level tests green (`test_m5_permissions`,
|
||
`test_m5_kick_ban_move_mute`, `test_m5_admin_accounts`, `test_m5_channel_crud`);
|
||
`vccli` can drive all moderation/admin/channel operations against a live server.
|
||
|
||
- [x] **Server-side moderation & permissions** — per-session `Permissions`, kick/ban/move/
|
||
server-mute, channel CRUD, DB schema v2 (`channels`, `bans`), BLAKE2b channel passwords.
|
||
- [x] **C ABI** — `vc_kick_user`, `vc_ban_user`, `vc_set_permission`, `vc_set_server_mute`,
|
||
`vc_move_user`, `vc_create_channel`, `vc_edit_channel`, `vc_delete_channel`,
|
||
`vc_create_account`, `vc_reset_password`, `vc_delete_account`, `vc_list_accounts`,
|
||
`vc_get_permissions`; events `VC_EVENT_GENERIC_RESULT`, `VC_EVENT_ACCOUNT_LIST`.
|
||
- [x] **Four M5 tests** passing — `ctest --preset dev` 21/21.
|
||
- [x] **vccli** M5 flags: `--kick`, `--ban`, `--move`, `--server-mute`/`-unmute`/`-deafen`/
|
||
`-undeafen`, `--set-permission`, channel CRUD, account CRUD, `--username`/`--password`.
|
||
- [x] **All three client UIs** (Windows WinForms, macOS AppKit, iOS SwiftUI) expose the full
|
||
M5 moderation and admin surface.
|
||
- [x] **Docs** — `docs/protocol.md`, `docs/security.md` kept in sync.
|
||
- [x] **DRED/audio-quality polish** — done (2026-06-20). `bool dred` added to `AudioConfig`
|
||
proto (field 11) and `vc_audio_config` C ABI. Encoder: `OPUS_SET_DRED_DURATION(2)` when
|
||
enabled (20 ms of ML redundancy per packet). Decoder: `OpusDREDDecoder` + per-stream
|
||
`OpusDRED` scratch pre-allocated; `JitterBuffer::try_copy_front_payload` peeks at the next
|
||
buffered packet on every PLC step; if DRED data is present, `opus_decoder_dred_decode`
|
||
reconstructs the lost frame — otherwise falls back to standard PLC. New test:
|
||
`test_dred_toggle` (ctest 22/22). Files: `voicecat.proto`, `voicecat.h`,
|
||
`opus_codec.{h,cpp}`, `audio_engine.{h,cpp}`, `client.cpp`, `session.{h,cpp}`.
|
||
- [x] **In-band FEC decoder wiring** — done (2026-06-22). The encoder set `OPUS_SET_INBAND_FEC`
|
||
all along, but the decoder never invoked it — the loss path went DRED → PLC, so FEC redundancy
|
||
was emitted (and paid for in bitrate) but never consumed. Wired the FEC recovery into
|
||
`AudioEngine::on_playback`'s loss branch between DRED and PLC: copy the next buffered packet
|
||
once, try DRED, else (if the stream negotiated FEC) `decode(next_pkt, …, fec=true)`, else PLC.
|
||
Added per-stream `RemoteStream::fec_enabled_`, captured from `OpusParams` in
|
||
`init_recv_stream`. Recovery priority is now **DRED → FEC → PLC**. ctest 27/27 green. Files:
|
||
`audio_engine.{h,cpp}`, `docs/voice.md`.
|
||
- [ ] **DRED toggle in client UIs** — expose the `dred` flag in all three channel-config UIs
|
||
so admins can enable it per channel. Windows: `ChannelEditForm` / `vc_channel_info.audio.dred`
|
||
checkbox. macOS AppKit: channel-edit sheet. iOS SwiftUI: channel-edit form. All three UIs
|
||
already have full channel CRUD wired; this is an additive checkbox on the existing audio-config
|
||
section. (Core/protocol/ABI all done — this is UI-only work.)
|
||
- [x] **macOS ScreenCaptureKit screen-audio** — done 2026-06-21. `ScreenAudioCapture.swift`
|
||
drives an `SCStream` (audio-only, `excludesCurrentProcessAudio`), converts Float32 →
|
||
int16 in the channel's mono/stereo mode, and calls `vc_stream_feed_pcm`. Capture starts on
|
||
the self `.streamStarted` event (when the effective config is known); wired into
|
||
`MainWindowController.screenAudioClicked()`.
|
||
- [x] **iOS ReplayKit Broadcast Extension** (`VoiceCatBroadcast`) — done 2026-06-21.
|
||
Forward-to-host design: the extension (`SampleHandler.swift`) captures `.audioApp`,
|
||
converts to 48 kHz int16 stereo, and writes a shared App Group SPSC ring
|
||
(`BroadcastAudioRing.swift`); the host's `BroadcastAudioPump` owns the `SCREEN_AUDIO`
|
||
stream and feeds via `vc_stream_feed_pcm` (single session, no creds on disk). UI is an
|
||
`RPSystemBroadcastPickerView` in `VoiceControlsView`. (Replaced the speculative
|
||
`BroadcastCredentials.swift` self-connecting design, now removed.)
|
||
- [x] **External PCM feed/tap API** (`vc_stream_feed_pcm` + `vc_set_pcm_sink`) — done
|
||
2026-06-20. Promotes `vc_test_inject_capture` (mono-only, TEST-ONLY) to a public API with
|
||
stereo support. Adds a symmetric PCM sink fired on the playback thread per decoded remote
|
||
stream. Full wrappers for Swift (`feedPcm`/`setPcmSink`) and C# (`StreamFeedPcm`/
|
||
`SetPcmSink`). Three new C++ ctests (`test_feed_pcm_round_trip`, `test_feed_pcm_stereo`,
|
||
`test_pcm_sink`), 4 Swift XCTest smoke tests, 4 C# xUnit smoke tests. Docs updated
|
||
(architecture.md §4 new subsection, voice.md §9 updated, protocol.md §8 explicit
|
||
no-protocol-change note, roadmap.md M5 entry). `ctest --preset dev` 23/23.
|
||
- [x] **macOS client UI overhaul** — done 2026-06-20. Mirrors the Windows client's UI
|
||
overhaul (toolbar, unified log, PM windows, channel counts, output volume, keyboard
|
||
shortcuts), adapted to Mac-native conventions:
|
||
- **NSToolbar**: Join Voice, Share Screen Audio, Mute, Deafen (SF Symbol toggle buttons),
|
||
and Output Volume slider (NSSlider 0–100, default 80). Voice actions + mute/deafen +
|
||
output volume moved out of the bottom voice panel into the toolbar. Bottom panel keeps
|
||
input-mode segmented control / VAD slider / PTT key / device picker / level meter.
|
||
- **Unified log**: chat `NSTextView` + activity `NSTableView` collapsed into a single
|
||
`NSTextView` — activity events in `secondaryLabelColor` (gray), chat in default color.
|
||
Removed `activityTableView` and `activityLog` array.
|
||
- **Private messaging**: scope dropdown removed; compose bar always sends to the current
|
||
channel. Each PM conversation opens in its own modeless `PrivateMessageWindowController`
|
||
(NSWindow). Incoming `.textMessage` with `.private` scope routed to the right window;
|
||
outgoing PMs echoed by server arrive through the same path. "Send Private Message…"
|
||
added to user context menu. "New Private Message…" (⌘⇧N) opens `UserPickerSheet`
|
||
listing all server users.
|
||
- **Channel counts**: outline view renders `"Name (n)"` with live user counts;
|
||
`refreshChannelTree()` called on `.userJoined`/`.userLeft` (was missing).
|
||
- **Voice menu** (⌘⇧V join/leave, ⌘⇧S share screen, ⌘⇧M mute, ⌘⇧D deafen) and **Messages
|
||
menu** (⌘⇧N new PM) added to `NSApp.mainMenu` via `NSMenuItem` key equivalents with
|
||
`[.command, .shift]` mask. Removed on `windowWillClose`. Mac-native: ⌘ not Ctrl, dispatched
|
||
by the responder chain (no custom key monitor needed).
|
||
- **Output volume**: `setOutputVolume(_:)` wrapper added to `VoiceCatClient.swift` (was
|
||
missing — the C ABI + C# wrapper shipped in commit 97fa659 but the Swift wrapper was
|
||
never added). Wired end-to-end: toolbar slider → `client.setOutputVolume(gain)`.
|
||
- **Part A (uncompiled Swift fix)**: the external PCM feed/tap Swift wrapper (commit
|
||
615d2a8) was never compiled — the local xcframework predating the `voicecat.h` PCM
|
||
additions. Fixed: rebuilt xcframework (regenerated module map), fixed `UInt`→`Int` type
|
||
mismatch in `feedPcm` (Swift imports `size_t` as `Int` not `UInt`), added
|
||
`VoiceCatPcmSinkCallback` typealias (Swift-idiomatic alias for the C `vc_pcm_sink_cb`
|
||
so consumers don't need to directly import `VoiceCatC`). `swift test` 10/10 green.
|
||
- **Audio settings moved to Settings window**: the bottom voice panel (input mode, VAD
|
||
slider, PTT key, device picker, level meter) was removed from the main window and moved
|
||
into a new `SettingsWindowController` — a modeless window opened via the app menu's
|
||
"Settings…" (⌘,) item. The main window is now just toolbar + channels + users + chat.
|
||
Source-of-truth for audio settings (`selectedInputMode`, `vadThresholdValue`,
|
||
`selectedInputDeviceId`, `pttKeyCode`) lives in `MainWindowController` so voice start can
|
||
apply them even before the settings window has been opened; `SettingsWindowController`
|
||
reads from and writes back to those properties and applies changes to the client
|
||
immediately when voice is active. The level meter is forwarded from
|
||
`MainWindowController.handleLevel` → `settingsWindowController.updateLevel(rms:)`.
|
||
`keyCodeName` helper deduplicated (was duplicated in `PttKeyCaptureSheet.swift` +
|
||
`MainWindowController.swift` — now shared from `MainWindowController.swift`).
|
||
- Files: `MainWindowController.swift` (overhauled), `PrivateMessageWindowController.swift`
|
||
(new), `UserPickerSheet.swift` (new), `SettingsWindowController.swift` (new),
|
||
`VoiceCatClient.swift` (setOutputVolume + VoiceCatPcmSinkCallback typealias + feedPcm
|
||
type fix), `ExternalPcmTests.swift` (use typealias), `PttKeyCaptureSheet.swift` (removed
|
||
duplicate `keyCodeName`), `VoiceCatMac.xcodeproj/project.pbxproj` (register 3 new files).
|
||
- **Platform-specific adaptations** (vs. Windows): `NSToolbar` instead of `ToolStrip`;
|
||
global menu bar + `NSMenuItem` key equivalents (⌘ not Ctrl, responder-chain dispatched);
|
||
PM windows as modeless `NSWindow`s; picker as Mac sheet; gray = `secondaryLabelColor`;
|
||
SF Symbols for toolbar icons.
|
||
|
||
---
|
||
|
||
## Decisions log
|
||
|
||
All architecture/scope decisions are settled and recorded in
|
||
[docs/roadmap.md §2 "Resolved decisions"](docs/roadmap.md) and reflected across `docs/`.
|
||
If you make a *new* decision, record it there and link it here.
|
||
|
||
---
|
||
|
||
## How to update this file
|
||
|
||
1. Check off tasks as you complete them; flip a milestone to `[x]` only when its **exit
|
||
criterion test** passes.
|
||
2. Keep the **"Where we left off / next action"** block at the top accurate — it's the first
|
||
thing the next agent reads.
|
||
3. When you start a milestone, copy its task list from `docs/roadmap.md` into a section here.
|