Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
95 Commits 7 Branches 0 Tags
9a20953c08d0f0608bbc6dc3a4f773f201f91844
Commit Graph
25 Commits
Author SHA1 Message Date
TalonandClaude Opus 4.8 b44a200b95 fix(audio): apply receive-side NR to stereo mic streams
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Details
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Details 
The per-listener noise-reduction toggle (vc_set_remote_stream) did nothing
on Windows/macOS/iOS. The decode loop gated the RNNoise pass on
dec_channels == 1 as a proxy for "this stream is voice" (assuming
stereo => screen-share). The stereo-mic capture commit broke that: a stereo
mic with send-side NR off transmits stereo Opus, so the receiver decoded
two channels and skipped NR entirely. gain/mute have no channel guard, which
is why only NR appeared broken.

Thread the stream kind through init_recv_stream into RemoteStream::is_voice
(set from si.kind() == STREAM_MIC), gate receive NR on is_voice instead of
channel count, and fold a stereo voice frame to mono -> denoise -> duplicate
back across both channels in place (symmetric with the send-side downmix;
RNNoise is mono-only). Screen-audio shares are never denoised.

New test test_recv_noise_reduction drives AudioEngine and asserts a stereo
voice stream's noise floor collapses with NR on (RMS 1046 -> 0.1) while a
screen-audio share stays unchanged. ctest --preset dev green 29/29.
Docs: voice.md section 10. Shared-core fix; clients need only a rebuild.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-23 21:11:03 +02:00
TalonandClaude Opus 4.8 bad9c7533a feat(audio): real noise suppression via vendored RNNoise (send + receive)
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Details
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Details 
The two-sided NR plumbing (RemoteStream::recv_ns + the per-listener
vc_set_remote_stream noise_reduction toggle) was wired but inert:
ApmProcessor::create() returned a no-op passthrough, because the
originally-planned webrtc-audio-processing has no working Windows/macOS
build. Drop in RNNoise as the real backend behind the same ApmProcessor
interface, lighting up both NR paths.

- Vendor RNNoise (BSD-3 + CC0) at third_party/rnnoise/ — the vcpkg port
  is !windows !arm, so it can't cover our primary targets. Shrunk int8
  model (78MB -> 11.7MB via upstream scripts/shrink_model.sh), built as a
  standalone C static lib with no RTCD (portable scalar path on x86,
  auto-NEON on arm64) under -DDISABLE_DEBUG_FLOAT. Model is baked in
  (rnnoise_create(NULL)); no runtime file.
- New RnnoiseProcessor (core/src/audio/apm_processor.cpp) selected by
  ApmProcessor::create() when VOICECAT_HAS_NS. Mono/48kHz/480-sample;
  our clock is fixed 48kHz and Opus frame sizes are multiples of 480, so
  no resampling. RT-safe: allocates at construction, lock-free in the
  capture/playback callbacks.
- Receive-side: lit up via the factory; gated to mono streams (a stereo
  stream is a screen-audio share, not voice).
- Send-side (new): vc_set_input_noise_reduction(client, enable) ABI +
  vc_client::mic_ns_, run before input gain/VAD in on_capture_frame. A
  stereo mic is downmixed to mono ONLY when NR is on — with NR off a
  stereo mic keeps full stereo (never collapse mic quality unasked).
- Enable C as a project language for the vendored lib.
- New noise_suppression test: white noise through ApmProcessor::create()
  drops ~99.9% RMS. ctest --preset dev green, 28/28. windows-client DLL
  builds clean with vc_set_input_noise_reduction exported, system-only deps.
- Docs synced: voice.md §10, tech-stack.md §1/§5, third_party/README.md,
  vcpkg.json note, PROGRESS.md, CLAUDE.md.

Client on/off UI toggles (Windows/macOS/iOS) are the remaining follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-23 13:30:54 +02:00
TalonandClaude Opus 4.8 7547b8e140 fix(audio): wire in-band FEC into the decoder loss path
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Details
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Details 
The encoder set OPUS_SET_INBAND_FEC, but the decoder never invoked FEC --
the loss path went DRED -> PLC, so FEC redundancy was emitted (and paid for
in bitrate) yet never consumed.

Wire FEC recovery into AudioEngine::on_playback between DRED and PLC: copy
the next buffered packet once, try DRED, else (if the stream negotiated FEC)
decode(next_pkt, ..., fec=true), else PLC. Recovery priority is now
DRED -> FEC -> PLC. Add per-stream RemoteStream::fec_enabled_, captured from
OpusParams in init_recv_stream. Docs (voice.md) updated to match.

ctest --preset dev: 27/27 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-22 20:18:29 +02:00
TalonandClaude Opus 4.8 ce2035f271 fix(audio): bound playout depth to stop voice latency ratcheting up
Build Linux Binaries / linux/amd64 (push) Has been cancelled
Details
Build Linux Binaries / linux/arm64 (push) Has been cancelled
Details 
Latency between speakers grew to multiple seconds and only reset on
rejoining voice. Root cause was the receiver playout logic, not the
codec settings: the playout clock free-ran in real time while the
sender omitted silence from its timestamps (and set no header flags),
and the only correction snapped the clock to the *oldest* buffered
frame — which could only ever add standing latency. target_depth_ms_
was computed but never enforced, so latency could only grow or reset.

Fix: bound playout against the stream's leading edge (newest frame).
(Re)seed to the leading edge on start/marker/starve (no prebuffer, so
latency stays low), and frame-skip catch-up trims any backlog beyond
target+hysteresis — the missing downward force.

Hardening: sender now stamps kFlagMarker (talkspurt start) and kFlagDtx,
consumed on recv for clean resync; adaptive late-drop window; EWMA
outlier rejection so silence gaps/stragglers don't poison the estimate;
duplicate counting and ring-underrun diagnostics.

New test_jitter_depth asserts depth stays bounded (<200ms) while
arrivals outrun playback. ctest --preset dev green (27/27).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-22 20:02:20 +02:00
Talon 6c17881cc0 feat(ios): real echo cancellation/NR via native Voice-Processing engine 
iOS "voice chat" had echo and no noise suppression: real iOS AEC/NS/AGC
come only from Apple's Voice-Processing I/O unit (VPIO), but the core
plays/captures via miniaudio's plain RemoteIO units, so .voiceChat mode
alone never engaged AEC.

Core (ABI PATCH 1->2):
- vc_set_mixed_output_sink + vc_set_external_playback. In external mode the
  AudioEngine opens no hardware playback device; a mixer-timer thread drives
  on_playback (decode+mix) on a ~20ms cadence and ships the final mix to the
  sink. start() also skips the hardware capture device when the MIC stream is
  external_feed (AudioParams.external_capture).
- New white-box test test_external_playback (drives the timer with no hw).

iOS/Swift:
- StreamDescriptor.externalFeed; VoiceCatClient.setMixedOutputSink /
  setExternalPlayback wrappers.
- IOSVoiceProcessingEngine: AVAudioEngine + setVoiceProcessingEnabled; mic
  tap -> feedPcm, mixed-sink lock-free ring -> AVAudioSourceNode (both share
  the VPIO unit so AEC has its reference signal).
- IOSAudioRouter.currentConfigUsesVoiceProcessing scopes VPIO to the AEC
  presets; SessionState join/leave + reconcileVoicePath() switch paths;
  Voice Chat defaults to speaker; Settings surfaces AEC/NS state.

Known: pending on-device verification; a few bugs to fix afterward.
 2026-06-22 02:38:01 +02:00
Talon 6071c8e238 fix(media): stop permanent voice loss after bad-network blip (protocol v2) 
A bad UDP packet on a flaky link could permanently wedge the voice path,
unrecoverable even across app restarts. Three defects:

1. Anti-replay window was advanced from the UNAUTHENTICATED header seq
   before the AEAD tag was checked, and not rolled back on failure. One
   corrupted/forged frame shoved recv_highest_ far ahead, after which every
   legitimate frame was rejected as "too old" forever. Reorder to
   replay-check -> authenticate -> update (RFC 3711 3.3); the window now
   moves only after a successful tag check.

2. The wire seq was only the low 16 bits of the nonce counter (zero-extended
   on receive). After 65,536 frames the nonce desynced and all frames failed
   auth. Widen the voice frame seq u16 -> u64 (header 14 -> 20 bytes). The
   core owns all UDP framing, so Swift/C# clients need only a rebuild. This
   is a versioned wire change: VOICECAT_PROTOCOL_VERSION 1 -> 2, handshake
   rejects on mismatch.

3. Server leaked per-session UDP state on disconnect; unregister_session now
   frees udp_endpoints_/udp_tokens_/ssrc_to_session_.

Also add rate-limited dropped-frame logging to MediaRelay so a wedged media
path is observable. New regression tests in test_media_aead.cpp cover the
poison (fails on old code) and the 16-bit wrap. ctest --preset dev
-E external_pcm: 22/22 pass (external_pcm aborts on a pre-existing CoreAudio
shutdown race, unrelated).
 2026-06-21 17:45:28 +02:00
TalonandClaude Sonnet 4.6 615d2a8e5f feat: external PCM feed/tap API (vc_stream_feed_pcm + vc_set_pcm_sink) 
Promotes vc_test_inject_capture (mono-only, TEST-ONLY) to a public,
stereo-capable production API and adds a symmetric PCM tap on the
receive side. Enables ReplayKit (iOS), ScreenCaptureKit (macOS), bots,
soundboards, and custom clients — all without a hardware audio device.

Core C++:
- voicecat.h: new vc_stream_feed_pcm, vc_pcm_sink_cb typedef,
  vc_set_pcm_sink; vc_test_inject_capture kept as deprecated alias
- audio_engine: stereo-aware inject_capture (channels param + ring
  reset on channel-count change); atomic pcm_sink_ fired per decoded
  frame in on_playback; RemoteStream carries user_id/stream_id for
  RT-safe sink metadata; init_recv_stream takes user_id+stream_id
- client.cpp: stream_feed_pcm / set_pcm_sink implementations;
  sync_remote_streams passes user_id/stream_id to init_recv_stream
- voicecat.cpp: trampolines + channels=1/2 validation

Tests: test_external_pcm (headless, 3 sub-tests: mono round-trip,
stereo feed L≠R, sink metadata+disable). ctest 23/23.

Swift: feedPcm / setPcmSink in VoiceCatClient.swift + 4 XCTest
smoke tests (ExternalPcmTests.swift).

C#: StreamFeedPcm / SetPcmSink in VoiceCatClient.cs + NativeMethods.cs
(vc_stream_feed_pcm unsafe P/Invoke, VcPcmSinkCallback delegate,
vc_set_pcm_sink via nint) + 4 xUnit smoke tests (ExternalPcmTests.cs).

Docs: architecture.md §4 new subsection, voice.md §9 updated
(macOS/iOS now reference vc_stream_feed_pcm), protocol.md §8 explicit
no-protocol-change note, roadmap.md M5 entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-20 17:52:09 +02:00
TalonandClaude Sonnet 4.6 97fa659422 feat(windows): UI overhaul -- toolbar, unified log, PM windows, channel counts, output volume 
- Voice actions (Join Voice, Share Screen Audio) moved to a ToolStrip toolbar and
  a new Voice menu in the menu bar; removed from the bottom voice panel
- Activity log and chat log collapsed into a single RichTextBox (rtbLog); activity
  events appear in gray, chat messages in default color
- Private messaging reworked: each conversation opens in its own modeless
  PrivateMessageForm instead of sharing the main chat log via a scope dropdown;
  cboScope removed; main compose bar always sends to the current channel
- New "Messages -> New Private Message..." menu item (Ctrl+P) opens a UserPickerDialog
  listing all connected server users (not just the current channel) so you can PM
  anyone on the server
- Channel tree now shows live user counts, e.g. "General (3)" -- counts sourced from
  the existing _users dictionary which already tracks all server users with channel IDs
- Global output volume slider (TrackBar, 0-100, default 80) added to the right panel;
  wired to new vc_set_output_volume C ABI function that applies a master gain multiplier
  in the audio engine playback callback after mixing all streams
- vc_set_output_volume added end-to-end: voicecat.h, audio_engine.h/.cpp,
  client.h/.cpp, voicecat.cpp, NativeMethods.cs, VoiceCatClient.cs
- Documented Windows PowerShell ctest requirement in AGENTS.md and CLAUDE.md:
  MinGW binaries exit 0xc0000139 in Git Bash; always run ctest/.exe via PowerShell

22/22 ctest green (PowerShell); dotnet build 0 warnings.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-20 14:24:54 +02:00
TalonandClaude Sonnet 4.6 fdcd8d1427 feat: DRED (Deep REDundancy) per-channel toggle 
Adds Opus 1.6 DRED support end-to-end: encoder embeds 20 ms of ML
redundancy in every packet when enabled; decoder recovers lost frames
from the next buffered packet's DRED extension rather than falling back
to PLC comfort noise.

Protocol: bool dred = 11 added to AudioConfig (backward-compatible,
defaults false). C ABI: int dred added to vc_audio_config. Encoder:
OPUS_SET_DRED_DURATION(2) when dred=true. Decoder: OpusDREDDecoder +
per-stream OpusDRED scratch pre-allocated off the RT thread;
JitterBuffer::try_copy_front_payload peeks at the next packet without
popping on every PLC step; opus_decoder_dred_decode reconstructs the
lost frame if DRED data is present, otherwise falls back to PLC.

New test: test_dred_toggle (22/22 ctest green).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-20 13:40:47 +02:00
Talon f3c1172a3e docs(ios): drop stale debug comments + TeamTalk refs from audio router 
After the stereo-mic/A2DP debugging settled, the iOS audio code carried
leftover TeamTalk5 comparison notes, source-line citations, TEMP DIAGNOSTIC
markers, and "this was the bug" narratives that no longer help. Reworded
those to state the current rules; kept the comments that document real
constraints (the setPreferredInputNumberOfChannels(2) trap, the re-entrancy
guard, the ma_context no-session-management config).

Comment-only — no behavior change. core builds, ctest --preset dev 21/21.
 2026-06-20 03:03:34 +02:00
TalonandClaude Opus 4.8 f1e1ef59ed fix(ios): stop miniaudio from clobbering AVAudioSession (stereo->A2DP output death) 
The real root cause of "selecting Stereo Mic kills headphone/A2DP output on Join
Voice." Every prior fix worked on the Swift IOSAudioRouter under the false premise
that "miniaudio does NOT touch AVAudioSession on iOS." It does: the core opened
devices via ma_device_init(nullptr, ...), and with a NULL context miniaudio 0.11.25
runs an iOS "hack" that sets the session category by device type, then
ma_context_init__coreaudio calls setCategory()+setActive() on every device open --
capture -> AVAudioSessionCategoryRecord with zero options. That wipes the
.playAndRecord category, the mode, and .allowBluetoothA2DP / .mixWithOthers /
.allowAirPlay that IOSAudioRouter had just configured, killing headphone/A2DP (and
even wired) output. Stereo presets break worst because they rely on the A2DP output
route the wipe removes. TeamTalk avoids this by opening RemoteIO/VPIO AudioUnits
directly and leaving the session entirely to the app.

Fix (core, cross-platform safe): AudioEngine now owns a ma_context built by
make_context_config() with coreaudio.sessionCategory = ma_ios_session_category_none
and noAudioSessionActivate/Deactivate = MA_TRUE, and routes all ma_device_init calls
(playback, capture, loopback) plus enumerate_devices through it. miniaudio no longer
touches AVAudioSession; IOSAudioRouter is the sole owner (the session is already
activated on connect in AppState before any device opens). Context is lazily inited
in start(), reused across restarts, uninited in ~AudioEngine.

Adds TEMP AudioSessionManager.logSessionState() diagnostics (after activate, on route
change, on .streamStarted) to verify on-device that the category stays
PlayAndRecord+allowBluetoothA2DP instead of flipping to Record. Remove once confirmed.

Windows: cmake --build --preset dev clean; ctest --preset dev 21/21.
iOS build + on-device verification pending on Mac.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-20 02:34:07 +02:00
TalonandClaude Sonnet 4.6 dcb7e6eeca fix(ios): fix stereo mic + A2DP output silence 
Three coordinated fixes for the bug where enabling stereo mic capture
caused all audio output (A2DP, speaker, wired) to go silent:

1. audio_engine.cpp — open playback before capture
   On iOS, starting the stereo capture AudioUnit can trigger an audio
   route reconfiguration that drops A2DP before the playback device has
   a chance to claim the route. Opening and starting the playback device
   first commits the output route (A2DP), so iOS is less likely to drop
   it when stereo capture activates afterward.

2. client.cpp — decouple set_capture_channels from engine restart
   Previously vc_set_capture_channels() stopped and restarted the engine
   immediately, which opened capture first (old ordering) and raced
   against the settling AVAudioSession route. Now it only stores the
   channel count; the caller (Swift via vc_audio_restart) controls when
   the engine restarts, after the route has settled.

3. IOSAudioRouter.swift — call audioRestart() after channel config
   selectCaptureChannels() and applyPreset() now call audioRestart()
   after applyConfiguration() + setCaptureChannels(). This is the
   vc_audio_restart() path that was added to the ABI in fdcc84f but
   never wired up in the Swift layer. The restart sees the stored
   channel count and reopens devices in the correct order (playback
   first, capture second).

The doStartMicStream path is unaffected: setCaptureChannels is called
before the server acknowledges the stream (engine not yet running), so
ensure_audio_running() picks up capture_channels=2 directly when the
stream is confirmed and opens with the right count from the start.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-19 17:39:23 +02:00
Talon 9fc51cffc4 feat(ios): audio overhaul, Join/Leave Voice, channel-id sync fix, stereo mic capture 
Three iOS client problems fixed plus a new core stereo-mic capture ABI:

1. Channel-id sync bug (mic button permanently dimmed): SessionState never
   synced currentChannelId from the self user's channelId on connect, so the
   mic button (gated on currentChannelId == 0) stayed dimmed. Added
   syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522);
   called from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult.
   Added applyServerMuteState() + serverMuted/serverDeafened to VoiceState.

2. Join/Leave Voice button: replaced icon-only mic toggle with explicit
   text button (parity with macOS). Mute/deafen disable when not in voice.

3. IOSAudioRouter.swift (new): full AVAudioSession routing layer — input
   port selection, built-in mic orientation/polar patterns, Bluetooth
   HFP/A2DP/Off modes, Standard/Raw mic processing, stereo capture, AirPlay,
   UserDefaults persistence. AudioSessionManager delegates to it.

4. Core stereo-mic capture (append-only ABI): vc_set_capture_channels()
   lets the core open the mic device in stereo (2-ch interleaved). LocalStream
   gains capture_channels; ensure_audio_running reads it; audio_engine.cpp
   capture_accum_ + on_capture updated to channel-aware accumulation. Test
   test_stereo_mic_capture (headless, L!=R stereo round-trip). Swift wrapper
   VoiceCatClient.setCaptureChannels.

5. Settings UI rework: AVAudioSession-derived input/output tree replaces
   miniaudio device picker.

6. iOS deployment target raised to 18.0 (Package.swift + project.pbxproj).
   swift-tools-version 6.0 with swiftLanguageModes .v5.

Docs: tech-stack.md, architecture.md, voice.md, roadmap.md, building.md
updated; stale 'vc_audio_suspend/resume deferred' claims corrected.

Verified: ctest --preset dev 21/21 green; swift test 6/6 green;
xcodebuild -target VoiceCatiOS -sdk iphonesimulator BUILD SUCCEEDED.
 2026-06-19 13:17:52 +02:00
Talon e26e7db5b1 feat(ios): ship iOS SwiftUI client (VoiceCatiOS) 
Full SwiftUI app at clients/apple/iOS/VoiceCatiOS.xcodeproj:
- 24 Swift source files: AppState + SessionState (@Observable @MainActor),
  AudioSessionManager (AVAudioSession owner + interruption/route handling),
  ServerListStore/SavedServer (App Group container + Keychain sharing),
  and 14 SwiftUI views covering the full feature set
- NavigationSplitView on iPad, TabView on iPhone (horizontalSizeClass)
- Channel tree via OutlineGroup, user list with context menu admin actions
- PTT via DragGesture(minimumDistance: 0) + @GestureState
- onEvent closures hop to MainActor via Task { @MainActor in ... }
- App Group: group.cat.voice.VoiceCat (shared with future ReplayKit extension)

C ABI: add vc_audio_suspend / vc_audio_resume (AudioEngine::suspend/resume)
called by AudioSessionManager on AVAudioSession interruption events.

XCFramework: add ios-arm64 and ios-arm64-simulator slices to build-xcframework.sh;
Package.swift gains .iOS(.v17) platform; CMakePresets.json adds apple-ios /
apple-ios-sim presets with arm64-ios / arm64-ios-simulator vcpkg triplets.

Verified: xcodebuild -target VoiceCatiOS -sdk iphonesimulator26.5 BUILD SUCCEEDED.
 2026-06-19 02:10:25 +02:00
Talon d397731db9 feat(audio): per-stream mix controls in Windows client + vc_get_remote_stream getter 
PerUserTuningDialog previously broadcast one gain/mute/NR set to *all* of a
user's streams, even though the core mixer (AudioEngine::RemoteStream) and
the C ABI (vc_set_remote_stream) were already per-stream. The UI had no
per-mix controls anywhere.

Reworks the dialog to enumerate ListUserStreams on open and render one row
per stream (kind + label + Gain + Mute + NR), each wiring only to its own
stream_id. Adds a read-back ABI counterpart, vc_get_remote_stream, so the
dialog opens at the listener's actual current per-stream settings (defaults
1.0/unmuted/NR-off) rather than always 100%. Additive ABI change only; no
existing symbols touched.

Tests: test_m3_multistream extended with getter round-trip assertions; new
C# smoke test exercises the full P/Invoke marshaling path with two clients.
Docs: voice.md §10 notes the getter. NR checkbox keeps its honest
'passthrough' label (NS DSP still unbuilt per §8).
 2026-06-18 02:06:44 +02:00
Talon 487a561963 fix(net): broadcast LEFT on disconnect, add keepalive/reaper, cap PLC hiss 
Three reported bugs traced to one root cause plus two missing designed features:

1. Stale users + eternal PLC hiss (root cause): ConnSession::close() silently
   erased dropped users without broadcasting UserEvent::LEFT, so peers never
   learned the user left and their audio engines never called remove_stream —
   Opus PLC synthesized comfort noise forever. Fix: broadcast_left() helper
   + close() broadcasts LEFT before erasing.

2. PLC cap (defense-in-depth): on_playback now caps pure PLC at ~2s, then
   emits digital silence so a stale stream can never hiss forever even if
   remove_stream is skipped. Resets automatically on fresh packets.

3. No timeout / no ping: client never sent Ping, server had no last_seen /
   reaper, so half-open connections (NAT timeout, wifi loss, sleep) left
   ghost users forever. Fix: client Ping every 15s with RTT measurement,
   ConnSession::last_seen bumped on every inbound TCP/UDP frame, steady_timer
   reaper sweeps every 15s and drops sessions older than 45s (configurable
   via server::Config).

4. UDP KEEPALIVE: client sends plaintext kFrameKeepalive every 5s; server
   bumps last_seen + echoes back. Keeps NAT bindings alive and lets media
   activity defer the reaper independently of TCP.

5. Graceful client disconnect: vc_disconnect() sends Disconnect{code=0} via
   a flag-based io-thread exit (no double-close race); server handles
   client-sent Disconnect with immediate close() + LEFT broadcast.

3 new tests: disconnect_left, plc_cap, reaper_timeout. 21/21 ctest green.

Docs: protocol.md §6/§7, voice.md §6, architecture.md §5, PROGRESS.md.
 2026-06-18 01:18:33 +02:00
Talon cccf085a87 fix(audio): stereo screen-audio loopback capture on Windows 
start_loopback_capture hardcoded channels=1, forcing miniaudio to downmix the system's stereo mix to mono before the encoder saw it -- on_capture_frame then upmixed L=R to produce fake stereo. Now the loopback device opens in the channel's mode (stereo when the channel is stereo), CaptureCallback carries an explicit channels param so the encoder gets real interleaved L/R, and a mono fallback covers unusual render endpoints. New test_loopback_stereo_capture asserts L!=R end-to-end; 18/18 ctest green.
 2026-06-17 23:27:59 +02:00
TalonandClaude Opus 4.8 a2f159e971 fix(audio): seed/re-sync playout clock so VAD/PTT gaps don't silence playback 
RemoteStream::playout_ts was seeded to 0 and only advanced inside the
decode loop (including on every PLC iteration), so it free-ran at ~1x
wall-clock regardless of whether the sender was transmitting. The
sender's frame timestamps only advance while it actually sends (the
VAD/PTT gate returns before ls.timestamp += samples). Across a late join
or any VAD/PTT silence gap the two clocks diverged without bound; once
past the jitter buffer's 500 ms late-drop window every real frame was
dropped-as-late (clock ahead) or never-due (clock behind) -> permanent
silence, while the talk indicator (driven by push_recv_frame, independent
of the jitter buffer) stayed lit.

Add JitterBuffer::peek_front_ts() (try-lock, RT-safe) and seed/re-sync
playout_ts to the earliest buffered frame on the first frame and whenever
it has drifted past +/-200/500 ms. This seeds startup and recovers after
every silence gap.

New regression test test_playout_resync free-runs the clock ~2 s past the
drop window, pushes a ts=0 frame, and asserts audible output: fails
(energy=0) without the fix, passes with it. ctest --preset m1-dev: 14/14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-17 12:24:39 +02:00
TalonandClaude Sonnet 4.6 7da0a02b3a fix(audio): buffer capture frames to Opus encoder's fixed frame size 
AudioEngine::on_capture() was passing miniaudio's hardware callback period
(commonly 480 samples / 10 ms on WASAPI shared mode) directly to opus_encode(),
which requires exactly frame_samples_ (960 for 20 ms @ 48 kHz). The mismatch
returned OPUS_BAD_ARG and silently dropped every real mic frame, while screen
share and injected test frames happened to be correctly sized and worked fine.

Fix: accumulate PCM in a pre-allocated CaptureAccum buffer (mirroring the
existing RemoteStream::ring fix on the playback side) and only call capture_cb_
when a full frame_samples_ chunk is ready. Same pattern applied to on_loopback().

Add test_capture_frame_accumulation() to verify the accumulator fires exactly the
right number of callbacks for misaligned chunk sizes (480, 240+720, 1920 samples).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-17 01:51:50 +02:00
TalonandClaude Sonnet 4.6 63b241cc2e feat(M4): Windows WinForms client, TOFU identity pinning, VAD threshold + always-on mode 
Core ABI extensions (voicecat.h):
- vc_list_channels / vc_list_users / vc_list_user_streams — pull-based snapshot getters
  for the channel-tree and user-list UI; session_model_mu_ guards cross-thread reads
- VC_EVENT_JOIN_RESULT / vc_join_channel — channel join with optional password
- VC_EVENT_SERVER_IDENTITY + vc_confirm_server_identity — TOFU gate that blocks io_thread_
  until the UI approves or rejects; pins TLS leaf-cert SHA-256 (not declared Ed25519)
- vc_get_server_identity_display — Ed25519 fingerprint for human-readable display only
- VC_INPUT_ALWAYS_ON = 2 in vc_input_mode — transmit unconditionally, no VAD gate
- vc_set_vad_threshold — live RMS threshold update (0.0–1.0); EnergyVadProcessor stores
  it atomically so the audio RT path reads without a lock

C++ implementation:
- SessionModel::apply_snapshot / apply_channel_event fixed to populate parent_id,
  password_protected, and max_users (were permanently zeroed)
- TlsContext::peer_cert_fingerprint — SHA-256 of peer leaf cert DER via mbedTLS
- TofuStore split into peek (read-only) + pin (write) so first-connect only persists
  after user approval; tofu_store_path in vc_config for per-user pin file location
- TcpAcceptor uses dual-stack IPv6+IPv4 fallback (fixes localhost → ::1 on Windows)
- windows-client CMake preset: Release shared DLL, static MinGW runtime, no tools/tests
- New C++ tests: test_channel_user_list_abi, test_tofu_flow (14/14 green)

Windows client (clients/windows/ — .NET 10 WinForms):
- VoiceCat.Interop: LibraryImport P/Invoke surface, UnmanagedCallersOnly callbacks,
  Channel<VoiceCatEvent> event delivery drained by 30ms WinForms Timer
- VoiceCat.App: ConnectDialog (saved servers, DPAPI password storage), ServerIdentity-
  Dialog (TOFU first-connect / mismatch warning), MainForm (channel TreeView, user
  ListBox, RichTextBox chat, voice controls, device pickers, VAD/PTT/always-on mode,
  per-user gain/mute/NR tuning, VAD sensitivity TrackBar, level meter ProgressBar)
- PttKeyCaptureDialog — focus-scoped PTT key capture (documented limitation)
- PerUserTuningDialog — real-time gain/mute/NR applied to all of a user's streams
- Accessibility: explicit AccessibleName/Description on every control, & mnemonics,
  Activity log ListBox as durable screen-reader record, AutomationNotification for
  curated live announcements

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-17 00:35:16 +02:00
TalonandClaude Sonnet 4.6 5be869c61a fix(audio): decouple Opus decode cadence from playback callback period 
on_playback() was passing miniaudio's hardware playback-callback frame
count to opus_decode()'s max_samples, instead of the decoder's fixed
frame size (960 samples @ 20ms/48kHz). Since real packets decode to
more samples than the (often smaller, e.g. ~480 on default low-latency
WASAPI) hardware period, opus_decode returned OPUS_BUFFER_TOO_SMALL on
nearly every callback -- packets were received/decrypted/jitter-buffered
correctly but never decoded into audible PCM. Result: control-plane
events and VAD worked, but zero audio in headphones.

mix_for_test()'s white-box test masked this since it always called
on_playback with frames == frame_samples, the one case where the bug
is invisible.

Fix: RemoteStream gained a small ring buffer (init_ring/push_ring/
pop_ring) that decouples decode cadence from playback-callback cadence.
on_playback now tops the ring up by decoding whole Opus frames (always
decoder.frame_samples(), never the hardware frame count) and drains
exactly what the callback asks for, silence-padding (PLC) on underrun.

Side effect: also fixes playout_ts, which was advancing by the wrong
unit (hardware frames instead of decoded samples) -- it now tracks
correctly against jitter-buffer timestamps.

ctest --test-dir build/m1-dev: 12/12 green.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 17:39:49 +02:00
TalonandClaude Sonnet 4.6 5f6c223526 feat: device enumeration, VAD/PTT input gate, stereo playback, WASAPI loopback 
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:

- Device enumeration (vc_list_devices) + input device selection
  (vc_set_input_device), backed by AudioEngine::enumerate_devices() via
  miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
  ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
  webrtc-audio-processing (the originally-planned APM) has no working
  Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
  abseil-cpp dependency), so VAD is a new lightweight, dependency-free
  energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
  interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
  stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
  stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
  miniaudio's loopback device type), replacing test-only injection as the
  production capture path.

Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).

Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.

Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.

Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 16:11:52 +02:00
TalonandClaude Sonnet 4.6 867557eda1 feat(M3): multi-stream & per-channel tuning 
Implements docs/roadmap.md M3: multiple concurrent streams per user (MIC +
SCREEN_AUDIO + AUX_DEVICE), independent per-stream receiver gain/mute/noise-
reduction, talk indicators, and enforced per-channel Opus configurability
(mono/stereo, bitrate, frame size, FEC/DTX, application).

Bugs fixed along the way (found while implementing, not pre-existing scope):
- Server hard-coded stream_id=1 for every announce, so a second stream from
  the same user silently overwrote the first in SessionRegistry::set_user_stream.
  Now a per-session counter (ConnSession::next_stream_id_); handle_stream_stop
  validates against announced_stream_ids_ before clearing.
- Client dropped mode/dtx/complexity/application from effective_audio even for
  the single M2 stream -- only sample_rate/bitrate_bps/frame_ms/fec were ever
  applied to OpusParams. Fixed on both the send (handle_stream_announce_result)
  and receive (sync_remote_streams) paths via a shared
  opus_params_from_audio_config() helper.
- OpusEncoder always used OPUS_APPLICATION_VOIP; added OpusParams::application
  and wired it through.
- on_playback's per-stream decode passed the wrong frame_size to opus_decode
  (total samples instead of samples-per-channel), which would have overflowed
  the decode buffer for any stereo stream.
- teardown_voice() raced when called concurrently from run_io()'s own cleanup
  and from disconnect() on a different thread -- both could see
  udp_thread_/talk_timer_thread_ as joinable() at once and race to join() the
  same std::thread (intermittent std::system_error under ctest). Fixed with a
  teardown_mu_ guard instead of carrying the flake forward.

New:
- Per-channel AudioConfig: SessionRegistry now seeds Lobby (mono/24kbps/VOIP/
  FEC+DTX) and a new "Music Room" channel (stereo/128kbps/AUDIO/no DTX);
  handle_stream_announce enforces the channel's config, clamping (not
  overriding) bitrate_bps to its ceiling.
- core/src/core/client.h/.cpp: local-stream state is now a
  std::unordered_map<int, LocalStream> keyed by vc_stream_kind, with
  request_id-correlated announce/result handling (request_id already
  round-tripped on the wire; just wasn't read before). on_capture_frame is
  kind-aware and upmixes mono capture to stereo when a stream's config calls
  for it. set_self_mute's mic_muted now only gates the MIC kind. NS is wired
  through set_remote_stream. New run_talk_timer() thread emits
  VC_EVENT_TALK_STATE from both remote and local edge detection.
- core/src/audio/audio_engine.h/.cpp: kind-keyed injection taps
  (inject_capture), stereo-to-mono downmix at the decode/mix boundary,
  RemoteStream gains recv_ns (lazy ApmProcessor) + noise_reduction_enabled
  and last_voice_ms/talking; new set_stream_noise_reduction() and
  poll_talk_transitions().
- core/src/session/session.h/.cpp: Stream now carries the full AudioConfig,
  not just sample_rate/frame_ms.
- New additive C ABI (core/include/voicecat.h): vc_audio_config +
  vc_get_stream_audio_config (effective Opus config for any stream you own or
  a peer's); vc_test_inject_capture (test-only synthetic PCM injection,
  clearly marked, mirrors AudioEngine::inject_capture).
- tests/test_m3_multistream.cpp: the M3 exit criterion through the real ABI
  (mirrors test_voice_client_abi.cpp's approach, not raw sockets) -- two
  concurrent local streams, independent gain/mute/NS control, per-channel
  config divergence via vc_get_stream_audio_config, talk indicators.

Explicitly out of scope for this pass (tracked in PROGRESS.md, not silently
dropped): VAD/PTT input gate + device enumeration; real WASAPI loopback
capture for SCREEN_AUDIO (synthetic injection only); true stereo playback
output (AudioEngine's mixer/output device stays mono -- Opus itself is fully
stereo-correct on the wire).

ctest --test-dir build/m1-dev: 11/11 green, verified across 3 consecutive
full-suite runs plus 8 standalone runs of the new test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 14:12:37 +02:00
TalonandClaude Sonnet 4.6 694494a5be feat(M2): UDP voice/media plane -- SFU relay, Opus, AEAD, jitter buffer 
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.

Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 01:31:14 +02:00
TalonandClaude Opus 4.8 b332b0972b scaffold: M0 skeleton + agent onboarding (build, architecture, progress) 
Turn the design into a buildable, dependency-free M0 skeleton plus the
onboarding layer so a new agent can pick up instantly.

Build system:
- CMake + CMakePresets (dev = no deps; server-release = vcpkg) + vcpkg.json
- Skeleton builds with just a C++20 compiler; deps stay off until needed
- .gitattributes (LF), .gitignore, .clang-format

Core (libvoicecat):
- core/include/voicecat.h: full C ABI (the client/server contract), stubbed
- core/proto/voicecat.proto: control-plane wire format, matches docs/protocol.md
- src/{net,crypto,codec,protocol,session,audio,core}: subsystem stubs that
  return VC_ERR_NOT_IMPLEMENTED, each pointing to its design doc
- server/ (voicecat-server) and tools/vccli/ link the core
- tests/: CTest smoke test asserting the C ABI contract (behavior, not just build)
- clients/{apple,windows}: M4 placeholders

Onboarding for agents:
- CLAUDE.md: hub — build/test commands, architecture at a glance, doc map, rules
- AGENTS.md: working method (behavior-driven; clean compile is the floor not the goal)
- PROGRESS.md: living tracker — M0 done, M1 task checklist, "where we left off"

Verified: cmake --preset dev && cmake --build --preset dev && ctest --preset dev → green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-15 21:09:09 +02:00