Implement system/desktop audio sharing on the Apple clients, feeding the
existing SCREEN_AUDIO Opus -> AEAD -> UDP path via vc_stream_feed_pcm. No
C++/protocol/codec changes -- the core was already ready (the Windows-only
loopback is #ifdef VOICECAT_HAS_LOOPBACK; off Windows the stream just waits
for fed PCM). Audio only; video is dropped.
macOS (in-process):
- ScreenAudioCapture.swift drives an audio-only SCStream
(excludesCurrentProcessAudio), converts Float32 -> int16 in the channel's
mono/stereo mode, and calls feedPcm. Capture starts on the self
.streamStarted event (effective config known then). Wired into
MainWindowController.screenAudioClicked().
iOS (forward-to-host, single session):
- VoiceCatBroadcast: a ReplayKit Broadcast Upload Extension consumes
.audioApp only, resamples to 48kHz int16 stereo (AVAudioConverter), and
writes a shared App Group SPSC ring (BroadcastAudioRing.swift). It does
not link libvoicecat.
- Host BroadcastAudioPump drains the ring (reacting to the extension's
Darwin notifications) and feeds the SCREEN_AUDIO stream it owns, downmixing
to mono when the channel is mono. Screen audio appears as a second stream
of the same user; no credentials persisted. UI is RPSystemBroadcastPicker
View in VoiceControlsView. Removes the speculative BroadcastCredentials.
Docs: voice.md s9, CLAUDE.md status, PROGRESS.md.
Adds Opus 1.6 DRED support end-to-end: encoder embeds 20 ms of ML
redundancy in every packet when enabled; decoder recovers lost frames
from the next buffered packet's DRED extension rather than falling back
to PLC comfort noise.
Protocol: bool dred = 11 added to AudioConfig (backward-compatible,
defaults false). C ABI: int dred added to vc_audio_config. Encoder:
OPUS_SET_DRED_DURATION(2) when dred=true. Decoder: OpusDREDDecoder +
per-stream OpusDRED scratch pre-allocated off the RT thread;
JitterBuffer::try_copy_front_payload peeks at the next packet without
popping on every PLC step; opus_decoder_dred_decode reconstructs the
lost frame if DRED data is present, otherwise falls back to PLC.
New test: test_dred_toggle (22/22 ctest green).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Diagnosed by comparing against TeamTalk5 (Client/iTeamTalk), which
achieves stereo mic + A2DP output. Five fixes:
1. configureStereoCapture now calls setPreferredInput +
setInputDataSource (mirroring TeamTalk5's SoundDevicesModel).
Previously omitted based on incorrect diagnosis that
setPreferredInput collapsed A2DP — the real culprit was
setPreferredInputNumberOfChannels(2), which neither project uses.
2. New C ABI: vc_audio_restart (full stop + re-init, unlike
suspend/resume which only stop/start). Swift wrapper added.
The withAudioSuspend wrapper that used it was removed after
on-device testing showed it killed all audio (including
VoiceOver) when switching presets — the core's
set_capture_channels handles engine restart internally.
3. Bluetooth options: Voice Chat preset now includes BOTH
.allowBluetoothHFP AND .allowBluetoothA2DP (matching TeamTalk5's
UtilSound.swift:228). Previously HFP-only blocked A2DP headphones.
4. Capture channels now reset when switching stereo→mono via
selectCaptureChannels/applyPreset. AudioSessionManager tracks
activeMicStreamId (set by SessionState on join/leave voice).
5. Docs synced: voice.md, tech-stack.md, architecture.md,
PROGRESS.md. Removed stale setPreferredInputNumberOfChannels(2)
references.
Verified: ctest --preset dev 21/21 green, iOS client builds.
Stereo mic + A2DP output still needs on-device debugging — the
core recipe is correct but iOS 26 route behavior requires
hands-on testing with a debugger.
Three iOS client problems fixed plus a new core stereo-mic capture ABI:
1. Channel-id sync bug (mic button permanently dimmed): SessionState never
synced currentChannelId from the self user's channelId on connect, so the
mic button (gated on currentChannelId == 0) stayed dimmed. Added
syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522);
called from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult.
Added applyServerMuteState() + serverMuted/serverDeafened to VoiceState.
2. Join/Leave Voice button: replaced icon-only mic toggle with explicit
text button (parity with macOS). Mute/deafen disable when not in voice.
3. IOSAudioRouter.swift (new): full AVAudioSession routing layer — input
port selection, built-in mic orientation/polar patterns, Bluetooth
HFP/A2DP/Off modes, Standard/Raw mic processing, stereo capture, AirPlay,
UserDefaults persistence. AudioSessionManager delegates to it.
4. Core stereo-mic capture (append-only ABI): vc_set_capture_channels()
lets the core open the mic device in stereo (2-ch interleaved). LocalStream
gains capture_channels; ensure_audio_running reads it; audio_engine.cpp
capture_accum_ + on_capture updated to channel-aware accumulation. Test
test_stereo_mic_capture (headless, L!=R stereo round-trip). Swift wrapper
VoiceCatClient.setCaptureChannels.
5. Settings UI rework: AVAudioSession-derived input/output tree replaces
miniaudio device picker.
6. iOS deployment target raised to 18.0 (Package.swift + project.pbxproj).
swift-tools-version 6.0 with swiftLanguageModes .v5.
Docs: tech-stack.md, architecture.md, voice.md, roadmap.md, building.md
updated; stale 'vc_audio_suspend/resume deferred' claims corrected.
Verified: ctest --preset dev 21/21 green; swift test 6/6 green;
xcodebuild -target VoiceCatiOS -sdk iphonesimulator BUILD SUCCEEDED.
When no simulator is booted, grep -oE finds no UUID and exits 1. With
set -euo pipefail, that non-zero exit propagates through the command
substitution and kills the script at the UDID= assignment before it can
fall through to the boot-one branch.
Fix: add || true to both find_sim and sim_name pipelines so they always
return 0 regardless of whether a match was found.
Also replace sleep 2 with xcrun simctl bootstatus <UDID> -b, which blocks
until the simulator is fully booted — more reliable than a fixed delay
and faster when the simulator starts quickly.
Update docs/building.md §9 to show the bootstatus command.
scripts/build-ios-client.sh: builds VoiceCatiOS.app for iOS simulator
- calls build-xcframework.sh --all (all 3 slices required for iOS sim slice)
- xcodebuild -target VoiceCatiOS with SYMROOT=OBJROOT to co-locate SPM
and app build products (fixes "unable to resolve module dependency" error)
- stages result to dist/ios-client/
scripts/run-ios-simulator.sh: installs and launches on an iPhone simulator
- finds a booted simulator or boots the first available iPhone sim
- opens Simulator.app, installs via simctl install, launches via simctl launch
- --build flag to build first; --log to stream app logs; --device / --udid to
pin a specific simulator
scripts/build-all.sh: add --ios-client opt-in flag (macOS only)
docs/building.md:
- add iOS entry to quick-nav table
- update §6 (apple platform): remove "scaffolding" caveat now that iOS slices
are validated; trim to essentials + pointer to build-xcframework.sh
- add §9 (iOS client SwiftUI): XCFramework, xcodebuild command with rationale
for -target/-SYMROOT flags, run script usage, full simctl commands, Xcode UI
Add §7 (Windows client: dotnet build) and §8 (macOS client: XCFramework
+ xcodebuild) with the full build/run commands for each client platform.
Also add a quick-navigation table at the top of the doc.
Lays the groundwork for the macOS (AppKit) and iOS (SwiftUI) clients with a shared
Swift core wrapping the C ABI, mirroring the proven Windows VoiceCat.Interop layer.
Architecture decision: macOS UI = AppKit (not SwiftUI) for the most mature VoiceOver
accessibility story — same rationale as the Windows client's WinForms-over-WinUI-3
decision. iOS stays SwiftUI. Recorded in docs/roadmap.md §2.
Build infrastructure (Phase 0):
- clients/apple/scripts/build-xcframework.sh: runs cmake --preset apple-dev, merges
libvoicecat.a + 107 vcpkg static deps into a single ~30 MB fat static library
(libvoicecat-fat.a) via libtool -static (SPM binary targets link one .a per slice),
stages voicecat.h + a generated module.modulemap (module VoiceCatC) into the headers,
runs xcodebuild -create-xcframework -> clients/apple/VoiceCatCore.xcframework.
VoiceCatCore Swift Package (Phase 1):
- Package.swift: binary target (VoiceCatCoreXCF) + library (VoiceCatCore) + test target.
- Sources/VoiceCatCore/: 7 files mirroring the C# VoiceCat.Interop patterns adapted to
Swift native C interop — Enums (9 Swift mirrors of C enums, UInt32-backed), Config,
Event (copies ev.text to String inside the callback — the #1 lifetime rule), Models
(10 Swift value types), Marshaling (C arrays -> Swift + immediate vc_free_*),
Callbacks (@convention(c) + Unmanaged.passUnretained, the Swift analog of C#'s
[UnmanagedCallersOnly] + GCHandle), VoiceCatClient (owns vc_client* as OpaquePointer,
all 38 C ABI functions, deinit -> vc_client_destroy then frees config CStrings, event
delivery on main queue via coalesced DispatchQueue.main drain).
Tests — 6/6 green (swift test against a real voicecat-server):
- testConnectTofuAuthListChannelsRoundTrips, testAdminChannelCrudAccountCrudRoundTrips,
testScreenAudioStreamStartsAndStops, testPerStreamRecvControlsRoundTrip, plus two
static smoke tests. Catches Swift-specific interop bugs (@convention(c) callback
lifetime, Unmanaged pointer resolution, CString memory management, enum raw-value
bridging, struct layout) that C++ ctest cannot. C++ suite still 21/21 green.
Docs updated (house rule): tech-stack.md §2, architecture.md §4, roadmap.md M4 + §2,
clients/apple/README.md (full rewrite), PROGRESS.md, .gitignore.
macOS port groundwork — core, server, tools, and tests now build and run on
macOS 26.5 / Apple Silicon. ctest --preset dev green 21/21 (2 consecutive runs).
apple-dev produces valid arm64 libvoicecat.a + XCFramework for the Swift Package.
Three real cross-platform bugs found and fixed (all latent on Windows/Linux):
1. test_m2_voice.cpp POSIX branch missing <netdb.h> — Linux glibc transitively
includes it, macOS doesn't. Would fail on any strict POSIX system.
2. SIGPIPE killing processes on macOS — writing to a closed TCP socket raises
SIGPIPE by default (doesn't exist on Windows, benign on Linux). Fixed by
ignoring SIGPIPE in both core client init and server startup (POSIX-only,
#ifndef _WIN32). Production fix, not just tests.
3. Use-after-free of Asio's kqueue reactor on server shutdown — the
deterministic test_tofu_flow segfault. TcpServerConn's tls_read_loop runs on
a blocking-I/O thread; when Server::run() returned, io_context was destroyed
while those threads were still running. On macOS kqueue the reactor pointer
is null'd immediately -> segfault in socket.close(). Latent on Windows IOCP
and Linux epoll. Fix: TcpAcceptor now tracks connections; new shutdown()
closes all + joins threads before io is destroyed; Server::stop() now closes
acceptor + media_relay too (was just io.stop()).
Verified: dev + apple-dev presets build green, 21/21 tests pass, server starts
+ two vccli text chat over TLS (M1 on Mac), vccli --voice starts MIC stream via
CoreAudio (M2 protocol-level), vccli --list-devices enumerates CoreAudio
devices, xcodebuild -create-xcframework produces valid VoiceCatCore.xcframework.
No ABI or proto changes. Docs updated: building.md, clients/apple/README.md,
PROGRESS.md, CLAUDE.md status line.
Rationalize the preset set to match the project's actual state (past M5):
- Rename dev->skeleton (no-deps stub smoke), m1-dev->dev (default dev preset)
- Drop m2-dev (cache-identical to m1-dev)
- Add release preset (optimized + tests on, symbols kept)
- Strip server-release binaries (-s linker flag)
- Add apple-dev/apple-ios/apple-ios-sim scaffolding presets for XCFramework
Add cmake/voicecat-toolchain.cmake wrapper that auto-resolves the vcpkg
triplet from the host platform (x64-mingw-static/x64-linux/arm64-osx) so
the main presets work on Windows/Linux/macOS without per-OS variants.
Update all docs (building.md, CLAUDE.md, README.md, AGENTS.md, deployment.md,
tech-stack.md, client READMEs) and stale preset-name references in code
comments. No C++ behavior changes — the core was already portable.
PerUserTuningDialog previously broadcast one gain/mute/NR set to *all* of a
user's streams, even though the core mixer (AudioEngine::RemoteStream) and
the C ABI (vc_set_remote_stream) were already per-stream. The UI had no
per-mix controls anywhere.
Reworks the dialog to enumerate ListUserStreams on open and render one row
per stream (kind + label + Gain + Mute + NR), each wiring only to its own
stream_id. Adds a read-back ABI counterpart, vc_get_remote_stream, so the
dialog opens at the listener's actual current per-stream settings (defaults
1.0/unmuted/NR-off) rather than always 100%. Additive ABI change only; no
existing symbols touched.
Tests: test_m3_multistream extended with getter round-trip assertions; new
C# smoke test exercises the full P/Invoke marshaling path with two clients.
Docs: voice.md §10 notes the getter. NR checkbox keeps its honest
'passthrough' label (NS DSP still unbuilt per §8).
Three reported bugs traced to one root cause plus two missing designed features:
1. Stale users + eternal PLC hiss (root cause): ConnSession::close() silently
erased dropped users without broadcasting UserEvent::LEFT, so peers never
learned the user left and their audio engines never called remove_stream —
Opus PLC synthesized comfort noise forever. Fix: broadcast_left() helper
+ close() broadcasts LEFT before erasing.
2. PLC cap (defense-in-depth): on_playback now caps pure PLC at ~2s, then
emits digital silence so a stale stream can never hiss forever even if
remove_stream is skipped. Resets automatically on fresh packets.
3. No timeout / no ping: client never sent Ping, server had no last_seen /
reaper, so half-open connections (NAT timeout, wifi loss, sleep) left
ghost users forever. Fix: client Ping every 15s with RTT measurement,
ConnSession::last_seen bumped on every inbound TCP/UDP frame, steady_timer
reaper sweeps every 15s and drops sessions older than 45s (configurable
via server::Config).
4. UDP KEEPALIVE: client sends plaintext kFrameKeepalive every 5s; server
bumps last_seen + echoes back. Keeps NAT bindings alive and lets media
activity defer the reaper independently of TCP.
5. Graceful client disconnect: vc_disconnect() sends Disconnect{code=0} via
a flag-based io-thread exit (no double-close race); server handles
client-sent Disconnect with immediate close() + LEFT broadcast.
3 new tests: disconnect_left, plc_cap, reaper_timeout. 21/21 ctest green.
Docs: protocol.md §6/§7, voice.md §6, architecture.md §5, PROGRESS.md.
start_loopback_capture hardcoded channels=1, forcing miniaudio to downmix the system's stereo mix to mono before the encoder saw it -- on_capture_frame then upmixed L=R to produce fake stereo. Now the loopback device opens in the channel's mode (stereo when the channel is stereo), CaptureCallback carries an explicit channels param so the encoder gets real interleaved L/R, and a mono fallback covers unusual render endpoints. New test_loopback_stereo_capture asserts L!=R end-to-end; 18/18 ctest green.
A connected Windows client would randomly snap from its joined channel
back to Lobby. Root cause was a state-sync inconsistency, not a drop:
the server delivered self-initiated state changes (channel join/leave,
stream announce/stop) only as a private *Result to the actor and
broadcast the authoritative UserEvent::UPDATED to everyone else. The
core never applied the result to its SessionModel, so vc_list_users()
kept self in the old channel; the Windows HandleUserUpdated rebuilds
_currentChannelId from vc_list_users() on any user's UPDATED event, so
the next unrelated event surfaced the stale self-channel.
Fix, per the response-vs-broadcast contract now documented in
docs/protocol.md §6: the *Result is pure ack/correlation/actor-private
payload; the resulting state change is broadcast to every client
INCLUDING the actor, and clients apply it to their local model rather
than re-deriving own state from a *Result.
- server: join/leave/stream announce+stop broadcast with exclude=0
- server: text fan-out includes the sender (channel + private echo)
- core: response handlers no longer mutate session_model_
- windows: drop optimistic text echo; render own message via the relay
- docs/protocol.md §6: document the response-vs-broadcast contract
Registry-level admin broadcasts (move/mute/kick/channel CRUD) already
used exclude=0 and were correct. ctest build/m1-dev 18/18 green;
VoiceCat.App builds 0 warnings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- docs/building.md: explains what each CMake preset (dev, m1-dev, m2-dev,
server-release) is actually for, and how to build voicecat-server + vccli
for manual testing. Linked from CLAUDE.md's doc index.
- core/include/voicecat.h, core/src/voicecat.cpp, core/src/protocol/protocol.h,
server/src/main.cpp: doc-header comments still claimed M0-skeleton/stub
behavior (VC_ERR_NOT_IMPLEMENTED everywhere, "prints what it would do",
protobuf codegen "commented") that M1-M3 made real. Updated to describe
current behavior, with the dev-preset stub fallback noted explicitly where
it still applies.
Closes the three items PROGRESS.md's M3 section explicitly carried forward as
out of scope:
- Device enumeration (vc_list_devices) + input device selection
(vc_set_input_device), backed by AudioEngine::enumerate_devices() via
miniaudio's ma_context_get_devices. Device ids are opaque hex-encoded
ma_device_id strings.
- VAD/PTT send-side input gate (vc_set_input_mode, vc_set_push_to_talk).
webrtc-audio-processing (the originally-planned APM) has no working
Windows/MSVC build upstream (GCC-only Meson, unfinished MinGW support, hard
abseil-cpp dependency), so VAD is a new lightweight, dependency-free
energy/RMS processor (EnergyVadProcessor) behind the existing ApmProcessor
interface. Gating is MIC-only; SCREEN_AUDIO/AUX_DEVICE always bypass it.
- True stereo playback: AudioEngine's mixer and output device now carry
stereo end-to-end (mono streams upmix L=R) instead of downmixing decoded
stereo streams to mono before mixing.
- Real WASAPI loopback capture for SCREEN_AUDIO (Windows-only, via
miniaudio's loopback device type), replacing test-only injection as the
production capture path.
Also: vccli gains --list-devices, --input-device, --input-mode, and
--share-screen-audio flags, plus a stdin command loop (ptt on/off, mode
vad/ptt) for manual verification. New test_vad_ptt_devices.cpp covers all
four items (ABI-level + a white-box AudioEngine stereo-mix check).
Docs updated to match: voice.md, roadmap.md (decision-log entry superseding
the original webrtc-audio-processing choice), tech-stack.md, README.md,
architecture.md, CLAUDE.md, PROGRESS.md.
Still explicitly out of scope, documented not silently dropped: real
webrtc-audio-processing/AEC (no AEC/NS/AGC exists at all yet), macOS/iOS
SCREEN_AUDIO capture, process-specific loopback, and a pre-existing
RT-thread rule violation in the capture path that predates this work.
Verified: ctest 12/12 green across 3 consecutive full-suite runs (both dev
and m1-dev presets build clean); test_vad_ptt_devices passed 5 consecutive
standalone runs; manually verified live (vccli --list-devices against real
hardware, vccli --voice --input-mode vad streaming without incident).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the full voice pipeline: 14-byte binary frame header, ChaCha20-Poly1305
AEAD keyed from the TLS exporter, libopus encode/decode with FEC/PLC/DTX,
an adaptive per-ssrc jitter buffer, a miniaudio capture/playback engine, an
APM passthrough stub, and the UdpBinding/StreamAnnounce signaling chain
wired through ConnSession/SessionRegistry into a new server-side SFU
(MediaRelay) that decrypts and re-encrypts frames per channel member.
Exit criterion verified: test_m2_voice — two headless clients relay 50
encrypted Opus frames through the server; ctest --preset m1-dev is 9/9
green. Also corrects protocol.md's UdpBinding diagram, which described the
UDP-side binding packet as AEAD-sealed when it is in fact a plaintext
bootstrap frame (separate from the TCP/TLS UdpBinding ack).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
