Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ios): audio overhaul, Join/Leave Voice, channel-id sync fix, stereo mic capture

Browse Source 
Three iOS client problems fixed plus a new core stereo-mic capture ABI:

1. Channel-id sync bug (mic button permanently dimmed): SessionState never
   synced currentChannelId from the self user's channelId on connect, so the
   mic button (gated on currentChannelId == 0) stayed dimmed. Added
   syncSelfChannel() (mirrors macOS MainWindowController.swift:461,491,522);
   called from init/.channelList/.userJoined/.userLeft/.userUpdated/.joinResult.
   Added applyServerMuteState() + serverMuted/serverDeafened to VoiceState.

2. Join/Leave Voice button: replaced icon-only mic toggle with explicit
   text button (parity with macOS). Mute/deafen disable when not in voice.

3. IOSAudioRouter.swift (new): full AVAudioSession routing layer — input
   port selection, built-in mic orientation/polar patterns, Bluetooth
   HFP/A2DP/Off modes, Standard/Raw mic processing, stereo capture, AirPlay,
   UserDefaults persistence. AudioSessionManager delegates to it.

4. Core stereo-mic capture (append-only ABI): vc_set_capture_channels()
   lets the core open the mic device in stereo (2-ch interleaved). LocalStream
   gains capture_channels; ensure_audio_running reads it; audio_engine.cpp
   capture_accum_ + on_capture updated to channel-aware accumulation. Test
   test_stereo_mic_capture (headless, L!=R stereo round-trip). Swift wrapper
   VoiceCatClient.setCaptureChannels.

5. Settings UI rework: AVAudioSession-derived input/output tree replaces
   miniaudio device picker.

6. iOS deployment target raised to 18.0 (Package.swift + project.pbxproj).
   swift-tools-version 6.0 with swiftLanguageModes .v5.

Docs: tech-stack.md, architecture.md, voice.md, roadmap.md, building.md
updated; stale 'vc_audio_suspend/resume deferred' claims corrected.

Verified: ctest --preset dev 21/21 green; swift test 6/6 green;
xcodebuild -target VoiceCatiOS -sdk iphonesimulator BUILD SUCCEEDED.
This commit is contained in:
Talon
2026-06-19 13:17:52 +02:00
parent a10a18aebe
commit 9fc51cffc4
23 changed files with 861 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
core/src/audio/audio_engine.cpp
View File

@@ -150,12 +150,13 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
#ifdef VOICECAT_HAS_AUDIO
// Pre-allocate capture accumulators before the devices start so on_capture / on_loopback
// never allocate on the RT thread. count=0 means "empty"; the buf is sized to exactly one
// encoder frame so a memcpy into it can never overrun. The mic accumulator is mono
// (params_.capture_channels, always 1 in v1 — no stereo mic). The loopback accumulator is
// encoder frame so a memcpy into it can never overrun. The mic accumulator is sized to
// frame_samples_ * capture_channels (1 = mono, 2 = stereo interleaved — set via
// vc_set_capture_channels, e.g. iOS stereo built-in mic). The loopback accumulator is
// sized mono here as a safe default and re-sized to frame_samples_*channels in
// start_loopback_capture() once the screen stream's channel mode is known (off the RT
// thread, before the loopback device is started).
capture_accum_.buf.assign(static_cast<size_t>(frame_samples_), 0);
capture_accum_.buf.assign(static_cast<size_t>(frame_samples_) * p.capture_channels, 0);
capture_accum_.count = 0;
loopback_accum_.buf.assign(static_cast<size_t>(frame_samples_), 0);
loopback_accum_.count = 0;
@@ -428,20 +429,25 @@ void AudioEngine::on_capture(const int16_t* pcm, ma_uint32 frames) {
// then fire capture_cb_. WASAPI shared mode commonly delivers 480-sample (10 ms) callbacks
// regardless of the periodSizeInFrames hint above; passing a sub-frame chunk directly to
// opus_encode() returns OPUS_BAD_ARG (negative), silently dropping every mic frame.
// PCM here is interleaved across params_.capture_channels (1 = mono, 2 = stereo L/R —
// e.g. iOS stereo built-in mic via vc_set_capture_channels) — the accumulator was sized to
// frame_samples_*capture_channels in start(), so a memcpy into it can never overrun.
// capture_cb_ receives samples-per-channel (frame_samples_) and the channel count explicitly.
if (!capture_cb_ || frame_samples_ <= 0) return;
const int ch = std::max(1u, params_.capture_channels);
const int16_t* src = pcm;
auto remaining = static_cast<int>(frames);
auto remaining = static_cast<int>(frames) * ch;
const int full = frame_samples_ * ch;
while (remaining > 0) {
int space = frame_samples_ - capture_accum_.count;
int space = full - capture_accum_.count;
int copy = std::min(remaining, space);
std::memcpy(capture_accum_.buf.data() + capture_accum_.count, src,
static_cast<size_t>(copy) * sizeof(int16_t));
capture_accum_.count += copy;
src += copy;
remaining -= copy;
if (capture_accum_.count == frame_samples_) {
capture_cb_(0, capture_accum_.buf.data(), frame_samples_,
static_cast<int>(params_.capture_channels));
if (capture_accum_.count == full) {
capture_cb_(0, capture_accum_.buf.data(), frame_samples_, ch);
capture_accum_.count = 0;
}
}

31
core/src/audio/audio_engine.h
View File

@@ -82,7 +82,7 @@ class JitterBuffer {
// ── AudioParams ──────────────────────────────────────────────────────────────
struct AudioParams {
uint32_t sample_rate = 48000;
uint32_t capture_channels = 1; // no stereo capture device (mic) in this pass
uint32_t capture_channels = 1; // mic capture: 1 = mono, 2 = stereo (set via vc_set_capture_channels)
uint32_t playback_channels = 2; // true stereo output (see audio_engine.cpp on_playback)
uint32_t frame_ms = 20;
std::string capture_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
@@ -215,6 +215,35 @@ class AudioEngine {
}
}
}
// TEST-ONLY — stereo-aware variant: drives the capture accumulator with interleaved L/R
// PCM (channels=2) or mono (channels=1). Sizes the accumulator to frame_samples_*channels
// and invokes `cb` with the channel count passed through — mirrors feed_loopback_for_test.
// Use to verify stereo mic capture (vc_set_capture_channels → on_capture's accumulator).
void feed_capture_for_test(const int16_t* pcm, int frames_per_channel, int channels,
const CaptureCallback& cb) {
if (frame_samples_ <= 0 || !cb) return;
const int ch = std::max(1, channels);
const int full = frame_samples_ * ch;
if (static_cast<int>(capture_accum_.buf.size()) != full) {
capture_accum_.buf.assign(static_cast<size_t>(full), 0);
capture_accum_.count = 0;
}
const int16_t* src = pcm;
auto remaining = frames_per_channel * ch;
while (remaining > 0) {
int space = full - capture_accum_.count;
int copy = std::min(remaining, space);
std::memcpy(capture_accum_.buf.data() + capture_accum_.count, src,
static_cast<size_t>(copy) * sizeof(int16_t));
capture_accum_.count += copy;
src += copy;
remaining -= copy;
if (capture_accum_.count == full) {
cb(0, capture_accum_.buf.data(), frame_samples_, ch);
capture_accum_.count = 0;
}
}
}
#endif
#ifdef VOICECAT_HAS_LOOPBACK