Talon/voice-cat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ios): real echo cancellation/NR via native Voice-Processing engine

Browse Source 
iOS "voice chat" had echo and no noise suppression: real iOS AEC/NS/AGC
come only from Apple's Voice-Processing I/O unit (VPIO), but the core
plays/captures via miniaudio's plain RemoteIO units, so .voiceChat mode
alone never engaged AEC.

Core (ABI PATCH 1->2):
- vc_set_mixed_output_sink + vc_set_external_playback. In external mode the
  AudioEngine opens no hardware playback device; a mixer-timer thread drives
  on_playback (decode+mix) on a ~20ms cadence and ships the final mix to the
  sink. start() also skips the hardware capture device when the MIC stream is
  external_feed (AudioParams.external_capture).
- New white-box test test_external_playback (drives the timer with no hw).

iOS/Swift:
- StreamDescriptor.externalFeed; VoiceCatClient.setMixedOutputSink /
  setExternalPlayback wrappers.
- IOSVoiceProcessingEngine: AVAudioEngine + setVoiceProcessingEnabled; mic
  tap -> feedPcm, mixed-sink lock-free ring -> AVAudioSourceNode (both share
  the VPIO unit so AEC has its reference signal).
- IOSAudioRouter.currentConfigUsesVoiceProcessing scopes VPIO to the AEC
  presets; SessionState join/leave + reconcileVoicePath() switch paths;
  Voice Chat defaults to speaker; Settings surfaces AEC/NS state.

Known: pending on-device verification; a few bugs to fix afterward.
This commit is contained in:
Talon
2026-06-22 02:38:01 +02:00
parent e806b698ec
commit 6c17881cc0
19 changed files with 755 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
core/src/audio/audio_engine.cpp
View File

@@ -216,6 +216,10 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
}
ma_context* ctx = context_inited_ ? &context_ : nullptr;
// External playback (iOS VPIO): skip the hardware playback device entirely — a timer
// thread drives the mixer and the final mix goes to the Swift VPIO renderer (launched
// after the capture block below).
if (!external_playback_) {
// ── Playback device (opened first on iOS: commits the output route — e.g. A2DP —
// before the capture device starts. Starting stereo capture can trigger an iOS audio
// route reconfiguration; opening playback first ensures A2DP is already committed
@@ -252,7 +256,11 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
}
}
}
} // end if (!external_playback_)
// External capture (iOS VPIO / external feed): skip the hardware mic device — PCM is fed
// via inject_capture / vc_stream_feed_pcm. capture_cb_ (set above) still fires for fed frames.
if (!params_.external_capture) {
// ── Capture device (opened after playback so the output route is already committed) ──
ma_device_id cap_id{};
bool have_cap_id = !p.capture_device_id.empty() &&
@@ -277,6 +285,17 @@ bool AudioEngine::start(const AudioParams& p, CaptureCallback capture_cb) {
ma_device_uninit(&capture_device_);
}
}
} // end if (!params_.external_capture)
// External playback: launch the mixer-timer thread now that the engine is configured. It
// drives on_playback() (decode + mix all remote streams) every frame_ms and ships the final
// mix to mixed_sink_ for the Swift VPIO renderer. No hardware playback device exists.
if (external_playback_) {
mixer_scratch_.assign(
static_cast<size_t>(frame_samples_) * params_.playback_channels, 0);
mixer_timer_stop_.store(false, std::memory_order_release);
mixer_timer_thread_ = std::thread([this] { run_mixer_timer(); });
}
#endif // VOICECAT_HAS_AUDIO
return true;
@@ -322,6 +341,9 @@ void AudioEngine::stop() {
if (!running_.exchange(false)) return;
#ifdef VOICECAT_HAS_AUDIO
// External playback: stop + join the mixer-timer thread before tearing down state it reads.
mixer_timer_stop_.store(true, std::memory_order_release);
if (mixer_timer_thread_.joinable()) mixer_timer_thread_.join();
if (capture_started_) {
ma_device_stop(&capture_device_);
ma_device_uninit(&capture_device_);
@@ -338,6 +360,12 @@ void AudioEngine::stop() {
bool AudioEngine::suspend() {
#ifdef VOICECAT_HAS_AUDIO
if (!running_.load(std::memory_order_acquire)) return true;
// External playback: pause the mixer timer (there is no playback device to stop). This
// matches the VPIO renderer going down during an AVAudioSession interruption.
if (external_playback_) {
mixer_timer_stop_.store(true, std::memory_order_release);
if (mixer_timer_thread_.joinable()) mixer_timer_thread_.join();
}
bool ok = true;
if (capture_started_) ok &= (ma_device_stop(&capture_device_) == MA_SUCCESS);
if (playback_started_) ok &= (ma_device_stop(&playback_device_) == MA_SUCCESS);
@@ -350,6 +378,11 @@ bool AudioEngine::suspend() {
bool AudioEngine::resume() {
#ifdef VOICECAT_HAS_AUDIO
if (!running_.load(std::memory_order_acquire)) return true;
// External playback: relaunch the mixer timer (mixer_scratch_ is still sized from start()).
if (external_playback_ && !mixer_timer_thread_.joinable()) {
mixer_timer_stop_.store(false, std::memory_order_release);
mixer_timer_thread_ = std::thread([this] { run_mixer_timer(); });
}
bool ok = true;
if (capture_started_) ok &= (ma_device_start(&capture_device_) == MA_SUCCESS);
if (playback_started_) ok &= (ma_device_start(&playback_device_) == MA_SUCCESS);
@@ -515,6 +548,11 @@ void AudioEngine::set_pcm_sink(PcmSink cb, void* user) {
pcm_sink_.store(cb, std::memory_order_release);
}
void AudioEngine::set_mixed_output_sink(MixedSink cb, void* user) {
mixed_sink_user_.store(user, std::memory_order_relaxed);
mixed_sink_.store(cb, std::memory_order_release);
}
#ifdef VOICECAT_HAS_AUDIO
void AudioEngine::capture_data_cb(ma_device* dev, void* /*out*/,
@@ -707,6 +745,32 @@ void AudioEngine::on_playback(int16_t* out, ma_uint32 frames) {
#endif
}
// External-playback timer (iOS VPIO): with no hardware playback device to "pull" frames, this
// dedicated thread drives the mixer on a steady cadence. It is NOT a real-time audio thread (a
// plain timed worker, same class as vc_client::run_talk_timer), but it must still not allocate
// in the loop because on_playback() takes streams_mu_ via try_lock and runs the RT-safe decode
// path — so mixer_scratch_ is pre-sized in start(). Uses a deadline-based sleep to bound drift.
void AudioEngine::run_mixer_timer() {
using clock = std::chrono::steady_clock;
const uint32_t ch = params_.playback_channels;
const uint32_t spc = static_cast<uint32_t>(frame_samples_);
const auto period = std::chrono::milliseconds(params_.frame_ms);
auto next = clock::now();
while (!mixer_timer_stop_.load(std::memory_order_acquire)) {
on_playback(mixer_scratch_.data(), spc);
if (auto sink = mixed_sink_.load(std::memory_order_relaxed)) {
sink(mixed_sink_user_.load(std::memory_order_relaxed), mixer_scratch_.data(), spc, ch,
params_.sample_rate);
}
next += period;
// If we fell badly behind (e.g. the thread was descheduled), reset the deadline rather
// than spin to catch up — the VPIO renderer rides its own clock + jitter ring.
auto now = clock::now();
if (next < now) next = now + period;
std::this_thread::sleep_until(next);
}
}
#ifdef VOICECAT_HAS_LOOPBACK
void AudioEngine::loopback_data_cb(ma_device* dev, void* /*out*/, const void* in,
ma_uint32 frame_count) {

32
core/src/audio/audio_engine.h
View File

@@ -21,6 +21,7 @@
#include <memory>
#include <mutex>
#include <optional>
#include <thread>
#include <unordered_map>
#include <vector>
@@ -92,6 +93,10 @@ struct AudioParams {
uint32_t frame_ms = 20;
std::string capture_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
std::string playback_device_id; // "" = default; opaque id from AudioEngine::enumerate_devices
// External capture: the mic is fed via inject_capture/vc_stream_feed_pcm (e.g. iOS VPIO),
// so start() skips opening the hardware capture device. Derived from the MIC LocalStream's
// external_feed flag in vc_client::ensure_audio_running().
bool external_capture = false;
};
// One enumerated device, returned by AudioEngine::enumerate_devices(). `id` is an internal,
@@ -202,6 +207,18 @@ class AudioEngine {
using PcmSink = void(*)(void*, uint32_t, uint32_t, const int16_t*, size_t, uint32_t, uint32_t);
void set_pcm_sink(PcmSink cb, void* user);
// External mixed-output sink (iOS VPIO). Receives the FINAL mixed PCM (post output-volume,
// all remote streams summed) on the mixer-timer thread when external playback is enabled.
// Matching signature to vc_mixed_output_cb (cast at the C-ABI boundary). Pass nullptr to
// disable. Thread-safe (atomic store; the timer read is relaxed-load).
using MixedSink = void(*)(void*, const int16_t*, size_t, uint32_t, uint32_t);
void set_mixed_output_sink(MixedSink cb, void* user);
// External-playback mode (iOS VPIO): when enabled, start() does NOT open a hardware
// playback device; a timer thread drives the mixer (on_playback) on a ~20 ms cadence and
// ships the final mix to the mixed-output sink. Set before start() (or apply via restart).
void set_external_playback(bool enable) { external_playback_ = enable; }
// External PCM feed overload: stereo-aware variant of inject_capture. samples_per_channel
// is samples per channel; total samples written = samples_per_channel * channels.
void inject_capture(int kind, const int16_t* pcm, size_t samples_per_channel, int channels);
@@ -336,6 +353,14 @@ class AudioEngine {
bool capture_started_ = false;
bool playback_started_ = false;
// External-playback mode (iOS VPIO): no hardware playback device; this timer thread drives
// on_playback() on a ~20 ms cadence and delivers the final mix to mixed_sink_. mixer_scratch_
// is pre-sized in start() (frame_samples_ * playback_channels) so the loop never allocates.
std::thread mixer_timer_thread_;
std::atomic<bool> mixer_timer_stop_{false};
std::vector<int16_t> mixer_scratch_;
void run_mixer_timer();
#ifdef VOICECAT_HAS_LOOPBACK
// Desktop-audio loopback capture (SCREEN_AUDIO) — own lifecycle, decoupled from
// capture_device_/playback_device_ start/stop (a screen-share can start/stop independently
@@ -491,6 +516,13 @@ class AudioEngine {
std::atomic<PcmSink> pcm_sink_{nullptr};
std::atomic<void*> pcm_sink_user_{nullptr};
// External mixed-output sink + mode flag (iOS VPIO). mixed_sink_ is written by
// set_mixed_output_sink (any thread); read by run_mixer_timer via relaxed load.
// external_playback_ is read in start() to gate hardware-playback-device creation.
std::atomic<MixedSink> mixed_sink_{nullptr};
std::atomic<void*> mixed_sink_user_{nullptr};
bool external_playback_ = false;
#ifdef VOICECAT_HAS_OPUS
::OpusDREDDecoder* dred_dec_ = nullptr; // shared DRED decoder; null if unsupported
#endif