fix(net): broadcast LEFT on disconnect, add keepalive/reaper, cap PLC hiss

Three reported bugs traced to one root cause plus two missing designed features:

1. Stale users + eternal PLC hiss (root cause): ConnSession::close() silently
   erased dropped users without broadcasting UserEvent::LEFT, so peers never
   learned the user left and their audio engines never called remove_stream —
   Opus PLC synthesized comfort noise forever. Fix: broadcast_left() helper
   + close() broadcasts LEFT before erasing.

2. PLC cap (defense-in-depth): on_playback now caps pure PLC at ~2s, then
   emits digital silence so a stale stream can never hiss forever even if
   remove_stream is skipped. Resets automatically on fresh packets.

3. No timeout / no ping: client never sent Ping, server had no last_seen /
   reaper, so half-open connections (NAT timeout, wifi loss, sleep) left
   ghost users forever. Fix: client Ping every 15s with RTT measurement,
   ConnSession::last_seen bumped on every inbound TCP/UDP frame, steady_timer
   reaper sweeps every 15s and drops sessions older than 45s (configurable
   via server::Config).

4. UDP KEEPALIVE: client sends plaintext kFrameKeepalive every 5s; server
   bumps last_seen + echoes back. Keeps NAT bindings alive and lets media
   activity defer the reaper independently of TCP.

5. Graceful client disconnect: vc_disconnect() sends Disconnect{code=0} via
   a flag-based io-thread exit (no double-close race); server handles
   client-sent Disconnect with immediate close() + LEFT broadcast.

3 new tests: disconnect_left, plc_cap, reaper_timeout. 21/21 ctest green.

Docs: protocol.md §6/§7, voice.md §6, architecture.md §5, PROGRESS.md.
This commit is contained in:
2026-06-18 01:18:33 +02:00
parent cccf085a87
commit 487a561963
19 changed files with 1006 additions and 13 deletions

View File

@@ -59,10 +59,12 @@ void ConnSession::set_io(SendFn send_fn, CloseFn close_fn) {
}
void ConnSession::begin() {
touch_last_seen();
// Nothing to do at TCP level — wait for ClientHello
}
void ConnSession::on_frame(std::vector<uint8_t> frame) {
touch_last_seen(); // any inbound TCP activity keeps this session off the reaper's list
voicecat::v1::Envelope env;
if (!protocol::decode_envelope(frame, env)) return;
@@ -87,6 +89,12 @@ void ConnSession::on_frame(std::vector<uint8_t> frame) {
case voicecat::v1::Envelope::kPing:
handle_ping(env.ping());
break;
case voicecat::v1::Envelope::kDisconnect:
// Client-initiated graceful disconnect (code=0). Falls through to close() which
// broadcasts UserEvent::LEFT — same as a TCP drop, but immediate (no reaper/EOF
// wait). Gated on Authenticated so a pre-auth stray Disconnect can't skip cleanup.
if (st == State::Authenticated) handle_client_disconnect(env.disconnect());
break;
case voicecat::v1::Envelope::kLeaveChannel:
if (st == State::Authenticated)
handle_leave_channel();
@@ -172,7 +180,19 @@ void ConnSession::close() {
if (closed_.exchange(true)) return;
state_.store(State::Disconnecting, std::memory_order_release);
uint32_t uid = user_id_.load();
if (uid) registry_->remove_user(uid);
if (uid) {
// Broadcast LEFT BEFORE erasing the user, so remaining clients (and the audio
// engine's remove_stream path on each peer) learn about the departure. This
// covers ungraceful disconnects (TCP drop, crash, network loss) that previously
// silently erased the user from the registry without notifying anyone — which
// left stale users in peer client lists and kept Opus PLC hissing forever on
// peers whose remove_stream was never triggered. Mirrors kick_user's first half
// (session_registry.cpp kick_user). broadcast_left releases its shared_lock
// before remove_user acquires the unique_lock, so no deadlock; and send_envelope
// on this session is a no-op now that closed_ is true.
registry_->broadcast_left(uid, "");
registry_->remove_user(uid);
}
if (session_id_) registry_->unregister_session(session_id_);
if (close_fn_) close_fn_();
}
@@ -475,6 +495,15 @@ void ConnSession::handle_ping(const voicecat::v1::Ping& msg) {
send_envelope(env);
}
void ConnSession::handle_client_disconnect(const voicecat::v1::Disconnect& /*msg*/) {
// Graceful client-initiated disconnect (code=0). close() broadcasts UserEvent::LEFT and
// cleans up the registry — identical to the TCP-drop path, but triggered by the client's
// explicit goodbye so peers learn about the departure immediately (no reaper timeout,
// no EOF detection delay). Idempotent: the closed_ guard in close() handles the
// inevitable follow-up TCP EOF when the client closes its socket.
close();
}
void ConnSession::handle_udp_binding(uint64_t req_id, const voicecat::v1::UdpBinding& msg) {
if (msg.ack()) return; // server→client direction; ignore if echoed back

View File

@@ -15,6 +15,7 @@
#include <array>
#include <atomic>
#include <chrono>
#include <cstdint>
#include <functional>
#include <memory>
@@ -82,12 +83,24 @@ class ConnSession : public std::enable_shared_from_this<ConnSession> {
uint64_t session_id() const { return session_id_; }
uint32_t user_id() const { return user_id_; }
// Keepalive: bump last_seen to "now" on any inbound activity (TCP frame or UDP voice
// frame). The server's reaper sweeps sessions whose last_seen is older than the
// timeout (docs/protocol.md §7). Stored as steady_clock ms since epoch in an atomic so
// the reaper thread can read it lock-free.
void touch_last_seen() {
last_seen_ms_.store(std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now().time_since_epoch()).count(),
std::memory_order_release);
}
int64_t last_seen_ms() const { return last_seen_ms_.load(std::memory_order_acquire); }
private:
void handle_client_hello(uint64_t req_id, const voicecat::v1::ClientHello& msg);
void handle_auth_request(uint64_t req_id, const voicecat::v1::AuthRequest& msg);
void handle_join_channel(uint64_t req_id, const voicecat::v1::JoinChannelRequest& msg);
void handle_text_message(const voicecat::v1::TextMessage& msg);
void handle_ping(const voicecat::v1::Ping& msg);
void handle_client_disconnect(const voicecat::v1::Disconnect& msg);
void handle_udp_binding(uint64_t req_id, const voicecat::v1::UdpBinding& msg);
void handle_stream_announce(uint64_t req_id, const voicecat::v1::StreamAnnounce& msg);
void handle_stream_stop(const voicecat::v1::StreamStop& msg);
@@ -136,6 +149,11 @@ class ConnSession : public std::enable_shared_from_this<ConnSession> {
std::atomic<uint32_t> user_id_{0};
std::atomic<bool> closed_{false};
// Keepalive: steady_clock ms since epoch of the last inbound activity. Bumped by
// touch_last_seen() on every on_frame() and (via media_relay) on every UDP voice frame.
// The reaper (server.cpp) drops sessions whose last_seen is older than 45s.
std::atomic<int64_t> last_seen_ms_{0};
// M2 UDP / media
std::array<uint8_t, 16> udp_token_{};
mutable std::mutex udp_ep_mu_;

View File

@@ -130,7 +130,20 @@ void MediaRelay::on_udp_frame(const uint8_t* data, size_t len,
return;
}
// kFrameKeepalive or unknown: silently discard.
if (frame_type == voicecat::net::kFrameKeepalive) {
// Plaintext KEEPALIVE (docs/voice.md §6): identify the sender by its verified UDP
// endpoint, bump last_seen so the reaper doesn't drop a client whose TCP control
// channel is idle but whose media path is alive, and echo the keepalive back so the
// client can measure media-path RTT/loss independently of the TCP ping.
auto session = registry_->find_by_udp_endpoint(sender);
if (!session) return;
session->touch_last_seen();
// Echo back to the sender (same plaintext header).
udp_.send_to(data, len, sender);
return;
}
// Unknown frame type: silently discard.
}
} // namespace voicecat::server

View File

@@ -8,7 +8,10 @@
#include <asio.hpp>
#include <asio/signal_set.hpp>
#include <chrono>
#include <functional>
#include <memory>
#include <vector>
#include "conn_session.h"
#include "core/worker_pool.h"
@@ -148,6 +151,32 @@ int Server::run() {
io.stop();
});
// ── Keepalive reaper (docs/protocol.md §7) ─────────────────────────────────
// Sweeps every reaper_sweep_ms and drops any session whose last_seen is older than
// reaper_timeout_ms. Each close() broadcasts UserEvent::LEFT via the Tier 1 fix, so
// peers learn about the timeout exactly like a normal disconnect — their audio engines
// call remove_stream and stop PLC. This catches half-open connections (NAT timeout,
// wifi loss without RST, laptop sleep) that never produce a TCP EOF and would otherwise
// leave ghost users forever. Disabled when reaper_timeout_ms <= 0.
asio::steady_timer reaper_timer(io);
std::function<void()> arm_reaper;
if (cfg_.reaper_timeout_ms > 0) {
arm_reaper = [&] {
reaper_timer.expires_after(std::chrono::milliseconds(cfg_.reaper_sweep_ms));
reaper_timer.async_wait([&](std::error_code ec) {
if (ec || io.stopped()) return;
for (auto& sess : registry->find_stale_sessions(cfg_.reaper_timeout_ms)) {
std::printf("[server] reaper: dropping stale session %llu (user %u)\n",
static_cast<unsigned long long>(sess->session_id()),
sess->user_id());
sess->close();
}
arm_reaper();
});
};
arm_reaper();
}
std::printf("[voicecat-server] %s — TCP :%u UDP :%u\n",
cfg_.server_name.c_str(), bound, media_bound);
std::printf("[voicecat-server] fingerprint: %s\n",

View File

@@ -24,6 +24,13 @@ struct Config {
std::function<void(uint16_t)> on_ready;
// Called with the actual bound UDP media port once the relay is ready.
std::function<void(uint16_t)> on_media_ready;
// Keepalive reaper (docs/protocol.md §7): sweep every reaper_sweep_ms and drop any
// session whose last inbound TCP/UDP activity is older than reaper_timeout_ms. Defaults
// match the doc: 15s sweep, 45s timeout (3 missed 15s pongs). Set reaper_timeout_ms = 0
// to disable the reaper entirely (useful for tests that don't want it).
int64_t reaper_timeout_ms = 45000;
int64_t reaper_sweep_ms = 15000;
};
class Server {

View File

@@ -3,6 +3,7 @@
#ifdef VOICECAT_HAS_NET
#include <atomic>
#include <chrono>
#include <mutex>
#include <shared_mutex>
@@ -202,6 +203,22 @@ void SessionRegistry::broadcast_unlocked(const voicecat::v1::Envelope& env,
}
}
std::vector<std::shared_ptr<ConnSession>> SessionRegistry::find_stale_sessions(
int64_t max_age_ms) const {
std::shared_lock lk(mu_);
auto now_ms = std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now().time_since_epoch()).count();
std::vector<std::shared_ptr<ConnSession>> stale;
for (auto& [sid, weak] : sessions_) {
auto sess = weak.lock();
if (!sess) continue;
int64_t seen = sess->last_seen_ms();
if (seen == 0) continue; // not yet initialized — skip (shouldn't happen after begin())
if (now_ms - seen > max_age_ms) stale.push_back(sess);
}
return stale;
}
// ── Permissions ───────────────────────────────────────────────────────────────
void SessionRegistry::set_session_permissions(uint64_t session_id,
@@ -253,6 +270,11 @@ bool SessionRegistry::kick_user(uint32_t user_id, const std::string& reason) {
return true;
}
void SessionRegistry::broadcast_left(uint32_t user_id, const std::string& reason) {
std::shared_lock lk(mu_);
broadcast_unlocked(make_left_event(user_id, reason), /*exclude*/ 0);
}
bool SessionRegistry::ban_user(uint32_t user_id, const std::string& reason, int64_t expires_at) {
{
std::unique_lock lk(mu_);

View File

@@ -66,6 +66,12 @@ class SessionRegistry {
// Remove a user (called on disconnect after auth).
void remove_user(uint32_t user_id);
// Broadcast a UserEvent::LEFT for a user to all other sessions. Called by
// ConnSession::close() before remove_user() so remaining clients learn about an
// ungraceful disconnect (TCP drop, crash, network loss). Mirrors the first half of
// kick_user(). Takes the shared lock internally; safe to call from ConnSession::close.
void broadcast_left(uint32_t user_id, const std::string& reason);
// Move a user to a channel. Returns false if channel doesn't exist.
bool set_user_channel(uint32_t user_id, uint32_t channel_id);
@@ -82,6 +88,12 @@ class SessionRegistry {
// Broadcast an envelope to all sessions except the excluded one.
void broadcast(const voicecat::v1::Envelope& env, uint64_t exclude_session_id = 0) const;
// Return all sessions whose last_seen is older than max_age_ms (steady_clock ms), i.e.
// have not had any inbound TCP or UDP activity in that span. The reaper (server.cpp)
// calls close() on each — which broadcasts UserEvent::LEFT via the Tier 1 fix. Locks
// only to collect the list; close() runs outside the lock (mirrors kick_user's pattern).
std::vector<std::shared_ptr<ConnSession>> find_stale_sessions(int64_t max_age_ms) const;
private:
void broadcast_unlocked(const voicecat::v1::Envelope& env,
uint64_t exclude_session_id = 0) const;