fix(net): broadcast LEFT on disconnect, add keepalive/reaper, cap PLC hiss
Three reported bugs traced to one root cause plus two missing designed features:
1. Stale users + eternal PLC hiss (root cause): ConnSession::close() silently
erased dropped users without broadcasting UserEvent::LEFT, so peers never
learned the user left and their audio engines never called remove_stream —
Opus PLC synthesized comfort noise forever. Fix: broadcast_left() helper
+ close() broadcasts LEFT before erasing.
2. PLC cap (defense-in-depth): on_playback now caps pure PLC at ~2s, then
emits digital silence so a stale stream can never hiss forever even if
remove_stream is skipped. Resets automatically on fresh packets.
3. No timeout / no ping: client never sent Ping, server had no last_seen /
reaper, so half-open connections (NAT timeout, wifi loss, sleep) left
ghost users forever. Fix: client Ping every 15s with RTT measurement,
ConnSession::last_seen bumped on every inbound TCP/UDP frame, steady_timer
reaper sweeps every 15s and drops sessions older than 45s (configurable
via server::Config).
4. UDP KEEPALIVE: client sends plaintext kFrameKeepalive every 5s; server
bumps last_seen + echoes back. Keeps NAT bindings alive and lets media
activity defer the reaper independently of TCP.
5. Graceful client disconnect: vc_disconnect() sends Disconnect{code=0} via
a flag-based io-thread exit (no double-close race); server handles
client-sent Disconnect with immediate close() + LEFT broadcast.
3 new tests: disconnect_left, plc_cap, reaper_timeout. 21/21 ctest green.
Docs: protocol.md §6/§7, voice.md §6, architecture.md §5, PROGRESS.md.
This commit is contained in:
@@ -59,10 +59,12 @@ void ConnSession::set_io(SendFn send_fn, CloseFn close_fn) {
|
||||
}
|
||||
|
||||
void ConnSession::begin() {
|
||||
touch_last_seen();
|
||||
// Nothing to do at TCP level — wait for ClientHello
|
||||
}
|
||||
|
||||
void ConnSession::on_frame(std::vector<uint8_t> frame) {
|
||||
touch_last_seen(); // any inbound TCP activity keeps this session off the reaper's list
|
||||
voicecat::v1::Envelope env;
|
||||
if (!protocol::decode_envelope(frame, env)) return;
|
||||
|
||||
@@ -87,6 +89,12 @@ void ConnSession::on_frame(std::vector<uint8_t> frame) {
|
||||
case voicecat::v1::Envelope::kPing:
|
||||
handle_ping(env.ping());
|
||||
break;
|
||||
case voicecat::v1::Envelope::kDisconnect:
|
||||
// Client-initiated graceful disconnect (code=0). Falls through to close() which
|
||||
// broadcasts UserEvent::LEFT — same as a TCP drop, but immediate (no reaper/EOF
|
||||
// wait). Gated on Authenticated so a pre-auth stray Disconnect can't skip cleanup.
|
||||
if (st == State::Authenticated) handle_client_disconnect(env.disconnect());
|
||||
break;
|
||||
case voicecat::v1::Envelope::kLeaveChannel:
|
||||
if (st == State::Authenticated)
|
||||
handle_leave_channel();
|
||||
@@ -172,7 +180,19 @@ void ConnSession::close() {
|
||||
if (closed_.exchange(true)) return;
|
||||
state_.store(State::Disconnecting, std::memory_order_release);
|
||||
uint32_t uid = user_id_.load();
|
||||
if (uid) registry_->remove_user(uid);
|
||||
if (uid) {
|
||||
// Broadcast LEFT BEFORE erasing the user, so remaining clients (and the audio
|
||||
// engine's remove_stream path on each peer) learn about the departure. This
|
||||
// covers ungraceful disconnects (TCP drop, crash, network loss) that previously
|
||||
// silently erased the user from the registry without notifying anyone — which
|
||||
// left stale users in peer client lists and kept Opus PLC hissing forever on
|
||||
// peers whose remove_stream was never triggered. Mirrors kick_user's first half
|
||||
// (session_registry.cpp kick_user). broadcast_left releases its shared_lock
|
||||
// before remove_user acquires the unique_lock, so no deadlock; and send_envelope
|
||||
// on this session is a no-op now that closed_ is true.
|
||||
registry_->broadcast_left(uid, "");
|
||||
registry_->remove_user(uid);
|
||||
}
|
||||
if (session_id_) registry_->unregister_session(session_id_);
|
||||
if (close_fn_) close_fn_();
|
||||
}
|
||||
@@ -475,6 +495,15 @@ void ConnSession::handle_ping(const voicecat::v1::Ping& msg) {
|
||||
send_envelope(env);
|
||||
}
|
||||
|
||||
void ConnSession::handle_client_disconnect(const voicecat::v1::Disconnect& /*msg*/) {
|
||||
// Graceful client-initiated disconnect (code=0). close() broadcasts UserEvent::LEFT and
|
||||
// cleans up the registry — identical to the TCP-drop path, but triggered by the client's
|
||||
// explicit goodbye so peers learn about the departure immediately (no reaper timeout,
|
||||
// no EOF detection delay). Idempotent: the closed_ guard in close() handles the
|
||||
// inevitable follow-up TCP EOF when the client closes its socket.
|
||||
close();
|
||||
}
|
||||
|
||||
void ConnSession::handle_udp_binding(uint64_t req_id, const voicecat::v1::UdpBinding& msg) {
|
||||
if (msg.ack()) return; // server→client direction; ignore if echoed back
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
|
||||
#include <array>
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
@@ -82,12 +83,24 @@ class ConnSession : public std::enable_shared_from_this<ConnSession> {
|
||||
uint64_t session_id() const { return session_id_; }
|
||||
uint32_t user_id() const { return user_id_; }
|
||||
|
||||
// Keepalive: bump last_seen to "now" on any inbound activity (TCP frame or UDP voice
|
||||
// frame). The server's reaper sweeps sessions whose last_seen is older than the
|
||||
// timeout (docs/protocol.md §7). Stored as steady_clock ms since epoch in an atomic so
|
||||
// the reaper thread can read it lock-free.
|
||||
void touch_last_seen() {
|
||||
last_seen_ms_.store(std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now().time_since_epoch()).count(),
|
||||
std::memory_order_release);
|
||||
}
|
||||
int64_t last_seen_ms() const { return last_seen_ms_.load(std::memory_order_acquire); }
|
||||
|
||||
private:
|
||||
void handle_client_hello(uint64_t req_id, const voicecat::v1::ClientHello& msg);
|
||||
void handle_auth_request(uint64_t req_id, const voicecat::v1::AuthRequest& msg);
|
||||
void handle_join_channel(uint64_t req_id, const voicecat::v1::JoinChannelRequest& msg);
|
||||
void handle_text_message(const voicecat::v1::TextMessage& msg);
|
||||
void handle_ping(const voicecat::v1::Ping& msg);
|
||||
void handle_client_disconnect(const voicecat::v1::Disconnect& msg);
|
||||
void handle_udp_binding(uint64_t req_id, const voicecat::v1::UdpBinding& msg);
|
||||
void handle_stream_announce(uint64_t req_id, const voicecat::v1::StreamAnnounce& msg);
|
||||
void handle_stream_stop(const voicecat::v1::StreamStop& msg);
|
||||
@@ -136,6 +149,11 @@ class ConnSession : public std::enable_shared_from_this<ConnSession> {
|
||||
std::atomic<uint32_t> user_id_{0};
|
||||
std::atomic<bool> closed_{false};
|
||||
|
||||
// Keepalive: steady_clock ms since epoch of the last inbound activity. Bumped by
|
||||
// touch_last_seen() on every on_frame() and (via media_relay) on every UDP voice frame.
|
||||
// The reaper (server.cpp) drops sessions whose last_seen is older than 45s.
|
||||
std::atomic<int64_t> last_seen_ms_{0};
|
||||
|
||||
// M2 UDP / media
|
||||
std::array<uint8_t, 16> udp_token_{};
|
||||
mutable std::mutex udp_ep_mu_;
|
||||
|
||||
@@ -130,7 +130,20 @@ void MediaRelay::on_udp_frame(const uint8_t* data, size_t len,
|
||||
return;
|
||||
}
|
||||
|
||||
// kFrameKeepalive or unknown: silently discard.
|
||||
if (frame_type == voicecat::net::kFrameKeepalive) {
|
||||
// Plaintext KEEPALIVE (docs/voice.md §6): identify the sender by its verified UDP
|
||||
// endpoint, bump last_seen so the reaper doesn't drop a client whose TCP control
|
||||
// channel is idle but whose media path is alive, and echo the keepalive back so the
|
||||
// client can measure media-path RTT/loss independently of the TCP ping.
|
||||
auto session = registry_->find_by_udp_endpoint(sender);
|
||||
if (!session) return;
|
||||
session->touch_last_seen();
|
||||
// Echo back to the sender (same plaintext header).
|
||||
udp_.send_to(data, len, sender);
|
||||
return;
|
||||
}
|
||||
|
||||
// Unknown frame type: silently discard.
|
||||
}
|
||||
|
||||
} // namespace voicecat::server
|
||||
|
||||
@@ -8,7 +8,10 @@
|
||||
#include <asio.hpp>
|
||||
#include <asio/signal_set.hpp>
|
||||
|
||||
#include <chrono>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <vector>
|
||||
|
||||
#include "conn_session.h"
|
||||
#include "core/worker_pool.h"
|
||||
@@ -148,6 +151,32 @@ int Server::run() {
|
||||
io.stop();
|
||||
});
|
||||
|
||||
// ── Keepalive reaper (docs/protocol.md §7) ─────────────────────────────────
|
||||
// Sweeps every reaper_sweep_ms and drops any session whose last_seen is older than
|
||||
// reaper_timeout_ms. Each close() broadcasts UserEvent::LEFT via the Tier 1 fix, so
|
||||
// peers learn about the timeout exactly like a normal disconnect — their audio engines
|
||||
// call remove_stream and stop PLC. This catches half-open connections (NAT timeout,
|
||||
// wifi loss without RST, laptop sleep) that never produce a TCP EOF and would otherwise
|
||||
// leave ghost users forever. Disabled when reaper_timeout_ms <= 0.
|
||||
asio::steady_timer reaper_timer(io);
|
||||
std::function<void()> arm_reaper;
|
||||
if (cfg_.reaper_timeout_ms > 0) {
|
||||
arm_reaper = [&] {
|
||||
reaper_timer.expires_after(std::chrono::milliseconds(cfg_.reaper_sweep_ms));
|
||||
reaper_timer.async_wait([&](std::error_code ec) {
|
||||
if (ec || io.stopped()) return;
|
||||
for (auto& sess : registry->find_stale_sessions(cfg_.reaper_timeout_ms)) {
|
||||
std::printf("[server] reaper: dropping stale session %llu (user %u)\n",
|
||||
static_cast<unsigned long long>(sess->session_id()),
|
||||
sess->user_id());
|
||||
sess->close();
|
||||
}
|
||||
arm_reaper();
|
||||
});
|
||||
};
|
||||
arm_reaper();
|
||||
}
|
||||
|
||||
std::printf("[voicecat-server] %s — TCP :%u UDP :%u\n",
|
||||
cfg_.server_name.c_str(), bound, media_bound);
|
||||
std::printf("[voicecat-server] fingerprint: %s\n",
|
||||
|
||||
@@ -24,6 +24,13 @@ struct Config {
|
||||
std::function<void(uint16_t)> on_ready;
|
||||
// Called with the actual bound UDP media port once the relay is ready.
|
||||
std::function<void(uint16_t)> on_media_ready;
|
||||
|
||||
// Keepalive reaper (docs/protocol.md §7): sweep every reaper_sweep_ms and drop any
|
||||
// session whose last inbound TCP/UDP activity is older than reaper_timeout_ms. Defaults
|
||||
// match the doc: 15s sweep, 45s timeout (3 missed 15s pongs). Set reaper_timeout_ms = 0
|
||||
// to disable the reaper entirely (useful for tests that don't want it).
|
||||
int64_t reaper_timeout_ms = 45000;
|
||||
int64_t reaper_sweep_ms = 15000;
|
||||
};
|
||||
|
||||
class Server {
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
#ifdef VOICECAT_HAS_NET
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <mutex>
|
||||
#include <shared_mutex>
|
||||
|
||||
@@ -202,6 +203,22 @@ void SessionRegistry::broadcast_unlocked(const voicecat::v1::Envelope& env,
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<std::shared_ptr<ConnSession>> SessionRegistry::find_stale_sessions(
|
||||
int64_t max_age_ms) const {
|
||||
std::shared_lock lk(mu_);
|
||||
auto now_ms = std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now().time_since_epoch()).count();
|
||||
std::vector<std::shared_ptr<ConnSession>> stale;
|
||||
for (auto& [sid, weak] : sessions_) {
|
||||
auto sess = weak.lock();
|
||||
if (!sess) continue;
|
||||
int64_t seen = sess->last_seen_ms();
|
||||
if (seen == 0) continue; // not yet initialized — skip (shouldn't happen after begin())
|
||||
if (now_ms - seen > max_age_ms) stale.push_back(sess);
|
||||
}
|
||||
return stale;
|
||||
}
|
||||
|
||||
// ── Permissions ───────────────────────────────────────────────────────────────
|
||||
|
||||
void SessionRegistry::set_session_permissions(uint64_t session_id,
|
||||
@@ -253,6 +270,11 @@ bool SessionRegistry::kick_user(uint32_t user_id, const std::string& reason) {
|
||||
return true;
|
||||
}
|
||||
|
||||
void SessionRegistry::broadcast_left(uint32_t user_id, const std::string& reason) {
|
||||
std::shared_lock lk(mu_);
|
||||
broadcast_unlocked(make_left_event(user_id, reason), /*exclude*/ 0);
|
||||
}
|
||||
|
||||
bool SessionRegistry::ban_user(uint32_t user_id, const std::string& reason, int64_t expires_at) {
|
||||
{
|
||||
std::unique_lock lk(mu_);
|
||||
|
||||
@@ -66,6 +66,12 @@ class SessionRegistry {
|
||||
// Remove a user (called on disconnect after auth).
|
||||
void remove_user(uint32_t user_id);
|
||||
|
||||
// Broadcast a UserEvent::LEFT for a user to all other sessions. Called by
|
||||
// ConnSession::close() before remove_user() so remaining clients learn about an
|
||||
// ungraceful disconnect (TCP drop, crash, network loss). Mirrors the first half of
|
||||
// kick_user(). Takes the shared lock internally; safe to call from ConnSession::close.
|
||||
void broadcast_left(uint32_t user_id, const std::string& reason);
|
||||
|
||||
// Move a user to a channel. Returns false if channel doesn't exist.
|
||||
bool set_user_channel(uint32_t user_id, uint32_t channel_id);
|
||||
|
||||
@@ -82,6 +88,12 @@ class SessionRegistry {
|
||||
// Broadcast an envelope to all sessions except the excluded one.
|
||||
void broadcast(const voicecat::v1::Envelope& env, uint64_t exclude_session_id = 0) const;
|
||||
|
||||
// Return all sessions whose last_seen is older than max_age_ms (steady_clock ms), i.e.
|
||||
// have not had any inbound TCP or UDP activity in that span. The reaper (server.cpp)
|
||||
// calls close() on each — which broadcasts UserEvent::LEFT via the Tier 1 fix. Locks
|
||||
// only to collect the list; close() runs outside the lock (mirrors kick_user's pattern).
|
||||
std::vector<std::shared_ptr<ConnSession>> find_stale_sessions(int64_t max_age_ms) const;
|
||||
|
||||
private:
|
||||
void broadcast_unlocked(const voicecat::v1::Envelope& env,
|
||||
uint64_t exclude_session_id = 0) const;
|
||||
|
||||
Reference in New Issue
Block a user