Repository navigation
fix(cluster): survive leader death in a 3-node ring — per-peer outbound queues + self excluded from the peer set (#1168) - #1179
Merged
Conversation
…er starve heartbeats to live peers (#1168) Pre: ClusterNode::dispatch awaited each transport.send inline and sequentially, and TcpTransport::send_to opened a fresh TCP connection per message bounded by SEND_TIMEOUT=5s. A peer that accepts connections but never completes the handshake (a SIGTERM-graceful node still holding its listener) therefore consumed the full 5s of EVERY heartbeat round while the election timeout is 150-300ms — the live follower timed out, forced an election, and the new leader hit the same stall. Measured on rc.83 (issue #1168): kill the leader of a 3-node ring and the survivors churn through a new election every 1.5-11s indefinitely (94 terms in 4 minutes), never stabilising until the dead node rejoins. Fix, entirely inside TcpTransport (no signature changes; the Raft state machine, runner, and node dispatch are untouched): - TcpTransport::new pre-spawns one bounded queue (cap 256) and one sender task per configured peer. - ClusterTransport::send becomes an enqueue (try_send): it returns once the message is accepted for delivery, never after TCP I/O. Errors only for an unknown peer or a closed task; a full queue drops the message — Raft retransmits by design (the next AppendEntries resends from next_index), so dropping is the safe failure mode where buffering is not. - The per-peer task attempts one message per connection, and on failure drops that message and backs off on a doubling ladder (backoff_after_failures: 100ms -> 6.4s cap, pinned by unit test). A dead peer costs ~4 attempts in the first second, then one per 6.4s — not one per 50ms tick. - Logging is per state change: WARN once when a peer that was up goes down, INFO once when it answers again — not one WARN per tick for four minutes (the pre-fix outage produced thousands of identical lines). - send_to (the awaited one-shot path) remains for tests and one-shot callers, now delegating to a factored send_frame. Tests (all green; clippy clean): - backoff_ladder_doubles_then_caps (lib) - send_to_a_stalled_peer_returns_without_awaiting_the_tcp_exchange: a black-hole peer (accepts, never handshakes) must not delay send() past 2s — pre-fix it took the full 5s SEND_TIMEOUT. - a_stalled_peer_does_not_starve_a_healthy_peer: the miniature #1168 scenario — dispatching to a stalled peer then a healthy peer in the same batch delivers to the healthy peer immediately (pre-fix: gated behind the stalled peer's 5s timeout). - a_dead_peer_is_backed_off_not_retried_every_message: 100 sends across 1s produce <=8 connection attempts (ladder predicts ~4; pre-fix ~100). Ring-level before/after (kill-the-leader on the release binary) to be measured and posted on #1168 before merge. Files: - engine/crates/xerj-cluster/src/transport.rs (queues, task, ladder) - engine/crates/xerj-cluster/tests/transport_tests.rs (3 integration tests)
Symptom: a 3-node ring (full-membership `peers` on every node, the
convention documented in docs/clustering) churned through a new election
every few hundred ms FOREVER after any node died. rc.83 measurement:
94 terms in 4 minutes. With the per-peer outbound queues (previous
commit on this branch) the starvation went away — and the churn got
FASTER, 909 terms in 4 minutes, which is what pinned the root cause:
the transport fix was real but secondary.
Root cause: the documented convention lists the full membership on
every node, self included ("the current node's id is inferred from the
entry whose address matches its own"), and nothing removed the self
entry before the membership reached Raft.
- RaftNode computed cluster_size = peers.len()+1: a self entry made it
3 peers + 1 = 4 members of a 3-node ring, so majority() = 3 =
unanimity of the three real nodes — zero fault tolerance.
- Every node also sent its own RequestVote to itself over loopback TCP
(self was a routable peer in the transport map), and
handle_request_vote granted it (voted_for was already Some(self) from
start_election, and voted_for == candidate_id is a re-grant), so each
candidate counted its own vote TWICE: the in-memory self-vote plus
the loopback response. "Received vote votes=2 needed=3".
- After the leader died, each survivor could only "win" with
self + self + one cross-grant, so the surviving pair kept leapfrogging
terms — a new election every ~250 ms, indefinitely.
Diagnostic signature (RUST_LOG=xerj_cluster=debug, pre-fix ring):
Handling inbound Raft message node=19401 from=19401 <- self over TCP
Received vote votes=2 needed=3 <- unanimity math
Survivors started elections within ~200 µs of each other with identical
gap sequences — the churn signature, not jitter.
Fix — self is excluded at three layers, each warning when it fires:
- main.rs: peers.remove(node_id) once, after the map is parsed and
before TcpTransport::new (config validation enforces label==address,
so the map key is the address); the Raft peer_ids list filters self
as well.
- raft.rs RaftNode::new: filters p == id from the peers Vec, so
cluster_size/majority are computed over the REAL membership; adds a
peers() accessor for the assertion.
- transport.rs TcpTransport::new: filters the entry whose addr ==
listen_addr, so a node can no longer address itself over the wire.
Numbers (3-node ring on loopback, tick_ms=50, kill the leader, 240 s
observation; /tmp/clustest harness):
rc.83 binary: 94 terms / 4 min, churn never stops
transport fix only: 909 terms / 4 min, 623+645 elections
(starvation removed, the Raft-level
pathology runs at full speed)
this commit: new leader ~250 ms after the kill
(n3: one split-vote retry, term 3), then
ZERO elections for the remaining 239 s;
n2 never even timed out; unreachable
warns bounded at 2 total
Tests (all green, cargo test -p xerj-cluster: 25 incl. 3 new):
- raft::test_self_is_excluded_from_peers — full-membership input yields
peers [n2,n3] and majority() == 2, the real cluster's majority.
- transport::a_peers_entry_naming_self_is_excluded_from_the_transport —
send(self) errors "unknown peer", send_to(self) errors, the real peer
still receives term 7.
- transport::ring_with_full_membership_lists_survives_leader_death —
the acceptance scenario: three ClusterNodes over InMemoryBus, each
constructed with the FULL membership self-included exactly as the
server passes it; asserts exactly one leader before the failure,
exactly one new leader within 2 s of dropping the leader's runner,
and IDENTICAL terms 2 s later (election stability — the invariant
#1168 broke).
Known edge: a peer entry whose label differs from its address is
refused at config load (validate_peers: "the label must equal the
address"), so the label-based self match in main.rs is exhaustive for
every bootable config.
Files:
- engine/crates/xerj-server/src/main.rs (~2502) — self removed from the
peer map and the Raft peer_ids
- engine/crates/xerj-cluster/src/raft.rs::RaftNode::new — self filtered
from peers; peers() accessor
- engine/crates/xerj-cluster/src/transport.rs::TcpTransport::new — self
entry (addr == listen_addr) filtered with a warn
- engine/crates/xerj-cluster/tests/transport_tests.rs — 2 new tests
Ravandevil25
pushed a commit
to Ravandevil25/xerj
that referenced
this pull request
Oct 6, 2026
Two PRs, both born from the rc.83 three-node ring tests: - xerj-org#1179 (fix/cluster-dead-peer-starvation) closes xerj-org#1168: the full- membership [cluster] peers convention left self in the peer set, so a 3-node ring computed a 4-member majority (unanimity — zero fault tolerance) AND double-counted its own vote via the loopback re-grant. Self is excluded at three layers now. Measured on the /tmp/clustest 3-node loopback ring (tick_ms=50, kill the leader, 240 s): rc.83 churned 94 terms and never stopped; transport-fix-only churned FASTER (909 terms) which pinned the root cause; with self-exclusion a new leader lands ~250 ms after the kill and the ring then holds ZERO elections for 239 s. 25 tests green in xerj-cluster incl. the ring acceptance scenario. - xerj-org#1180 (fix/clustering-docs-honesty): the clustering page described an unshipped replication design (metadata replication, drain/ activate/remove endpoints, regions, NodeState, invented health output). Rewritten to what the ring does today, with the xerj-org#1168 numbers; docs-index cards corrected across 29 pages; flapping entry in troubleshooting is now "churn is a defect, file it"; upgrades page says full stop/start (the cluster wire has no version negotiation); llms-full.txt corrected on what rides the Raft wire. Roll: version 1.0.0-rc.84 -> 1.0.0-rc.85 (Cargo.toml + Cargo.lock 17 entries), 54 landing pages footer-rolled, CHANGELOG rc.85 section, ROADMAP review line re-verified live at cut time — ten open defects (the eleven from the rc.84 cut minus xerj-org#1168, closed by xerj-org#1179; all ten re-checked OPEN on the tracker minutes before this commit).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1168.
Two commits, one root cause found by their interaction:
acf3665aeper-peer outbound queues —ClusterNode::dispatchawaited everytransport.sendinline, and each send was a fresh connect+handshake bounded by the 5 sSEND_TIMEOUTwhile the election timeout is 150–300 ms. A dying-but-listening peer (SIGTERM: bound, silent) therefore starved heartbeats to the live peers. Now one bounded queue + one long-lived sender task per peer;sendis atry_sendenqueue; failures drop (Raft retransmits fromnext_index) and back off100 ms → 6.4 s; logging is per state change, not per tick.eb50a36d7self excluded from the peer set — the real root cause. The documented full-membershippeersconvention left each node's own entry in the set, so a 3-node ring computedcluster_size = 4→majority = 3= unanimity (zero fault tolerance), and every node sent its ownRequestVoteto itself over loopback TCP and granted it, counting its own vote twice. After a leader died, survivors "won" only via self + self + one cross-grant and leapfrogged terms forever. Fix at three layers (server config,RaftNode::new,TcpTransport::new), each warning when it fires. Full evidence and the debug-log signature in the issue comment.Measured, 3-node ring on loopback (
tick_ms=50, kill the leader, 240 s)Tests
cargo test -p xerj-cluster— 25 green, 5 new: backoff ladder pins; stalled-peer send returns without the TCP exchange; a stalled peer does not starve a healthy peer; dead peer is backed off (~4 attempts/s → ≤8 in 1 s, was ~100); self-entry excluded from the transport; ring acceptancering_with_full_membership_lists_survives_leader_death(full membership self-included as the server passes it → exactly one leader → drop the leader's runner → exactly one new leader within 2 s → identical terms 2 s later).The ES-YAML conformance gate is untouched by this change (cluster crate only; server change is config plumbing before the runner spawns).