8000 fix(cluster): survive leader death in a 3-node ring — per-peer outbound queues + self excluded from the peer set (#1168) by xerj-org · Pull Request #1179 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

fix(cluster): survive leader death in a 3-node ring — per-peer outbound queues + self excluded from the peer set (#1168) - #1179

Merged
xerj-org merged 2 commits into
mainfrom
fix/cluster-dead-peer-starvation
Oct 6, 2026
Merged

xerj-org merged 2 commits into
mainfrom
fix/cluster-dead-peer-starvation

Conversation

@xerj-org
@xerj-org xerj-org commented Oct 6, 2026
Copy link
Copy Markdown
Owner

Closes #1168.

Two commits, one root cause found by their interaction:

  1. acf3665ae per-peer outbound queues — ClusterNode::dispatch awaited every transport.send inline, and each send was a fresh connect+handshake bounded by the 5 s SEND_TIMEOUT while the election timeout is 150–300 ms. A dying-but-listening peer (SIGTERM: bound, silent) therefore starved heartbeats to the live peers. Now one bounded queue + one long-lived sender task per peer; send is a try_send enqueue; failures drop (Raft retransmits from next_index) and back off 100 ms → 6.4 s; logging is per state change, not per tick.

  2. eb50a36d7 self excluded from the peer set — the real root cause. The documented full-membership peers convention left each node's own entry in the set, so a 3-node ring computed cluster_size = 4 → majority = 3 = unanimity (zero fault tolerance), and every node sent its own RequestVote to itself over loopback TCP and granted it, counting its own vote twice. After a leader died, survivors "won" only via self + self + one cross-grant and leapfrogged terms forever. Fix at three layers (server config, RaftNode::new, TcpTransport::new), each warning when it fires. Full evidence and the debug-log signature in the issue comment.

Measured, 3-node ring on loopback (tick_ms=50, kill the leader, 240 s)

build result
rc.83 94 terms / 4 min, churn never stops
commit 1 only 909 terms / 4 min, 623+645 elections (starvation removed, Raft pathology at full speed — this is what pinned the root cause)
this PR new leader ~250 ms after the kill, then zero elections for the remaining 239 s

Tests

cargo test -p xerj-cluster — 25 green, 5 new: backoff ladder pins; stalled-peer send returns without the TCP exchange; a stalled peer does not starve a healthy peer; dead peer is backed off (~4 attempts/s → ≤8 in 1 s, was ~100); self-entry excluded from the transport; ring acceptance ring_with_full_membership_lists_survives_leader_death (full membership self-included as the server passes it → exactly one leader → drop the leader's runner → exactly one new leader within 2 s → identical terms 2 s later).

The ES-YAML conformance gate is untouched by this change (cluster crate only; server change is config plumbing before the runner spawns).

…er starve heartbeats to live peers (#1168)

Pre: ClusterNode::dispatch awaited each transport.send inline and
sequentially, and TcpTransport::send_to opened a fresh TCP connection per
message bounded by SEND_TIMEOUT=5s. A peer that accepts connections but
never completes the handshake (a SIGTERM-graceful node still holding its
listener) therefore consumed the full 5s of EVERY heartbeat round while
the election timeout is 150-300ms — the live follower timed out, forced
an election, and the new leader hit the same stall. Measured on rc.83
(issue #1168): kill the leader of a 3-node ring and the survivors churn
through a new election every 1.5-11s indefinitely (94 terms in 4
minutes), never stabilising until the dead node rejoins.

Fix, entirely inside TcpTransport (no signature changes; the Raft state
machine, runner, and node dispatch are untouched):

- TcpTransport::new pre-spawns one bounded queue (cap 256) and one
  sender task per configured peer.
- ClusterTransport::send becomes an enqueue (try_send): it returns once
  the message is accepted for delivery, never after TCP I/O. Errors only
  for an unknown peer or a closed task; a full queue drops the message —
  Raft retransmits by design (the next AppendEntries resends from
  next_index), so dropping is the safe failure mode where buffering is
  not.
- The per-peer task attempts one message per connection, and on failure
  drops that message and backs off on a doubling ladder
  (backoff_after_failures: 100ms -> 6.4s cap, pinned by unit test). A
  dead peer costs ~4 attempts in the first second, then one per 6.4s —
  not one per 50ms tick.
- Logging is per state change: WARN once when a peer that was up goes
  down, INFO once when it answers again — not one WARN per tick for four
  minutes (the pre-fix outage produced thousands of identical lines).
- send_to (the awaited one-shot path) remains for tests and one-shot
  callers, now delegating to a factored send_frame.

Tests (all green; clippy clean):
- backoff_ladder_doubles_then_caps (lib)
- send_to_a_stalled_peer_returns_without_awaiting_the_tcp_exchange: a
  black-hole peer (accepts, never handshakes) must not delay send() past
  2s — pre-fix it took the full 5s SEND_TIMEOUT.
- a_stalled_peer_does_not_starve_a_healthy_peer: the miniature #1168
  scenario — dispatching to a stalled peer then a healthy peer in the
  same batch delivers to the healthy peer immediately (pre-fix: gated
  behind the stalled peer's 5s timeout).
- a_dead_peer_is_backed_off_not_retried_every_message: 100 sends across
  1s produce <=8 connection attempts (ladder predicts ~4; pre-fix ~100).

Ring-level before/after (kill-the-leader on the release binary) to be
measured and posted on #1168 before merge.

Files:
- engine/crates/xerj-cluster/src/transport.rs (queues, task, ladder)
- engine/crates/xerj-cluster/tests/transport_tests.rs (3 integration tests)
Symptom: a 3-node ring (full-membership `peers` on every node, the
convention documented in docs/clustering) churned through a new election
every few hundred ms FOREVER after any node died. rc.83 measurement:
94 terms in 4 minutes. With the per-peer outbound queues (previous
commit on this branch) the starvation went away — and the churn got
FASTER, 909 terms in 4 minutes, which is what pinned the root cause:
the transport fix was real but secondary.

Root cause: the documented convention lists the full membership on
every node, self included ("the current node's id is inferred from the
entry whose address matches its own"), and nothing removed the self
entry before the membership reached Raft.

- RaftNode computed cluster_size = peers.len()+1: a self entry made it
  3 peers + 1 = 4 members of a 3-node ring, so majority() = 3 =
  unanimity of the three real nodes — zero fault tolerance.
- Every node also sent its own RequestVote to itself over loopback TCP
  (self was a routable peer in the transport map), and
  handle_request_vote granted it (voted_for was already Some(self) from
  start_election, and voted_for == candidate_id is a re-grant), so each
  candidate counted its own vote TWICE: the in-memory self-vote plus
  the loopback response. "Received vote votes=2 needed=3".
- After the leader died, each survivor could only "win" with
  self + self + one cross-grant, so the surviving pair kept leapfrogging
  terms — a new election every ~250 ms, indefinitely.

Diagnostic signature (RUST_LOG=xerj_cluster=debug, pre-fix ring):
  Handling inbound Raft message node=19401 from=19401   <- self over TCP
  Received vote votes=2 needed=3                        <- unanimity math
Survivors started elections within ~200 µs of each other with identical
gap sequences — the churn signature, not jitter.

Fix — self is excluded at three layers, each warning when it fires:
- main.rs: peers.remove(node_id) once, after the map is parsed and
  before TcpTransport::new (config validation enforces label==address,
  so the map key is the address); the Raft peer_ids list filters self
  as well.
- raft.rs RaftNode::new: filters p == id from the peers Vec, so
  cluster_size/majority are computed over the REAL membership; adds a
  peers() accessor for the assertion.
- transport.rs TcpTransport::new: filters the entry whose addr ==
  listen_addr, so a node can no longer address itself over the wire.

Numbers (3-node ring on loopback, tick_ms=50, kill the leader, 240 s
observation; /tmp/clustest harness):
  rc.83 binary:                94 terms / 4 min, churn never stops
  transport fix only:          909 terms / 4 min, 623+645 elections
                               (starvation removed, the Raft-level
                               pathology runs at full speed)
  this commit:                 new leader ~250 ms after the kill
                               (n3: one split-vote retry, term 3), then
                               ZERO elections for the remaining 239 s;
                               n2 never even timed out; unreachable
                               warns bounded at 2 total

Tests (all green, cargo test -p xerj-cluster: 25 incl. 3 new):
- raft::test_self_is_excluded_from_peers — full-membership input yields
  peers [n2,n3] and majority() == 2, the real cluster's majority.
- transport::a_peers_entry_naming_self_is_excluded_from_the_transport —
  send(self) errors "unknown peer", send_to(self) errors, the real peer
  still receives term 7.
- transport::ring_with_full_membership_lists_survives_leader_death —
  the acceptance scenario: three ClusterNodes over InMemoryBus, each
  constructed with the FULL membership self-included exactly as the
  server passes it; asserts exactly one leader before the failure,
  exactly one new leader within 2 s of dropping the leader's runner,
  and IDENTICAL terms 2 s later (election stability — the invariant
  #1168 broke).

Known edge: a peer entry whose label differs from its address is
refused at config load (validate_peers: "the label must equal the
address"), so the label-based self match in main.rs is exhaustive for
every bootable config.

Files:
- engine/crates/xerj-server/src/main.rs (~2502) — self removed from the
  peer map and the Raft peer_ids
- engine/crates/xerj-cluster/src/raft.rs::RaftNode::new — self filtered
  from peers; peers() accessor
- engine/crates/xerj-cluster/src/transport.rs::TcpTransport::new — self
  entry (addr == listen_addr) filtered with a warn
- engine/crates/xerj-cluster/tests/transport_tests.rs — 2 new tests
@cla-bot cla-bot Bot added the cla-signed label Oct 6, 2026
@xerj-org
xerj-org merged commit f32f939 into main Oct 6, 2026
22 checks passed
Ravandevil25 pushed a commit to Ravandevil25/xerj that referenced this pull request Oct 6, 2026
Two PRs, both born from the rc.83 three-node ring tests:

- xerj-org#1179 (fix/cluster-dead-peer-starvation) closes xerj-org#1168: the full-
  membership [cluster] peers convention left self in the peer set, so a
  3-node ring computed a 4-member majority (unanimity — zero fault
  tolerance) AND double-counted its own vote via the loopback
  re-grant. Self is excluded at three layers now. Measured on the
  /tmp/clustest 3-node loopback ring (tick_ms=50, kill the leader,
  240 s): rc.83 churned 94 terms and never stopped; transport-fix-only
  churned FASTER (909 terms) which pinned the root cause; with
  self-exclusion a new leader lands ~250 ms after the kill and the
  ring then holds ZERO elections for 239 s. 25 tests green in
  xerj-cluster incl. the ring acceptance scenario.

- xerj-org#1180 (fix/clustering-docs-honesty): the clustering page described
  an unshipped replication design (metadata replication, drain/
  activate/remove endpoints, regions, NodeState, invented health
  output). Rewritten to what the ring does today, with the xerj-org#1168
  numbers; docs-index cards corrected across 29 pages; flapping entry
  in troubleshooting is now "churn is a defect, file it"; upgrades
  page says full stop/start (the cluster wire has no version
  negotiation); llms-full.txt corrected on what rides the Raft wire.

Roll: version 1.0.0-rc.84 -> 1.0.0-rc.85 (Cargo.toml + Cargo.lock 17
entries), 54 landing pages footer-rolled, CHANGELOG rc.85 section,
ROADMAP review line re-verified live at cut time — ten open defects
(the eleven from the rc.84 cut minus xerj-org#1168, closed by xerj-org#1179; all ten
re-checked OPEN on the tracker minutes before this commit).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cluster: killing the leader in a 3-node ring causes perpetual election churn — no stable leader until the dead node rejoins

1 participant

0