Repository navigation
Conversation
…s the in-sync set
Root cause
----------
`replicate_write` (engine/crates/xerj-api/src/cluster_shard.rs) runs after
the primary has applied the op. With number_of_replicas=1,
`replica_quorum_acks(1) = 1`. When the single replica failed the copy,
`acked.len() (0) < needed (1)` returned `replica_nack` BEFORE the code that
proposes `SetShardInSync`, which only ran for `needed <= acked < targets`.
The replica stayed in the in-sync set, `backfill_out_of_sync_replicas`
skipped it (it only backfills copies that are out of sync), so it never
healed. Reads and failover then trusted a copy missing every refused write.
The early `targets.len() < needed` return had the same hole: replicas still
listed in sync were neither asked nor dropped. Bulk, update,
update_by_query and delete_by_query all go through `replicate_write`, so
they shared the bug.
Fix
---
Invariant: a copy that did not confirm a write the primary applied leaves
the in-sync set before the request returns, success or failure.
- `publish_in_sync(state, index, shard, nodes)`: the SetShardInSync
proposal plus the post-apply check, factored out.
- `mark_stale`: whenever acked < targets (including acked empty), publish
the confirmed subset (possibly empty = primary only) first, log WARN,
then return Ok if acked >= needed, else the same 503 replica_nack. The
reason text now says the copy was marked stale and will be backfilled.
If the shrink itself fails (no leader / commit timeout) it logs ERROR
"replica may be stale and is still in sync" and returns 503 quorum_nack.
No retry loop.
- `targets.len() < needed`: still copy to the replicas that are listed in
sync and drop the ones that do not confirm, then the same quorum_nack.
- `mark_shard_copies_stale`: where the primary applied a write but could
not even build the copy (own doc unreadable / not JSON / index closed)
in local_update, replicate_bulk_result and copy_updated_document, drop
every in-sync replica of that shard.
- Bulk: per-item `replicate_write` means the first failing item of a shard
shrinks the set; later items of that shard see the shrunk set, so it is
one proposal per shard per bulk request in the common case.
- Backfill: unchanged. The replica's runner calls
`backfill_out_of_sync_replicas` every 200 ms
(xerj-cluster/src/runner.rs REPLICA_BACKFILL_INTERVAL); an empty in-sync
overlay does not short-circuit it (router `get_replicas` returns the
empty overlay, the replica is not in it, so it asks the primary), and
`backfill_replica` re-adds it with SetShardInSync after the copy.
- Quorum and the 503 itself are unchanged (write availability is a
separate roadmap item).
Test
----
New: `a_replica_that_misses_a_refused_write_leaves_the_in_sync_set_and_is_backfilled`
(tests/cluster_metadata_replicates.rs, InMemoryBus, 3 nodes, rep 1): 200
writes to one shard with the replica's shard handler cleared -> all 503,
primary holds 200; the replica is out of the in-sync set on all 3 members;
heal -> backfill copies 200 and re-adds it; primary stopped -> search
total 200 and all 200 GETs succeed through the replica.
before: FAILED "member 2 still lists n1 in sync for shard 2 after it
missed 200 writes: [\"n1\"]"
after: 20/20 passes in a loop; whole file 10/10 runs.
Adjusted: `a_quorum_of_copies_acks_when_one_of_two_replicas_does_not` now
waits for both replicas to leave the set after the refused write and for
replica_b to be backfilled back in before its second write (before, the
refused write left both stale replicas listed in sync).
cargo test --release -p xerj-api: 1076 passed, 0 failed (117 binaries).
Lab (3 real nodes, ports 195x0, wiki 3 shards rep 1, 120k-doc bulk via
n1/n2, kill -9 n3 (replica of shard 1, primary n2) 5 s into ingest, restart
after 30 s; scripts in xerj-cluster-lab/stale/scripts/lab.sh):
before (origin/main 7458a22): n2/wiki 39,980 vs n3/.xerj_replica_wiki_1
13,178 -> diff 26,802, unchanged for 150 s after rejoin. Killing
n2: search total 93,438 -> 66,636 (-26,802) with _shards.failed 0.
after: n2/wiki 39,980 vs n3/.xerj_replica_wiki_1 39,980 -> diff 0 at
the t+20 s sample after n3 came up. Killing n2: 92,408 -> 92,408
(0 change). Killing n1: 0 change in both runs.
ES-YAML conform
905A
ance (private port 19700, throwaway data dir):
1448 passed / 0 failed / 10 skipped.
Reference coding: `xerj code elasticsearch "in-sync allocation ids mark
stale on replica failure"` could not run (no reference node is serving on
:9200 in this session; corpus list shows 0 of 21 loaded, all >30 days
old). Approach follows the well-known ES design (ReplicationOperation fails
the shard copy and removes its allocation id from the in-sync set via the
master before acking the write) from memory, APPROACH ONLY; no code copied.
Residual (not this bug, seen in the after run): kill -9 of a PRIMARY
between its local apply and the replica copy left 34 docs on n3/wiki that
n1/.xerj_replica_wiki_2 never got (12,142 vs 12,108). Those writes were
never answered; the primary died before it could mark anything stale. That
needs a primary-replica resync on restart/promotion and is a separate item.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
replicate_write(engine/crates/xerj-api/src/cluster_shard.rs) runs afterthe primary has applied the op. With number_of_replicas=1,
replica_quorum_acks(1) = 1. When the single replica failed the copy,acked.len() (0) < needed (1)returnedreplica_nackBEFORE the code thatproposes
SetShardInSync, which only ran forneeded <= acked < targets.The replica stayed in the in-sync set,
backfill_out_of_sync_replicasskipped it (it only backfills copies that are out of sync), so it never
healed. Reads and failover then trusted a copy missing every refused write.
The early
targets.len() < neededreturn had the same hole: replicas stilllisted in sync were neither asked nor dropped. Bulk, update,
update_by_query and delete_by_query all go through
replicate_write, sothey shared the bug.
Fix
Invariant: a copy that did not confirm a write the primary applied leaves
the in-sync set before the request returns, success or failure.
publish_in_sync(state, index, shard, nodes): the SetShardInSyncproposal plus the post-apply check, factored out.
mark_stale: whenever acked < targets (including acked empty), publishthe confirmed subset (possibly empty = primary only) first, log WARN,
then return Ok if acked >= needed, else the same 503 replica_nack. The
reason text now says the copy was marked stale and will be backfilled.
If the shrink itself fails (no leader / commit timeout) it logs ERROR
"replica may be stale and is still in sync" and returns 503 quorum_nack.
No retry loop.
targets.len() < needed: still copy to the replicas that are listed insync and drop the ones that do not confirm, then the same quorum_nack.
mark_shard_copies_stale: where the primary applied a write but couldnot even build the copy (own doc unreadable / not JSON / index closed)
in local_update, replicate_bulk_result and copy_updated_document, drop
every in-sync replica of that shard.
replicate_writemeans the first failing item of a shardshrinks the set; later items of that shard see the shrunk set, so it is
one proposal per shard per bulk request in the common case.
backfill_out_of_sync_replicasevery 200 ms(xerj-cluster/src/runner.rs REPLICA_BACKFILL_INTERVAL); an empty in-sync
overlay does not short-circuit it (router
get_replicasreturns theempty overlay, the replica is not in it, so it asks the primary), and
backfill_replicare-adds it with SetShardInSync after the copy.separate roadmap item).
Test
New:
a_replica_that_misses_a_refused_write_leaves_the_in_sync_set_and_is_backfilled(tests/cluster_metadata_replicates.rs, InMemoryBus, 3 nodes, rep 1): 200
writes to one shard with the replica's shard handler cleared -> all 503,
primary holds 200; the replica is out of the in-sync set on all 3 members;
heal -> backfill copies 200 and re-adds it; primary stopped -> search
total 200 and all 200 GETs succeed through the replica.
before: FAILED "member 2 still lists n1 in sync for shard 2 after it
missed 200 writes: ["n1"]"
after: 20/20 passes in a loop; whole file 10/10 runs.
Adjusted:
a_quorum_of_copies_acks_when_one_of_two_replicas_does_notnowwaits for both replicas to leave the set after the refused write and for
replica_b to be backfilled back in before its second write (before, the
refused write left both stale replicas listed in sync).
cargo test --release -p xerj-api: 1076 passed, 0 failed (117 binaries).
Lab (3 real nodes, ports 195x0, wiki 3 shards rep 1, 120k-doc bulk via
n1/n2, kill -9 n3 (replica of shard 1, primary n2) 5 s into ingest, restart
after 30 s; scripts in xerj-cluster-lab/stale/scripts/lab.sh):
before (origin/main 7458a22): n2/wiki 39,980 vs n3/.xerj_replica_wiki_1
13,178 -> diff 26,802, unchanged for 150 s after rejoin. Killing
n2: search total 93,438 -> 66,636 (-26,802) with _shards.failed 0.
after: n2/wiki 39,980 vs n3/.xerj_replica_wiki_1 39,980 -> diff 0 at
the t+20 s sample after n3 came up. Killing n2: 92,408 -> 92,408
(0 change). Killing n1: 0 change in both runs.
ES-YAML conformance (private port 19700, throwaway data dir):
1448 passed / 0 failed / 10 skipped.
Reference coding:
xerj code elasticsearch "in-sync allocation ids mark stale on replica failure"could not run (no reference node is serving on:9200 in this session; corpus list shows 0 of 21 loaded, all >30 days
old). Approach follows the well-known ES design (ReplicationOperation fails
the shard copy and removes its allocation id from the in-sync set via the
master before acking the write) from memory, APPROACH ONLY; no code copied.
Residual (not this bug, seen in the after run): kill -9 of a PRIMARY
between its local apply and the replica copy left 34 docs on n3/wiki that
n1/.xerj_replica_wiki_2 never got (12,142 vs 12,108). Those writes were
never answered; the primary died before it could mark anything stale. That
needs a primary-replica resync on restart/promotion and is a separate item.
🤖 Generated with Claude Code