feat(cluster): acknowledge document writes at a quorum of copies by xerj-org · Pull Request #2303 · xerj-org/xerj

xerj-org · 2026-10-23T23:34:01Z

What this changes, and why

Part of #1170. This is the quorum-ack half of step 2. It does not finish the issue.

On current main, before this branch:

Cluster metadata (create, delete, mapping, settings) already goes through the Raft log (feat(cluster): replicate index metadata through the Raft log #2257).
Document writes are routed to the primary (feat(cluster): route document writes and search to shard primaries #2271) and copied to every replica named by index.number_of_replicas at create (feat(cluster): replicate acknowledged document writes onto shard replicas #2287). The caller was not acknowledged until every placed replica confirmed. One replica that did not answer was HTTP 503 replica_not_acknowledged, even when the primary and another replica had the write. That is not a quorum.
Read routing / fan-out for GET /{index}/_doc/{id} and a single-index search is already in (feat(cluster): route document writes and search to shard primaries #2271, feat(cluster): replicate acknowledged document writes onto shard replicas #2287). Search still does not merge aggregations, suggest, collapse, rescore, profile, search_after, or scroll, and an unscoped or wildcard search stays on the node that received it.
Snapshot install is not implemented. The server still does not open FileRaftLog. A joining or restarted node does not catch up.

WalReplicator's ReplicationMode::Quorum is a majority of the replica list alone (replicas.len() / 2 + 1). On two replicas that is both of them, so wiring it would not have changed a 3-node ring. The document path uses a majority of the copies placed at create: copies / 2 + 1, and the primary already counts.

placed replicas	copies	quorum	replica acks required
0	1	1	0
1	2	2	1
2	3	2	1

With two replicas, one confirmation is enough. The replica that does not confirm is removed from the in-sync set by ClusterCommand::SetShardInSync before the caller is acknowledged. Later GET and search failover use that set, not the create-time list. The placed count does not shrink, so the primary alone is still not a quorum of three copies. A replica that fell out is not added back: nothing replays the writes it missed.

number_of_replicas 0 or 1 is unchanged. [cluster] enabled = false returns before any of this. There is still no wait_for_active_shards, no replica promotion, and no snapshot catch-up.

What remains for #1170

Snapshot install so a joining or restarted node catches up (step 4). The Raft log in the server is still in memory. Restart drops the shard plan and the in-sync set.
wait_for_active_shards.
Changing number_of_replicas after create does not move or backfill copies.
No replica promotion. A write still needs the primary.
_bulk, _update, update-by-query, and native /v1 ingest do not route or replicate.
Search still does not merge aggregations, suggest, collapse, rescore, profile, search_after, or scroll. Unscoped /_search, comma lists, and wildcards stay on the receiving node.
A replica removed from the in-sync set stays out for the life of the process.

Evidence

rustc 1.97.1, from engine/, cargo +1.97.1

cargo fmt --all -- --check                                          exit 0
cargo clippy -j 8 -p xerj-cluster -p xerj-engine -p xerj-api \
  --all-targets -- -D warnings                                      exit 0
cargo test -j 8 -p xerj-cluster --lib                               67 passed
cargo test -j 8 -p xerj-cluster                                     exit 0
  (integration binary: 10 passed; doc-tests: 2 passed, 1 ignored)
cargo test -j 8 -p xerj-engine --lib                                822 passed
cargo test -j 8 -p xerj-engine --test shard_router_write_path       2 passed
cargo test -j 8 -p xerj-api --test cluster_metadata_replicates      6 passed in 1.35s

a_quorum_of_copies_acks_when_one_of_two_replicas_does_not on an in-process 3-node ring with number_of_replicas: 2:

Both replica handlers cleared: PUT is 503 replica_not_acknowledged (the primary alone is not a quorum).
One replica handler restored: PUT succeeds. The live replica store has the document. The stopped replica's store does not.
Every member's in-sync list for that shard becomes that one replica. Placement stays 2.
After the primary's handler is cleared, GET and match_all from the stale member return the document with _index orders.

The same test failed while replicate_write returned on the first replica error, which is the previous behaviour:

primary plus one replica is a quorum of three copies: 503 Service Unavailable
{"error":{"type":"replica_not_acknowledged","reason":"primary applied the write on [orders] but replica [n2] did not acknowledge it: shard call to n2 failed: no shard handler registered on that member. ..."},"status":503}

Restoring the quorum tally made that test pass again (0.20s). The existing ring tests still pass, including number_of_replicas: 1 (that one replica must confirm) and cluster disabled (the document stays on the receiving node, no replica store, search total 1).

No benchmark was run. Single-node requests return before this path because has_cluster() is false. That is a control-flow claim, not a measured latency.

Claim	Test	Result
One of two replicas is enough to ack	`a_quorum_of_copies_acks_when_one_of_two_replicas_does_not`	passed
Primary alone, with two replicas placed, is 503	same test, both handlers cleared	passed
A missed replica is not what a later read uses	same test, GET and search after the primary handler is cleared	passed
`number_of_replicas` 1 still needs that replica; single-node stays local	`an_acknowledged_write_is_on_the_replica_and_readable_after_the_primary_stops`, `cluster_disabled_create_stays_on_the_node_that_received_it`	passed
Returning on the first replica error rejects a one-of-two ack	temporary revert of the tally, same test	failed as quoted above

Checks

cargo fmt --all (CI runs rustfmt --check and clippy -D warnings)
Scoped release build of the crates touched: cargo build --release -j 32 -p <crate>
cargo test -p xerj-cluster, cargo test -p xerj-engine --lib, cargo test -p xerj-engine --test shard_router_write_path, and cargo test -p xerj-api --test cluster_metadata_replicates
ES-YAML conformance suite at 0 failed
New ES-compatible behaviour has a matching YAML case under engine/tests/es-compat-yaml/yaml/
Docs updated (CHANGELOG.md, landing/docs/clustering.html)
Applicable audit in docs/CONTRIBUTION_REVIEW.md
Operational-recipe audit: N/A. The clustering page edits are a description of current behaviour. The existing bootstrap TOML blocks were not changed and were not executed.

Not run: a release build, cargo test -p xerj-api beyond cluster_metadata_replicates, and the ES-YAML suite. The suite boots one node with cluster disabled; the new branch returns before routing when no handle is attached. No new YAML case, because the behaviour needs a ring. A real TCP three-node boot was not run; the ring test is in-process. The path where a quorum of replicas confirm but the in-sync Raft proposal then fails was not forced.

Review notes: the effective diff is the replica ack tally and the in-sync publish. Single-node create and document writes still return before replicate_write. A write is acknowledged only after the in-sync entry has been applied on the primary when the set actually shrinks. A 503 does not shrink the set. The in-sync set is invocation-local in the same way as the shard plan: the server Raft log is still in memory. Failure of the in-sync proposal does not roll back the primary write; the response stays 503 and a retry of the index is the recovery. No performance number is claimed.

Provenance

Written by: AI agent (Cursor Grok 4.7), in this cloud-agent run. The account that opens the pull request is the human who can be asked about it.
Verified (commands run, output observed): the cargo fmt, clippy, and test commands listed above, on rustc 1.97.1, including the six-test ring file and the failing run under the first-replica-error mutation.
Assumed (not tested, and what would falsify it): a TCP three-node boot behaves like the in-process ring. A write acknowledged before any in-sync replica has the document, or a GET after the primary stops that returns the replica which missed the write, would falsify the quorum claim. A single-node _doc or _search failure in the ES-YAML suite would falsify the "single-node unchanged" claim. No Co-Authored-By trailer was added.

Part of #1170. A write on a shard with two replicas waited for both. One replica that did not answer turned an otherwise stored document into HTTP 503, even though the primary and the other replica had it. The quorum is a majority of the copies placed at create (copies / 2 + 1). The primary already counts, so one of two replica confirmations is enough. The replica that misses that write is removed from the in-sync set by a Raft entry before the caller is acknowledged, and later reads do not use it. The placed count does not shrink, so the primary alone is still not a quorum of three copies. number_of_replicas 0 or 1 is unchanged. Single-node mode never enters this path. Nothing replays the writes a dropped replica missed. Snapshot install, wait_for_active_shards, and replica promotion stay open. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>

CHANGELOG.md was the only conflict. Both Unreleased Added entries are kept: the quorum document-write acknowledgement from this branch, and the DeepSeek Harness profile bundle that landed on main. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>

xerj-org · 2026-10-24T02:01:58Z

Xerj Merger made the fix: document writes in a cluster are now acknowledged once a majority of the placed copies (primary included, copies/2+1) confirm, instead of requiring every replica; one slow or missing replica no longer turns a quorum-held write into a 503. Adds cluster tests and updates the clustering docs. Part of #1170 (issue stays open).

Motivation: #2308 (semantic wildcard skip) and #2303 (cluster quorum acks) landed on main after this branch, and the changelog also picked up the #1122 large-JSON ingest bound. The branch no longer merged cleanly. Resolution: keep both Unreleased Fixed entries — the proxy semantic resume (#2306) and the large-JSON bound (#1122). The Rust overlap in xerj-autoindex and xerj-engine auto-merged. The proxy-width resume patch is unchanged against main, and main's quorum, wildcard-skip, and ingest-bound behavior is unchanged. No .github files were edited. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>

Nine pull requests landed on main after the rc.99 cut. They are cited in [1.0.0-rc.99] - 2026-10-23, and [Unreleased] stays empty. - #2245 fixes #2234: _refresh no longer waits on a background merge. - #2250 fixes #1612: autoindex estimates neural embedding time, and docs say --embed-mode is fixed at ingest. - #2296 closes #2294: a persisted index.default_pipeline runs again after a clean restart. - #2298: DeepSeek Harness profile bundle in dsh/, README links, and landing/brandbook/xerj-mark-400.png. dsh/package.json is 1.0.0-rc.99. - #2300 and #2301: Cline is marked tested in the EN/KO/ZH README tables, with a field report. The run was against v1.0.0-rc.98. - #2302 improves #1122, which stays open: a bulk reserves memory before parsing, and uncompressed JSON over 64 MiB is streamed. - #2303 is part of #1170, which stays open: a document write is acknowledged once a quorum of copies confirms. - #2308 fixes #2305: a semantic query over a wildcard, alias, or comma list skips indices that have no semantic_text field. The window since v1.0.0-rc.98 is 91 merged pull requests. The 18 crate versions in engine/Cargo.lock stay 1.0.0-rc.99. Landing footers that this merge touched stay V1.0.0-RC.99. /product now says rc.99 lists xerj hub next to xerj ask when the build has a hub signing key (#2282); the rc.98 capture of that page is otherwise unchanged. #2312 merged to corpus-hub, not main, so it is not in this cut. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>

The only conflict was CHANGELOG.md. engine.rs and landing/llms-full.txt auto-merged. Both behaviours stay. #1981 (this branch): a flat RSS at or above the watermark releases the breaker after 2 seconds and does not re-arm inside the 4 MiB band. While that latch stays engaged, drain requests double from 2 seconds up to 15 minutes unless RSS falls by more than 4 MiB. corpus index reports eta_quality=stalled at 0/N and after 10 minutes of silence on a phase with no denominator. main, kept alongside that: - #2296 / #2294: an existing index's persisted index.default_pipeline is copied from settings.json back into the in-memory map on boot, failed-index retry, and snapshot restore. - #2303: a single-document write is acknowledged at a quorum of copies, with the replica-sync lock on the engine. - #2250 / #1612: neural autoindex prints an estimated embedding time, and --embed-mode stays fixed at ingest. llms-full.txt keeps that text and the 0/N stall sentence. - #2308 / #2305 and #2311 came in with no overlap on the drain schedule. CHANGELOG keeps the #1981 entry and main's #2294 default_pipeline entry. No .github file was edited. Verified (rustc 1.97.1 (8bab26f4f 2026-07-14), from engine/): - cargo +1.97.1 fmt -p xerj-engine -p xerj-autoindex -- --check: exit 0 - RUST_MIN_STACK=32MiB cargo +1.97.1 test --no-fail-fast -j 2 -p xerj-engine -p xerj-autoindex -- --test-threads=2: exit 0, 3118 passed, 0 failed, 29 ignored. Includes flat_rss_above_the_watermark_releases_and_does_not_rearm, drain_requests_back_off_when_rss_does_not_fall, rss_noise_around_a_flat_footprint_still_releases, commit_starvation_holds_a_flat_footprint_until_commit_returns, and a_phase_at_zero_reports_a_stall_once_nothing_completes. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>

cla-bot Bot added the cla-signed label Oct 23, 2026

xerj-org merged commit a28166e into main Oct 24, 2026
24 checks passed

cursor Bot mentioned this pull request Oct 24, 2026

Blog: r 8000 c.99 smart, resource-aware autoindex and more harness tools #2295

Merged

cursor Bot mentioned this pull request Oct 24, 2026

Release v1.0.0-rc.99 #2297

Merged

Navigation Menu

Repository navigation

feat(cluster): acknowledge document writes at a quorum of copies - #2303

feat(cluster): acknowledge document writes at a quorum of copies#2303
xerj-org merged 2 commits into
mainfrom
cursor/cluster-quorum-acks-b915

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

What this changes, and why

What remains for #1170

Evidence

Checks

Provenance

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants