Repository navigation
Conversation
Root cause. ShardRouter::route_doc hashed every index with one
cluster-wide width (`num_shards`). install_shard_plan / assign_one_shard
(and update_from_metadata) widened that width to the widest plan seen,
so creating one index with more shards (a node started with four
members proposes a 4-shard plan; `old` had 3) re-bucketed every older
index. Acknowledged ids then routed to a shard that did not hold them
(404 on GET, members disagreeing), and route_write returned "write
locally" for ids that landed on shard 3, which `old` has no plan entry
for, so those writes sat outside the plan where search never looked.
Fix.
- xerj-cluster router: per-index width `index_shards`, set from each
index's own assignments in assign / update_from_metadata (also takes
IndexMetadata.num_shards when larger), removed in clear_index.
route_doc uses it; only an unplanned index uses the fallback
`num_shards`, so single-node routing is unchanged. New `Route` enum
(NoPlan / Owned / Unowned) via `route()`, `index_num_shards()`.
- xerj-engine: install_shard_plan and assign_one_shard no longer touch
the global width. route_write and primary_of now return
Result<Option<..>, ShardUnassigned>: a planned shard with no primary
is an error, never "write locally".
- xerj-api cluster_shard: index/get/delete/update dispatch, the
forwarded update handler, replicate_write, bulk_refused_here and
bulk_on_primaries answer 503 shard_not_assigned_exception (bulk: a
per-item 503) for an unassigned shard instead of storing locally.
No caller iterates 0..num_shards for search fan-out, health or backfill:
fan-out uses search_targets / shards_owned_by from the routing table.
Before/after (in-process 3-node InMemoryBus ring,
tests/cluster_per_index_shard_width.rs): create `old` (3 shards, 1
replica), bulk 2,000 docs, then commit a 4-shard `new`.
before: 994 of 6,000 (member, id) GETs of acked `old` ids missed
(16.6%) right after `new` was created.
after: 0 of 6,000 missed; _search total 2,000 on every member; 500
more writes through every member -> total 2,500, 0 of 7,500
GETs missed.
Tests added: router unit tests (10k ids of a 3-shard index unchanged
after a 4-shard index is installed; a plan omitting shard 2 routes
Unowned), engine test (route_write/primary_of Err for the gap, Ok(None)
for an unplanned index), and an API test (503
shard_not_assigned_exception for PUT and bulk on every member, nothing
stored locally). test_router_update_from_metadata now asserts the
per-index width instead of cluster-wide widening.
Out of scope: documents already misplaced on a running cluster are not
moved.
Reference-coding: skipped. `xerj code` needs the :9200 reference
server, which was unreachable from this lane, and no quickwit or
meilisearch clone is in ~/.xerj-code/corpora. The change is confined
to this repo's own router (per-index width map + routing enum).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (data loss)
ShardRouter::route_dochashed every index with one cluster-wide width.install_shard_planandassign_one_shardwidened it to the widest plan seen. When one index got more shards (a node started with four members proposes a 4-shard plan), every older index was re-bucketed:route_write(the older index has no shard 3 in its plan), so they were stored outside the plan, where search never lookedFix
router.rs: per-index widthindex_shards, set from each index's own assignments (assign,update_from_metadata, which also takesIndexMetadata.num_shardswhen larger) and removed inclear_index.route_docuses it. Only an index with no plan falls back to the oldnum_shards, so single-node routing is unchanged. Adds aRouteenum (NoPlan/Owned/Unowned) viaroute(), plusindex_num_shards().engine.rs:install_shard_planandassign_one_shardno longer widen a global width.route_writeandprimary_ofnow returnResult<Option<..>, ShardUnassigned>: a planned shard with no primary is an error and never means "write locally".cluster_shard.rs: index, get, delete and update dispatch, the forwarded update handler,replicate_write,bulk_refused_hereandbulk_on_primariesnow answer 503shard_not_assigned_exceptionfor an unassigned shard (bulk returns a 503 per item) instead of storing the document locally.search_targets/shards_owned_by). None of them loop over0..num_shards, so nothing else needed the per-index width.Before / after
Test: in-process 3-node
InMemoryBusring (tests/cluster_per_index_shard_width.rs). Createold(3 shards, 1 replica), bulk 2,000 docs, then commit a 4-shardnew.oldids that missed_searchtotal on each memberTests
Unowned.test_router_update_from_metadatanow checks the per-index width. It used to check that the cluster-wide width grew.shard_router_write_path.rs):route_writeandprimary_ofreturnErrfor the missing shard, andOk(None)for an index with no plan._bulkto an unassigned shard get 503shard_not_assigned_exceptionon every member, and nothing is stored locally.cargo test --release -j 16 -p xerj-cluster -p xerj-api: 1203 passed, 0 failed.Out of scope
This does not repair documents already misplaced on a running cluster. Ids written under the widened width stay on the shard they were written to. Fixing that needs a re-route or reindex pass, which should be its own change.
The lab ring repro (
xerj-cluster-labS4) was not re-run. All numbers above come from the in-process ring.Reference-coding
Skipped.
xerj codeneeds the reference server on :9200, which this lane could not reach, and no quickwit or meilisearch clone exists under~/.xerj-code/corpora. The change stays inside this repo's own router (a per-index width map and a routing enum).🤖 Generated with Claude Code