Repository navigation
fix(es-compat): _delete_by_query/_update_by_query through an alias touched only the first member - #453
fix(es-compat): _delete_by_query/_update_by_query through an alias touched only the first member#453xerj-org wants to merge 3 commits into
Conversation
…and called it done Closes #450. Reproduced on main, exactly as filed. Three indices of ten documents behind one alias `tri`: POST /tri/_delete_by_query {"query":{"match_all":{}}} -> HTTP 200 {"total":10,"deleted":10,"batches":1,"failures":[]} -> idx-a: 0 idx-b: 10 idx-c: 10 (20 of 30 never considered) Nothing in that body distinguishes it from a correct run: the counts are internally consistent, `failures` is empty, and the status is 200. `_update_by_query` is the same shape — it reported `"updated":10` and left twenty documents untouched. Both handlers began with `Engine::get_index` on the raw path segment, and `get_index` resolves an alias to `aliased.first()` — one member, silently. That is the read truncation of #433 on a destructive path, and it is worse: a short read is recoverable by reading again, a short delete leaves the caller believing the operation completed. Both now resolve the selector with `resolve_index_selector`, the same expansion `search_impl` uses, open and authorize every member up front so a refusal fails the request before anything is written, and run over each. Alias filters are collected the way `search_impl` collects them and AND-ed into the query as a `bool.filter`, so deleting through a filtered alias reaches only what that alias can see — expanding to members without the filter would have deleted the whole corpus, a worse bug than the one being fixed. A member that fails becomes an entry in `failures` carrying its index name rather than vanishing; if every member fails, the first error is returned unchanged so the request keeps its real status code. When the selector resolves to exactly one index the runner's value is returned untouched, so every existing single-index path is byte-identical. After: POST /tri/_delete_by_query -> {"total":30,"deleted":30,"batches":3} 0 of 30 left POST /tri/_update_by_query -> {"total":30,"updated":30} 10+10+10 touched POST /lowtri/_delete_by_query (alias filtered to n<=3) -> {"total":9,"deleted":9} 21 survive, 0 with n<=3 POST /idx-b/_delete_by_query -> {"total":1,"deleted":1,"batches":1} unchanged POST /no-such-index/... -> 404 Five tests in `by_query_through_an_alias_touches_every_member.rs`, asserting on ground truth in each concrete index rather than on the reported counts — the reported counts are what made this invisible. Fail-before proved by reverting only the source hunk: the three tests that encode the defect fail, and the two no-regression guards (single index, unknown index) pass, because that behaviour was already correct. Gates: fmt clean, clippy `-D warnings` clean, and the full xerj-api suite green in both the dev and ci-test profiles under rustc 1.97.1. The other `get_index` read-side callers named in the issue — `_explain`, `_segments`, `_cat/segments`, `_eql/search`, `_async_search`, `_mget`, `_flush` — are untouched here and belong with #433/#451.
Independent panel (head
|
…t fixed
Refuted 3 of 3. Both must-fixes are regressions this change introduced, and I
reproduced each against a live node before touching anything.
1. A FILTERED ALIAS CREATED THE CANONICAL WAY LOST ITS FILTER, so the new
expansion deleted everything behind it.
`POST /_aliases {"actions":[{"add":{…,"filter":…}}]}` is the API every
Elasticsearch filtered-alias example uses. `AliasActionParams` had no
`filter` field, so the filter was accepted, answered `acknowledged: true`,
and thrown away. `by_query_targets` then found no recorded filter and deleted
every member in full:
this branch POST /lowtri/_delete_by_query -> total 30, deleted 30, 0 left
parent same request -> total 10, deleted 10, 20 left
same node, alias made with PUT /{i}/_alias -> total 9, deleted 9, 21 left
My own commit message said expanding without the filter "would have deleted
the whole corpus, a worse bug than the one being fixed". It did. And my test
helper carries a comment noting that `_aliases` does not store the filter —
I found the gap while writing tests, routed around it, and never asked what
the expansion would do when it met one.
`POST /_aliases` now REFUSES an action carrying `filter`, naming the route
that does persist it. Silently ignoring it is the one option that cannot be
made safe; accepting it means storing metadata this route has never stored,
which is a change to `_search` semantics and belongs in its own PR.
2. `_all` AND `*` BECAME A CLUSTER-WIDE DELETE AT `write` PRIVILEGE.
They 404'd before, because `get_index("_all")` matched nothing.
`resolve_index_selector` honours them with no hidden/system exclusion, so one
request emptied every index the caller could see. Measured on this branch: 46
documents across 17 indices at HTTP 200, including `.xerj_dashboards` and
`.xerj_magic_links`, with `.xerj_users`, `.xerj_api_tokens`, `.xerj_sessions`,
`.xerj_audit` and `.xerj_cluster_state` in range. `required_privilege` gives
this op segment `Privilege::WriteIndex`, while the pre-existing `DELETE /*`
needs `Privilege::AdminIndex` — so emptying the cluster went from `manage` to
`write`.
I called this "the same expansion `search_impl` uses". It is only the
name-resolution half, without the `expand_wildcards`, `ignore_unavailable`
and `allow_no_indices` handling that makes wildcards safe on the read side.
Both by-query verbs now refuse `_all`, `*` and globs with a 400 that names
the offending pattern, the way ES gates the same shape behind
`action.destructive_requires_name`.
Both are the same mistake wearing different clothes: I took a resolver built for
reads and pointed it at a destructive verb without asking what its permissiveness
means when the verb is DELETE.
Two new tests, each encoding one regression — a refused wildcard must leave all
30 documents, and a refused `_aliases` filter must leave no alias behind while
the working route still deletes exactly 9. Seven tests total, all green.
Gates: fmt clean, clippy --workspace --all-targets -D warnings clean, full
xerj-api suite green under rustc 1.97.1 in the ci-test profile.
Known cosmetic: both refusals surface as `search_phase_execution_exception`,
which is what `XerjError::invalid_query` maps to. The status is right (400) and
the reason string is specific; the ES type name for these would be
`illegal_argument_exception`. Left alone rather than widening the error mapping
in a PR about destructive correctness.
The should-fixes the panel raised on top of these — alias-of-alias still
reproducing #450, a closed member being emptied where `_search` skips it,
per-member filters AND-ed globally, `timed_out` hardcoded, and the scripted
budget becoming per-member — are not addressed here and the PR stays a draft
until they are.
…efusal guarded the wrong door
Refuted again. The previous revision made `POST /_aliases` reject an action
carrying a `filter`, on the reasoning that accepting-and-ignoring is the one
option that cannot be made safe. That was right about the danger and wrong about
where it lives.
The refusal blocks the caller who DECLARES a filter. It does nothing about the
shape ES clients actually send, because the canonical repoint idiom carries no
filter on the `add` at all:
POST /_aliases {"actions":[{"remove":{old}},{"add":{new}}]}
Nothing to refuse, so it is accepted at 200 — and the alias lands on a backing
index with no metadata, silently unfiltered. With the by-query paths now
honouring alias filters, the next `_delete_by_query` through that alias empties
the whole index. A reviewer measured it: 10 of 10 destroyed where the filter
meant 3.
It also removed a capability without replacing it. `PUT /{index}/_alias/{alias}`
cannot do the `remove` half, so after that change there was NO route able to
atomically repoint a filtered alias — and my error message pointed at exactly
that route.
So the filter is stored now, mirroring the `attach` closure `put_alias` has had
all along, and `remove` clears the metadata so a later re-add of the same name
cannot inherit a filter nobody asked for. Verified end to end on a live node:
PUT /old-1/_alias/cur {filter n<=3} -> filter present
POST /_aliases remove old-1 / add new-1 +filter -> 200 (was 400)
GET /_alias/cur -> filter followed the repoint
POST /cur/_delete_by_query {match_all} -> total 3, deleted 3
GET /new-1/_count -> 7 surviving
The test that encoded the refusal is replaced by one that encodes the storage
and the repoint, including that the alias leaves the old index and that an
unfiltered expansion would have emptied the new one. 7 tests, all green.
Gates: fmt clean, clippy --workspace --all-targets -D warnings clean, full
xerj-api suite green under rustc 1.97.1 in BOTH the dev and ci-test profiles.
STILL OPEN, and why this stays a draft — a second must-fix I have not addressed:
this PR makes the by-query verbs the only routes that expand an alias to every
member, while `_search` and `_count` still read one. The standard pre-flight for
a delete is to run the same query as a search first, so that preview now
under-reports by the member count: search says 5, the delete removes 15. That is
#450 inverted, and it punishes precisely the operator who checks before
deleting. The right resolution is for the READ paths to expand too — which is
#433/#449 — not for this path to go back to touching one member. Sequencing
those is the next decision, and I would rather state it than paper over it.
Independent verification — BLOCKING data-loss regression, NOT merging3 skeptics (correctness / regression-risk·data-safety / evidence-holds) in worktrees under 1.97.1. The data-safety lens found a restart-triggered over-delete; I confirmed it in the engine source. Verdicts verbatim: correctness — refuted:false, blocking:false.
regression-risk (data-safety / over-deletion) — refuted:true, blocking:true.
evidence-holds — refuted:false, blocking:false.
Confirmed independently
Decision: do NOT merge. Blocked on a durability prerequisite.The in-memory-only filter store defeats this PR's core guarantee on a destructive path: after any restart, The in-memory fix (hit all members + AND the alias filter + symmetric update path) is correct and valuable — it just needs alias filter metadata to be durable first. Blocked on the root-cause issue below. Non-blocking nit from the evidence lens: no filtered |
… and _terms_enum Part of xerj-org#449's class A, split out because it is small, additive and provable on its own. `resolve_indices_for_op` had no alias branch at all. Its concrete-name arm checked the name against the list of real indices and 404'd otherwise, so an alias was simply "not an index". Measured on a three-member alias `tri` over idx-a/idx-b/idx-c: POST /tri/_refresh -> 404 POST /idx-a/_refresh -> 200 POST /tri/_forcemerge -> 404 POST /idx-a/_forcemerge -> 200 POST /tri/_cache/clear -> 404 POST /idx-a/_cache/clear -> 200 POST /tri/_terms_enum -> 404 POST /idx-a/_terms_enum -> 200 An alias is a valid index selector for every one of these in Elasticsearch. The new branch expands to EVERY member, not `aliased.first()`. That distinction is the reason this is worth doing carefully rather than quickly: a maintenance op that silently addressed one member of a multi-index alias is the same defect class as xerj-org#450 on the destructive side, and a `_refresh` reaching one of three members makes the next search non-deterministic with nothing in the response admitting it. After: POST /tri/_refresh -> 200, _shards {"total":3,"successful":3,"failed":0} A concrete index name still wins over an alias of the same name, matching ES, where the two namespaces cannot collide. The missing-name rule is deliberately unchanged — ES defaults `ignore_unavailable=false`, so a typo must fail the whole request rather than silently doing less than asked: POST /no-such-thing/_refresh -> 404 POST /idx-a,typo/_refresh -> 404 POST /idx-*/_refresh -> 200 (wildcard expansion, unchanged) Two tests. Fail-before proved by reverting ONLY the source hunk: `maintenance_endpoints_accept_an_alias_and_reach_every_member` FAILED, `an_unknown_name_still_fails_the_whole_request` passed — the second is the no-regression guard, so it passing is the point. Gates: fmt clean, clippy --workspace --all-targets -D warnings clean, full xerj-api suite green under rustc 1.97.1 in BOTH the dev and ci-test profiles. NOT fixed here, and xerj-org#449 stays open: the other resolver class, where `Engine::get_index` resolves an alias to `aliased.first()` and endpoints answer from one member instead of 404ing — `_explain`, `_segments`, `_cat/segments`, `_eql/search`, `_async_search`, `_mget` and `_flush`. Those return 200 with a partial answer, which is a different and harder problem than returning 404 on a valid name, and it needs the same "what does this op mean across members" judgement that xerj-org#453 is still drafted over.
…erj-org#524) The alias->members topology is persisted to aliases.json, but each alias's filter lived only in the in-memory index_alias_metadata map and was never written to disk. So a filtered alias came back after a restart with its members but no filter: _search/_count through it silently returned the whole backing corpus instead of the slice the filter names, and once the by-query paths honour the filter (xerj-org#453), a _delete_by_query through such an alias would empty the whole backing index. Persist the map to an alias_metadata.json sidecar and reload it on boot, mirroring flush_aliases/load_persisted_aliases. A separate file (not a wider aliases.json) keeps that file's shape unchanged and is downgrade-safe — an older binary ignores it — the same call the ingest-pipeline-refusals sidecar makes. flush_alias_metadata runs after every mutation of the map (put-alias attach, create-time aliases, index delete); load_alias_metadata runs next to load_persisted_aliases on boot. Test: create a filtered alias, drop the engine, open a fresh one on the same data dir, assert the filter is still there. Proven fail-before by reverting only the source hunks — the alias comes back as {"orders":{"aliases": {"gold":{}}}}, filter gone. Verified under rustc 1.97.1: fmt, clippy -D warnings, build --profile ci-test, and the test under both dev and ci-test.
|
Closing as superseded — the bug this draft targets (#450) is already fixed on The final fan-out landed in #552 (commit This draft ( If a gap in the merged fix is later found, it should be a fresh PR against the current |
Closes #450.
Reproduced on main, as filed
Three indices of ten documents behind one alias
tri:Nothing distinguishes that from a correct run: counts internally consistent,
failuresempty, status 200._update_by_queryis the same —"updated":10, twenty documents untouched.Cause, confirmed: both handlers began with
Engine::get_indexon the raw path segment, andget_indexresolves an alias toaliased.first()(engine.rs:3112) — one member, silently.The fix
Both handlers now resolve the selector with
resolve_index_selector— the same expansionsearch_impluses — open and authorize every member up front so a refusal fails the request before anything is written, and run over each.Alias filters are collected the way
search_implcollects them and AND-ed in as abool.filter. This mattered: expanding an alias to its members without the filter would delete the whole corpus — a worse bug than the one being fixed.A failing member becomes a
failuresentry carrying its index name rather than vanishing. If every member fails, the first error is returned unchanged so the request keeps its real status code. When the selector resolves to exactly one index the runner’s value is returned untouched, so every existing single-index path is byte-identical.After
/tri/_delete_by_querytotal:30, deleted:30, batches:3/tri/_update_by_querytotal:30, updated:30/lowtri/_delete_by_query(alias filteredn<=3)total:9, deleted:9n<=3/idx-b/_delete_by_querytotal:1, deleted:1, batches:1/no-such-index/_delete_by_query404Tests
Five tests in
by_query_through_an_alias_touches_every_member.rs, asserting on ground truth in each concrete index rather than on the reported counts — the reported counts are exactly what made this invisible.Fail-before proved by reverting only the source hunk, keeping the tests:
The two that pass are the no-regression guards (single index, unknown index) — that behaviour was already correct, which is the point of including them.
Gates
cargo fmt --all --checkclean,clippy --workspace --all-targets -D warningsclean, and the fullxerj-apisuite green in both the dev and ci-test profiles under rustc 1.97.1 (216 lib + all integration targets).Out of scope
The other
get_indexread-side callers the issue names —_explain,_segments,_cat/segments,_eql/search,_async_search,_mget,_flush— are untouched here and belong with #433/#451.Not verified
Whether the detached
wait_for_completion=falsepath has an independent test; I exercised it by inspection only (it takes the samerun_*_overcall as the synchronous path).