8000 fix(es-compat): _delete_by_query/_update_by_query through an alias touched only the first member by xerj-org · Pull Request #453 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

fix(es-compat): _delete_by_query/_update_by_query through an alias touched only the first member - #453

Closed
xerj-org wants to merge 3 commits into
mainfrom
fix/by-query-alias-members
Closed

xerj-org wants to merge 3 commits into
mainfrom
fix/by-query-alias-members

Conversation

@xerj-org
Copy link
Copy Markdown
Owner

Closes #450.

Reproduced on main, as filed

Three indices of ten documents behind one alias tri:

POST /tri/_delete_by_query {"query":{"match_all":{}}}
  -> HTTP 200 {"total":10,"deleted":10,"batches":1,"failures":[]}
  -> idx-a: 0   idx-b: 10   idx-c: 10        20 of 30 never considered

Nothing distinguishes that from a correct run: counts internally consistent, failures empty, status 200. _update_by_query is the same — "updated":10, twenty documents untouched.

Cause, confirmed: both handlers began with Engine::get_index on the raw path segment, and get_index resolves an alias to aliased.first() (engine.rs:3112) — one member, silently.

The fix

Both handlers now resolve the selector with resolve_index_selector — the same expansion search_impl uses — open and authorize every member up front so a refusal fails the request before anything is written, and run over each.

Alias filters are collected the way search_impl collects them and AND-ed in as a bool.filter. This mattered: expanding an alias to its members without the filter would delete the whole corpus — a worse bug than the one being fixed.

A failing member becomes a failures entry carrying its index name rather than vanishing. If every member fails, the first error is returned unchanged so the request keeps its real status code. When the selector resolves to exactly one index the runner’s value is returned untouched, so every existing single-index path is byte-identical.

After

request result ground truth
/tri/_delete_by_query total:30, deleted:30, batches:3 0 of 30 left
/tri/_update_by_query total:30, updated:30 10+10+10 touched
/lowtri/_delete_by_query (alias filtered n<=3) total:9, deleted:9 21 survive, 0 with n<=3
/idx-b/_delete_by_query total:1, deleted:1, batches:1 unchanged
/no-such-index/_delete_by_query 404 —

Tests

Five tests in by_query_through_an_alias_touches_every_member.rs, asserting on ground truth in each concrete index rather than on the reported counts — the reported counts are exactly what made this invisible.

Fail-before proved by reverting only the source hunk, keeping the tests:

failures:
    a_filtered_alias_deletes_only_the_documents_it_selects
    delete_by_query_through_an_alias_deletes_from_every_member
    update_by_query_through_an_alias_updates_every_member
test result: FAILED. 2 passed; 3 failed

The two that pass are the no-regression guards (single index, unknown index) — that behaviour was already correct, which is the point of including them.

Gates

cargo fmt --all --check clean, clippy --workspace --all-targets -D warnings clean, and the full xerj-api suite green in both the dev and ci-test profiles under rustc 1.97.1 (216 lib + all integration targets).

Out of scope

The other get_index read-side callers the issue names — _explain, _segments, _cat/segments, _eql/search, _async_search, _mget, _flush — are untouched here and belong with #433/#451.

Not verified

Whether the detached wait_for_completion=false path has an independent test; I exercised it by inspection only (it takes the same run_*_over call as the synchronous path).

…and called it done

Closes #450.

Reproduced on main, exactly as filed. Three indices of ten documents behind one
alias `tri`:

  POST /tri/_delete_by_query {"query":{"match_all":{}}}
    -> HTTP 200 {"total":10,"deleted":10,"batches":1,"failures":[]}
    -> idx-a: 0   idx-b: 10   idx-c: 10      (20 of 30 never considered)

Nothing in that body distinguishes it from a correct run: the counts are
internally consistent, `failures` is empty, and the status is 200.
`_update_by_query` is the same shape — it reported `"updated":10` and left
twenty documents untouched.

Both handlers began with `Engine::get_index` on the raw path segment, and
`get_index` resolves an alias to `aliased.first()` — one member, silently.
That is the read truncation of #433 on a destructive path, and it is worse: a
short read is recoverable by reading again, a short delete leaves the caller
believing the operation completed.

Both now resolve the selector with `resolve_index_selector`, the same expansion
`search_impl` uses, open and authorize every member up front so a refusal fails
the request before anything is written, and run over each. Alias filters are
collected the way `search_impl` collects them and AND-ed into the query as a
`bool.filter`, so deleting through a filtered alias reaches only what that alias
can see — expanding to members without the filter would have deleted the whole
corpus, a worse bug than the one being fixed.

A member that fails becomes an entry in `failures` carrying its index name
rather than vanishing; if every member fails, the first error is returned
unchanged so the request keeps its real status code. When the selector resolves
to exactly one index the runner's value is returned untouched, so every existing
single-index path is byte-identical.

After:

  POST /tri/_delete_by_query   -> {"total":30,"deleted":30,"batches":3}  0 of 30 left
  POST /tri/_update_by_query   -> {"total":30,"updated":30}              10+10+10 touched
  POST /lowtri/_delete_by_query (alias filtered to n<=3)
                               -> {"total":9,"deleted":9}   21 survive, 0 with n<=3
  POST /idx-b/_delete_by_query -> {"total":1,"deleted":1,"batches":1}    unchanged
  POST /no-such-index/...      -> 404

Five tests in `by_query_through_an_alias_touches_every_member.rs`, asserting on
ground truth in each concrete index rather than on the reported counts — the
reported counts are what made this invisible. Fail-before proved by reverting
only the source hunk: the three tests that encode the defect fail, and the two
no-regression guards (single index, unknown index) pass, because that behaviour
was already correct.

Gates: fmt clean, clippy `-D warnings` clean, and the full xerj-api suite green
in both the dev and ci-test profiles under rustc 1.97.1.

The other `get_index` read-side callers named in the issue — `_explain`,
`_segments`, `_cat/segments`, `_eql/search`, `_async_search`, `_mget`, `_flush`
— are untouched here and belong with #433/#451.
@cla-bot cla-bot Bot added the cla-signed label Aug 17, 2026
@xerj-org
Copy link
Copy Markdown
Owner Author

Independent panel (head 5ef0aa4) — 3 of 3 refuted

Converted to draft. Three skeptics, each told to REFUTE, distinct lenses. All three refuted, and they converged on two must-fixes that are regressions this change causes. Verbatim below.

453-dataloss — refuted: True

Drove two real nodes side by side — one built from PR head 5ef0aa4 (port 29453), one from its parent 8d7fc76 (port 29463), both ci-test profile under rustc 1.97.1, own data dirs, disk_flood_stage_percent=0. Claims 1, 3, 4 and 5 all hold: 30/30 delete and 30/30 update through tri; a PUT-created filtered alias deletes exactly 9 and leaves 21 with none at n<=3; a single-index and a single-member-alias request return byte-identical bodies; unknown index still 404s; fmt clean, clippy -p xerj-api --all-targets -D warnings clean, 5/5 new tests green in ci-test. I also could not break the things the lens most suspected: resolve_index_selector dedupes, so tri,idx-a, tri2,idx-*,_all and idx-* never double-visit (a +1000 script applied exactly once, batches==distinct members); bool.filter composes correctly with a user bool{should,minimum_should_match} and with knn (pre-filter binds, 0 survivors inside the alias filter, 14 outside); wait_for_completion=false really does run every member and stores the merged body in _tasks/{id} (6/6/6); per-document partial failure surfaces with the member name attached and deleted < total.

But the destructive lens found two regressions this diff causes, both reproduced live and both A/B'd against the parent commit. (1) A filtered alias created the canonical ES way — POST /_aliases {"actions":[{"add":{...,"filter":...}}]} — has its filter silently discarded (AliasActionParams has no filter field), so the new expansion deletes the ENTIRE corpus behind the alias: 30 of 30 on the PR vs 10 of 30 on main, HTTP 200, failures:[]. This is precisely the "worse bug than the one being fixed" the commit message claims to have avoided, and the PR's own test helper carries a comment admitting the _aliases route does not store filters. (2) POST /_all/_delete_by_query and POST /*/_delete_by_query 404 on main and now empty every index the caller can see — 22 indices / 20,058 docs in one request, including .xerj_users, .xerj_api_tokens, .xerj_sessions, .xerj_audit, .xerj_cluster_state and .xerj-memory-* — at Privilege::WriteIndex, where the pre-existing DELETE /* requires Privilege::AdminIndex. Below those: #450 still reproduces verbatim through one extra level of alias indirection, a closed member is silently emptied where _search through the same alias skips it, one unopenable member 404s the whole request, and merge_by_query hardcodes timed_out:false.

Findings

[must-fix] A filtered alias created via POST /_aliases loses its filter, so the new expansion deletes the whole corpus behind it (30/30 vs 10/30 on main)

by_query_targets collects alias filters from state.engine.index_alias_metadata, which is only ever written by put_alias (PUT /{index}/_alias/{name}) and by index-create-time aliases. post_aliases never writes it, and AliasActionParams (es_compat.rs:19181) declares only index and alias with no deny_unknown_fields — so a filter in an add action is accepted, answered acknowledged:true, and thrown away. apply_alias_filters then gets an empty vec and returns the user's query untouched, and every member is deleted in full.

This inverts the PR's central safety claim. The commit says "expanding to members without the filter would have deleted the whole corpus, a worse bug than the one being fixed" — that is exactly what happens for aliases created through the canonical ES atomic alias API, which is the API the Elasticsearch docs use for every filtered-alias example. The PR's own test helper routes around it with the comment "_aliases with a filter in the add action does not store the filter; the per-index alias route does", so the gap was known and the tests were written to avoid it rather than to cover it.

Blast radius grew with the fix: on main the same request deleted only aliased.first(); now it deletes every member. Fix options: teach post_aliases to persist filter/routing/is_write_index into index_alias_metadata (the real fix, since _search is equally blind today), or refuse a _delete_by_query/_update_by_query whose selector names an alias for which any member has no recorded metadata — never silently widen.

Two live nodes, identical requests. PR head 5ef0aa4 (:29453):
  POST /_aliases {"actions":[{"add":{"index":"idx-a","alias":"lowtri","filter":{"range":{"n":{"lte":3}}}}}, ... idx-b, idx-c]}
    -> 200 {"acknowledged":true}
  GET /_alias/lowtri -> {"idx-c":{"aliases":{"lowtri":{}}},"idx-a":{"aliases":{"lowtri":{}}},"idx-b":{"aliases":{"lowtri":{}}}}   <- filter gone
  before: a=10 b=10 c=10
  POST /lowtri/_delete_by_query {"query":{"match_all":{}}}
    -> 200 {"total":30,"deleted":30,"batches":3,"failures":[]}
  after:  a=0 b=0 c=0            <- 30 of 30 destroyed; intent was 9

Parent 8d7fc76 (:29463), byte-identical request sequence:
  POST /lowtri/_delete_by_query -> 200 {"total":10,"deleted":10,"batches":1,"failures":[]}
  after:  a=0 b=10 c=10          <- 10 of 30

Control on the SAME PR node, same filter, only the creation route changed:
  PUT /idx-{a,b,c}/_alias/lowput {"filter":{"range":{"n":{"lte":3}}}}
  GET /_alias/lowput -> filter present on all three
  POST /lowput/_delete_by_query {"match_all":{}} -> {"total":9,"deleted":9,"batches":3}
  after: a=7 b=7 c=7

Ruled out wrong reasons: identical filter JSON in both arms, so not a typo; `GET /_alias/lowtri` proves the alias exists on all three members, so not a failed alias creation; auth disabled on both nodes, so not authz; both nodes started at 10/10/10 and both returned 200 with non-zero `deleted`, so not a disk block. Source: /tmp/w453-overdel/repo/engine/crates/xerj-api/src/es_compat.rs:19181 (AliasActionParams), :19198-19266 (post_aliases, no index_alias_metadata write), :24222 (by_query_targets), :24266 (apply_alias_filters).

[must-fix] _delete_by_query / _update_by_query gained _all and *, which 404'd on main — one write-privileged request now empties every index on the node, system indices included

by_query_targets swaps Engine::get_index for resolve_index_selector, which honours _all, * and globs (es_compat.rs:19297-19344). On the parent commit those selectors reached get_index("_all"), found neither alias nor index, and 404'd. The PR turns them into a cluster-wide destructive operation.

Three things make it worse than the pre-existing DELETE /*:

  1. Privilege. required_privilege (authz.rs:506) maps an op segment that is neither a POST_READ_OP nor an ADMIN_OP to Privilege::WriteIndex; DELETE /{index} has no op segment and falls to the None => Privilege::AdminIndex arm. So emptying every index used to need manage and now needs only write.
  2. No hidden/system exclusion. resolve_index_selector globs over raw list_indices() output with no dot-prefix or hidden filter, so * matches .xerj_users, .xerj_api_tokens, .xerj_sessions, .xerj_passkeys, .xerj_idp_config, .xerj_audit, .xerj_cluster_state. ES 8 marks its system indices hidden and expand_wildcards defaults to open, so * does not match them there. Note is_reserved_index only guards the .xerj-memory- prefix (xerj-common/src/types.rs:190), not the .xerj_ console/auth stores, so an unscoped principal has write on them.
  3. The commit calls this "the same expansion search_impl uses", but it is only the name-resolution half — none of the expand_wildcards / ignore_unavailable / allow_no_indices handling at es_compat.rs:8684-8730 comes with it.

At minimum: exclude dot-prefixed/hidden indices from wildcard expansion on the by-query endpoints, and require a concrete name (or an explicit opt-in, the way ES's action.destructive_requires_name works) before _all/* is accepted on a destructive path.

Parent 8d7fc76 (:29463):
  POST /_all/_delete_by_query {"match_all":{}} -> 404 index_not_found_exception "no such index [_all]"
  POST /*/_delete_by_query    {"match_all":{}} -> 404 index_not_found_exception "no such index [*]"
  counts unchanged: a=10 b=10 c=10

PR head 5ef0aa4 (:29453), non-destructive breadth probe first (query matches nothing; `batches` == members visited):
  POST /_all/_delete_by_query {"query":{"term":{"tag":"__no_such_value__"}}} -> 200 {"total":0,"deleted":0,"batches":17}
  _cat/indices: 17 indices, 14 of them dot/system

Then for real:
  .xerj_dashboards = 15 docs, .xerj_magic_links = 1 doc before
  POST /*/_delete_by_query {"query":{"match_all":{}}} -> 200 {"total":34,"deleted":34,"batches":17,"failures":[]}
  .xerj_dashboards = 0, .xerj_magic_links = 0, idx-a/b/c = 0

And with the reserved namespace present (auth-disabled node, so index_guard imposed no filter — with auth on, `list_indices()` is guard-filtered and an unscoped principal would not see `.xerj-memory-*`):
  PUT /.xerj-memory-probe, 1 doc
  POST /*/_delete_by_query {"match_all":{}} -> 200 {"total":20058,"deleted":20058,"batches":22}
  .xerj-memory-probe count: 1 -> 0

Calibration, ruling out over-claiming: `DELETE /*` on the parent node already wipes everything (17 indices -> 0, then `DELETE /_all` -> 0), so cluster-wide destruction is not novel to xerj — but it sits behind `AdminIndex`, and this PR opens the same outcome at `WriteIndex` on an endpoint that previously refused the selector outright.

[should-fix] #450 still reproduces verbatim through one extra level of alias indirection — the fix resolves exactly one level and then falls back to get_index's .first()

resolve_index_selector pushes an alias's targets without re-resolving them (es_compat.rs:19333-19338), and by_query_targets then calls Engine::get_index(&name) on each — which is the very function whose aliased.first() behaviour (engine.rs:3112-3124) is the root cause of #450. If a target is itself an alias, the truncation returns intact. merge_by_query's results.len() == 1 shortcut then hands the single member's body back untouched, so the response is byte-identical to the pre-fix bug report.

Engine::add_alias (engine.rs:2828) performs no existence or type check on its target, so POST /_aliases {"add":{"index":"tri2","alias":"meta"}} where tri2 is an alias is accepted with acknowledged:true — ES rejects this with invalid_alias_name_exception. Worse, GET /_alias/meta returns {} because get_aliases enumerates indices, so the operator cannot even see the alias exists.

The fix's invariant "every member is visited" is not enforced structurally; it depends on alias targets never being aliases, which nothing in the codebase enforces. Either resolve transitively (with a cycle guard) or refuse an alias-of-alias at creation.

PR head 5ef0aa4 (:29453), idx-a/b/c at 10 docs each behind alias `tri2`:
  POST /_aliases {"actions":[{"add":{"index":"tri2","alias":"meta"}}]} -> 200 {"acknowledged":true}
  GET /_alias/meta -> {}                        <- invisible alias
  before: a=10 b=10 c=10
  POST /meta/_delete_by_query {"query":{"match_all":{}}}
    -> 200 {"took":0,"timed_out":false,"total":10,"deleted":10,"batches":1,"version_conflicts":0,"noops":0,"failures":[]}
  after:  a=0 b=10 c=10

That body is bit-for-bit the shape the commit message quotes as the defect being closed: internally consistent counts, empty `failures`, HTTP 200, and 20 of 30 documents never considered.

Ruled out wrong reasons: `tri2` itself works correctly on the same node in the same session (POST /tri2/... deletes 30/30), so this is the extra indirection level and not a broken alias; `batches:1` confirms exactly one member was visited, matching `get_index`'s `.first()`.

[should-fix] A closed index is silently emptied through an alias, where _search through the same alias deliberately skips it

by_query_targets never consults state.engine.closed_indices. search_impl does, at es_compat.rs:8684-8730: a closed member matched by a wildcard is dropped, and a closed index named explicitly returns index_closed_exception 400.

The hole exists on the parent for the direct single-index path too (I verified POST /idx-c/_delete_by_query empties a closed idx-c on the PR node), so the root defect is pre-existing. What this PR adds is reach: the closed index is now a target the caller never named and cannot see in the response. The result is a live contradiction created by this diff — POST /tri/_search skips closed idx-c while POST /tri/_delete_by_query empties it — under a commit message that describes the change as adopting "the same expansion search_impl uses". Closing an index is the ES idiom for "take this offline / do not touch it"; emptying it with failures:[] at HTTP 200 gives the operator no signal at all.

PR head 5ef0aa4 (:29453), alias `tri` over idx-a(10) idx-b(9) idx-c(9):
  POST /idx-c/_close -> 200 {"acknowledged":true,"indices":{"idx-c":{"closed":true}}}
  POST /idx-c/_search  -> 400 {"type":"index_closed_exception","reason":"closed","index":"idx-c"}
  POST /tri/_search    -> 200 hits.total.value = 10        <- closed member skipped
  POST /tri/_delete_by_query {"match_all":{}}
                       -> 200 {"total":28,"deleted":28,"batches":3,"failures":[]}
  POST /idx-c/_open; refresh
  after: a=0 b=0 c=0                                        <- the CLOSED index was emptied

Pre-existing-vs-new check on the same node: POST /idx-c/_close then POST /idx-c/_delete_by_query {"match_all":{}} -> {"total":10,"deleted":10}, idx-c count 0. So the direct path was already broken; the alias expansion is what turns "the one index you named" into "every closed member of every alias".

[should-fix] One unopenable member fails the entire request, so an alias created before its index (a pattern post_aliases explicitly supports) becomes undeleteable

by_query_targets opens every member up front and propagates the first get_index error, which is deliberate and fail-closed. But Engine::add_alias performs no existence check, and post_aliases's own comment says "ES lets an alias be attached to a name before the index exists, and clients rely on that ordering". So the supported create-alias-then-create-index sequence leaves an alias with a member that cannot be opened, and every _delete_by_query/_update_by_query through that alias 404s — naming an index the caller never mentioned — until the missing member is created. The same shape blocks the alias whenever any member is in a failed/unopenable state.

On the parent this request succeeded against whichever member aliased.first() returned. The new behaviour loses no data (safe direction), but it is an availability regression on a supported workflow, and the 404 body is confusing because it names a member rather than the selector. Consider skipping unopenable members into failures with the operation still running over the rest, or at least reporting the selector alongside the failing member.

PR head 5ef0aa4 (:29453), idx-a has 10 docs:
  POST /_aliases {"actions":[{"add":{"index":"idx-a","alias":"pre"}},{"add":{"index":"idx-future","alias":"pre"}}]}
    -> 200 {"acknowledged":true}      (idx-future does not exist)
  POST /pre/_delete_by_query {"query":{"term":{"n":101}}}
    -> 404 {"type":"index_not_found_exception","reason":"no such index [idx-future]", ... ,"status":404}
  idx-a count after: 10 (unchanged)

Ruled out wrong reasons: the same node deletes correctly through the 3-member `tri` alias in the same session, so this is the dangling member and not a broken alias path; the error names `idx-future`, i.e. it came from the per-member `get_index` in `by_query_targets` and not from selector resolution. Source: es_compat.rs:24248-24262; engine.rs:2828 (add_alias, no existence check); es_compat.rs:19212-19215 (the comment describing the supported ordering).

[should-fix] merge_by_query hardcodes "timed_out": false, discarding a member's own report that its run was cut short

run_update_by_query breaks out of its hit loop when Instant::now() >= deadline and returns "timed_out": timed_out — with the comment "The loop has to stop on its own, and say that it did rather than reporting timed_out: false over a run that was cut short" (es_compat.rs:24585-24588, :24668). merge_by_query never reads that field from any member; the merged body writes the literal false (es_compat.rs:24319) and sums the counters. A truncated destructive run over a multi-member alias therefore reports timed_out:false, failures:[], HTTP 200 — the exact "a partial run must never read as a complete one" property this PR exists to establish, defeated by the PR's own merge function. The single-index path is unaffected because results.len() == 1 returns the member body untouched, so the divergence appears only when the selector expands.

One-line fix: timed_out |= v.get("timed_out").and_then(Value::as_bool).unwrap_or(false) and emit that.

Honest caveat on reachability: I could not drive this over HTTP. SCRIPTED_UPDATE_BUDGET_MS is 30s and the search caps at 10,000 hits, so a member needs ~3ms/doc to trip it; on this node a scripted _update_by_query over 10,000 hits took 211ms. It is reachable on a slower node (sync WAL, contention, a script priced cheaply by the work counter but costing milliseconds — the exact shape MAX_EVAL_SLICE's own doc comment says the deadline exists to bound), but I am reporting this as proven by inspection, not by repro.

es_compat.rs:24316-24329 (merged body):
    json!({
        "took": took,
        "timed_out": false,        <- literal; no member's value is ever consulted
        "total": total,
        affected: done,
        ...
    })

versus es_compat.rs:24660-24672 (single-member body, returned untouched when results.len()==1):
    json!({
        "took": took,
        "timed_out": timed_out,    <- true when the loop broke on the deadline
        ...
    })

and es_compat.rs:24585-24588, the break that sets it:
    if Instant::now() >= deadline {
        timed_out = true;
        break;
    }

Measured throughput ruling out a cheap repro: POST /big-a/_update_by_query over 12,000 docs (10,000 hits after the size cap) with a 3-statement script -> {"took":211,"total":10000,"updated":10000}.

[nit] The 30s scripted-update budget silently became per-member instead of per-request

SCRIPTED_UPDATE_BUDGET_MS is documented as "Wall-clock ceiling for the Painless work one scripted-update request may do ... It bounds the request as a whole" (es_compat.rs:24496-24502), and run_update_by_query computes let deadline = Instant::now() + Duration::from_millis(SCRIPTED_UPDATE_BUDGET_MS) afresh on entry. run_update_by_query_over now calls it once per member in a loop, so a request over an N-member alias gets N x 30s of Painless. With _all/* newly accepted (see the wildcard finding), N is the whole node — on the 22-index node I ran, that is 11 minutes of scripted work from one HTTP request, and with wait_for_completion=false it is detached. Compute the deadline once in run_update_by_query_over and thread it in, or update the doc comment to say what the bound now is.

/tmp/w453-overdel/repo/engine/crates/xerj-api/src/es_compat.rs:24482-24493 (run_update_by_query_over loops over members) calling :24568 (`let deadline = Instant::now() + Duration::from_millis(SCRIPTED_UPDATE_BUDGET_MS)`), against the constant's own doc at :24496-24502.

[nit] The index_guard module's spawn audit table still describes the old single-handle shape

crates/xerj-engine/src/index_guard.rs carries a table listing every non-test tokio::spawn in the workspace and why it is or is not a door, and its row for these handlers still reads "es_compat::delete_by_query / update_by_query (wait_for_completion=false) | same: one already-authorized Index handle, no Engine". The PR updated the two inline comments at the spawn sites to say "handles" plural but not this table. The conclusion is still correct — I confirmed the spawned task owns Vec<(String, Arc<Index>)> and holds no Engine — but the table is the canonical audit that future spawns are checked against, so it should not drift.

/tmp/w453-overdel/repo/engine/crates/xerj-engine/src/index_guard.rs, the audit table under "### The audit", row for `es_compat::delete_by_query` / `update_by_query`. Updated spawn-site comments are at es_compat.rs:24152-24157 and :24466-24467.

[nit] Merged-body cosmetics: took is a max not a sum, batches counts members, and an alias filter is duplicated once per backing index

Three harmless-but-untidy details in the new code. (a) took = took.max(sum(&v,"took")) — members run sequentially, so the wall-clock cost of the request is the sum, and the reported took under-reports it. (b) batches sums to one per member, so it reports member count rather than scroll batches; the commit message presents batches:3 for a 3-member alias as expected, so this may be intentional, but it is not ES's meaning of the field. (c) by_query_targets iterates every backing index of the alias and pushes the filter each time it finds one, producing bool.filter:[f,f,f] for a 3-member alias — semantically identical, just three redundant clause evaluations per document, and it charges three times against MAX_CLAUSE_COUNT. It faithfully mirrors search_impl's loop at es_compat.rs:9163-9176, so fixing it is a two-call-site change if anyone cares.

/tmp/w453-overdel/repo/engine/crates/xerj-api/src/es_compat.rs:24296 (`took = took.max(...)`), :24299 (`batches += sum(&v, "batches")`), :24236-24246 (filter collection loop). Observed live: POST /lowput/_delete_by_query over a 3-member filtered alias -> {"total":9,"deleted":9,"batches":3}.

453-evidence — refuted: True

Claim 3 as literally stated survives every probe I could build: I recorded (HTTP status, body) for 27 single-index _delete_by_query/_update_by_query calls — happy paths plus 8 distinct error shapes (script CPU/limit refusal, unknown pipeline, unrunnable pipeline, malformed query, missing query field 422, missing content-type 415, unknown index 404, read_only block) and the detached ?wait_for_completion=false path read back through _tasks/{id} — at HEAD 5ef0aa4 and with ONLY the source hunk reverted (git show HEAD~1:engine/crates/xerj-api/src/es_compat.rs). After scrubbing took/task-key/node/timestamps, single_index_paths.json and detached_paths.json are byte-identical between the two trees. Issue #123's status/body coupling holds everywhere, including the new multi-member all-error path (HTTP 400 carrying status:400). I ruled out the "wrong reason" both ways: the harness is the same binary/profile/toolchain with one file swapped, and it does produce diffs — on the selector cases, where behaviour genuinely changed. Claim 6 (fail-before) reproduces exactly: 3 of 5 new tests fail with the hunk reverted, and they are the three named. Claim 4 holds. merge_by_query's error detection has no swallow hole — EsErrorResponse (error.rs:71-73) and script_limit_error_value both always carry error AND status, so the "has status, no error" gap I hunted for cannot occur; refuted. Mixed success/failure across members surfaces honestly (a read_only member produced 200, total:12 deleted:6, six failures[] entries each tagged "index":"pb"; pre-fix the same request reported total:6 deleted:6 failures:[]). GATES, rustc 1.97.1: cargo fmt --all --check clean; cargo clippy --profile ci-test --workspace --all-targets -- -D warnings clean. ci-test workspace suite: 108 binaries, 2835 passed / 1 failed / 23 ignored. Dev profile (with --no-fail-fast): 133 binaries, 3129 passed / 1 failed / 30 ignored, plus one aborted binary. Both failures are environmental and I proved it rather than assuming: (a) xerj-server::brain_cli::brain_end_to_end_five_note_fixture dies on _bulk: HTTP 429 — the disk flood-stage watermark on this 98%-full box — and it fails identically with the PR's source hunk reverted; the file contains zero references to _delete_by_query/_update_by_query/_alias. (b) xerj-engine::painless_script_limits::script_failure_is_not_cached_or_replayed aborts with fatal runtime error: stack overflow under dev only (it is 10/10 green in ci-test) — the documented debug-frame-size behaviour, in a crate that does not depend on xerj-api at all. Two aarch64/disk workarounds were needed to get the dev profile to run and I am disclosing them: RUSTFLAGS=-C target-feature=+fp16 (third-party gemm-f16 0.19.0 fails to assemble fmul v0.8h with "instruction requires: fullfp16"; it fails before any workspace crate compiles) and CARGO_PROFILE_DEV_DEBUG=0 (a full-DWARF dev tree exceeded the free disk; opt-level 0, debug-assertions, overflow-checks and panic=unwind are unchanged, so test semantics are stock dev). WHERE THE CLAIM IS TOO NARROW: "byte-identical" is a statement about response bytes only. On the same single-index path the change costs 2.8x–5.7x more wall time and scales with the number of indices in the process, and separately the PR silently turns _all/* from a 404 into a destructive sweep that includes the .xerj_* console/auth indices. Both are below the resolution of a body diff, which is exactly why I looked for them.

Findings

[must-fix] _all and * now silently delete from .xerj_* system indices (console users, API tokens, sessions) — they were a 404 before

by_query_targets (es_compat.rs:24232) resolves the path segment with resolve_index_selector (:19297), whose _all/* branch expands to every index Engine::list_indices() returns with no hidden-index rule. The sibling resolver in the same file, resolve_indices_for_op (:19378-19386), has exactly the guard that is missing here, and its own comment says why: POST /*/_close must never sweep in .xerj_users-class system indices. ES also excludes hidden indices from default */_all expansion, so this deviates from the compatibility target the PR is written against. It is reachable by an ordinary API key, not just a superuser: RESERVED_INDEX_PREFIX is .xerj-memory- (xerj-common/src/types.rs:190), so .xerj_users is NOT reserved, Principal::Unscoped::allows_index returns true for it (auth.rs:103), and index_guard::visible therefore keeps it in list_indices(). The blast radius is the whole xerj-console-api::indices::ALL list — .xerj_users, .xerj_api_tokens, .xerj_sessions, .xerj_passkeys, .xerj_idp_config, .xerj_audit, … The PR's commit message shows the author was alert to the mirror hazard ("expanding to members without the filter would have deleted the whole corpus, a worse bug than the one being fixed") but _all/* are mentioned nowhere in the message and covered by none of the five new tests. Honest mitigating context for whoever weighs this: DELETE /* already reaches .xerj_users today (I checked: 200 {"acknowledged":true}, then .xerj_users 404s), so a comparable door pre-exists — but DELETE /* reads as "destroy everything" whereas POST /*/_delete_by_query reads as a routine cross-index row purge, and POST /*/_close refuses wildcards outright with 400 "Wildcard expressions or all indices are not allowed", which shows the project's stance on wildcards for destructive index ops.

Same request, same harness, only es_compat.rs swapped. BEFORE (HEAD~1 source): `POST /_all/_delete_by_query {"query":{"match_all":{}}}` -> 404 `{"error":{..."reason":"no such index [_all]"...},"status":404}`, `.xerj_users` count stays 1. AFTER (5ef0aa4): -> 200 `{"took":..,"timed_out":false,"total":7,"deleted":7,"batches":2,"version_conflicts":0,"noops":0,"failures":[]}`, `.xerj_users` count 1 -> 0 and the user index `one` 6 -> 0. Identical result for `POST /*/_delete_by_query` (404 `no such index [*]` before; 200 total:7 deleted:7 after). Full snapshots: /tmp/w453-regress/before/all_and_star.json vs /tmp/w453-regress/after/all_and_star.json. Control that the door is new and not pre-existing elsewhere: `POST /*/_close` -> 400 `illegal_argument_exception: Wildcard expressions or all indices are not allowed`, `.xerj_users` untouched.

[should-fix] The 30 s scripted-update budget is now per-member, and its own doc comment says it bounds the request as a whole

run_update_by_query_over (es_compat.rs:24482-24491) calls run_update_by_query once per member, and run_update_by_query mints a fresh deadline on every call: let deadline = Instant::now() + Duration::from_millis(SCRIPTED_UPDATE_BUDGET_MS) (:24568). The constant's doc comment (:24496-24501) states the invariant this breaks in as many words — "It bounds the request as a whole — every statement, every document — rather than any single evaluation, which is the only bound that survives _update_by_query's hit loop and its detached wait_for_completion=false form" — and the comment is left unedited by the PR. After this change one HTTP request buys 30 s x member-count of Painless CPU. That multiplier used to be capped at 1 because the endpoint could only ever address one index; it is now capped by the alias size, and by finding #1 it is capped by the total index count, since _all and * are newly reachable. The detached form makes it worse: ?wait_for_completion=false returns {"task":...} immediately and the whole N x 30 s runs unattended on the runtime.

es_compat.rs:24482-24491 (`for (name, idx) in indices { let r = run_update_by_query(idx, search_req, script.clone(), pipeline.clone()).await; ... }`) against es_compat.rs:24568 (`let deadline = Instant::now() + Duration::from_millis(SCRIPTED_UPDATE_BUDGET_MS);`) and the unchanged contract at :24496-24501. Verified by inspection; I did not build a 30 s+ live repro, and I am flagging that rather than implying I did.

[should-fix] merge_by_query hardcodes timed_out: false and never reads any member's timed_out — a truncated multi-index _update_by_query reads as a complete run

run_update_by_query can and does report truncation: its hit loop sets timed_out = true and breaks when the deadline passes, and it returns "timed_out": timed_out (es_compat.rs:24669). merge_by_query (:24281) accumulates took, total, the affected count, batches, version_conflicts, noops and failures from each member (:24296-24312) but emits a literal "timed_out": false (:24319) and never looks at v["timed_out"] for any member. So a multi-member _update_by_query in which one member's hit loop was cut short by the budget answers HTTP 200 with timed_out:false and an empty failures array over a run that did not finish. That is the exact failure mode this PR exists to eliminate — the commit message's own words are "the entire point of #450 is that a partial run must never read as a complete one" — reintroduced one layer up, in the merge the PR adds. Combined with finding #2 (per-member budget) this is the only signal a caller would have that a member was cut short, and it is discarded.

es_compat.rs:24319 `"timed_out": false,` inside the `json!` that `merge_by_query` returns, versus es_compat.rs:24669 `"timed_out": timed_out,` in the per-member body it is merging. No read of the `"timed_out"` key appears anywhere between :24281 and :24327. Verified by inspection; not reproduced live because tripping it needs >30 s of Painless CPU per member.

[should-fix] Every by-query request, including plain single-index ones, now pays an O(all indices) Index::stats() sweep — measured 2.8x at 0 bystanders and 5.7x at 150

by_query_targets calls resolve_index_selector unconditionally (es_compat.rs:24232), and that helper's first statement is state.engine.list_indices().await (:19297-19303) regardless of whether the selector is a plain concrete name. Engine::list_indices (engine.rs:3297-3312) awaits entry.value().stats().await for every index in the process, and Index::stats (index.rs:18166+) takes the schema read lock, snapshots the segment list, runs aggregated_field_encodings() and builds HashSets over RawString values per field. search_impl — the function this PR says it mirrors — does not do this: it gates the identical enumeration behind let needs_resolve = raw_names.iter().any(|n| *n == "_all" || n.contains('*')); (es_compat.rs:8608), so a plain name skips it entirely. The fix would be to apply that same short-circuit (or resolve_indices_for_op's is_selector test) before enumerating. This is invisible to a response-body diff, which is why the byte-identical claim does not cover it, and it lands on a data-plane write endpoint that clients call in loops.

Same harness binary, ci-test profile, rustc 1.97.1, 30 iterations after 3 warmups, before/after runs back to back on the same box, differing only in es_compat.rs. Per single-index `POST /target/_delete_by_query` with a term query matching nothing: 0 bystander indices — before 20 us, after 56 us. 150 bystander indices (20 docs each) — before 36 / 20 / 36 us across three runs, after 205 / 117 / 206 us. The before-side slope over 0->150 bystanders is ~16 us (environmental); the after-side slope is ~150 us, i.e. ~1 us per index in the process, which matches per-index `stats()` and rules out a codegen artifact (a codegen difference would not scale with bystander count). Harness: /tmp/w453-regress/zz_perf_diff.rs.

[should-fix] A comma selector mixing a filtered alias with a concrete index silently under-deletes the concrete index and reports a complete run

by_query_targets collects alias filters into one flat Vec<Value> (es_compat.rs:24237-24249) and apply_alias_filters (:24266-24271) AND-s all of them into a single bool.filter applied to every member. There is no per-member association, so a filter belonging to one alias is applied to indices that alias does not cover. POST /plain,low/_delete_by_query {"match_all":{}} — plain named directly with no filter, low a filter alias over fa,fb — answers HTTP 200 {"total":9,"deleted":9,"failures":[]} and leaves 7 of plain's 10 documents alive. Internally consistent counts, empty failures, 200: the exact signature #450 describes. I checked whether this is inherited rather than introduced, and it is — GET /plain,low/_search returns 6 instead of 13, so search_impl's collection has the same quirk and the PR's claim to mirror it is accurate. What the PR introduces is the reachability on a destructive endpoint: /plain,low/_delete_by_query was a flat 404 before this change. A related shape from the same flattening: an alias whose members carry DIFFERENT filters AND-s them into a contradiction, so POST /split/_delete_by_query {"match_all":{}} (n<=3 on da, n>=8 on db) deleted 0 of 20 and reported {"total":0,"deleted":0,"failures":[]} at 200 — a complete no-op indistinguishable from success. _search on split also returns 0, so that one is likewise mirrored.

`POST /plain,low/_delete_by_query {"query":{"match_all":{}}}` -> 200 `{"took":0,"timed_out":false,"total":9,"deleted":9,"batches":3,"version_conflicts":0,"noops":0,"failures":[],...}`; then `GET /plain/_count` -> 7 (was 10, caller asked for match_all), `GET /fa/_count` -> 7 (correct). Comparison probe: `POST /plain,low/_search {"size":0}` -> `hits.total.value` 6, not 13, confirming the collection is inherited from search_impl. Split-filter probe: `POST /split/_delete_by_query {"query":{"match_all":{}}}` -> 200 `{"total":0,"deleted":0,"failures":[]}`, `da` 10/10 and `db` 10/10 survive. Harness: /tmp/w453-regress/zz_regress_diff2.rs.

[nit] The merged _update_by_query body drops the deleted key that the single-index body carries

run_update_by_query returns "deleted": 0 alongside "updated" (es_compat.rs:24670), matching ES, which always includes deleted on an _update_by_query response. merge_by_query builds its output from a fixed key list that includes only affected (es_compat.rs:24316-24326), so the merged body has no deleted at all. A client that reads resp["deleted"] on _update_by_query works on every single-index call and raises a KeyError the first time the selector expands to two or more members — a shape difference that depends on cluster topology rather than on the request.

Single-index `POST /sa/_update_by_query` body keys: [took, timed_out, total, updated, deleted, batches, version_conflicts, noops, failures, throttled_millis, requests_per_second, throttled_until_millis]. Two-member alias `POST /duo/_update_by_query` body keys: the same list minus `deleted`. Diff computed in-test: `KEYS PRESENT ON SINGLE BUT ABSENT ON MULTI: ["deleted"]`.

[nit] failures[] mixes two entry shapes and is unbounded now that _all can expand the member set

merge_by_query emits member-level failures as {"index": name, "cause": err} (es_compat.rs:24297) and per-document failures as the runner's {"id":..,"cause":..} with index injected (:24306-24311), so one array carries two shapes and a consumer cannot destructure it uniformly; ES's failures entries have a single shape (index/shard/status/cause). Separately, each member can contribute up to size: 10000 per-document failure objects and there is no cap on the merged array, so with _all newly reachable (finding #1) one refused request can serialize 10000 x index-count failure objects into a single response body.

Observed both shapes in one run: a read_only member produced `{"id":"pb-1","cause":{"reason":"index [pb] is blocked for read_only operations"},"index":"pb"}` entries, while the member-error branch at es_compat.rs:24297 produces `{"index":name,"cause":err}`. The 10000-per-member bound is the `"size": 10000` in the search body both handlers build (es_compat.rs:24132, :24405).

453-regression — refuted: True

The fix is real and its headline claims reproduce, but claim 2 is false in general and the change opens two new data-loss paths that were 404s on main, so I cannot call the evidence sound as written.\n\nWhat I reproduced myself, not from the PR table. On main-source (es_compat.rs reverted to HEAD~1, tests kept) POST /tri/_delete_by_query {\"match_all\":{}} over three ten-document indices answers 200 {\"total\":10,\"deleted\":10,\"batches\":1,\"failures\":[]} and leaves idx-a=0 idx-b=10 idx-c=10 — the filed defect exactly, and nothing in that body distinguishes it from a complete run. _update_by_query reports updated:10 with touched=[10,0,0]. On head: 30/30 with 0/0/0, and 30/30 with [10,10,10]. Single index /idx-b term n=5 -> total 1, deleted 1, batches 1, 10/9/10 on both sides; /no-such-index -> 404 on both. Fail-before redone independently: reverting only the source hunk fails exactly the three named tests and passes the two guards. fmt RC=0, clippy -D warnings RC=0, full xerj-api suite in ci-test RC=0 (31 result blocks, zero failures). So claims 1, 3, 4, 6 and the ci-test half of 5 all hold.\n\nWhere it breaks. (1) by_query_targets uses resolve_index_selector, the one selector in the tree with no hidden-index rule — so POST /*/_delete_by_query and /_all/_delete_by_query now empty .xerj_* system indices and .xerj-memory-* brains (verified: 34/34 deleted, .xerj_secrets 4->0, .xerj-memory-bob-edges 3->0), while POST /*/_refresh on the same build still correctly skips them and both selectors were a 404 on main. (2) POST /_aliases never records an alias filter (AliasActionParams has no such field), so a filtered alias built the canonical way now deletes 30 of 30 where 9 is correct — up from 10 on main. That is verbatim the outcome the commit message says it avoided ("a worse bug than the one being fixed"), and the new test helper documents the gap in a comment and routes around it via PUT /{index}/_alias. Two more silent-partial shapes follow from filters being AND-ed globally rather than per member: an alias with different filters per index deletes nothing at 200/deleted:0/failures:[], and an alias filter leaks onto indices named concretely in the same selector.\n\nReading the tests adversarially. They do assert per-index ground truth, and they would catch "members visited but the filter dropped" and "counts summed while one index touched". They cannot catch a mangled query: every multi-member case sends match_all, and the only selective query is single-index, which returns through the len()==1 early-out. I built the broken implementation the brief asked for — discard the caller's query whenever indices.len() > 1 — and it passes all five tests while POST /tri/_delete_by_query {\"term\":{\"n\":5}} destroys the entire 30-document corpus.\n\nAuthorization holds — that claim survived everything I threw at it. With an auth-enabled router, a key scoped to one member of a three-member alias gets 403 naming the member it cannot reach and nothing is deleted (3/3/3 intact); a scoped POST /*/_delete_by_query removes only the caller's own index; Engine::get_index runs index_guard::visible on each concrete name after alias resolution, so an alias cannot launder access.\n\nMethod notes. Fail-before and "reproduce on main" are the same experiment run once, which is why the main numbers and the three failures come from one build with only es_compat.rs moved. The sabotage run was checked both ways — 5/5 green AND the corpus destroyed — so it did not pass for want of compiling. The one thing I could not verify is the dev-profile full-suite gate: cargo test -p xerj-api stalls at script_limits_http (0 CPU ticks over 20s, 48 sleeping threads). I ruled the PR out as the cause — that target passes 3/3 in isolation in both profiles, the full ci-test suite is green, and identically stuck processes from two other worktrees were already on this box at load 16 / 98% disk. The PR's own five tests do pass in the dev profile."

Findings

[must-fix] POST //_delete_by_query and /_all/delete_by_query now empty .xerj system indices and .xerj-memory-* brains — both were a 404 on main

by_query_targets resolves the selector with resolve_index_selector, which — unlike resolve_indices_for_op, the selector used by _refresh/_close/_forcemerge — has NO hidden-index exclusion. resolve_indices_for_op carries the rule explicitly (if n.starts_with('.') && !part.starts_with('.') { continue }) with the comment "POST /*/_close must never sweep in .xerj_users-class system indices"; Engine::is_system_index (name.starts_with(".xerj_")) enforces the same rule for snapshot/restore, and xerj-console-api owns .xerj_users/.xerj_api_tokens/.xerj_sessions/.xerj_audit under that prefix, created on first boot. So the PR moves the single most destructive verb in the API onto the one resolver that does not implement the project's own wildcard rule. Before this PR both selectors were a plain 404, so this is not a pre-existing hole — it is newly reachable. It is reachable by any superuser, which is the default principal when no admin key is configured. Fix: apply the same dot-prefix exclusion in resolve_index_selector's _all/*/glob branches, or have by_query_targets filter with is_system_index.

Same build, same test app. `POST /*/_refresh` -> 200 {"_shards":{"total":3,...}} (3 user indices; .xerj_secrets correctly skipped). `POST /*/_delete_by_query {"match_all":{}}` -> 200 {"total":34,"deleted":34,"batches":4,"failures":[]}, and .xerj_secrets went 4 -> 0. `POST /_all/_delete_by_query` -> identical (34/34, secrets 0). With es_compat.rs reverted to HEAD~1 (main) and nothing else changed: both -> 404 index_not_found_exception, .xerj_secrets still 4. Reproduced with the reserved agent-memory namespace too: admin `POST /*/_delete_by_query` on an auth-enabled router emptied .xerj-memory-bob-edges (3 -> 0). File: engine/crates/xerj-api/src/es_compat.rs:24232 (by_query_targets -> resolve_index_selector at :19297), contrast :19380-19388 (resolve_indices_for_op).

[must-fix] A filtered alias created with POST /_aliases now deletes the whole corpus (30 of 30, correct is 9) — the exact outcome the commit message claims to have prevented

The alias-filter safety net reads state.engine.index_alias_metadata, which is only written by put_alias (PUT /{index}/_alias/{alias}) and the create-index aliases block. post_aliases never writes it, and AliasActionParams is { index: String, alias: String } — serde silently drops a filter key in an add action. So for a filtered alias built the ES-canonical, atomic way, alias_filters comes back empty, apply_alias_filters returns the query untouched, and every member is swept. The commit message says: "expanding to members without the filter would have deleted the whole corpus, a worse bug than the one being fixed" — that is precisely what happens on this route. The PR turns a 1-of-N over-delete into an N-of-N over-delete. The new test file's own helper documents the gap ("_aliases with a filter in the add action does not store the filter; the per-index alias route does") and routes around it rather than covering or guarding it, so claim 2 of the PR is true only for aliases create 8000 d through one of the two routes. Fix: add filter/routing/is_write_index to AliasActionParams and persist into index_alias_metadata in post_aliases (small), or refuse to expand an alias whose stored metadata is absent.

POST /_aliases {"actions":[{"add":{"index":"idx-a","alias":"lowtri","filter":{"range":{"n":{"lte":3}}}}}, ...idx-b, idx-c]} -> 200 {"acknowledged":true}. GET /idx-a/_alias -> {"idx-a":{"aliases":{"lowtri":{}}}} (filter not stored). Then POST /lowtri/_delete_by_query {"match_all":{}} -> HEAD: 200 {"total":30,"deleted":30,"batches":3,"failures":[]}, idx-a=0 idx-b=0 idx-c=0. MAIN (es_compat.rs reverted): 200 {"total":10,"deleted":10,"batches":1}, idx-a=0 idx-b=10 idx-c=10. Correct is 9 deleted / 7-7-7 left, which is what the PR's own test gets because it builds the alias with PUT /{m}/_alias/{alias}. Files: engine/crates/xerj-api/src/es_compat.rs:19181-19184 (AliasActionParams), :19198-19268 (post_aliases), :24236-24248 (filter collection); test helper comment at engine/crates/xerj-api/tests/by_query_through_an_alias_touches_every_member.rs:79.

[should-fix] Per-member alias filters are AND-ed globally, so an alias with a different filter per index deletes nothing and reports 200 / deleted:0 / failures:[]

by_query_targets pushes every member's filter into one flat Vec and apply_alias_filters emits bool.filter: [f1, f2, ...] — an intersection across indices, not a per-index scope. ES applies each index's alias filter to that index only. The result is a silent no-op reported as a clean, complete run: total 0, deleted 0, empty failures, HTTP 200. That is exactly the #450 failure shape ("a partial run must never read as a complete one") reintroduced on a different axis. The commit says the filters are "collected the way search_impl collects them", which is accurate — search_impl at :9163-9196 has the identical flat AND — but on the read side that under-returns and on this side it under-deletes while claiming success. Fix: run each member with only its own filter, i.e. carry Vec<(name, Arc, Option)> and build the search request per member.

Alias `split` created with PUT /idx-a/_alias/split {"filter":{"range":{"n":{"lte":3}}}} and PUT /idx-b/_alias/split {"filter":{"range":{"n":{"gte":8}}}}, 10 docs each. POST /split/_delete_by_query {"match_all":{}} -> 200 {"total":0,"deleted":0,"batches":2,"version_conflicts":0,"noops":0,"failures":[]}; idx-a=10 idx-b=10 idx-c=10. Correct is 3 from idx-a + 3 from idx-b = 6 deleted, leaving 7/7/10. (On main the same call deleted all 10 of idx-a — also wrong, differently.) engine/crates/xerj-api/src/es_compat.rs:24236-24248 and :24266-24271.

[should-fix] An alias filter leaks onto indices named concretely in the same selector, silently under-deleting them while reporting success

alias_filters is collected from any alias appearing anywhere in the comma list, then applied to the whole expanded set. An index named directly in the selector should be deleted in full — it was not addressed through the alias, so the alias's filter has no business scoping it. The caller gets a 200 with an internally consistent deleted count and no failures, and is left with documents it asked to delete. Same silent-partial class as the bug being fixed. Note this selector shape was a 404 on main, so it is newly reachable. Fix is the same as the previous finding: scope filters per member rather than per request.

Alias `lowtri` (filter n<=3) on idx-a and idx-b only; idx-c has no alias. POST /lowtri,idx-c/_delete_by_query {"match_all":{}} -> HEAD: 200 {"total":9,"deleted":9,"batches":3,"failures":[]}; idx-a=7 idx-b=7 idx-c=7. Correct: 3+3+10 = 16 deleted, leaving 7/7/0. MAIN: 404 index_not_found_exception [lowtri,idx-c], nothing deleted. engine/crates/xerj-api/src/es_compat.rs:24236-24248.

[should-fix] The five new tests cannot see whether the caller's query survives the multi-member path — a sabotaged build that deletes the entire corpus passes all five

Every multi-member assertion in the new file uses {"match_all":{}}: delete_by_query_through_an_alias_deletes_from_every_member, update_by_query_through_an_alias_updates_every_member and a_filtered_alias_deletes_only_the_documents_it_selects all send match_all, and the only selective query in the file ({"term":{"n":5}}) is on a single concrete index, which returns through merge_by_query's len()==1 early-out and never exercises the new code. So the suite is blind to the entire class of "members are visited but the query is mangled". I proved this by construction rather than by inspection. Add one multi-member case with a selective query (assert 9/9/9 per index for term n=5 over tri), and one with a selective query through the filtered alias.

Sabotage applied to es_compat.rs only: in both handlers, `if indices.len() > 1 { json!({"match_all":{}}) }` substituted for the caller's query before apply_alias_filters. Result: `cargo test --profile ci-test --test by_query_through_an_alias_touches_every_member` -> 5 passed; 0 failed. Same build: POST /tri/_delete_by_query {"query":{"term":{"n":5}}} -> 200 {"total":30,"deleted":30,"batches":3,"failures":[]} with idx-a=0 idx-b=0 idx-c=0 — the whole 30-document corpus destroyed by a request that should have removed 3. The real head build answers 200 {"total":3,"deleted":3} / 9-9-9 on the same request, so the implementation is correct here; the tests simply cannot tell. Source restored byte-for-byte afterwards (git status clean at 5ef0aa4).

[should-fix] A script-limit refusal degrades from 400 with no deletion count to 200 with deleted:0 once the selector is an alias, breaking the invariant an existing test asserts

run_delete_by_query fails closed on results.script_failure and returns {"error":..,"status":400}. On the single-index path by_query_response lifts that to a real 400 and the body carries no deleted key — script_limits_http::delete_by_query_refuses_a_selection_a_script_limit_truncated asserts exactly that ("a refused delete must not also report a deletion count"). Through a multi-member alias, merge_by_query's all-failed early-out only fires when EVERY member errored; one member that never evaluated the script (empty, or no candidate docs) returns a success value, so the merged body has no status, gets HTTP 200, and reports total/deleted counts alongside the exception buried in failures[]. The refusal contract is per-index only. Related: the merge is also non-transactional with no pre-validation, so a query-level refusal can now land after earlier members have already been deleted.

Alias `pair` over idx-empty (0 docs) and idx-full (1 doc), same over-depth minimum_should_match_script used by script_limits_http. POST /idx-full/_delete_by_query -> 400 {"error":{"type":"script_exception",...},"status":400}, no `deleted` key. POST /pair/_delete_by_query, identical body -> 200 OK {"took":0,"timed_out":false,"total":0,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"failures":[{"index":"idx-full","cause":{"type":"script_exception",...}}],...}. engine/crates/xerj-api/src/es_compat.rs:24281-24330 (merge_by_query) and :24196-24204 (by_query_response).

[nit] _update_by_query's merged body drops the deleted field the single-index path and ES both emit

merge_by_query builds a fixed object with only the affected key, so deleted (always present, usually 0, in run_update_by_query's own result) disappears the moment the selector resolves to more than one index. A client reading body["deleted"] on _update_by_query gets a value for a concrete index and undefined for an alias. Add both keys unconditionally.

Single member (main, /tri): {"total":10,"updated":10,"deleted":0,"batches":1,...}. Three members (head, /tri): {"took":0,"timed_out":false,"total":30,"updated":30,"batches":3,"version_conflicts":0,"noops":0,"failures":[],"throttled_millis":0,"requests_per_second":-1,"throttled_until_millis":0} — no `deleted`. engine/crates/xerj-api/src/es_compat.rs:24317-24329.

[nit] Pre-existing hardcoded size:10000 makes the same silent short delete the PR exists to eliminate, on the same handler, and it is now multiplied per member

Both handlers build the selection with a fixed "size": 10000 and report "batches": 1, with no scroll. A single index holding more than 10k matches reports a complete run and leaves the remainder. This is untouched by the PR and pre-existing, but it is the identical failure signature the issue is about (200, total == deleted, failures empty, nothing to distinguish it from a complete run), and the merge now sums per-member caps so a 3-member alias can report 30000/30000 while 3 x N documents survive. Worth resolving alongside #450 rather than after it, since the fix's whole thesis is that a partial run must not read as a complete one.

Index `big` bulk-loaded with 10,050 docs (verified GET /big/_count -> 10050). POST /big/_delete_by_query {"match_all":{}} -> 200 {"took":80,"timed_out":false,"total":10000,"deleted":10000,"batches":1,"version_conflicts":0,"noops":0,"failures":[]}; GET /big/_count afterwards -> 50. engine/crates/xerj-api/src/es_compat.rs:24132 and :24405.

[nit] Every _delete_by_query / _update_by_query request now pays a full Engine::list_indices(), which awaits stats() per index

resolve_index_selector unconditionally calls state.engine.list_indices().await, and that iterates the whole indices DashMap awaiting entry.value().stats() on each one, even when the path segment is a single concrete name that the old code resolved with one map lookup. On a node with many indices this is a new per-request cost on a write verb. Take the cheap path w

(truncated at GitHub's comment limit; both must-fixes are stated in full below.)

My read — this is worse than what it fixes, and I am drafting it

Two regressions, both mine, both reproduced live against the parent commit:

1. A filtered alias created the canonical ES way now deletes everything. POST /_aliases {"actions":[{"add":{...,"filter":...}}]} silently discards the filter — AliasActionParams has no filter field and no deny_unknown_fields, so it answers acknowledged:true and throws it away. by_query_targets then finds no recorded filter, apply_alias_filters returns the query untouched, and every member is deleted in full:

PR head 5ef0aa4   POST /lowtri/_delete_by_query -> {"total":30,"deleted":30}   a=0 b=0 c=0
parent 8d7fc76    same request                  -> {"total":10,"deleted":10}   a=0 b=10 c=10
same PR node, alias created via PUT /{index}/_alias -> {"total":9,"deleted":9}  a=7 b=7 c=7

My commit message says expanding without the filter "would have deleted the whole corpus, a worse bug than the one being fixed". That is exactly what it does for the API the Elasticsearch docs use in every filtered-alias example. Worse: my own test helper carries a comment saying _aliases does not store the filter — I discovered the gap while writing the tests, routed around it, and never asked what my new expansion would do when it hit one.

2. _all and * became cluster-wide destructive operations at write privilege. They 404'd on main (get_index("_all") matched nothing). resolve_index_selector honours them, with no hidden/dot-prefix filter, so one request empties every index the caller can see — .xerj_users, .xerj_api_tokens, .xerj_sessions, .xerj_audit, .xerj_cluster_state, .xerj-memory-* included. The pre-existing DELETE /* requires Privilege::AdminIndex; this path resolves to Privilege::WriteIndex. I described it as "the same expansion search_impl uses" — it is only the name-resolution half, without any of the expand_wildcards / ignore_unavailable / allow_no_indices handling that makes it safe there.

Both are the same mistake in different clothes: I took a resolver built for reads and pointed it at a destructive path without asking what its permissiveness means when the verb is DELETE.

Next revision, narrow and fail-closed rather than clever: by-query resolves concrete names and aliases only — _all, * and globs go back to refusing — and POST /_aliases rejects an add carrying filter instead of silently dropping it, so the silent-drop that makes expansion dangerous cannot happen. The remaining should-fixes (alias-of-alias, closed members, per-member filters AND-ed globally, timed_out hardcoded, the per-member script budget) get addressed on top.

…t fixed

Refuted 3 of 3. Both must-fixes are regressions this change introduced, and I
reproduced each against a live node before touching anything.

1. A FILTERED ALIAS CREATED THE CANONICAL WAY LOST ITS FILTER, so the new
   expansion deleted everything behind it.

   `POST /_aliases {"actions":[{"add":{…,"filter":…}}]}` is the API every
   Elasticsearch filtered-alias example uses. `AliasActionParams` had no
   `filter` field, so the filter was accepted, answered `acknowledged: true`,
   and thrown away. `by_query_targets` then found no recorded filter and deleted
   every member in full:

     this branch   POST /lowtri/_delete_by_query -> total 30, deleted 30, 0 left
     parent        same request                  -> total 10, deleted 10, 20 left
     same node, alias made with PUT /{i}/_alias   -> total 9,  deleted 9,  21 left

   My own commit message said expanding without the filter "would have deleted
   the whole corpus, a worse bug than the one being fixed". It did. And my test
   helper carries a comment noting that `_aliases` does not store the filter —
   I found the gap while writing tests, routed around it, and never asked what
   the expansion would do when it met one.

   `POST /_aliases` now REFUSES an action carrying `filter`, naming the route
   that does persist it. Silently ignoring it is the one option that cannot be
   made safe; accepting it means storing metadata this route has never stored,
   which is a change to `_search` semantics and belongs in its own PR.

2. `_all` AND `*` BECAME A CLUSTER-WIDE DELETE AT `write` PRIVILEGE.

   They 404'd before, because `get_index("_all")` matched nothing.
   `resolve_index_selector` honours them with no hidden/system exclusion, so one
   request emptied every index the caller could see. Measured on this branch: 46
   documents across 17 indices at HTTP 200, including `.xerj_dashboards` and
   `.xerj_magic_links`, with `.xerj_users`, `.xerj_api_tokens`, `.xerj_sessions`,
   `.xerj_audit` and `.xerj_cluster_state` in range. `required_privilege` gives
   this op segment `Privilege::WriteIndex`, while the pre-existing `DELETE /*`
   needs `Privilege::AdminIndex` — so emptying the cluster went from `manage` to
   `write`.

   I called this "the same expansion `search_impl` uses". It is only the
   name-resolution half, without the `expand_wildcards`, `ignore_unavailable`
   and `allow_no_indices` handling that makes wildcards safe on the read side.
   Both by-query verbs now refuse `_all`, `*` and globs with a 400 that names
   the offending pattern, the way ES gates the same shape behind
   `action.destructive_requires_name`.

Both are the same mistake wearing different clothes: I took a resolver built for
reads and pointed it at a destructive verb without asking what its permissiveness
means when the verb is DELETE.

Two new tests, each encoding one regression — a refused wildcard must leave all
30 documents, and a refused `_aliases` filter must leave no alias behind while
the working route still deletes exactly 9. Seven tests total, all green.

Gates: fmt clean, clippy --workspace --all-targets -D warnings clean, full
xerj-api suite green under rustc 1.97.1 in the ci-test profile.

Known cosmetic: both refusals surface as `search_phase_execution_exception`,
which is what `XerjError::invalid_query` maps to. The status is right (400) and
the reason string is specific; the ES type name for these would be
`illegal_argument_exception`. Left alone rather than widening the error mapping
in a PR about destructive correctness.

The should-fixes the panel raised on top of these — alias-of-alias still
reproducing #450, a closed member being emptied where `_search` skips it,
per-member filters AND-ed globally, `timed_out` hardcoded, and the scripted
budget becoming per-member — are not addressed here and the PR stays a draft
until they are.
…efusal guarded the wrong door

Refuted again. The previous revision made `POST /_aliases` reject an action
carrying a `filter`, on the reasoning that accepting-and-ignoring is the one
option that cannot be made safe. That was right about the danger and wrong about
where it lives.

The refusal blocks the caller who DECLARES a filter. It does nothing about the
shape ES clients actually send, because the canonical repoint idiom carries no
filter on the `add` at all:

  POST /_aliases {"actions":[{"remove":{old}},{"add":{new}}]}

Nothing to refuse, so it is accepted at 200 — and the alias lands on a backing
index with no metadata, silently unfiltered. With the by-query paths now
honouring alias filters, the next `_delete_by_query` through that alias empties
the whole index. A reviewer measured it: 10 of 10 destroyed where the filter
meant 3.

It also removed a capability without replacing it. `PUT /{index}/_alias/{alias}`
cannot do the `remove` half, so after that change there was NO route able to
atomically repoint a filtered alias — and my error message pointed at exactly
that route.

So the filter is stored now, mirroring the `attach` closure `put_alias` has had
all along, and `remove` clears the metadata so a later re-add of the same name
cannot inherit a filter nobody asked for. Verified end to end on a live node:

  PUT /old-1/_alias/cur {filter n<=3}          -> filter present
  POST /_aliases  remove old-1 / add new-1 +filter -> 200 (was 400)
  GET /_alias/cur                              -> filter followed the repoint
  POST /cur/_delete_by_query {match_all}       -> total 3, deleted 3
  GET /new-1/_count                            -> 7 surviving

The test that encoded the refusal is replaced by one that encodes the storage
and the repoint, including that the alias leaves the old index and that an
unfiltered expansion would have emptied the new one. 7 tests, all green.

Gates: fmt clean, clippy --workspace --all-targets -D warnings clean, full
xerj-api suite green under rustc 1.97.1 in BOTH the dev and ci-test profiles.

STILL OPEN, and why this stays a draft — a second must-fix I have not addressed:
this PR makes the by-query verbs the only routes that expand an alias to every
member, while `_search` and `_count` still read one. The standard pre-flight for
a delete is to run the same query as a search first, so that preview now
under-reports by the member count: search says 5, the delete removes 15. That is
#450 inverted, and it punishes precisely the operator who checks before
deleting. The right resolution is for the READ paths to expand too — which is
#433/#449 — not for this path to go back to touching one member. Sequencing
those is the next decision, and I would rather state it than paper over it.
@xerj-org
Copy link
Copy Markdown
Owner Author

Independent verification — BLOCKING data-loss regression, NOT merging

3 skeptics (correctness / regression-risk·data-safety / evidence-holds) in worktrees under 1.97.1. The data-safety lens found a restart-triggered over-delete; I confirmed it in the engine source. Verdicts verbatim:

correctness — refuted:false, blocking:false.

CLAIM CONFIRMED across all four sub-claims by code trace + the 7 new tests passing on CI toolchain 1.97.1 (0 failed).

(a) All members: by_query_targets (es_compat.rs ~24276) calls resolve_index_selector, which expands an alias name to every backing member (line 19401-19406: if let Some(targets)=aliases.get(part){for n in targets.value(){out.push}}). Each member is opened via get_index up front (fail-before-write). run_delete_by_query_over / run_update_by_query_over iterate ALL members and merge_by_query the results. A 3-member alias resolves to 3 and each is deleted/updated. Tests delete_by_query_through_an_alias_deletes_from_every_member and update_by_query_through_an_alias_updates_every_member pass.

(b) Filter combination is safe: apply_alias_filters returns json!({"bool":{"must":[query],"filter":alias_filters}}) — the user query is a must clause, alias filters are filter clauses. This is b 10000 yte-identical placement to search_impl (lines 9191-9196), so knn pre-filter semantics match. Neither the user query nor the alias filter can be dropped: query is always the sole must element, filters are the filter array. by_query_targets collects filters from index_alias_metadata[backing][alias]["filter"] — the exact shape set_alias_metadata writes and search_impl reads. Test a_filtered_alias_deletes_only_the_documents_it_selects passes (asserts the alias filter is honoured, not the whole corpus).

(c) Plain index unchanged: a non-alias name has no aliases.get(part) entry, so alias_filters stays empty and apply_alias_filters returns the query verbatim; a single-member merge_by_query returns that one result unchanged. Test a_plain_single_index_request_is_unchanged passes.

(d) Metadata: set_alias_metadata writes json!({"filter":f}) ONLY when filter is a non-empty object, else calls clear_alias_metadata; the add path in post_aliases calls set_alias_metadata(add.filter), the remove path calls clear_alias_metadata. AliasActionParams gained filter: Option<Value>. Test post_aliases_stores_the_filter_and_a_repoint_keeps_it passes (filter survives the atomic add and a remove+add repoint).

Symmetry: update path got the SAME treatment — run_update_by_query_over exists and update_by_query applies apply_alias_filters identically to delete. No half-fixed path.

#449 interaction (CLEAN merge) verified benign: the by-query path uses its own filter-collection loop reading the same metadata shape as the #449-modified search_impl, and runs the pre-combined search_req directly (not through search_impl), so there is no double filter application. Bonus safety: wildcard/_all selectors are refused on these destructive verbs (test a_wildcard_or_all_selector_is_refused_rather_than_expanded), preventing _all/_delete_by_query from emptying system indices.

Nothing to block: this is a correct data-loss fix that over-deletes nothing and applies alias filters on every path.

regression-risk (data-safety / over-deletion) — refuted:true, blocking:true.

OVER-DELETE PATH (blocking, data loss): alias filters are stored ONLY in the in-memory index_alias_metadata DashMap (engine.rs:646, inited empty at :851) and are NEVER persisted or restored. Boot runs only load_persisted_aliases() (engine.rs:981), and aliases.json is serialized as HashMap<String,Vec> — alias→members topology ONLY, no filters (flush_aliases, engine.rs:1698-1703). add_alias flushes topology (engine.rs:2835); the PR's set_alias_metadata (es_compat.rs:19216) writes only the volatile map. After ANY restart, a genuinely-filtered N-member alias resolves to all members (topology survived) but by_query_targets collects alias_filters=[] (es_compat.rs:24336-24344), apply_alias_filters (:24365) returns the raw query unchanged, and POST /{alias}/_delete_by_query {"match_all":{}} deletes every doc in every member — the exact 30-of-30 (filter meant 9) data-loss #453 set out to prevent, now reachable via a routine restart and strictly WORSE than pre-#453 (which only touched aliased.first()). The new 364-line test never restarts the engine, so it stays green while this path is open. Other sub-questions clean: (b) match_all/absent query keep the filter ANDed when metadata present; (c) plain-index deletes get no filter (aliases-map gated, no over-delete); (d) task and non-task paths both call *_over(&indices) identically over all members; (e) merge_by_query surfaces per-member errors into failures (no hidden partial success); (f) no #449 interaction — search_impl (9163) and by_query_targets (24336) are separate handlers with identical collection logic, no double-apply. But the non-durable filter store defeats the PR's core safety guarantee on a destructive path = blocking.

evidence-holds — refuted:false, blocking:false.

Verified empirically in worktree on v453 (rustc 1.97.1). WITH fix: all 7 tests in by_query_through_an_alias_touches_every_member.rs PASS (10.54s). FAIL-BEFORE via git apply -R of the es_compat.rs diff only (kept the test): 5/7 fail exactly as the claim requires — under-delete guarded (delete_by_query_through_an_alias_deletes_from_every_member: idx-b still 10, only first member idx-a deleted), under-update guarded (update_by_query_through_an_alias_updates_every_member: idx-b not updated), over-delete/filter guarded (a_filtered_alias_deletes_only_the_documents_it_selects: deleted all 10 of idx-a ignoring filter → 20 remain not 21, deleted 10 not 9), plus POST /_aliases persistence (post_aliases_stores_the_filter_and_a_repoint_keeps_it) and the _all//idx- refusal. File restored afterward. Coverage of the four items: (i) multi-member delete COVERED; (ii) filtered-slice delete (the over-delete guard) COVERED; (iv) POST /_aliases filter persistence COVERED including the remove+add repoint idiom and a filter-honoring delete. GAP on (iii): _update_by_query is covered ONLY for the multi-member/under-update mode — there is NO filtered _update_by_query test, so the over-update (filtered alias emptying an index via update) mode is never exercised; the over-delete mode is guarded for delete alone. This is a test-coverage gap (nit), not a data-loss defect — the code fix itself works, so non-blocking. Claim substantially holds; the only overstatement is "covers _update_by_query as well" for the over-delete mode specifically.

Confirmed independently

index_alias_metadata (engine.rs:655, holds alias filters) is initialised empty (860) and is never serialized — a grep for any flush/persist of it returns nothing. Only aliases.json (topology, name→members) is persisted (flush_aliases/load_persisted_aliases). So across a restart the alias keeps its members but loses its filter.

Decision: do NOT merge. Blocked on a durability prerequisite.

The in-memory-only filter store defeats this PR's core guarantee on a destructive path: after any restart, POST /{filtered-alias}/_delete_by_query {"match_all":{}} deletes every document in every member (30 of 30 where the filter meant 9) — strictly worse than the pre-#453 bug, which only touched the first member's 10. The 364-line test never restarts the engine, so it stays green.

The in-memory fix (hit all members + AND the alias filter + symmetric update path) is correct and valuable — it just needs alias filter metadata to be durable first. Blocked on the root-cause issue below. Non-blocking nit from the evidence lens: no filtered _update_by_query test (over-update mode unexercised).

buger pushed a commit to probelabs/xerj that referenced this pull request Aug 21, 2026
… and _terms_enum

Part of xerj-org#449's class A, split out because it is small, additive and provable on
its own.

`resolve_indices_for_op` had no alias branch at all. Its concrete-name arm
checked the name against the list of real indices and 404'd otherwise, so an
alias was simply "not an index". Measured on a three-member alias `tri` over
idx-a/idx-b/idx-c:

  POST /tri/_refresh      -> 404      POST /idx-a/_refresh      -> 200
  POST /tri/_forcemerge   -> 404      POST /idx-a/_forcemerge   -> 200
  POST /tri/_cache/clear  -> 404      POST /idx-a/_cache/clear  -> 200
  POST /tri/_terms_enum   -> 404      POST /idx-a/_terms_enum   -> 200

An alias is a valid index selector for every one of these in Elasticsearch.

The new branch expands to EVERY member, not `aliased.first()`. That distinction
is the reason this is worth doing carefully rather than quickly: a maintenance
op that silently addressed one member of a multi-index alias is the same defect
class as xerj-org#450 on the destructive side, and a `_refresh` reaching one of three
members makes the next search non-deterministic with nothing in the response
admitting it. After:

  POST /tri/_refresh -> 200, _shards {"total":3,"successful":3,"failed":0}

A concrete index name still wins over an alias of the same name, matching ES,
where the two namespaces cannot collide.

The missing-name rule is deliberately unchanged — ES defaults
`ignore_unavailable=false`, so a typo must fail the whole request rather than
silently doing less than asked:

  POST /no-such-thing/_refresh -> 404
  POST /idx-a,typo/_refresh    -> 404
  POST /idx-*/_refresh         -> 200   (wildcard expansion, unchanged)

Two tests. Fail-before proved by reverting ONLY the source hunk:
`maintenance_endpoints_accept_an_alias_and_reach_every_member` FAILED,
`an_unknown_name_still_fails_the_whole_request` passed — the second is the
no-regression guard, so it passing is the point.

Gates: fmt clean, clippy --workspace --all-targets -D warnings clean, full
xerj-api suite green under rustc 1.97.1 in BOTH the dev and ci-test profiles.

NOT fixed here, and xerj-org#449 stays open: the other resolver class, where
`Engine::get_index` resolves an alias to `aliased.first()` and endpoints answer
from one member instead of 404ing — `_explain`, `_segments`, `_cat/segments`,
`_eql/search`, `_async_search`, `_mget` and `_flush`. Those return 200 with a
partial answer, which is a different and harder problem than returning 404 on a
valid name, and it needs the same "what does this op mean across members"
judgement that xerj-org#453 is still drafted over.
buger pushed a commit to probelabs/xerj that referenced this pull request Aug 21, 2026
…erj-org#524)

The alias->members topology is persisted to aliases.json, but each alias's
filter lived only in the in-memory index_alias_metadata map and was never
written to disk. So a filtered alias came back after a restart with its
members but no filter: _search/_count through it silently returned the whole
backing corpus instead of the slice the filter names, and once the by-query
paths honour the filter (xerj-org#453), a _delete_by_query through such an alias would
empty the whole backing index.

Persist the map to an alias_metadata.json sidecar and reload it on boot,
mirroring flush_aliases/load_persisted_aliases. A separate file (not a wider
aliases.json) keeps that file's shape unchanged and is downgrade-safe — an
older binary ignores it — the same call the ingest-pipeline-refusals sidecar
makes. flush_alias_metadata runs after every mutation of the map (put-alias
attach, create-time aliases, index delete); load_alias_metadata runs next to
load_persisted_aliases on boot.

Test: create a filtered alias, drop the engine, open a fresh one on the same
data dir, assert the filter is still there. Proven fail-before by reverting
only the source hunks — the alias comes back as {"orders":{"aliases":
{"gold":{}}}}, filter gone. Verified under rustc 1.97.1: fmt, clippy
-D warnings, build --profile ci-test, and the test under both dev and ci-test.
@xerj-org
Copy link
Copy Markdown
Owner Author

Closing as superseded — the bug this draft targets (#450) is already fixed on main.

The final fan-out landed in #552 (commit 71d77913, "fan _delete_by_query/_update_by_query out to every alias member (#450)"), with the shared resolve_by_query_targets resolver (es_compat.rs:24475) now driving both delete_by_query and update_by_query; #553 (2eec70a7) added the filtered-alias by-query slice test and the _update_by_query deleted:0 aggregate. #450 is closed.

This draft (fix/by-query-alias-members) is the earlier, still-evolving attempt at the same fix — its three commits (deleted one member → the alias expansion was more dangerous than the bug → store the alias filter instead of refusing it) are exactly the design iteration that #552 settled. It now conflicts (mergeStateStatus DIRTY) because main already rewrote these functions, and its repro test duplicates the merged coverage. Nothing here is unique to salvage; closing rather than rebasing a redundant draft.

If a gap in the merged fix is later found, it should be a fresh PR against the current resolve_by_query_targets, not a rebase of this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_delete_by_query and _update_by_query through a multi-index alias silently touch only the first member and report success

1 participant

0