Repository navigation
fix(xerj-autoindex): resume cleanup 404 is an answer; an uncountable node is UNKNOWN — not "wrote 8000 no new records" - #1174
Merged
Merged
Conversation
…an uncountable node is UNKNOWN — not "wrote no new records" The 2026-10-06 overnight failure of the standing xerj-search rebuild (issue #1173) ended in two lies from our own tooling: 1. The per-file cleanup before a replacement POSTs /{index}/_delete_by_query against the index the journal names. A namespace swap mid-run retires that index, the node answers 404, and the client classified the response as an Abort — killing a run whose goal state (no stale documents) already held: a missing index cannot hold any. The fix gives delete_by_query_pass the same rule count_endpoint, cat_indices_json and search_present already follow: 404 IS the answer, "deleted nothing", answered on the first request — convergent is not a thing to retry. 2. corpus index's legacy arm then compared before/after counts and printed "autoindex failed ... and wrote no new records" — while the after-count had come back UNKNOWN from the node the failed run had just left unhealthy. An absence the node never confirmed is not ours to claim. The decision is now a named three-state verdict (LegacyFailureVerdict: Salvaged / CountUnknown / WroteNothing) so the honest-claims rule the public docs are held to is pinned by a test rather than by message wording: UNKNOWN prints UNKNOWN and names --fresh; "wrote no new records" is only printed when both counts are known. Verification: xerj-autoindex lib tests 1255 passed / 0 failed (includes the two new tests: delete_by_query_on_a_missing_index_is_already_clean_not_an_abort — 404 answered once, no retry, Ok — and a_failed_run_the_node_cannot_count_is_unknown_not_wrote_nothing); clippy clean; fmt clean. Addresses the honesty half of #1173. The resume-throughput crawl (~45 KB/s vs fresh) is the other half and stays open there — this lands the two fixes that changed a recoverable state into a false report, nothing about replay speed. Files: engine/crates/xerj-autoindex/src/esclient.rs (delete_by_query_pass), engine/crates/xerj-autoindex/src/xc.rs (legacy_failure_verdict + the legacy failure arm).
# Conflicts: # CHANGELOG.md
Owner
Author
…esty # Conflicts: # CHANGELOG.md
Ravandevil25
pushed a commit
to Ravandevil25/xerj
that referenced
this pull request
Oct 6, 2026
Version roll 1.0.0-rc.83 → 1.0.0-rc.84. Engine window (all merged to main before this cut): - xerj-org#1172 (fix/cluster-topology-honesty, closes xerj-org#1169): cluster endpoints stopped fabricating topology — health reports real configured membership, wait_for_nodes times out honestly (408 + timed_out), upper bounds enforced, _cat/nodes lists every configured member, _cluster/state reports the actual cluster address; multinode smoke skips with a recorded reason instead of passing vacuously. Also the xerj-org#1171 config half: peer-label≠address and cluster.port collisions refused at load. - xerj-org#1175 (fix/cluster-degraded-health, closes xerj-org#1171 with xerj-org#1172): the transport boot outcome recorded on AppState (ClusterTransportStatus Disabled/Running/Degraded) before the first router exists; degraded boot → health yellow, number_of_nodes 1, additive xerj_cluster_transport:"degraded" marker; unopenable indices (red) still outrank the marker. - xerj-org#1174 (fix/corpus-index-resume, xerj-org#1173 honesty half — throughput half stays open): xerj corpus index stopped reporting a converged corpus as a failed crawl; resume path re-crawls only what's missing. - xerj-org#1167 (docs): hub counts un-staled — llms.txt:51 and the corpus-hub blog note said 75 live / 104 backlog from 2026-10-02; now 100/119. - xerj-org#1163 (thomas-villani, docs): EPUB/notebook answer pages recaptured on the rc.83 build; \x08 man-page fact-check alias fix + self-test. Mechanics of this commit: - engine/Cargo.toml version → 1.0.0-rc.84; cargo update -w rolled 17 Cargo.lock entries. - landing/: 91 V1.0.0-RC.83 footer spans → V1.0.0-RC.84 across 54 files (0 old spans remain); landing/index.html data-latest-tag span → v1.0.0-rc.84. - CHANGELOG: [Unreleased] rolled into [1.0.0-rc.84] - 2026-10-06 with all five window PRs cited under Fixed. - ROADMAP: Last-reviewed line → rc.84; open-defects shortlist re-verified live at cut time — eleven open (xerj-org#1091 xerj-org#1092 xerj-org#1094 xerj-org#1100 xerj-org#1146 xerj-org#1147 xerj-org#1158 xerj-org#1122 carried forward, xerj-org#1168 xerj-org#1170 cluster liveness/leader-wiring, xerj-org#1173 throughput half); xerj-org#1169/xerj-org#1171 moved to the CHANGELOG; GA-window rc.84 paragraph added. ES-YAML gate on the window PRs: 0 failed (452/452 re-run after one non-reproducing co-tenant flake).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The 2026-10-06 overnight failure of the standing
xerj-searchrebuild (#1173) ended in two false reports from our own tooling:/{index}/_delete_by_queryagainst the index the journal names. A namespace swap mid-run retires that index, the node answers 404, and the client classified the response asAbort— killing a run whose goal state (no stale documents) already held. A missing index cannot hold any.corpus index's legacy arm then compared before/after counts and printed "autoindex failed … and wrote no new records" — while the after-count had come back UNKNOWN from the node the failed run had just left unhealthy. The run had written records; the node never said otherwise; the tool claimed it anyway.What changed
delete_by_query_passgets the rulecount_endpoint,cat_indices_jsonandsearch_presentalready follow: 404 IS the answer — "deleted nothing" — returned asOk(0)on the first request, because convergent is not a thing to retry. Every other status keeps the refusal error.LegacyFailureVerdict(Salvaged/CountUnknown/WroteNothing), so the honest-claims boundary is pinned by a test rather than by message wording:--freshVerification
xerj-autoindexlib tests 1255 passed / 0 failed, including the two new tests:delete_by_query_on_a_missing_index_is_already_clean_not_an_abort— 404 answered once, no retry,Oka_failed_run_the_node_cannot_count_is_unknown_not_wrote_nothing— all three verdicts, UNKNOWN from either sideScope
Addresses the honesty half of #1173. The resume-throughput crawl (~45 KB/s vs fresh, the replay path re-applying the un-committed tail) stays open on the issue — this changes what the tool says, not how fast it replays.
Files:
engine/crates/xerj-autoindex/src/esclient.rs(delete_by_query_pass),engine/crates/xerj-autoindex/src/xc.rs(legacy_failure_verdict+ the legacy failure arm).