8000 fix(xerj-autoindex): resume cleanup 404 is an answer; an uncountable node is UNKNOWN — not "wrote no new records" by xerj-org · Pull Request #1174 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

fix(xerj-autoindex): resume cleanup 404 is an answer; an uncountable node is UNKNOWN — not "wrote 8000 no new records" - #1174

Merged
xerj-org merged 4 commits into
mainfrom
fix/corpus-resume-honesty
Oct 6, 2026
Merged

xerj-org merged 4 commits into
mainfrom
fix/corpus-resume-honesty

Conversation

@xerj-org
@xerj-org xerj-org commented Oct 6, 2026
Copy link
Copy Markdown
Owner

Why

The 2026-10-06 overnight failure of the standing xerj-search rebuild (#1173) ended in two false reports from our own tooling:

  1. A converged state was fatal. The per-file cleanup before a replacement POSTs /{index}/_delete_by_query against the index the journal names. A namespace swap mid-run retires that index, the node answers 404, and the client classified the response as Abort — killing a run whose goal state (no stale documents) already held. A missing index cannot hold any.
  2. An absence was fabricated. corpus index's legacy arm then compared before/after counts and printed "autoindex failed … and wrote no new records" — while the after-count had come back UNKNOWN from the node the failed run had just left unhealthy. The run had written records; the node never said otherwise; the tool claimed it anyway.

What changed

  • delete_by_query_pass gets the rule count_endpoint, cat_indices_json and search_present already follow: 404 IS the answer — "deleted nothing" — returned as Ok(0) on the first request, because convergent is not a thing to retry. Every other status keeps the refusal error.
  • The legacy-arm failure decision is now a named three-state verdict, LegacyFailureVerdict (Salvaged / CountUnknown / WroteNothing), so the honest-claims boundary is pinned by a test rather than by message wording:
    • both counts known and grew → salvaged (unchanged behavior)
    • any count UNKNOWN → prints UNKNOWN (not absent) and names --fresh
    • both counts known, no growth → "wrote no new records (N -> M)" — the absence claim is earned, and now carries the numbers

Verification

  • xerj-autoindex lib tests 1255 passed / 0 failed, including the two new tests:
    • delete_by_query_on_a_missing_index_is_already_clean_not_an_abort — 404 answered once, no retry, Ok
    • a_failed_run_the_node_cannot_count_is_unknown_not_wrote_nothing — all three verdicts, UNKNOWN from either side
  • clippy clean, fmt clean.

Scope

Addresses the honesty half of #1173. The resume-throughput crawl (~45 KB/s vs fresh, the replay path re-applying the un-committed tail) stays open on the issue — this changes what the tool says, not how fast it replays.

Files: engine/crates/xerj-autoindex/src/esclient.rs (delete_by_query_pass), engine/crates/xerj-autoindex/src/xc.rs (legacy_failure_verdict + the legacy failure arm).

…an uncountable node is UNKNOWN — not "wrote no new records"

The 2026-10-06 overnight failure of the standing xerj-search rebuild
(issue #1173) ended in two lies from our own tooling:

1. The per-file cleanup before a replacement POSTs
   /{index}/_delete_by_query against the index the journal names. A
   namespace swap mid-run retires that index, the node answers 404,
   and the client classified the response as an Abort — killing a run
   whose goal state (no stale documents) already held: a missing index
   cannot hold any. The fix gives delete_by_query_pass the same rule
   count_endpoint, cat_indices_json and search_present already follow:
   404 IS the answer, "deleted nothing", answered on the first request
   — convergent is not a thing to retry.

2. corpus index's legacy arm then compared before/after counts and
   printed "autoindex failed ... and wrote no new records" — while the
   after-count had come back UNKNOWN from the node the failed run had
   just left unhealthy. An absence the node never confirmed is not
   ours to claim. The decision is now a named three-state verdict
   (LegacyFailureVerdict: Salvaged / CountUnknown / WroteNothing) so
   the honest-claims rule the public docs are held to is pinned by a
   test rather than by message wording: UNKNOWN prints UNKNOWN and
   names --fresh; "wrote no new records" is only printed when both
   counts are known.

Verification: xerj-autoindex lib tests 1255 passed / 0 failed
(includes the two new tests:
delete_by_query_on_a_missing_index_is_already_clean_not_an_abort —
404 answered once, no retry, Ok — and
a_failed_run_the_node_cannot_count_is_unknown_not_wrote_nothing);
clippy clean; fmt clean.

Addresses the honesty half of #1173. The resume-throughput crawl
(~45 KB/s vs fresh) is the other half and stays open there — this
lands the two fixes that changed a recoverable state into a false
report, nothing about replay speed.

Files: engine/crates/xerj-autoindex/src/esclient.rs
(delete_by_query_pass), engine/crates/xerj-autoindex/src/xc.rs
(legacy_failure_verdict + the legacy failure arm).
@cla-bot cla-bot Bot added the cla-signed label Oct 6, 2026
@xerj-org
xerj-org commented Oct 6, 2026
Copy link
Copy Markdown
Owner Author

Maintainer note: merged main into this branch (931f82e) to resolve the CHANGELOG [Unreleased] collision with #1172 — both ### Fixed entries kept, #1172's first. No code changes; CI re-running on the merge.

@xerj-org
xerj-org merged commit 856ff6a into main Oct 6, 2026
22 checks passed
Ravandevil25 pushed a commit to Ravandevil25/xerj that referenced this pull request Oct 6, 2026
Version roll 1.0.0-rc.83 → 1.0.0-rc.84.

Engine window (all merged to main before this cut):
- xerj-org#1172 (fix/cluster-topology-honesty, closes xerj-org#1169): cluster endpoints
  stopped fabricating topology — health reports real configured
  membership, wait_for_nodes times out honestly (408 + timed_out),
  upper bounds enforced, _cat/nodes lists every configured member,
  _cluster/state reports the actual cluster address; multinode smoke
  skips with a recorded reason instead of passing vacuously. Also the
  xerj-org#1171 config half: peer-label≠address and cluster.port collisions
  refused at load.
- xerj-org#1175 (fix/cluster-degraded-health, closes xerj-org#1171 with xerj-org#1172): the
  transport boot outcome recorded on AppState
  (ClusterTransportStatus Disabled/Running/Degraded) before the first
  router exists; degraded boot → health yellow, number_of_nodes 1,
  additive xerj_cluster_transport:"degraded" marker; unopenable indices
  (red) still outrank the marker.
- xerj-org#1174 (fix/corpus-index-resume, xerj-org#1173 honesty half — throughput half
  stays open): xerj corpus index stopped reporting a converged corpus
  as a failed crawl; resume path re-crawls only what's missing.
- xerj-org#1167 (docs): hub counts un-staled — llms.txt:51 and the corpus-hub
  blog note said 75 live / 104 backlog from 2026-10-02; now 100/119.
- xerj-org#1163 (thomas-villani, docs): EPUB/notebook answer pages recaptured
  on the rc.83 build; \x08 man-page fact-check alias fix + self-test.

Mechanics of this commit:
- engine/Cargo.toml version → 1.0.0-rc.84; cargo update -w rolled 17
  Cargo.lock entries.
- landing/: 91 V1.0.0-RC.83 footer spans → V1.0.0-RC.84 across 54
  files (0 old spans remain); landing/index.html data-latest-tag span
  → v1.0.0-rc.84.
- CHANGELOG: [Unreleased] rolled into [1.0.0-rc.84] - 2026-10-06 with
  all five window PRs cited under Fixed.
- ROADMAP: Last-reviewed line → rc.84; open-defects shortlist
  re-verified live at cut time — eleven open (xerj-org#1091 xerj-org#1092 xerj-org#1094 xerj-org#1100
  xerj-org#1146 xerj-org#1147 xerj-org#1158 xerj-org#1122 carried forward, xerj-org#1168 xerj-org#1170 cluster
  liveness/leader-wiring, xerj-org#1173 throughput half); xerj-org#1169/xerj-org#1171 moved to
  the CHANGELOG; GA-window rc.84 paragraph added.

ES-YAML gate on the window PRs: 0 failed (452/452 re-run after one
non-reproducing co-tenant flake).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0