Repository navigation
fix(autoindex): fail fast when the node disk is at flood stage - #2087
Merged
Merged
Conversation
Motivation ---------- A graph-path `xerj autoindex` re-run resumes the journaled plan. Files added to the folder after that plan was frozen were stored as junk and the run finished `reason=completed-with-junk` (exit 3). That is the everyday "unusable input, still success" signal, so the new files never landed and the index looked complete. `--fresh` rebuilt the plan and indexed them. Field report on v1.0.0-rc.97: seven post-plan files, exit 3, `--fresh` then exit 0. The graph path cannot safely re-cluster a frozen plan. Dataset identity and graph anchors are committed with it, and `reconcile_plan` is the `--no-graph` route: it projects additions onto frozen schemas and fails the whole run closed when a file needs a new dataset. Wiring that into the graph path would turn "new shape" from a recorded skip into an abort. Resume therefore keeps the frozen plan and reports the skip as its own state. Fix --- Post-plan files stay in the plan so a later deletion can still sweep their catalog row, with status `stale-plan` (legacy journals that used status `skipped` plus the "not in the frozen resume plan" reason are promoted once). They are excluded from `files_junk`. `skipped_appeared` counts them on every resume while they remain on disk. The terminal reason is `stale-resume-plan` (exit 3, so callers that branch on 0/3 still see a finished-but-incomplete run) and the summary names `--fresh`. Real junk and the alias-sweep failure are unchanged when no such files are present. Evidence -------- New test `a_resume_whose_plan_predates_new_files_reports_a_stale_plan_not_junk` failed before the fix with `files_junk` 2 (both added files) and `skipped_appeared` 2. After the fix, `cargo +1.97.1 test -p xerj-autoindex --lib -- --test-threads=4`: 1524 passed, 0 failed, 2 ignored. `cargo +1.97.1 fmt --all -- --check` clean. `cargo +1.97.1 clippy -p xerj-autoindex --all-targets -- -D warnings` clean. `cargo +1.97.1 build --release -j 4 -p xerj-autoindex` finished in 5m 14s. ES-YAML not run: no engine or wire-protocol change. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Motivation ---------- On v1.0.0-rc.97 a data volume at 98% used sat over the default 95% disk flood-stage watermark, so the node refused every write. `xerj autoindex` still walked, hashed, scanned and prepared for about ten minutes and only then learned the first bulk was blocked. Raising `limits.disk_flood_stage_percent` to 99 unblocked the node. Issue #823 already made the write-block message name the free space; this is the fail-fast half of that report (#2077). Root cause ---------- The watermark is the node's data volume, measured by the engine governor against the filesystem that holds the data directory. Autoindex is an HTTP client and never asked. The block is enforced only on the write path (`read_only_allow_delete`, HTTP 429), which runs after prepare. Fix --- Immediately after the readiness ping, and before catalog search, journal preflight, object download or walk, autoindex GETs `/_cat/allocation?format=json`. The JSON now also carries the effective gate the governor already uses: `disk.flood_stage`, `disk.threshold_enabled`, and `disk.flood_blocked`. The text cat table is unchanged. Used percent uses the same u128 formula as the governor. The run stops when the node reports the volume at or over that threshold (inclusive, matching the latch) or when the write block is still engaged under the 1% hysteresis. The error names free space, used percent, the threshold, `limits.disk_flood_stage_percent` (0 disables it), and the live `PUT /_cluster/settings` override. A missing `disk.flood_stage` is not guessed at 95: an older XERJ, Elasticsearch, a disabled watermark, the 99 workaround, and the HTTP test doubles all continue. A probe transport error or a non-2xx response does not abort a healthy run. Evidence -------- The new run-level test points autoindex at a file, so a walk that starts fails with "is not a directory". With the probe call removed, that test failed before any flood text: thread 'flood::tests::autoindex_refuses_before_walk_only_when_the_node_reports_flood_stage' panicked at flood.rs:232: assertion failed: message.contains("limits.disk_flood_stage_percent"): /tmp/.tmptsPUha/not-a-folder is not a directory Restored, the same test passes: the flooded node (98% used, 2.0gb free, threshold 95) is refused with the override text, the only requests are `GET /` and `GET /_cat/allocation?format=json`, and no `.autoindex.lock` is created. A healthy 10% response reaches the walk. `cargo +1.97.1 test --locked -p xerj-autoindex --lib -- --test-threads=8`: 1527 passed, 0 failed, 2 ignored. `cargo +1.97.1 test --locked -p xerj-engine --lib disk_flood`: 3 passed. `disk_threshold_disabled_releases_and_suppresses_block`: 1 passed. `cargo +1.97.1 test --locked -p xerj-api --test flood_stage_allocation`: 1 passed (live node JSON `disk.flood_stage` is "95"; the text table does not contain "flood"). `cargo +1.97.1 fmt -p xerj-autoindex -p xerj-engine -p xerj-api -- --check` clean. Clippy `-D warnings` clean on xerj-autoindex `--all-targets`, xerj-engine `--lib`, and xerj-api `--lib --test flood_stage_allocation`. Not run: `cargo fmt --all`, scoped release builds, the full engine and api suites, and the ES-YAML runner. No YAML case references `/_cat/allocation`; the text line is unchanged and the new JSON keys are additive. CI runs the suite. Fixes #2077. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
A dry run writes nothing, and its plan is what a user short on disk needs.
4 of 8 tasks
# Conflicts: # CHANGELOG.md
xerj-org
marked this pull request as ready for review
October 21, 2026 18:23
|
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Xerj Merger.
|
Owner
Author
|
Xerj Merger made the fix: added commit 97f9181, so |
xerj-org
pushed a commit
that referenced
this pull request
Oct 21, 2026
autoindex hashed the folder before anyone could see whether the index would fit. A digit-leading prefix was rejected only after that hash (#2057), exit 3 did not name the skipped files (#2056), and the same folder under a second prefix was silent (#2078). The free-disk budget reuses the one pre-walk GET /_cat/allocation from the flood-stage probe. It does not issue a second allocation request. Already-over-flood still stops before the walk. Under the watermark the budget is free bytes above the reported flood reserve; the threshold is not guessed when the node omits it. --max-index-size can lower that budget. Per-kind ratios are dataset-index store sizes measured 2026-10-21 with xerj v1.0.0-rc.97 (default lexical feature-hash; the neural embedder was not enabled). A kind with no row stays unmeasured and does not refuse the run. When the default plan does not fit, stdout is a JSON plan list and the process exits 4 until --plan or --yes names a plan that fits. MCP xerj_index_budget returns the same estimate and does not index. Corpus index still drops CSV/HDF/simulation files before the estimate when skip_corpus_data is set. The budget then sees the files that will actually be hashed. Verified after rebasing onto origin/main (which already has #2086 and - test -p xerj-autoindex --lib -- --test-threads=8: 1545 passed, 0 failed, 2 ignored, finished in 34.26s - test -p xerj-mcp: lib 102 passed; published schema drift 4 passed; hub_tools 1 passed; search_exact_phrase 1 passed - test -p xerj-api --test flood_stage_allocation: 1 passed - clippy -p xerj-autoindex -p xerj-mcp -p xerj-console-api --all-targets -- -D warnings: exit 0 - check -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api: exit 0 - fmt -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api Not run: ES-YAML. This commit does not change the engine wire. The allocation JSON lives in the parent commit, which is the unmerged flood-stage probe (#2087). A release rebuild of this tree. Workspace-wide cargo test. xerj-console-api tests. cargo fmt --all.
xerj-org
added a commit
that referenced
this pull request
Oct 21, 2026
* feat(autoindex): budget the index before hashing autoindex hashed the folder before anyone could see whether the index would fit. A digit-leading prefix was rejected only after that hash (#2057), exit 3 did not name the skipped files (#2056), and the same folder under a second prefix was silent (#2078). The free-disk budget reuses the one pre-walk GET /_cat/allocation from the flood-stage probe. It does not issue a second allocation request. Already-over-flood still stops before the walk. Under the watermark the budget is free bytes above the reported flood reserve; the threshold is not guessed when the node omits it. --max-index-size can lower that budget. Per-kind ratios are dataset-index store sizes measured 2026-10-21 with xerj v1.0.0-rc.97 (default lexical feature-hash; the neural embedder was not enabled). A kind with no row stays unmeasured and does not refuse the run. When the default plan does not fit, stdout is a JSON plan list and the process exits 4 until --plan or --yes names a plan that fits. MCP xerj_index_budget returns the same estimate and does not index. Corpus index still drops CSV/HDF/simulation files before the estimate when skip_corpus_data is set. The budget then sees the files that will actually be hashed. Verified after rebasing onto origin/main (which already has #2086 and - test -p xerj-autoindex --lib -- --test-threads=8: 1545 passed, 0 failed, 2 ignored, finished in 34.26s - test -p xerj-mcp: lib 102 passed; published schema drift 4 passed; hub_tools 1 passed; search_exact_phrase 1 passed - test -p xerj-api --test flood_stage_allocation: 1 passed - clippy -p xerj-autoindex -p xerj-mcp -p xerj-console-api --all-targets -- -D warnings: exit 0 - check -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api: exit 0 - fmt -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api Not run: ES-YAML. This commit does not change the engine wire. The allocation JSON lives in the parent commit, which is the unmerged flood-stage probe (#2087). A release rebuild of this tree. Workspace-wide cargo test. xerj-console-api tests. cargo fmt --all. * autoindex budget: calibrate code and prose on real repository content The synthetic distinct-token trees under-predicted real .rs and .md indices by about 3.5x (97 files / 5.1 MB of code: model 4.3 MB, measured 15.3 MB). Replace the code and prose points with real-content runs on rc.97; data, pdf and unindexed points stay synthetic. * autoindex budget: a named --plan is the override when the estimate is wrong Naming a plan with --plan (or typing it at the prompt) now runs it even when the estimate says it does not fit, with a warning in the continue note. The estimate errs high after the real-content calibration, and the flood-stage guard still stops writes. The implicit default and --yes keep their behaviour. Also fixes the stray ' . ' in the default-plan line. * autoindex: don't tell agents to DELETE /<prefix>-* for a duplicate folder The same-folder-under-another-prefix note (#2078) suggested "DELETE /<other>-* to drop the other copy". A prefix can hold other folders (the unit test's own fixture has ax indexing /data/proj and /data/old), and the node accepts that wildcard delete, so following the hint destroys indices this run never touched. That is the failure in the #2114 field report. The note now names --prefix <other> and says both copies are kept. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Xerj Merger <merger@xerj.org>
This was referenced Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2077.
What this changes, and why
On v1.0.0-rc.97 a data volume at 98% used was already over the default 95% disk flood-stage watermark, so the node refused every write.
xerj autoindexstill walked, hashed, scanned and prepared for about ten minutes, and only then learned the first bulk was blocked. Raisinglimits.disk_flood_stage_percentto 99 unblocked the node. Issue #823 already made the write-block message name the free space; this is the fail-fast half of that report.The watermark is the node's data volume (the governor's
statvfsof the data directory), and autoindex is an HTTP client that may not be on that machine. The block is enforced only on the write path, which runs after prepare. Autoindex never asked.Immediately after the readiness ping, and before catalog search, journal preflight, object download or walk, autoindex reads
GET /_cat/allocation?format=json. That JSON now also carries the gate the governor already uses:disk.flood_stage,disk.threshold_enabled, anddisk.flood_blocked. The text cat table is unchanged.The run stops when the node reports the volume at or over that threshold (inclusive, matching the latch) or when the write block is still engaged under the 1% hysteresis. The error names the free space, the used percent, the threshold,
limits.disk_flood_stage_percent(0 disables it), and the livePUT /_cluster/settingsoverride. A missingdisk.flood_stageis not guessed at 95, so an older XERJ, Elasticsearch, a disabled watermark, the 99 workaround, and the HTTP test doubles all continue. A probe transport error or a non-2xx response does not abort a healthy run.Evidence
The new run-level test points autoindex at a file, so a walk that starts fails with
is not a directory. With the probe call removed, that test failed before any flood text:Restored, the same test passes: a flooded node (98% used, 2.0gb free, threshold 95) is refused with the override text, the only requests are
GET /andGET /_cat/allocation?format=json, and no.autoindex.lockis created. A healthy 10% response reaches the walk.The API test boots a node with the default config.
GET /_cat/allocation?format=jsonreturnsdisk.flood_stage"95"; the plain text table contains the node name and does not containflood.cargo +1.97.1 fmt -p xerj-autoindex -p xerj-engine -p xerj-api -- --checkwas clean. Clippy-D warningswas clean onxerj-autoindex --all-targets,xerj-engine --lib, andxerj-api --lib --test flood_stage_allocation. Toolchain: rustc 1.97.1.Checks
cargo fmt --all(CI runsrustfmt --checkandclippy -D warnings)cargo build --release -j 32 -p <crate>cargo test -p <crate>passes (cargo checkis not sufficient — test code is interleaved with production code)cargo run --release -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml), or: not applicable because this change is docs/landing-onlyengine/tests/es-compat-yaml/yaml/docs/CONTRIBUTION_REVIEW.mddocs/CONTRIBUTION_REVIEW.md. Record a negative control that exits non-zero with no artifact extracted, installed, published, or served. Keep verification and its action in one block; rerun the control with the failing exit discarded and with the next documented block appended, and require that neither reaches an artifact or unsafe/privileged path. List tested variants and unsupported variants separately, and verify every publication copy. Otherwise mark this itemN/Awith a reason.Not run:
cargo fmt --all(fmt--checkwas scoped to the three touched crates). Scoped release builds. The fullxerj-engineandxerj-apisuites (governor disk tests and the new allocation test were run; the autoindex lib suite was run in full). The ES-YAML runner. No YAML case references/_cat/allocation; the text line is unchanged and the new JSON keys are additive and XERJ-specific, so no new YAML case was added. CI runs rustfmt, clippy, and ES-YAML. Operational-document audit is N/A: no recipe or operational doc changed. Failure atomicity: the probe is beforeJournal::preflight, so a refusal creates no state dir and no lock (the run-level test asserts no.autoindex.lock).cargo test -p xerj-autoindexabove is the lib suite (1527 passed). Integration tests underengine/crates/xerj-autoindex/tests/were not run separately.Provenance
/tmp/.tmptsPUha/not-a-folder is not a directory); the same test passing after the call was restored; autoindex lib 1527 passed / 0 failed / 2 ignored; enginedisk_flood3 passed anddisk_threshold_disabled_releases_and_suppresses_block1 passed; APIflood_stage_allocation1 passed; scoped fmt--checkand clippy-D warningsclean on the invocations named above.cargo fmt --alland the scoped release builds compile (falsified by rustfmt or a release-only error); a node whose sampler has not yet latcheddisk.flood_blockedstill fails fast fromdisk.percentwhen used is already at the threshold (covered by the parser unit test, not by a live delayed-sampler run). A node that omitsdisk.flood_stageis left alone on purpose.