feat(autoindex): budget the index before hashing by xerj-org · Pull Request #2102 · xerj-org/xerj

xerj-org · 2026-10-21T17:34:46Z

What this changes, and why

xerj autoindex hashed and embedded a folder before anyone could see whether the index would fit. A digit-leading --prefix was rejected only after that hash (#2057). Exit 3 did not name the skipped files, and a PDF reuse-fallback line did not say the PDF was still indexed (#2056). The same folder under a second prefix was silent (#2078).

After the walk, and before hashing, autoindex now estimates the index from measured per-kind store-size ratios and compares that with the free space left under the node's flood-stage watermark. The budget reads the allocation row already fetched by the pre-walk probe. It does not issue a second GET /_cat/allocation. A node that omits disk.flood_stage does not get a guessed 95% margin. --max-index-size can lower the budget. When the default plan does not fit, stdout is a JSON plan list and the process exits 4 until --plan or --yes names a plan that fits. An unmeasured kind does not refuse the run. MCP xerj_index_budget returns the same estimate and does not index.

Fixes #2056. Fixes #2057. Fixes #2078.

This branch is stacked on the unmerged flood-stage probe. Commit ce41dea5 is #2087 rebased onto current main. The budget work is only ce41dea5..HEAD. Do not merge this draft until #2087 is on main and that parent commit is dropped:

git fetch origin main
git rebase --onto origin/main ce41dea5

#2086 (stale resume plan) and #2085 (MCP node-down hint) are already on main. This branch does not redo them. #2006 (--walk-only) was not folded in; the existing walk is the counting pass.

Evidence

The estimator's ratios are dataset-index store.size values measured 2026-10-21 with /workspace/engine/target/release/xerj reporting xerj v1.0.0-rc.97, started --insecure (lexical feature-hash; --embed-mode neural was not passed). Host: Linux 6.12.94+ x86_64, 4 cores, 16387280 KiB RAM. Harness (not committed): python3 /tmp/calibrate_budget.py. Recorded stdout: /tmp/xerj-cal3-results.jsonl. autoindex-catalog store size was excluded because it moved by tens of KB in both directions between runs. The two PDF points are about 1 KB and 10 KB and have a negative per-byte slope, so the fitter uses the steeper measured ratio. A kind with no row stays unmeasured.

After rebasing onto origin/main (cargo +1.97.1 from engine/, --offline, -j 4):

cargo test -p xerj-autoindex --lib -- --test-threads=8
# test result: ok. 1545 passed; 0 failed; 2 ignored; finished in 34.26s
# includes budget unit tests and
# incremental_reconcile_http_tests::delete_gate::a_budget_too_small_for_the_default_plan_accepts_a_smaller_one

cargo test -p xerj-mcp
# lib 102 passed; published schema drift 4 passed;
# hub_tools 1 passed; search_exact_phrase 1 passed

cargo test -p xerj-api --test flood_stage_allocation
# 1 passed (this test belongs to the stacked #2087 commit)

cargo clippy -p xerj-autoindex -p xerj-mcp -p xerj-console-api --all-targets -- -D warnings
# Finished dev profile, exit 0

cargo check -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api
# Finished dev profile, exit 0

cargo fmt -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api

The end-to-end test builds a tree with main.rs and a 2000-row csv, asks for an estimate with no_semantic and an unbounded --max-index-size, caps the budget at the code-and-docs estimate, and checks that a run with no --plan returns exit 4 with no data bulk. A second run with --plan code-and-docs returns 0 and indexes main.rs and not big.csv.

Checks

cargo fmt --all (CI runs rustfmt --check and clippy -D warnings)
Scoped release build of the crates touched: cargo build --release -j 32 -p <crate>
cargo test -p <crate> passes (cargo check is not sufficient — test code is interleaved with production code)
ES-YAML conformance suite at 0 failed (cargo run --release -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml), or: not applicable because this change is docs/landing-only
New ES-compatible behaviour has a matching YAML case under engine/tests/es-compat-yaml/yaml/
Docs updated if user-visible behaviour changed
For non-trivial changes: the applicable audit in docs/CONTRIBUTION_REVIEW.md
For non-trivial changes: if this PR changes an operational document or runnable recipe, complete the operational audit in docs/CONTRIBUTION_REVIEW.md. Record a negative control that exits non-zero with no artifact extracted, installed, published, or served. Keep verification and its action in one block; rerun the control with the failing exit discarded and with the next documented block appended, and require that neither reaches an artifact or unsafe/privileged path. List tested variants and unsupported variants separately, and verify every publication copy. Otherwise mark this item N/A with a reason.

Not run: cargo fmt --all (fmt was cargo +1.97.1 fmt -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api). Scoped release build of this tree (the calibration binary was an earlier release build of xerj v1.0.0-rc.97, not a rebuild after the final client). Workspace-wide cargo test. xerj-console-api tests (clippy compiled the crate). ES-YAML. The budget commit does not change the engine wire. The allocation JSON is the parent commit, which is #2087; that commit's own message says the ES-YAML runner was not run there either, and no YAML case references /_cat/allocation. No new YAML case in this PR.

landing/llms.txt and landing/llms-full.txt were not edited. Open PRs #1909 and #1925 already edit landing/llms.txt, and only one open PR may edit that file. The published tool count there is still short by xerj_index_budget. The generated schema landing/docs/agents/schemas/mcp-tools.json was updated, and published_schema_drift passed (19 tools with hub, 16 without). CLI help and CHANGELOG.md were updated.

Operational-recipe audit: N/A. This change adds CLI flags and help text, not a new runnable recipe, install path, or publication copy.

Contribution review

Ancestry: merge-base is current origin/main. Unique commits are ce41dea5 (fix(autoindex): fail fast when the node disk is at flood stage #2087, temporary) and the budget commit. fix(autoindex): a stale resume plan is not junk (#2076) #2086 and fix(mcp): say how to start a node when XERJ_URL is down #2085 are on main and were not replayed. The walk keeps skip_corpus_data (corpus index, corpus index: skip bulky data files (CSV/HDF/test output) by default for reference corpora #1985) and the budget estimates the files left after that filter.
Catalog mapping is unchanged. skipped is added to the run document only when junk is non-empty, and the human summary prints at most 20 skipped: lines after the one-line ok: headline.
A plan whose keep list is empty returns exit 4 and does not continue into a deletion. Accepted plan drops are not recorded as junk.
A reduced plan on a folder that is already fully indexed can look like deletions to the existing delta check. The delta checker was not changed.
--max-minutes 0 still only disables the time gate. The disk budget stays on.
--yes when the default plan does not fit accepts the recommended plan that fits. If none fit, the run still exits 4.
A TTY-chosen plan is not remembered by --watch: later passes clone the original config. --plan and --yes persist because they are on the config. Local --watch allows the new flags; object-storage --watch and map/status refuse them, same as --max-minutes.
The overlap warning is inside the non-continuing-journal block, so a resume of an already- 8000 indexed second prefix does not warn again, and the flood probe's two-GET tests stay exact.
Brain index config leaves max_index_bytes and budget_plan unset. A brain folder that does not fit exits 4.

Provenance

Written by: AI agent (Cursor Grok 4.7), in a cloud agent session. The account that opens this pull request is the record of accountability. No Co-authored-by trailer on the budget commit.
Verified (commands run, output observed): the cargo commands in Evidence, on the rebased tree. Calibration store sizes are the points in engine/crates/xerj-autoindex/src/budget.rs (MEASURED), taken from the release binary and harness named there. xerj --version on that binary printed xerj v1.0.0-rc.97.
Assumed (not tested, and what would falsify it): that these ratios hold on another machine, another version, or with --embed-mode neural (the calibration note says they do not; a neural run was not done). That a reduced plan on an already-indexed folder surfaces as a deletion delta (read from the existing checker, not exercised). That --watch forgets a TTY plan (follows from later passes cloning the original config; no watch run was done). ES-YAML stays at 0 failed with the stacked allocation JSON (not run here).

…#2103) The budget work in #2102 measured dataset-index store size with a release binary. This is the required one-file report for that run. Co-authored-by: Cursor Agent <cursoragent@cursor.com>

main serves eighteen tools now and xerj-org#2102 adds a nineteenth; the tested row is about the client registering the server, not the count.

* docs(readme): list 26 supported agent harnesses (#1991) Motivation: xerj init writes configs for three clients and prints setup lines for three more, while comparable tools publish a table of 20 or more, and readers use that list to decide whether a tool fits their setup. The gap is documentation, not code. What changed: a Supported agents section in the README, one row per client with its config file, config key, a note and a tested mark, the non-JSON shapes as snippets, and the same 26-row list on the docs CLI page. Verification: xerj v1.0.0-rc.97 on Windows answered initialize and tools/list over stdio, and OpenCode 2.1.2 registered the binary and reported connected after opencode mcp add wrote the project config. The other 25 rows are transcribed from each client's own documentation and are marked untested. Aider, T3 Code and Grok Bot are listed as excluded, with the reason. Signed-off-by: kclamano <klamano23@gmail.com> * docs: don't pin the MCP tool count in the supported-agents text main serves eighteen tools now and #2102 adds a nineteenth; the tested row is about the client registering the server, not the count. --------- Signed-off-by: kclamano <klamano23@gmail.com> Co-authored-by: Xerj Merger <merger@xerj.org>

autoindex hashed the folder before anyone could see whether the index would fit. A digit-leading prefix was rejected only after that hash (#2057), exit 3 did not name the skipped files (#2056), and the same folder under a second prefix was silent (#2078). The free-disk budget reuses the one pre-walk GET /_cat/allocation from the flood-stage probe. It does not issue a second allocation request. Already-over-flood still stops before the walk. Under the watermark the budget is free bytes above the reported flood reserve; the threshold is not guessed when the node omits it. --max-index-size can lower that budget. Per-kind ratios are dataset-index store sizes measured 2026-10-21 with xerj v1.0.0-rc.97 (default lexical feature-hash; the neural embedder was not enabled). A kind with no row stays unmeasured and does not refuse the run. When the default plan does not fit, stdout is a JSON plan list and the process exits 4 until --plan or --yes names a plan that fits. MCP xerj_index_budget returns the same estimate and does not index. Corpus index still drops CSV/HDF/simulation files before the estimate when skip_corpus_data is set. The budget then sees the files that will actually be hashed. Verified after rebasing onto origin/main (which already has #2086 and - test -p xerj-autoindex --lib -- --test-threads=8: 1545 passed, 0 failed, 2 ignored, finished in 34.26s - test -p xerj-mcp: lib 102 passed; published schema drift 4 passed; hub_tools 1 passed; search_exact_phrase 1 passed - test -p xerj-api --test flood_stage_allocation: 1 passed - clippy -p xerj-autoindex -p xerj-mcp -p xerj-console-api --all-targets -- -D warnings: exit 0 - check -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api: exit 0 - fmt -p xerj-autoindex -p xerj-mcp -p xerj-server -p xerj-console-api Not run: ES-YAML. This commit does not change the engine wire. The allocation JSON lives in the parent commit, which is the unmerged flood-stage probe (#2087). A release rebuild of this tree. Workspace-wide cargo test. xerj-console-api tests. cargo fmt --all.

The synthetic distinct-token trees under-predicted real .rs and .md indices by about 3.5x (97 files / 5.1 MB of code: model 4.3 MB, measured 15.3 MB). Replace the code and prose points with real-content runs on rc.97; data, pdf and unindexed points stay synthetic.

… wrong Naming a plan with --plan (or typing it at the prompt) now runs it even when the estimate says it does not 8000 fit, with a warning in the continue note. The estimate errs high after the real-content calibration, and the flood-stage guard still stops writes. The implicit default and --yes keep their behaviour. Also fixes the stray ' . ' in the default-plan line.

…lder The same-folder-under-another-prefix note (#2078) suggested "DELETE /<other>-* to drop the other copy". A prefix can hold other folders (the unit test's own fixture has ax indexing /data/proj and /data/old), and the node accepts that wildcard delete, so following the hint destroys indices this run never touched. That is the failure in the #2114 field report. The note now names --prefix <other> and says both copies are kept.

cla-bot · 2026-10-21T23:04:30Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Xerj Merger.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

xerj-org · 2026-10-21T23:04:46Z

Xerj Merger made the fix: I rebased onto main (--onto origin/main ce41dea5, now on top of #2087, #2108 and #2109), reviewed the change and added three commits.

Rebase conflict (lib.rs). #2087 now lets --dry-run through at the flood stage. I kept that and the PR's single probe: flood::probe runs once, its refusal applies only when !cfg.dry_run, and disk.volume still feeds the budget gate.

b9a64d0: recalibrated code and prose on real content. The MEASURED code and prose points came from small synthetic distinct-token trees, and they under-predict real repositories by about 3.5x. Measured with rc.97 on copies of this repository:

Code, semantic: 97 .rs files / 5.10 MB. The old model predicted 4.3 MB; the measured index was 15.3 MB.
Prose, semantic: 169 .md files / 2.97 MB. The old model predicted 4.65 MB; the measured index was 15.9 MB.

The new points are 5/97 .rs files and 24/169 .md files, at the pre-merge peak store.size (the size the disk actually has to hold) with timed runs. With these points the fit's per-file intercept is negative, so the fitter uses the steeper measured ratio, which errs high. Data, pdf and unindexed stay synthetic, and CALIBRATION_NOTE says so.

38a573a: a named --plan is the override. Before this, a plan named with --plan was refused whenever the estimate said it didn't fit, so a wrong estimate had no way around it (--max-index-size can only lower the budget). A named plan, whether passed with --plan or typed at the prompt, now runs with a warning. The flood-stage guard still stops writes. The implicit default (exit 4 plus plan JSON) and --yes (the recommended plan that fits) are unchanged. The help text and CHANGELOG are updated, there's a new unit test, and the stray . in the default-plan line is fixed.

dcdcd5b: removed a destructive hint. The new same-folder-under-another-prefix note (#2078) told agents to DELETE /<other>-* to drop the other copy. A prefix can hold other folders: the unit test's own fixture has ax indexing both /data/proj and /data/old. The node also accepts that wildcard delete, so following the hint would destroy indices this run never touched. That's the failure mode in the #2114 field report. The note now points to --prefix <other> and says both copies are kept.

Other review points, unchanged:

Exit 4 carries the plan JSON with decision_required, plans[], recommended and accept.
The TTY prompt asks once per line, up to 3 tries.
A digit-leading --prefix is rejected through IndexName::validate.
--yes now means "take the recommended reduced plan" when the default doesn't fit. That's a behaviour change for --yes callers, and it's stated in the note.

Local CI:

Full workspace run on the rebased tree with the recalibration: cargo fmt --check OK, clippy --workspace --all-targets -D warnings OK, cargo test --profile ci-test --workspace -- --test-threads=2: 5771 passed, 0 failed.
After the last two commits: fmt and workspace clippy OK, and cargo test --profile ci-test -p xerj-autoindex -p xerj-server -p xerj-mcp: 1844 passed, 0 failed.
The final rebase over docs: sign the CLA by adding AK-Lmn to .contributors #2108/docs: list 26 supported agent harnesses (#1991) #2109 brought in docs only.

The PR touches no workflow files. Squash-merging.

#2102 added xerj_index_budget, so main serves nineteen MCP tools (sixteen with no hub key). llms.txt and llms-full.txt said eighteen/fifteen and README/cli.html said seventeen/fourteen. Name the tool and fix the counts. Co-authored-by: Xerj Merger <merger@xerj.org>

cla-bot Bot added the cla-signed label Oct 21, 2026

xerj-org mentioned this pull request Oct 21, 2026

docs(feedback): autoindex store-size calibration #2103

Merged

3 of 8 tasks

xerj-org mentioned this pull request Oct 21, 2026

fix(autoindex): fail fast when the node disk is at flood stage #2087

Merged

4 of 8 tasks

cursoragent and others added 4 commits October 21, 2026 16:04

xerj-org force-pushed the cursor/autoindex-budget-bead branch from 300cc12 to dcdcd5b Compare October 21, 2026 23:04

cla-bot Bot removed the cla-signed label Oct 21, 2026

xerj-org marked this pull request as ready for review October 21, 2026 23:04

xerj-org merged commit c0091f3 into main Oct 21, 2026
2 of 4 checks passed

xerj-org mentioned this pull request Oct 21, 2026

docs: count xerj_index_budget in the MCP tool lists #2118

Merged

Navigation Menu

Repository navigation

feat(autoindex): budget the index before hashing - #2102

feat(autoindex): budget the index before hashing#2102
xerj-org merged 4 commits into
mainfrom
cursor/autoindex-budget-bead

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

What this changes, and why

Evidence

Checks

Contribution review

Provenance

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants