Repository navigation
feat(autoindex): add --walk-only so a large tree can be mapped without hashing - #2006
Merged
Merged
Conversation
…t hashing Motivation ---------- `xerj autoindex --dry-run` is the command an agent reaches for before indexing a drive it has not seen. After the walk it hashes every admitted file. A field report on rc.97 (issue #1912, mirror #1884) measured that hash on a 500 GB Windows tree at about 6 MB/s with "eta unknown", so the dry run itself was roughly a day. The walk's file counts, sizes and ignore-rule summary were already the useful part, and `autoindex map` cannot answer until a real index exists. Root cause ---------- `run_index_report_inner` always continues from the walk into `content::resolve_reporting`, which opens and xxh3-hashes every admitted byte, even when `--dry-run` will throw the digests away after printing the plan. There was no flag that stopped between those two steps. Fix --- `--walk-only` reuses that walk and `IgnoreReport::summary_lines`. It returns immediately after them, before the empty-folder short-circuit and before hashing. The same `record_*` counters now also keep a per-top-level-folder breakdown (`by_top`), so the per-folder ignore lines are not a second walk. Admitted files are grouped by the first path component of their portable relative path; bytes are metadata sizes of admitted files only. Ignored trees stay counts, not byte totals, because the deep count does not sum sizes. The flag implies `--dry-run` so the existing "index nothing" guards stay armed (pending generation, object plan-only, no publish). `--dry-run` alone is unchanged and still hashes, sniffs and prints the plan. `--walk-only` with `--watch`, an explicit `--max-minutes`/`--approve`/ `--yes`, `--label`, `map` or `status` is refused rather than silently ignored. Help and the changelog say the walk still reads ignore files and that hashing and sniffing are what is skipped. Evidence -------- `cargo +1.97.1 fmt -p xerj-autoindex -p xerj-server` `cargo +1.97.1 test --locked -p xerj-autoindex --lib walk_only_is_accepted_and_dry_run_still_hashes` — 1 passed `cargo +1.97.1 test --locked -p xerj-autoindex --lib walk_only_summarises_without_opening_file_contents` — 1 passed `cargo +1.97.1 clippy --locked -p xerj-autoindex --all-targets -- -D warnings` — exit 0 The temp-dir test watches the payload files with inotify (IN_OPEN | IN_ACCESS) and asserts the walk-plus-summary records no open or read, and that the rendered summary does not contain the payload bytes. Fixes #1912. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
3 of 8 tasks
4 of 8 tasks
# Conflicts: # engine/crates/xerj-autoindex/src/cli.rs
xerj-org
marked this pull request as ready for review
October 21, 2026 23:53
|
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Xerj Merger.
|
Owner
Author
|
Xerj Merger made the fix: I merged main, which now has #2087 and #2102. One conflict in |
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1912
What this changes, and why
xerj autoindex --dry-runwalks a tree and then hashes every admitted file. On a large drive that hash is the slow part: the field report behind #1912 (rc.97, mirror #1884) measured about 6 MB/s witheta unknownon a 500 GB Windows tree (219,916 files, 507,811 MB), so the dry run itself was roughly a day. The walk's file counts, sizes and ignore-rule summary were already the useful overview, andautoindex mapcannot answer until a real index exists.--walk-onlystops after that same walk. It prints each top-level folder's admitted file count and metadata size, what the ignore rules would drop and which rule, and the totals. It returns beforecontent::resolve_reporting, so it does not hash or sniff file contents and it indexes nothing.--dry-runalone is unchanged: it still hashes, sniffs and prints the plan.The flag implies
--dry-run, so the existing "index nothing" guards stay armed (a pending generation, an object plan-only result, no publish). Combining it with--watch, an explicit--max-minutes/--approve/--yes,--label,maporstatusis refused. Ignore files are still read, as on every walk; help says that rather than claiming file contents are never opened.Per-folder ignore lines come from the same
record_*counters the walk already updates (IgnoreReport.by_top), not a second walk.IgnoreReport::summary_linesis printed verbatim. Ignored trees are counts, not byte totals: the deep count does not sum sizes.Evidence
The temp-dir test builds a small tree (admitted files, a
*.logignored by.gitignore, and a file inside a prunedtarget/), walks it with deep counting, and summarises. On Linux it watches the payload files with inotifyIN_OPEN|IN_ACCESSand asserts the walk-plus-summary records no open or read, and that the rendered text does not contain the payload bytes.Earlier in the same session, before a help/changelog wording edit that only changed string literals, these also passed on Rust 1.97.1 with
--locked:ignore_rules::tests(18),walk::(23),cli::tests(69). The parser test above was re-run after that wording edit.Checks
cargo fmtfor the crates touched (cargo +1.97.1 fmt -p xerj-autoindex -p xerj-server). Notcargo fmt --all.cargo build --release -j 32 -p <crate>cargo test -p xerj-autoindex --libtests listed above. Not the full crate suite.xerj autoindex --helpdocuments--walk-only, and CHANGELOG.md has an Unreleased bullet.landing/llms.txtwas left alone.docs/CONTRIBUTION_REVIEW.md: product contract (--dry-runstill hashes; walk-only implies dry-run so pending/object guards stay armed), no new durable field, return is before hashing and before any publish, machine-readable--jsonuses the same summary struct, and the inotify test pins "did not open contents" rather than only a help string.Not run:
cargo fmt --all;cargo build --release -j 32 -p xerj-autoindex; the fullcargo test -p xerj-autoindexsuite (lib plus integration targets); the ES-YAML runner.Provenance
--locked. The inotify assertion is insidewalk_only_summarises_without_opening_file_contentsand that test passed on this Linux host.xerj autoindex <folder> --walk-onlyagainst a running node was not executed, so the early return inrun_index_report_inner(stdout summary, stderr note,xerj-donereasonwalk-only, exit 0) is covered by reading the call site, not by a process test. A hang, a hash progress line, or a non-zero exit on an empty or normal folder would falsify that. The non-Linux arm of the content test does not install an inotify watch; only this Linux run proved the files were not opened.