Repository navigation
fix(autoindex): open section overlap on a word - #1700
Merged
Merged
Conversation
A retrieval section repeats 200 bytes of the previous paragraph so a match on the boundary stays in both records. That tail was a raw byte slice, so a markdown section could begin in the middle of a token. On tokio, xerj search for "cargo install --locked cargo-fuzz" printed docs/contributing/pull-requests.md:205 starting "le to use in tests" — the overlap had landed inside "available". The overlap tail now walks back at most 64 bytes to the previous word, or forward to the next word when the token is longer than that window. A run of text with no whitespace is unchanged. Verified: cargo +1.97.1 test --locked -p xerj-autoindex --lib section_tests (17 passed). cargo +1.97.1 clippy --locked -p xerj-autoindex --all-targets -- -D warnings (exit 0). Rebased onto main after #1730 reverted #1623, which had defined WORD_SNAP_BYTES; the constant now lives with this fix, and the hard-split test that came with #1623 is dropped.
xerj-org
force-pushed
the
cursor/section-overlap-word-3c2c
branch
from
October 20, 2026 06:52
52116b9 to
bec73b6
Compare
This was referenced Oct 20, 2026
Owner
Author
|
Xerj Merger made the fix: reviewed. The 200-byte paragraph-overlap tail now starts on a word: it walks back up to 64 bytes to whitespace, or forward past a longer token. Text with no whitespace is unchanged, and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
A retrieval section repeats 200 bytes of the previous paragraph so a match that sits on the boundary is still in both records. That tail was a raw byte slice snapped only to a char boundary. The hard-split path already cuts on whitespace; the paragraph-overlap path did not.
On a full-history tokio index (
xerj autoindex /tmp/tokio --prefix tokio, 861 files, 11,246 records),xerj search --prefix tokio "cargo install --locked cargo-fuzz"printeddocs/contributing/pull-requests.md:205starting withle to use in tests. The stored section body began inside the wordavailable. The same shape showed up on the previous section of that file (he Cargo features).The overlap tail now walks back at most 64 bytes to the previous word, or forward to the next word when the token is longer than that window. Text with no whitespace is unchanged, so an unbroken CJK or
zrun still splits on a char boundary.Evidence
Before: the tokio search above (binary
xerj v1.0.0-rc.96at/tmp/xerj-main, nodehttp://127.0.0.1:9210) returned a passage whose_passage.textstartedle to usewithstart_offset0.After:
paragraph_overlap_opens_on_a_wordfails on the oldtail(the sliceutilities available…opened atilable) and passes after the snap.section_tests(17) still pass, including the pre-#239 byte-for-byte oracle, which calls the sametail.Fresh index of the same tree, for context, not as a before/after of this fix (the fix is in the splitter, and the indexed node was the previous binary):
xerj-done ok=true exit=0 reason=completed wall=7.0s files=861 records=11246 datasets=1.Checks
cargo fmt --all(CI runsrustfmt --checkandclippy -D warnings)cargo build --release -j 32 -p <crate>cargo test -p <crate>passes (cargo checkis not sufficient — test code is interleaved with production code)cargo run --release -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml), or: not applicable because this change is docs/landing-onlyengine/tests/es-compat-yaml/yaml/docs/CONTRIBUTION_REVIEW.mddocs/CONTRIBUTION_REVIEW.md. Record a negative control that exits non-zero with no artifact extracted, installed, published, or served. Keep verification and its action in one block; rerun the control with the failing exit discarded and with the next documented block appended, and require that neither reaches an artifact or unsafe/privileged path. List tested variants and unsupported variants separately, and verify every publication copy. Otherwise mark this itemN/Awith a reason.Not run:
cargo fmt --all(rustfmt--checkon the one edited file, edition 2021, toolchain 1.97.1). Scoped release build not run; tests and clippy used the dev profile. ES-YAML not run: the sectioner does not change the wire. No YAML case: not an ES request-shape change. Docs not updated: the overlap length is not a documented contract. Operational-recipe audit N/A: no runnable recipe or operational doc changed.Provenance
section_tests17 passed; clippy-D warningsexit 0; fresh autoindexwall=7.0s files=861 records=11246available to userather thanle to use. Falsified ifparagraph_overlap_opens_on_a_wordfails or a spaced paragraph section still begins mid-token.