Repository navigation
fix(autoindex): don't emit a section that the next one repeats whole - #1160
Merged
xerj-org merged 1 commit intoOct 5, 2026
Merged
Conversation
`for_each_section` packs paragraphs into ~2 KB sections and starts each new section with the last SECTION_OVERLAP (200) bytes of the previous one. When a short paragraph (typically a heading) was followed by one too big to join it, the short one was flushed as a section on its own, and the next section then began with ALL of it as overlap. The record carried no text of its own: a duplicate that competes for a result slot with the real section. Seen on a Gutenberg EPUB as "CHAPTER 26. Knights and Squires." alone, the next section repeating it; the same shape occurs in every family that sections through this function (PDF pages, DOCX, PPTX slides, txt prose, HTML, EPUB). Fix (extract/mod.rs): a section is flushed only if it is longer than SECTION_OVERLAP, i.e. only if its overlap tail is not the whole of it. A shorter one stays in `cur` and leads the next section, which is then at most 2 * SECTION_CHARS + 202 bytes before the existing hard-split path applies, as for any long paragraph. Nothing is dropped: the text is in the next section as before, once instead of twice. The byte-for-byte oracle test (`legacy_split_sections`, kept from the xerj-org#239 cost rewrite) gets the same one-line rule, so it still checks that the streaming and the simple implementations agree; its case list adds a short paragraph before long ones at SECTION_OVERLAP - 1, = and + 1. Test: a_section_the_next_would_repeat_whole_is_not_emitted, watched failing before the fix (first section was just the heading) and passing after. `cargo test -p xerj-autoindex --lib`: 1231 passed, 2 ignored; clippy -D warnings clean (rust:latest Docker). Files: engine/crates/xerj-autoindex/src/extract/mod.rs
Merged
4 tasks done
Ravandevil25
pushed a commit
to Ravandevil25/xerj
that referenced
this pull request
Oct 6, 2026
- engine 1.0.0-rc.82 -> 1.0.0-rc.83 (Cargo.toml + cargo update -w, 17 lock entries) - landing: data-latest-tag + all 91 footer brand spans -> RC.83 (the xerj-org#1155 guard now enforces this at cut) - CHANGELOG: rc.83 section — xerj-org#1155 (version-roll guard + xxhash 0.8.19), xerj-org#1153 (xerj-org#1145/xerj-org#1148), xerj-org#1156 (xerj-org#1136), xerj-org#1154, xerj-org#1159 (EPUB), xerj-org#1160 (redundant-overlap section), xerj-org#1161 via xerj-org#1162 (notebooks + data-file naming) - ROADMAP: cut-roll — shortlist re-verified live: 8 open (xerj-org#1091 xerj-org#1092 xerj-org#1094 xerj-org#1100 xerj-org#1146 xerj-org#1147 xerj-org#1158 xerj-org#1122); xerj-org#1136/xerj-org#1145/xerj-org#1148 closed in-window, de-linked here and moved to CHANGELOG; rc.83 paragraph in the GA window record
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
for_each_section(extract/mod.rs) packs paragraphs into sections of about 2 KB, and starts each section with the lastSECTION_OVERLAP(200) bytes of the previous one. Take a short paragraph, typically a heading, followed by one too big to join it. The short paragraph was flushed as a section on its own, and the next section then began with all of it as overlap. The result was a record with no text of its own, competing for a result slot with the real section.We found it on a Gutenberg EPUB (#1159's probe), where "CHAPTER 26. Knights and Squires." was a record by itself and the next record repeated it. The same shape occurs in every format that sections through this function: PDF pages, DOCX, PPTX slides, plain-text prose, HTML and EPUB.
Fix: a section is flushed only when it is longer than
SECTION_OVERLAP, that is, only when its overlap tail is not the whole section. A shorter one stays in the buffer and leads the next section. Nothing is dropped: that text appears in the next section as before, but once instead of twice. The combined section is at most2 * SECTION_CHARS + 202bytes before the existing hard-split path applies, the same bound as for any long paragraph.The byte-for-byte oracle test (
legacy_split_sections, kept from #239's cost rewrite) gets the same one-line rule, so it still checks that the streaming and simple implementations agree. Its cases now include a short paragraph before long ones, atSECTION_OVERLAP - 1,SECTION_OVERLAPandSECTION_OVERLAP + 1bytes.Evidence
Checks
cargo fmt --all, clippy-D warningscargo test -p xerj-autoindex --libNot run: the release build, and a re-index of a real corpus to count how many records disappear. On #1159's Moby-Dick probe, 2 of 858 records had this shape.
Provenance
rust:latestLinux container.