Repository navigation
docs: EPUB and Jupyter notebooks; recapture two answers; fix man-page fact-check aliases - #1163
Merged
Merged
Conversation
…aries 5d566a1 (xerj-org#1130) added the "Man pages (roff man(7))" THING row with aliases written by a scripted edit that decoded `\b` into the control character U+0008. The row read r"\x08man ?pages?\x08" etc., so none of its five aliases could match an article (re.search('\x08man ?pages?\x08', 'how to search man pages') is None): the GREEN row and its scope gate were never applied to a man-page article. Fix: the five aliases now hold `\b` word boundaries. Guard: `factcheck.py --self-test` now fails on any THING alias, or any rule pattern/context/requires, that holds a control character (tab and newline allowed in rule patterns). On origin/main it reports the five man-page aliases ("5 self-test failure(s)"); after this fix it passes ("self-test OK: 56 rules, ... 35 THING rows"). Written by an AI coding agent (Claude Code, Claude Opus 5.5), run and reviewed by Tom Villani (@thomas-villani). The agent also wrote the defect, in xerj-org#1130. Files: scripts/seo/claims_rules.py, scripts/seo/factcheck.py
…pped-files answers Follow-up to xerj-org#1159 (EPUB), xerj-org#1160 and xerj-org#1162 (notebooks, data-file hints). None of them is in the v1.0.0-rc.82 release (cut before they merged), so this should merge with or after the next release. - README format list, llms.txt extractor list, llms-full.txt § formats: EPUB (one record per chapter, toc-anchor split, DRM chapters skipped) and Jupyter notebooks (any kernel, one record per heading section, outputs capped, images/HTML skipped, nothing executed). - Fact-check matrix (claims_rules.py): - Jupyter notebooks AMBER -> GREEN, cite extract/ipynb.rs, gate scoped to .ipynb and saying what is not done (images/HTML outputs, execution, >64 MiB, .Rmd/.qmd/Jupytext not read as notebooks). - Ebooks (EPUB) RED -> GREEN, cite extract/epub.rs, gate: DRM skipped, image-only books junk, no .mobi/.azw/.fb2. - Parquet row widened to Parquet/Arrow/HDF5/NumPy/pickle: still RED (no reader), mech says they are now named with an export hint. - FC-THING-RED rewrite text: of the four extractors with the clearest demand, only OCR/images is still missing. - fixtures: good_jupyter_amber.md (expected Jupyter AMBER; its body is now false) becomes good_slack_amber.md on a st 8000 ill-AMBER row; new good_epub_answer.md and good_ipynb_answer.md. - content/answers/search-jupyter-notebook-cells.md: rewritten from a new capture (RUN-C). The old page described the generic JSON path (one doc per cell, "XERJ is not notebook-aware"), which is no longer what happens. New capture: 2 notebooks + the .py beside them -> 1 index (jp-docs, 11 docs); `okapi_transform` hits 3 sections + the .py; the printed accuracy hits "Okapi retrieval sweep > Run the sweep", cell 3; a mid-cell heading gets locator c1.2-s0; image, figure stand-in and HTML output queries return 0 hits. - content/answers/why-autoindex-skipped-files.md: the refused-file example was books/manual.epub -> "binary content (zip)" (stale since xerj-org#1114, false since xerj-org#1159). Recaptured on a rebuilt 26-file fixture: the ignore-rule accounting reproduces the published lines exactly; the second refusal is now data/events.parquet with the export hint; xerj-done line as captured. - Generated: landing/answers pages + index regenerated by build_articles. Captures: quick-profile Linux build of main 0618596 (reports v1.0.0-rc.82; neither feature is in that release - both pages say so), clean data dir. Fixture generators and raw capture output kept by the author outside the repo. Gates: factcheck --self-test OK (35 THING rows); --fixture-check 0 false negatives / 0 false positives (15/15 good fixtures clean); --fail-on error 0 ERROR; build_articles --check current; gen_sitemap --check ok; landing-constants-guard passes; seo_lint only the sitemap.stale CRLF artifact also present on main. Written by an AI coding agent (Claude Code, Claude Opus 5.5), run and reviewed by Tom Villani (@thomas-villani).
xerj-org
approved these changes
Oct 6, 2026
Owner
There was a problem hiding this comment.
Reviewed in full — this is the right shape and the aliases fix is a real catch.
What I checked:
- The two GREEN flips (notebooks, EPUB) cite
extract/ipynb.rs:1/extract/epub.rs:1— both exist on main since #1162/#1159, and the gate text is scoped the way our claims rules demand (does / does-not in the same breath: nbformat 3+4 with the 40-line/4KiB output cap and no execution; EPUB 2/3 with DRM-skip and no OCR).--check-matrixreporting drift here is by design. - The Parquet row keeps RED (still no reader) and only records what #1162 actually changed — sniff naming with an export hint, pickle never loaded.
- The man-page alias bug: the row held literal backspace bytes where
\bword boundaries were intended, so four aliases matched nothing — and the new self_test compiles every alias and rejects control characters, so the class cannot come back silently. Good fix. - The rewrite correction (
only OCR/images is still missing) matches the matrix after this PR.
One thing before merge, and it is not yours: your branch predates main's rustfmt fix for the exact line your diff context shows — order.rs:228, the band list my #1162 rebase left mis-indented, fixed on main in f24580d. Rebase onto main (or let the merge button update the branch) and Format + Clippy goes green — no code change needed on your side. Everything else on this PR is green. CLA shows signed from your earlier PRs; if the check re-fires after the rebase it should pick it up.
The only failing check on this PR: rustfmt wants the two trailing match arms joined. No semantic change; verified with cargo fmt -p xerj-autoindex.
8000
Ravandevil25
pushed a commit
to Ravandevil25/xerj
that referenced
this pull request
Oct 6, 2026
Version roll 1.0.0-rc.83 → 1.0.0-rc.84. Engine window (all merged to main before this cut): - xerj-org#1172 (fix/cluster-topology-honesty, closes xerj-org#1169): cluster endpoints stopped fabricating topology — health reports real configured membership, wait_for_nodes times out honestly (408 + timed_out), upper bounds enforced, _cat/nodes lists every configured member, _cluster/state reports the actual cluster address; multinode smoke skips with a recorded reason instead of passing vacuously. Also the xerj-org#1171 config half: peer-label≠address and cluster.port collisions refused at load. - xerj-org#1175 (fix/cluster-degraded-health, closes xerj-org#1171 with xerj-org#1172): the transport boot outcome recorded on AppState (ClusterTransportStatus Disabled/Running/Degraded) before the first router exists; degraded boot → health yellow, number_of_nodes 1, additive xerj_cluster_transport:"degraded" marker; unopenable indices (red) still outrank the marker. - xerj-org#1174 (fix/corpus-index-resume, xerj-org#1173 honesty half — throughput half stays open): xerj corpus index stopped reporting a converged corpus as a failed crawl; resume path re-crawls only what's missing. - xerj-org#1167 (docs): hub counts un-staled — llms.txt:51 and the corpus-hub blog note said 75 live / 104 backlog from 2026-10-02; now 100/119. - xerj-org#1163 (thomas-villani, docs): EPUB/notebook answer pages recaptured on the rc.83 build; \x08 man-page fact-check alias fix + self-test. Mechanics of this commit: - engine/Cargo.toml version → 1.0.0-rc.84; cargo update -w rolled 17 Cargo.lock entries. - landing/: 91 V1.0.0-RC.83 footer spans → V1.0.0-RC.84 across 54 files (0 old spans remain); landing/index.html data-latest-tag span → v1.0.0-rc.84. - CHANGELOG: [Unreleased] rolled into [1.0.0-rc.84] - 2026-10-06 with all five window PRs cited under Fixed. - ROADMAP: Last-reviewed line → rc.84; open-defects shortlist re-verified live at cut time — eleven open (xerj-org#1091 xerj-org#1092 xerj-org#1094 xerj-org#1100 xerj-org#1146 xerj-org#1147 xerj-org#1158 xerj-org#1122 carried forward, xerj-org#1168 xerj-org#1170 cluster liveness/leader-wiring, xerj-org#1173 throughput half); xerj-org#1169/xerj-org#1171 moved to the CHANGELOG; GA-window rc.84 paragraph added. ES-YAML gate on the window PRs: 0 failed (452/452 re-run after one non-reproducing co-tenant flake).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge timing: please merge this with or after the next release. EPUB (#1159), the section fix (#1160) and notebooks plus data-file hints (#1162) all merged after v1.0.0-rc.82 was cut.
landing/deploys on merge, so merging this earlier would makellms.txtadvertise formats the published binary cannot read yet. The two rewritten answer pages say which build they were captured on.What this changes, and why
Three commits.
1.
fix(seo): the man-page fact-check aliases never matched. In my docs commit for #1130 (5d566a1), a scripted edit decoded\binto the control character U+0008, so the "Man pages" THING row held\x08man ?pages?\x08and could not match any article. The aliases now use real\bword boundaries.factcheck.py --self-testnow fails if any alias or rule pattern holds a control character. On main it reports these 5 aliases; after the fix it passes.2.
docs: EPUB, notebooks and data files.llms.txtextractor list and thellms-full.txtformats section gain EPUB and Jupyter notebooks, each with what is not done.extract/ipynb.rs).extract/epub.rs).good_jupyter_amber.mdasserted a now-false behavior, so it moves to Slack exports (still AMBER) asgood_slack_amber.md. Newgood_epub_answer.mdandgood_ipynb_answer.md.answers/search-jupyter-notebook-cells, rewritten from a new capture. The old page documented the generic-JSON path: one document per cell, and "XERJ is not notebook-aware". The new capture, on a fixture of 2 notebooks plus the.pybeside them:jp-docs, with 11 docs;okapi_transformhits 3 sections and the.py;c1.2-s0;answers/why-autoindex-skipped-files, recaptured. Its refused-file example wasbooks/manual.epub→binary content (zip), which has been stale since fix(autoindex): name Office/OpenDocument/EPUB zips instead of advising unzip #1114 and false since feat(autoindex): extract EPUB books, one record per chapter in reading order #1159. I rebuilt the 26-file fixture: the ignore-rule accounting reproduces the published lines exactly, and the second refusal is nowdata/events.parquet, carrying the export hint.answers/index.jsonwere regenerated bybuild_articles.py.3.
chore(seo): sitemaplastmod, regenerated in its own commit.Evidence
Captures: a
--profile quickLinux build of main06185966(it reports v1.0.0-rc.82), on a clean data dir.Checks
.github/workflows/seo.yml, plus--fixture-checkandlanding-constants-guard(above).landing/llms.txt: this one (checked: no other open PR touches it)..gitignore:165(git check-ignore -v).Provenance