Repository navigation
feat(autoindex): Jupyter notebooks as heading sections; name Parquet/Arrow/NumPy/HDF5/pickle files with an export hint - #1161
Closed
thomas-villani wants to merge 2 commits into
Closed
thomas-villani wants to merge 2 commits into
thomas-villani wants to merge 2 commits into
Conversation
…ction
Motivation: autoindex had no notebook support; a .ipynb sniffed as JSON
and the generic JSON extractor made one record per CELL (75 for one
Python Data Science Handbook notebook), each repeating ~12 notebook-
metadata fields (kernelspec, language_info, a VS Code interpreter hash),
with code kept as an array of line strings and outputs as raw JSON. And
because datasets cluster on field names, every notebook became a dataset
of its own: nb-json-pandas-groupby, nb-json-ipynb-simple, ... Measured on
a v1.0.0-rc.82-era quick build against a folder of two notebooks.
What it does (extract/ipynb.rs, new; any kernel):
- Sections: cells grouped as a reader sees them. A section starts at
each Markdown ATX heading (also mid-cell; never inside a ``` / ~~~
fence; `#hashtag` and 4-space-indented lines are not headings) and
holds every cell to the next one. A section that is only its heading
line (`# Title` straight before `## Intro`) folds into the next.
- Fields: `title` (metadata.title, else the first level-1 heading, else
the first heading, else the file stem), `author` (metadata.authors),
`language` (kernelspec.language, language_info.name, else the kernel
name: python3 -> python, ir -> R, julia-* -> julia, deno ->
typescript), `heading`, `heading_path` (" > "-joined, when nested),
`cell` (1-based first cell), `section`, `body`. Locator c{cell}-s{j},
or c{cell}.{k}-s{j} for the k-th section starting in one cell (locators
key record ids, so they must be unique).
FieldOrigin::Extractor, so a folder of notebooks is one dataset.
- Body: Markdown as written; code fenced in the kernel's language; text
outputs after their cell in an unlabelled fence: stream text with ANSI
colors stripped and `\r` progress redraws collapsed, text/plain
results, errors as `ename: evalue`. Each output is capped at 40 lines
/ 4 KiB with "… (N more lines)". Image outputs, the text/plain
stand-in beside them (`<Figure size …>`) and HTML-only outputs are
not indexed. Nothing is executed.
- nbformat 3 (worksheets, `input`, `heading` cells, pyout/pyerr) too.
- Junk: invalid JSON, or no text. Whole-file read under the existing
64 MiB cap; MAX_RECORDS_PER_FILE and a 64 MiB text cap set
`truncated`.
Detection (sniff step 2b, before the JSON classifier, by content):
`looks_like_notebook` — first key `"cells"` with a `"cell_type"` in the
prefix (Jupyter and VS Code sort keys), or first key `nbformat` /
`nbformat_minor` / `metadata` (Colab, nbformat 3) with an nbformat
version number and a cells/worksheets list. JSON that merely has a
`cells` key stays Json (tested through `sniff`).
Wiring: Family::Ipynb (as_str "ipynb", is_document); dispatch in
extract/mod.rs; SourceAndDocs band in both order.rs functions; full-read
family in estimate::exact_scan_bytes; whole-file size cap in scan_file.
Evidence (rust:latest container, Linux):
- 8 tests in extract::ipynb: sections, heading paths, mid-cell headings,
fences, outputs (stream/result/image/html/error), ANSI and redraws,
output caps, nbformat 3, R kernel + metadata title/authors, Colab
banner/kernel-name/heading-only fold, `_` kept in heading text, junk,
and detection (positive: Jupyter, Colab, v3; negative: plain JSON
with `cells`, arrays, YAML).
- cargo test -p xerj-autoindex --lib: 1238 passed, 2 ignored; clippy
-D warnings clean.
- 7 real notebooks via a temporary probe test (not committed), all
sniffed as ipynb, 0 junk: three Python Data Science Handbook chapters,
the TensorFlow Keras classification tutorial (Colab), IRkernel's R demo
(`language: R`, R output and errors after their cells), all2md's
fixture, and a 76-record project notebook.
End to end (quick-profile Linux build of this branch, clean data dir,
`xerj autoindex` on the 7 notebooks + 12 data files): all notebooks in
ONE index (nb-docs, 144 docs = 137 sections + 7 file records; each file
holds exactly its extractor's record count + 1), e.g. heading_path
"Aggregation and Grouping > groupby: Split, Apply, Combine > The GroupBy
Object", cell 34; a match on R output ("deep error") returns the R
notebook with language R. The first E2E run caught sections that start
in the same cell sharing a locator (76 sections -> 10 docs for one
notebook); fixed, with a uniqueness assertion in the tests.
Not covered: JavaScript/Julia kernels on real files (the language comes
from the same metadata keys); notebooks over 64 MiB (embedded images)
are junk; Jupytext/marimo .py notebooks remain code, .Rmd/.qmd unchanged.
Written by an AI coding agent (Claude Code, Claude Opus 5.5), run and
reviewed by Tom Villani (@thomas-villani).
Files: engine/crates/xerj-autoindex/src/extract/ipynb.rs (new),
extract/mod.rs, sniff.rs, order.rs, estimate.rs, lib.rs
…d say how to index them A data-science folder's binary data files were junked as "binary content (unknown)" (measured: a Parquet file on the same quick build), which tells their owner nothing. Reading these formats would add large format libraries (arrow/parquet, hdf5) for files their owners can export in one line, so — as xerj-org#1114 did for office containers — the sniffer now names them and the junk reason names the export. Nothing is ever opened. Signatures (MAGIC_TABLE, sniff.rs), each from real files: - Parquet `PAR1`, qualified (printable): the first page header must follow, a Thrift compact struct whose first two fields are i32s, i.e. 0x15 at bytes 4 and 6 (pyarrow snappy/zstd/none, and a Chroma store). - Arrow IPC / Feather v2 `ARROW1\0\0`, NumPy `\x93NUMPY`, HDF5 `\x89HDF\r\n\x1a\n`: each holds a byte text cannot contain. - Python pickle protocols 2-5: PROTO 0x80 + version 2..5, and for 4/5 the FRAME opcode 0x95 (protocols 0/1 have no header: not claimed). - `.npz`: a zip whose members are all `.npy` (zip_container_kind). The printable-signature invariant test covers the new `PAR1` row. Advice (`unsupported_data_advice`, chained after the document advice in scan_file): "unsupported data format (Parquet): autoindex has no reader for it — export it to CSV or JSON Lines to index it now", likewise for Arrow/Feather, NumPy (npy/npz) and HDF5. A pickle gets "autoindex never loads pickles (loading one runs code chosen by whoever wrote the file) — export the object to CSV or JSON from code you trust". Tests (sniff::data_container_tests): real-file prefixes for every kind, two pickle negatives, an npz through the real zip path, the advice text. `cargo test -p xerj-autoindex --lib`: 1241 passed, 2 ignored; clippy -D warnings clean (rust:latest Docker). Files: engine/crates/xerj-autoindex/src/sniff.rs, lib.rs
Ravandevil25
pushed a commit
to Ravandevil25/xerj
that referenced
this pull request
Oct 6, 2026
- engine 1.0.0-rc.82 -> 1.0.0-rc.83 (Cargo.toml + cargo update -w, 17 lock entries) - landing: data-latest-tag + all 91 footer brand spans -> RC.83 (the xerj-org#1155 guard now enforces this at cut) - CHANGELOG: rc.83 section — xerj-org#1155 (version-roll guard + xxhash 0.8.19), xerj-org#1153 (xerj-org#1145/xerj-org#1148), xerj-org#1156 (xerj-org#1136), xerj-org#1154, xerj-org#1159 (EPUB), xerj-org#1160 (redundant-overlap section), xerj-org#1161 via xerj-org#1162 (notebooks + data-file naming) - ROADMAP: cut-roll — shortlist re-verified live: 8 open (xerj-org#1091 xerj-org#1092 xerj-org#1094 xerj-org#1100 xerj-org#1146 xerj-org#1147 xerj-org#1158 xerj-org#1122); xerj-org#1136/xerj-org#1145/xerj-org#1148 closed in-window, de-linked here and moved to CHANGELOG; rc.83 paragraph in the GA window record
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
Two commits for data-science folders.
1. Jupyter notebooks (
extract/ipynb.rs, any kernel). autoindex had no notebook support. A.ipynbsniffed as JSON, and the generic JSON extractor did this with it:Now cells are grouped the way a reader sees them, one record per Markdown-heading section:
title,author, and the kernel'slanguage, so R, Julia and JavaScript notebooks say what they are.heading,heading_path(e.g. "Aggregation and Grouping > groupby: Split, Apply, Combine > The GroupBy Object"), andcell, the 1-based first cell, for finding it in Jupyter.\rprogress bars are collapsed to what the terminal finally showed.ename: evalue.cellskey stays JSON.2. Data files named, with an export hint. Parquet, Arrow/Feather, NumPy (
.npy/.npz), HDF5 and pickle files used to be junked as "binary content (unknown)". Reading them would add large format libraries, so, as #1114 did for office containers, the sniffer now names them, and the junk reason says how to make them indexable:Pickles are never loaded, because unpickling runs code chosen by whoever wrote the file, and their reason says so. Every s 8000 ignature comes from real files: pyarrow (snappy, zstd and uncompressed), a Chroma Parquet store,
numpy.save/savez, h5py, andpickle.dumpsprotocols 2–5. Parquet's printablePAR1is qualified by the Thrift page header that follows it, as the magic table's invariant test requires.Evidence
End to end: a
--profile quickLinux build of this branch, a clean data dir, andxerj autoindexon 7 real notebooks plus 12 data files (exit 3, junk recorded):The notebooks: three Python Data Science Handbook chapters, TensorFlow's Keras classification tutorial (Colab), IRkernel's R demo, all2md's fixture, and a 76-section project notebook. The real files caught five bugs, each fixed with a test before this PR:
_stripped from heading text;Checks
cargo fmt --all, clippy-D warningscargo test -p xerj-autoindex --libllms.txtformat lists and a fact-check row, after feat(autoindex): extract EPUB books, one record per chapter in reading order #1159's follow-up (only one PR editsllms.txtat a time)sniff).Provenance
ipynb.py(source joining, kernel language, output types) and is re-implemented for autoindex's record model.rust:latest).languagecomes out empty would disprove this..Rmdand.qmdfiles, which are unchanged by this PR.