Repository navigation
feat(autoindex): extract Jupyter notebooks per heading section; name Parquet/Arrow/NumPy/HDF5/pickle files with an export hint (maintainer rebase of #1161) - #1162
Merged
Conversation
…ction
Motivation: autoindex had no notebook support; a .ipynb sniffed as JSON
and the generic JSON extractor made one record per CELL (75 for one
Python Data Science Handbook notebook), each repeating ~12 notebook-
metadata fields (kernelspec, language_info, a VS Code interpreter hash),
with code kept as an array of line strings and outputs as raw JSON. And
because datasets cluster on field names, every notebook became a dataset
of its own: nb-json-pandas-groupby, nb-json-ipynb-simple, ... Measured on
a v1.0.0-rc.82-era quick build against a folder of two notebooks.
What it does (extract/ipynb.rs, new; any kernel):
- Sections: cells grouped as a reader sees them. A section starts at
each Markdown ATX heading (also mid-cell; never inside a ``` / ~~~
fence; `#hashtag` and 4-space-indented lines are not headings) and
holds every cell to the next one. A section that is only its heading
line (`# Title` straight before `## Intro`) folds into the next.
- Fields: `title` (metadata.title, else the first level-1 heading, else
the first heading, else the file stem), `author` (metadata.authors),
`language` (kernelspec.language, language_info.name, else the kernel
name: python3 -> python, ir -> R, julia-* -> julia, deno ->
typescript), `heading`, `heading_path` (" > "-joined, when nested),
`cell` (1-based first cell), `section`, `body`. Locator c{cell}-s{j},
or c{cell}.{k}-s{j} for the k-th section starting in one cell (locators
key record ids, so they must be unique).
FieldOrigin::Extractor, so a folder of notebooks is one dataset.
- Body: Markdown as written; code fenced in the kernel's language; text
outputs after their cell in an unlabelled fence: stream text with ANSI
colors stripped and `\r` progress redraws collapsed, text/plain
results, errors as `ename: evalue`. Each output is capped at 40 lines
/ 4 KiB with "… (N more lines)". Image outputs, the text/plain
stand-in beside them (`<Figure size …>`) and HTML-only outputs are
not indexed. Nothing is executed.
- nbformat 3 (worksheets, `input`, `heading` cells, pyout/pyerr) too.
- Junk: invalid JSON, or no text. Whole-file read under the existing
64 MiB cap; MAX_RECORDS_PER_FILE and a 64 MiB text cap set
`truncated`.
Detection (sniff step 2b, before the JSON classifier, by content):
`looks_like_notebook` — first key `"cells"` with a `"cell_type"` in the
prefix (Jupyter and VS Code sort keys), or first key `nbformat` /
`nbformat_minor` / `metadata` (Colab, nbformat 3) with an nbformat
version number and a cells/worksheets list. JSON that merely has a
`cells` key stays Json (tested through `sniff`).
Wiring: Family::Ipynb (as_str "ipynb", is_document); dispatch in
extract/mod.rs; SourceAndDocs band in both order.rs functions; full-read
family in estimate::exact_scan_bytes; whole-file size cap in scan_file.
Evidence (rust:latest container, Linux):
- 8 tests in extract::ipynb: sections, heading paths, mid-cell headings,
fences, outputs (stream/result/image/html/error), ANSI and redraws,
output caps, nbformat 3, R kernel + metadata title/authors, Colab
banner/kernel-name/heading-only fold, `_` kept in heading text, junk,
and detection (positive: Jupyter, Colab, v3; negative: plain JSON
with `cells`, arrays, YAML).
- cargo test -p xerj-autoindex --lib: 1238 passed, 2 ignored; clippy
-D warnings clean.
- 7 real notebooks via a temporary probe test (not committed), all
sniffed as ipynb, 0 junk: three Python Data Science Handbook chapters,
the TensorFlow Keras classification tutorial (Colab), IRkernel's R demo
(`language: R`, R output and errors after their cells), all2md's
fixture, and a 76-record project notebook.
End to end (quick-profile Linux build of this branch, clean data dir,
`xerj autoindex` on the 7 notebooks + 12 data files): all notebooks in
ONE index (nb-docs, 144 docs = 137 sections + 7 file records; each file
holds exactly its extractor's record count + 1), e.g. heading_path
"Aggregation and Grouping > groupby: Split, Apply, Combine > The GroupBy
Object", cell 34; a match on R output ("deep error") returns the R
notebook with language R. The first E2E run caught sections that start
in the same cell sharing a locator (76 sections -> 10 docs for one
notebook); fixed, with a uniqueness assertion in the tests.
Not covered: JavaScript/Julia kernels on real files (the language comes
from the same metadata keys); notebooks over 64 MiB (embedded images)
are junk; Jupytext/marimo .py notebooks remain code, .Rmd/.qmd unchanged.
Written by an AI coding agent (Claude Code, Claude Opus 5.5), run and
reviewed by Tom Villani (@thomas-villani).
Files: engine/crates/xerj-autoindex/src/extract/ipynb.rs (new),
extract/mod.rs, sniff.rs, order.rs, estimate.rs, lib.rs
…d say how to index them A data-science folder's binary data files were junked as "binary content (unknown)" (measured: a Parquet file on the same quick build), which tells their owner nothing. Reading these formats would add large format libraries (arrow/parquet, hdf5) for files their owners can export in one line, so — as #1114 did for office containers — the sniffer now names them and the junk reason names the export. Nothing is ever opened. Signatures (MAGIC_TABLE, sniff.rs), each from real files: - Parquet `PAR1`, qualified (printable): the first page header must follow, a Thrift compact struct whose first two fields are i32s, i.e. 0x15 at bytes 4 and 6 (pyarrow snappy/zstd/none, and a Chroma store). - Arrow IPC / Feather v2 `ARROW1\0\0`, NumPy `\x93NUMPY`, HDF5 `\x89HDF\r\n\x1a\n`: each holds a byte text cannot contain. - Python pickle protocols 2-5: PROTO 0x80 + version 2..5, and for 4/5 the FRAME opcode 0x95 (protocols 0/1 have no header: not claimed). - `.npz`: a zip whose members are all `.npy` (zip_container_kind). The printable-signature invariant test covers the new `PAR1` row. Advice (`unsupported_data_advice`, chained after the document advice in scan_file): "unsupported data format (Parquet): autoindex has no reader for it — export it to CSV or JSON Lines to index it now", likewise for Arrow/Feather, NumPy (npy/npz) and HDF5. A pickle gets "autoindex never loads pickles (loading one runs code chosen by whoever wrote the file) — export the object to CSV or JSON from code you trust". Tests (sniff::data_container_tests): real-file prefixes for every kind, two pickle negatives, an npz through the real zip path, the advice text. `cargo test -p xerj-autoindex --lib`: 1241 passed, 2 ignored; clippy -D warnings clean (rust:latest Docker). Files: engine/crates/xerj-autoindex/src/sniff.rs, lib.rs
Merged
4 tasks done
Ravandevil25
pushed a commit
to Ravandevil25/xerj
that referenced
this pull request
Oct 6, 2026
- engine 1.0.0-rc.82 -> 1.0.0-rc.83 (Cargo.toml + cargo update -w, 17 lock entries) - landing: data-latest-tag + all 91 footer brand spans -> RC.83 (the xerj-org#1155 guard now enforces this at cut) - CHANGELOG: rc.83 section — xerj-org#1155 (version-roll guard + xxhash 0.8.19), xerj-org#1153 (xerj-org#1145/xerj-org#1148), xerj-org#1156 (xerj-org#1136), xerj-org#1154, xerj-org#1159 (EPUB), xerj-org#1160 (redundant-overlap section), xerj-org#1161 via xerj-org#1162 (notebooks + data-file naming) - ROADMAP: cut-roll — shortlist re-verified live: 8 open (xerj-org#1091 xerj-org#1092 xerj-org#1094 xerj-org#1100 xerj-org#1146 xerj-org#1147 xerj-org#1158 xerj-org#1122); xerj-org#1136/xerj-org#1145/xerj-org#1148 closed in-window, de-linked here and moved to CHANGELOG; rc.83 paragraph in the GA window record
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebase of #1161 by @thomas-villani — the head branch lives on his fork, which maintainers cannot update after #1159 landed into the same wiring tables. Both commits are his, rebased onto main (@ebe35c4f5): the sniff/order/estimate/lib arms now carry Epub and Ipynb side by side. cargo test -p xerj-autoindex: 1252 passed, 0 failed.
Closes #1161. Original description follows.
One record per Markdown-heading section (never inside a code fence), code fenced in the kernel's language, text outputs capped and ANSI-stripped, images skipped, nothing executed; plus Parquet/Arrow/NumPy/HDF5/pickle files named and told how to export themselves for indexing.