8000 feat(autoindex,xerj-common): xerj corpus build — harvested-record packs + add --from consumption (M1-M3) by xerj-org · Pull Request #1046 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

feat(autoindex,xerj-common): xerj corpus build — harvested-record packs + add --from consumption (M1-M3) - #1046

Merged
xerj-org merged 6 commits into
mainfrom
feat/corpus-builder
Sep 27, 2026
Merged

xerj-org merged 6 commits into
mainfrom
feat/corpus-builder

Conversation

@xerj-org
Copy link
Copy Markdown
Owner

What

xerj corpus build — a universal, incremental builder for harvested-record corpora (advisories, datasets, any structured records — not source code), plus the consumption half: xerj corpus add --from <pack> materializes a portable pack as a searchable corpus with per-hit licence warnings intact.

This is the records-pack half of tracker #1030 (ROADMAP.md — "a corpus hub of signed, pre-indexed packs"): the licence-safe half, since a pack of already-published advisory records carries no upstream-code licence questions.

Milestones 1–3 of the approved arc (M4 suggester, M5 rust-vulns showcase pack, M6 signing/publishing follow in their own PRs).

The commits

Commit What
be1d4841b (M1) harvest module: recipe TOML (strict validation), normalize, content-addressed store, union-find identity, cluster merge, deterministic pack emit; xerj corpus build CLI
5bd29f38a (M2) Network sources: http-zip + git adapters (promisor shallow checkout), upstream watermarks (skip walk AND prune when unchanged), defensive zip extraction
44d6b870f (M3) corpus add --from <pack-dir|manifest.json|pack.zip>: checksum-verify on read, per-slug materialization, corpus.json kind extension, one-kind-per-name refusal both directions
91a3e33ad fixups from the binary smoke: wrapper-dir descent in pack zips, share it: names the zip not the dead staging dir

Design decisions

  • Recipe-driven, never domain-coded. Field mappings, identity edges, merge precedence live in a commented TOML that ships verbatim inside the pack as provenance. The tool is licence-agnostic: licence is record data the pack author owns, carried (and warned about at query time), never enforced.
  • File presence is the dedup. Normalized envelopes are content-addressed (store/<xxh3_128>.json); an unchanged input rebuilds byte-identical shard files. Watermarks (git HEAD, ETag-style markers) are an optimization on top — correctness never depends on them.
  • Identity is union-find over declared edge values (e.g. id + aliases[] each=true): alias cycles merge for free; a CVE↔commit many-to-many stays derived edges, never identity nodes.
  • Determinism is load-bearing: shard assignment xxh3_64(id) % shards, id-lexicographic order, uniform key set on every record (absent → null/[]) — so engine _ids and dataset clustering stay stable across rebuilds.
  • The consumption seam is the per-slug layout (corpora/<name>/<slug>/records.jsonl): xerj code derives a hit's licence from the first locator segment, so a pack's source slugs key exactly like cloned repos always did — per-hit licence warnings work with zero query-side changes (pinned by a direct test against manifest::licence_map).
  • Strict pack verification on 8000 read: format_version checked before any hashing, every SHA256SUMS entry re-hashed, manifest-vs-SUMS cross-check, line counts vs manifest. Unsigned checksums cannot catch a fully self-consistent rewrite — detached signatures are M6, by design.

Verification

  • xerj-autoindex --lib: 1111 passed / 0 failed (at -j 4; one -j 32 run tripped the known co-tenant flake and was clean on rerun)
  • xerj-common --lib: 165 passed / 0 failed (manifest kind extension pins git-shaped bytes unchanged)
  • clippy -D warnings clean, rustfmt applied
  • Binary smoke end-to-end: corpus build demo → wrapper-shaped zip (python zipfile, demo/… entries) → add --from zip materializes the corpus with the alias-merged record (sources:["a","b"]) and kind:"harvested" corpus.json
  • ES-YAML conformance suite untouched (no engine-crate changes — xerj-autoindex/xerj-common only); CI verifies the 0-failed gate

Known limits (honest)

  • Freshness is operational: the 30-day staleness refusal will apply to these corpora; the scheduled rebuild (M6) is load-bearing, not decoration.
  • Cluster-id churn on source promotion is accepted and counted; an add-time hint to re-index --fresh follows in M5.
  • v1 ships checksums, not signatures — key distribution is the hard half and lands in M6 with --verify-sig.

Next (separate PRs)

M4 suggester (mapping.suggested.json / relations.jsonl, reusing crate::infer) → M5 tools/packs/rust-vulns first real build (OSV crates.io + RustSec + GHSA) → M6 GitHub Actions scheduled builds, detached signature, freshness badge.

…acks (M1)

ROADMAP #1030 names portable corpus packs as unbuilt; the only corpus
form today is git-cloned source code (`xerj corpus add|index`). There is
no way to build a corpus from harvested RECORDS (advisories, datasets,
any structured data), no cross-source identity resolution, and no
portable artifact another tool can consume.

This is M1 of the approved design (plan: immutable-wobbling-flask): the
domain core + `dir` sources. A recipe (TOML, human-authored, licence
reasoning in comments, ships verbatim in the pack) declares sources,
identity edges, merge precedence, and derived ops — the tool stays
domain-agnostic and licence-agnostic by directive; licence rides as
record data, never enforced.

New module engine/crates/xerj-autoindex/src/harvest/:
- recipe.rs   strict TOML parse/compile (deny_unknown_fields; unknown
              kinds name the planned network ones loudly; regexes
              compiled at load; RESERVED_KEYS protected)
- normalize.rs format adapters: `osv` (curated: affected[] ranges
              flattened into shared introduced/fixed lists — RustSec's
              packed and GHSA's split range shapes unify; ecosystem_specific
              /affects/functions → affected_functions, the only structured
              source of vulnerable function paths; database_specific is
              NEVER read — the schema marks it internal-use-only) and
              `flat` (mirrors extract::flatten_object exactly, verified
              against extract/mod.rs:252)
- store.rs    content-addressed envelope store: file presence IS the
              skip-already-stored dedup (same content ⇒ same xxh3-128
              key ⇒ convergent skip), atomic tmp+rename writes, per-slug
              prune. Canonical JSON sorts keys EXPLICITLY — the workspace
              enables serde_json preserve_order (xerj-common), so Map is
              an IndexMap and to_string would bake in insertion order
- identity.rs union-find over declared edge VALUES (alias chains and
              cycles collapse free; CVE↔commit stays a derived edge,
              never identity — a shared commit must not fuse two CVEs);
              merge = scalar first-non-empty under precedence, array
              union (typing respected: array anywhere ⇒ array), licence
              = restricted-member veto via xccode::licence::is_restricted
              else precedence-first
- pack.rs     deterministic emit: id-sorted, xxh3_64(id)%shards, uniform
              key set on EVERY record (absent array keys [], scalars
              null) so dataset clustering and _ids stay stable across
              rebuilds; manifest.json (format_version 1, kind harvested,
              per-file sha256, per-source new/skipped/pruned counts) +
              recipe.toml + SHA256SUMS
- mod.rs      `xerj corpus build <name> [--recipe] [--fresh]` CLI,
              builds live under ~/.xerj-code/builds/<name>/ (NOT corpora/
              — those children are corpus names), minimal glob matcher

Wiring: one dispatcher arm in xc.rs run_corpus_cli + CORPUS_USAGE line;
toml.workspace dep added (already in workspace). main.rs untouched.

Tests: 30 new (unit per module + end-to-end): fixture recipe → build →
GHSA+RUSTSEC merge via CVE alias → uniform keys → manifest structure →
re-run asserts new==0 and byte-identical shard files, generation
advances → upstream file deleted ⇒ pruned==1. Full crate suite green
(1096 passed, 0 failed). Binary smoke-tested by hand (build, rerun,
SHA256SUMS verify). ES-YAML untouched — gate unaffected.

Known scope (deliberate, per plan): network sources (http-zip/git),
watermarks, pack consumption (`corpus add --from <pack>`), suggester,
signing are M2–M6. http-zip/git kinds parse but refuse with the planned
message.

Files:
- engine/crates/xerj-autoindex/src/harvest/{mod,recipe,normalize,store,identity,pack}.rs (new)
- engine/crates/xerj-autoindex/src/xc.rs (dispatcher + usage)
- engine/crates/xerj-autoindex/src/lib.rs (pub mod harvest)
- engine/crates/xerj-autoindex/Cargo.toml (+toml)
…termarks (M2)

M1 (e8e2ecd5) built harvested packs from dir sources only; a real recipe
needs to pull from upstream. This adds the three source kinds the plan
names and the incremental machinery around them: a recipe can now mix
local fixtures (dir), public dump URLs (http-zip), and git repositories
at a pinned sha or tracking HEAD (git).

What changed:

- harvest/httpget.rs (new): blocking download with the esclient retry
  shape — 3 attempts, 1s exponential backoff, first-error-wins (a later
  attempt against a dead endpoint reports less than "HTTP 404"), atomic
  .part+rename, streamed via resp.take() under a 4 GiB cap so a runaway
  URL can never buffer unbounded in memory.
- harvest/source.rs (new): the adapters.
  * http-zip → download + extract into <build>/cache/<slug>. No
    watermark: the zip is re-downloaded every run and the
    content-addressed store dedups what did not change (correctness
    lives there; watermarks are only an optimization).
  * git → promisor (blob:none) shallow checkout at `rev`, reusing
    crate::xc::git/checkout_at_sha. Watermark = HEAD sha; a matching
    previous watermark means unchanged, and the orchestrator then skips
    the walk AND the prune (pruning needs the full live key set —
    skipping the walk but pruning would delete everything).
  * zip extraction is defensive by default: 2 GiB decompressed cap
    (declared entry sizes are untrusted), symlinks skipped+counted, and
    sanitize_entry_path rejects the zip-slip family — "..", absolute
    paths, and windows-style names (":", "\\") on EVERY os, because a
    pack may be built anywhere. Dest is cleared before extract so an
    upstream rename never survives as a stale duplicate.
- harvest/mod.rs: harvest loop routes through source::fetch; watermark
  saved only after a source's harvest fully succeeded (a failed run
  must re-walk); unchanged sources keep their store contents untouched
  and the pack rebuilds identically from the store; report.json gains
  watermark/unchanged per source (uniform schema — no
  skip_serializing_if, false serializes as false).
- harvest/pack.rs: SourceStat carries watermark/unchanged.
- harvest/recipe.rs: per-kind validation — dir needs recipe-relative
  path and refuses url/rev; http-zip needs http(s) url and refuses
  path/rev; git needs url (remote or local path) and refuses path.
- xc.rs: git()/checkout_at_sha() pub(crate) for reuse.

Tests: 36 harvest (was 30) / 1102 lib (was 1096), 0 failed. New: httpget
round-trip + 404-retry over a one-shot TcpListener (fake-node pattern);
zip extract/re-place; zip-slip counted-not-written; git source
watermark lifecycle (first fetch / unchanged skip / new-commit
re-activate) over a local origin; recipe network-kind validation.

Findings worth keeping:
- serde skip_serializing_if on `unchanged` omitted false from
  report.json/manifest — sile
8000
ntly breaking the uniform-schema invariant
  this subsystem exists to enforce. Removed; false is data too.
- The zip-slip test initially caught only 2 of 3 hostile entries:
  "C:/windows/evil.txt" is all-Normal path components on linux.
  Character-level rejection closes it.
- A skipped-walk source must also skip prune. The M1 e2e test proved
  records survive via the store; the failure mode was designed against,
  not observed — the watermark-skip path is new surface.

Known scope: M3 (pack consumption via `corpus add --from`), M4
(suggester), M5 (rust-vulns showcase), M6 (publish/sign) land on their
own branches per the approved plan. Reference-code mandate skipped for
this milestone — the standing corpora and the :9200 reference server
were wiped in the 2026-09-26 sandbox re-tool; the retry shape was
carried from esclient (in-repo) instead.

Files:
- engine/crates/xerj-autoindex/src/harvest/httpget.rs (new)
- engine/crates/xerj-autoindex/src/harvest/source.rs (new)
- engine/crates/xerj-autoindex/src/harvest/mod.rs
- engine/crates/xerj-autoindex/src/harvest/pack.rs
- engine/crates/xerj-autoindex/src/harvest/recipe.rs
- engine/crates/xerj-autoindex/src/xc.rs
…k consumption (M3)

M1/M2 build portable packs; nothing could consume one. `xerj corpus add
--from` now accepts a pack directory, the pack's own manifest.json, or a
pack.zip: the pack is checksum-verified end to end, then materialized as
a corpus `corpus index` can index unchanged.

What changed:

- harvest/pack.rs: `read_manifest(pack_dir)` — the verification reader.
  Unknown `format_version` refused BEFORE any hashing (a future format
  must be refused on the version alone); every SHA256SUMS line re-hashed;
  manifest-vs-sums disagreement named; per-file record counts checked.
  File names from the manifest are gated to plain names (no separators,
  `..`, drive letters) before ever being joined onto the pack dir — the
  zip-slip family, applied to JSON.
- xc.rs: `--from` routing — a dir with a harvested manifest.json, a
  manifest.json of that shape, or a `.zip` (staged through a tempdir via
  the M2 zip extractor, never build state) dispatches to
  `run_corpus_add_pack`; everything else falls to the hub-manifest arm
  unchanged. A path that LOOKS like a pack but is broken is an error,
  never a silent fallthrough (the hub arm would emit a confusing
  `repos[]` error for the same file).
- Materialization: records are partitioned into
  `corpora/<name>/<slug>/records.jsonl` by each record's winning
  `source`, lines copied VERBATIM (the pack line is canonical JSON),
  re-sorted by id per slug (shard files interleave ids; the corpus file
  should diff clean against the pack's own ordering). The per-slug layout
  is load-bearing: `xerj code` derives a hit's licence from the first
  locator segment (passage::locator_repo), so `rustsec/records.jsonl`
  keys licence_map exactly the way a cloned `rustsec/` repo always did —
  per-hit licence warnings work with zero query-side changes. Pinned by
  test against manifest::licence_map itself.
- corpus.json extension (xerj-common): `write_corpus_manifest_kind`
  emits an optional `"kind"` right after `"corpus"`; `None` is
  byte-for-byte the pinned git-corpus format (test-pinned), and
  `CorpusManifest` reads `kind` tolerantly so every historical manifest
  still loads. Pseudo-repos are the pack's sources: repo=slug,
  licence=source licence, sha=watermark, files/bytes=counted from the
  materialized records.
- One kind per name, both directions: the git arm refuses to clone onto
  a harvested name before touching anything; the pack arm refuses a name
  owned by a cloned-repo corpus. Re-adding a harvested corpus is a
  refresh: every dir the previous manifest owned is dropped and
  regenerated (a first draft only removed renamed slugs — a re-add would
  have APPENDED and duplicated every line; caught while writing the
  re-add test).

Tests: xerj-autoindex 1111/0 (was 1102; +5 pack-reader, +4 add-path),
xerj-common xccode::manifest 5/0 (was 4; +1 kind round-trip). New: pack
round-trip; tampered file NAMED; unknown format_version; truncated SUMS;
manifest/sums disagreement; the three `--from` pack spellings sniffed
(dir / manifest.json / zip) with hub-manifest and broken-dir cases;
filesystem e2e build→add→licence_map→re-add-no-dup→mix-refusal;
git-arm refusal; zip staging incl. a non-pack zip failing loudly.

Findings worth keeping:
- An attacker who rewrites records + manifest + SUMS consistently
  passes checksum verification — that is correct for unsigned checksums
  and is exactly what M6's detached signature over SHA256SUMS is for.
  The test comments say so where the trap is.
- `resolve_pack_source` sniffs `kind` on a JSON file before the hub arm
  reads it: two untrusted-input readers for two different documents,
  routed by content, not extension.

Known scope: M4 (suggester), M5 (rust-vulns showcase), M6 (publish +
sign) follow on their own branches. `corpus index` and `xerj code` are
deliberately untouched — dataset clustering over the uniform key set and
licence_map over slug dirs are the seams they already provide. Docs
(SKILL.md, README, ROADMAP records-pack half) land with M5/M6 per plan.
Reference-code mandate skipped — consumption of this repo's own M1 pack
format; the standing corpora/:9200 reference server remain wiped from
the 2026-09-26 sandbox re-tool.

Files:
- engine/crates/xerj-autoindex/src/harvest/pack.rs
- engine/crates/xerj-autoindex/src/xc.rs
- engine/crates/xerj-common/src/xccode/manifest.rs
A zip of the pack DIRECTORY — how every zip-of-a-directory tool packs,
and what the binary smoke actually produced — puts the pack under a
wrapper (`demo.zip` → `demo/manifest.json`). read_manifest then failed
with "cannot read manifest.json / Is a directory" because the staging
root held one child dir, not the pack.

After extraction, when the staging root has no manifest.json, descend
exactly one level IF the single child dir holds one. Deeper nesting is
not a shape this tool emits and fails as before — the descent is a
consumption convenience, not a search.

Also from the smoke: the closing "share it:" line named the staging
dir for the zip spelling — a tempdir deleted before the line even
prints. run_corpus_add_pack now takes the path to NAME (the pack dir
or the zip the user actually holds), so the re-share hint points at
something that exists.

The unit tests had only zipped the flat shape, which is why this lived
until the smoke: a_pack_zip_stages_and_materializes now zips BOTH
shapes (wrapper `demo/…` and bare contents) through a zip_up() helper
and asserts each materializes 2 records with kind=harvested; the
non-pack zip still fails with exit 2.

Verified by the M3 binary smoke: build → wrapper zip (python zipfile,
`demo/…` entries) → add --from zip materializes corpora/demo with the
alias-merged record (sources:["a","b"]) and kind=harvested corpus.json.
xerj-autoindex --lib xc:: 20/0, clippy clean, rustfmt applied.
@cla-bot cla-bot Bot added the cla-signed label Sep 27, 2026
…1046)

The arc commits were verified with `clippy --lib` only; CI's Format +
Clippy job runs with --tests, which caught two lints in test code:

- xc.rs CodeHomeGuard: the tuple field holds the ENV_LOCK guard purely
  for its Drop side effect (release on scope exit) and is never read —
  dead_code fires. #[expect(dead_code)] documents exactly that and
  will un-lint itself if the field ever gains a reader.
- recipe.rs minimal-recipe test used assert_eq!(bool, true) — clippy's
  bool_assert_comparison wants assert!(..).

No production code changed; behavior identical. xerj-autoindex
clippy --lib --tests -D warnings clean, harvest suite 42/0.
… inventory

PR #1046's Build + Test failed on
every_outbound_client_in_the_engine_is_on_the_published_list: M2 added
xerj-autoindex/src/harvest/httpget.rs, a reqwest::blocking client that
fetches http-zip sources — a new outbound client under engine/crates the
published list did not account for. The inventory test did exactly what
it exists for.

Classification: Role::Client("`xerj corpus build`"), with a new row in
docs/RERANK.md "Every way data leaves a XERJ node". The fetch is
read-only — HTTPS GETs to the URLs the operator's recipe names (plus
the git clone/fetch a git source shells out to, which the marker list
does not track; the doc row names it anyway so the operator sees the
whole picture). Records travel INTO the machine; nothing is sent
upstream, so xerj_rerank::DATA_EGRESS ("what a node SENDS") and the
air-gapped recipe rows are deliberately unchanged — same treatment as
`xerj autoindex s3://`, the closest precedent.

egress_inventory 4/4. No production code changed.
@xerj-org
xerj-org merged commit 258d56f into main Sep 27, 2026
22 checks passed
neoneye pushed a commit to agent-memory-atlas-archive/xerj-org--xerj that referenced this pull request Sep 29, 2026
…eduled rust-vulns publish (M6)

The publish half of the corpus-builder arc. Records packs (M1-M3) shipped
checksums, which prove integrity; nothing proved ORIGIN — an attacker who
rebuilds a pack self-consistently (tampered records, honestly rewritten
SHA256SUMS) passes every checksum read_manifest verifies. The prior-art
survey found no vulnerability database (Trivy DB, Grype/vunnel, PrimeVul,
VulnRAG) shipping a consumer-verifiable signature, so this is an open lane
rather than table stakes.

Engine (harvest/sign.rs, ring 0.17 — already in the lockfile via rustls,
zero new native build):

- generate_keypair / sign_pack / verify_pack: raw 64-byte detached
  ed25519 over SHA256SUMS. Deliberately boring format — no armored
  envelope, any ed25519 implementation can verify the bytes.
- SIG_NAME is intentionally NOT listed in SHA256SUMS (signing a list
  that contains the signature is circular); read_manifest re-hashes
  only the files SUMS names, so the .sig rides in a dir or zip without
  breaking readers.
- CLI: `xerj corpus keygen --out <prefix>` (refuses overwrite, creates
  the parent dir — the documented first run names tools/packs/keys/<n>,
  which does not exist yet), `xerj corpus sign <pack-dir> --key <seed>`,
  and `xerj corpus add --from <pack> --verify-sig <pubkey-file>` —
  verified BEFORE read_manifest and materialization: a failed signature
  leaves no corpus directory at all (pinned by the CLI test: wrong key
  → exit 2, corpus dir absent; right key installs; unsigned pack + no
  flag still installs, with a stderr nudge).
- The load-bearing test: a self-consistent rebuild (records edited,
  SUMS rewritten honestly) passes read_manifest's checksums and FAILS
  the signature — exactly the gap M3's pack.rs test comment deferred
  to "a later milestone".
- ring 0.17 API realities for the next person: from_seed_unchecked
  takes the RAW 32-byte seed (no pkcs8 DER needed),
  UnparsedPublicKey::verify takes plain &[u8] (there is no
  UnparsedSignature type), public_key() is on the signature::KeyPair
  trait.

Trust model (docs/CORPUS_PACKS.md states it in full): the pubkey
travels OUT OF BAND — committed at tools/packs/keys/rust-vulns.pub,
never inside the pack (a key shipped beside its own signature verifies
nothing). build ≠ publish: `corpus build` never signs; the seed exists
only as the PACK_SIGNING_SEED CI secret (set 2026-09-27 from the
keygen output; the local .key was deleted after the smoke run below).
v1 is a single release key; the rotation procedure is written down in
tools/packs/keys/README.md before it is needed.

Publishing (.github/workflows/pack-publish.yml): daily 04:23 UTC +
workflow_dispatch, concurrency-grouped (rolling tag, never two publishes
at once), builds the pack from the committed recipe, signs with the
secret, then VERIFIES ITS OWN OUTPUT against the committed .pub via the
consumer's exact command before publishing to the rolling prerelease
tag pack-rust-vulns (zip + loose SUMS + .sig + shields freshness json).
A half-rotated key fails the build instead of shipping an unverifiable
pack. Freshness is an operational promise the schedule keeps: the
pack's own 30-day staleness refusal applies to our published corpus
exactly as it does to a user's reference corpora.

Verified locally, the full production loop with the shipped keypair:
keygen → sign on the real rust-vulns pack (generation 2) →
`corpus add --verify-sig` with the committed .pub installs 1,950
records → wrong .pub → exit 2, nothing materialized. Gates: rustfmt
clean, clippy --lib --tests -D warnings clean, xerj-autoindex lib
suite 1129 passed / 0 failed / 2 ignored, pack_claims 3/3 (the README
numbers stay pinned to the measured pack-stats.json). ES-YAML
untouched — no engine query path touched.

Docs/GTM: docs/CORPUS_PACKS.md (format, trust model, honest limits —
records-not-index, not-a-security-product, cron-lag freshness);
rust-vulns README gets the workflow + freshness badges, the
consume-the-signed-pack block, and the rotation section; SKILL.md
gains --verify-sig; ROADMAP's corpus-hub item rolls to "built and
signed" (still unbuilt: the hub, the pre-indexed half xerj-org#1030); blog
post landing/blog/one-vulnerability-three-names.html (+ blog index
card, sitemap, pagedata entry) — every count in it is the committed
measured pack-stats.json, and the prior-art signature table is phrased
as our survey; CHANGELOG [Unreleased] Added entry citing xerj-org#1046/xerj-org#1047.

Files:
- engine/crates/xerj-autoindex/src/harvest/sign.rs (new; 5 unit tests)
- engine/crates/xerj-autoindex/src/harvest/mod.rs (pipeline note)
- engine/crates/xerj-autoindex/src/xc.rs (--verify-sig wiring + test)
- engine/crates/xerj-autoindex/Cargo.toml, engine/Cargo.toml (ring)
- .github/workflows/pack-publish.yml (new)
- tools/packs/keys/{rust-vulns.pub,README.md} (new)
- tools/packs/rust-vulns/README.md, tools/xerj-code/SKILL.md
- docs/CORPUS_PACKS.md (new), ROADMAP.md, CHANGELOG.md
- landing/blog/{one-vulnerability-three-names.html,index.html},
  landing/sitemap.xml, scripts/seo/pagedata.py
xerj-org added a commit that referenced this pull request Sep 30, 2026
…d packs shipped in rc.78

Both agent-facing index files still said corpus packs were "planned, no
code" (landing/llms.txt:50, and the NOT-IMPLEMENTED list in
landing/llms-full.txt). False since v1.0.0-rc.78 (2026-09-29): PRs
#1046/#1047 (builder + rust-vulns showcase) and #1048 (signing/publish),
hardened by #1049-#1051, shipped `xerj corpus build` — a declarative
recipe becomes a checksummed portable pack of records (format_version
refusal, licence + provenance on every record), `corpus add --from`
verifies per-file SHA-256 and `--verify-sig` a detached ed25519
signature over SHA256SUMS, and the first pack is rebuilt, signed and
published daily as a dated GitHub Release.

The correction claims exactly what shipped, no more: it says plainly that
the PRE-INDEXED half (an already-indexed bundle you mount instead of
spending your own CPU-hours) and a hub directory beyond the first pack
remain unbuilt — matching ROADMAP.md's corpus-hub row, which was rolled
in #1053.

Edits follow each file's own precedent for status rolls so nothing is
restructured: llms.txt rewrites the one clause in the not-implemented
caveat using the same inline pattern the line already uses for the
v1.0.0-rc.77 object-storage roll, and adds docs/CORPUS_PACKS.md to the
"file that proves it" tail; llms-full.txt keeps the log-mode bullet,
points it at a new "left this list in v1.0.0-rc.78" paragraph modelled
on the object-storage one, and keeps the list's 2026-09-26 re-check date
(the new paragraph carries its own 2026-09-29 date instead of claiming a
full re-check) and its ~76-char wrap.

Known lag, deliberately untouched here: the linked answer page
landing/answers/does-xerj-support-s3-alerting-plugins.md (and its .html
twin) still carries the old "no code exists" wording; it needs its own
surgical PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0