Repository navigation
feat(autoindex): signed corpus packs — keygen/sign/--verify-sig + scheduled rust-vulns publish (M6) - #1048
Merged
Conversation
…eduled rust-vulns publish (M6) The publish half of the corpus-builder arc. Records packs (M1-M3) shipped checksums, which prove integrity; nothing proved ORIGIN — an attacker who rebuilds a pack self-consistently (tampered records, honestly rewritten SHA256SUMS) passes every checksum read_manifest verifies. The prior-art survey found no vulnerability database (Trivy DB, Grype/vunnel, PrimeVul, VulnRAG) shipping a consumer-verifiable signature, so this is an open lane rather than table stakes. Engine (harvest/sign.rs, ring 0.17 — already in the lockfile via rustls, zero new native build): - generate_keypair / sign_pack / verify_pack: raw 64-byte detached ed25519 over SHA256SUMS. Deliberately boring format — no armored envelope, any ed25519 implementation can verify the bytes. - SIG_NAME is intentionally NOT listed in SHA256SUMS (signing a list that contains the signature is circular); read_manifest re-hashes only the files SUMS names, so the .sig rides in a dir or zip without breaking readers. - CLI: `xerj corpus keygen --out <prefix>` (refuses overwrite, creates the parent dir — the documented first run names tools/packs/keys/<n>, which does not exist yet), `xerj corpus sign <pack-dir> --key <seed>`, and `xerj corpus add --from <pack> --verify-sig <pubkey-file>` — verified BEFORE read_manifest and materialization: a failed signature leaves no corpus directory at all (pinned by the CLI test: wrong key → exit 2, corpus dir absent; right key installs; unsigned pack + no flag still installs, with a stderr nudge). - The load-bearing test: a self-consistent rebuild (records edited, SUMS rewritten honestly) passes read_manifest's checksums and FAILS the signature — exactly the gap M3's pack.rs test comment deferred to "a later milestone". - ring 0.17 API realities for the next person: from_seed_unchecked takes the RAW 32-byte seed (no pkcs8 DER needed), UnparsedPublicKey::verify takes plain &[u8] (there is no UnparsedSignature type), public_key() is on the signature::KeyPair trait. Trust model (docs/CORPUS_PACKS.md states it in full): the pubkey travels OUT OF BAND — committed at tools/packs/keys/rust-vulns.pub, never inside the pack (a key shipped beside its own signature verifies nothing). build ≠ publish: `corpus build` never signs; the seed exists only as the PACK_SIGNING_SEED CI secret (set 2026-09-27 from the keygen output; the local .key was deleted after the smoke run below). v1 is a single release key; the rotation procedure is written down in tools/packs/keys/README.md before it is needed. Publishing (.github/workflows/pack-publish.yml): daily 04:23 UTC + workflow_dispatch, concurrency-grouped (rolling tag, never two publishes at once), builds the pack from the committed recipe, signs with the secret, then VERIFIES ITS OWN OUTPUT against the committed .pub via the consumer's exact command before publishing to the rolling prerelease tag pack-rust-vulns (zip + loose SUMS + .sig + shields freshness json). A half-rotated key fails the build instead of shipping an unverifiable pack. Freshness is an operational promise the schedule keeps: the pack's own 30-day staleness refusal applies to our published corpus exactly as it does to a user's reference corpora. Verified locally, the full production loop with the shipped keypair: keygen → sign on the real rust-vulns pack (generation 2) → `corpus add --verify-sig` with the committed .pub installs 1,950 records → wrong .pub → exit 2, nothing materialized. Gates: rustfmt clean, clippy --lib --tests -D warnings clean, xerj-autoindex lib suite 1129 passed / 0 failed / 2 ignored, pack_claims 3/3 (the README numbers stay pinned to the measured pack-stats.json). ES-YAML untouched — no engine query path touched. Docs/GTM: docs/CORPUS_PACKS.md (format, trust model, honest limits — records-not-index, not-a-security-product, cron-lag freshness); rust-vulns README gets the workflow + freshness badges, the consume-the-signed-pack block, and the rotation section; SKILL.md gains --verify-sig; ROADMAP's corpus-hub item rolls to "built and signed" (still unbuilt: the hub, the pre-indexed half #1030); blog post landing/blog/one-vulnerability-three-names.html (+ blog index card, sitemap, pagedata entry) — every count in it is the committed measured pack-stats.json, and the prior-art signature table is phrased as our survey; CHANGELOG [Unreleased] Added entry citing #1046/#1047. Files: - engine/crates/xerj-autoindex/src/harvest/sign.rs (new; 5 unit tests) - engine/crates/xerj-autoindex/src/harvest/mod.rs (pipeline note) - engine/crates/xerj-autoindex/src/xc.rs (--verify-sig wiring + test) - engine/crates/xerj-autoindex/Cargo.toml, engine/Cargo.toml (ring) - .github/workflows/pack-publish.yml (new) - tools/packs/keys/{rust-vulns.pub,README.md} (new) - tools/packs/rust-vulns/README.md, tools/xerj-code/SKILL.md - docs/CORPUS_PACKS.md (new), ROADMAP.md, CHANGELOG.md - landing/blog/{one-vulnerability-three-names.html,index.html}, landing/sitemap.xml, scripts/seo/pagedata.py
The arc's [Unreleased] entry pointed forward to an unnamed M6 PR while that PR was being opened; it is #1048. One-line fixup on the same branch so the entry lands in main already accurate.
This was referenced Sep 27, 2026
xerj-org
added a commit
that referenced
this pull request
Sep 30, 2026
…d packs shipped in rc.78 Both agent-facing index files still said corpus packs were "planned, no code" (landing/llms.txt:50, and the NOT-IMPLEMENTED list in landing/llms-full.txt). False since v1.0.0-rc.78 (2026-09-29): PRs #1046/#1047 (builder + rust-vulns showcase) and #1048 (signing/publish), hardened by #1049-#1051, shipped `xerj corpus build` — a declarative recipe becomes a checksummed portable pack of records (format_version refusal, licence + provenance on every record), `corpus add --from` verifies per-file SHA-256 and `--verify-sig` a detached ed25519 signature over SHA256SUMS, and the first pack is rebuilt, signed and published daily as a dated GitHub Release. The correction claims exactly what shipped, no more: it says plainly that the PRE-INDEXED half (an already-indexed bundle you mount instead of spending your own CPU-hours) and a hub directory beyond the first pack remain unbuilt — matching ROADMAP.md's corpus-hub row, which was rolled in #1053. Edits follow each file's own precedent for status rolls so nothing is restructured: llms.txt rewrites the one clause in the not-implemented caveat using the same inline pattern the line already uses for the v1.0.0-rc.77 object-storage roll, and adds docs/CORPUS_PACKS.md to the "file that proves it" tail; llms-full.txt keeps the log-mode bullet, points it at a new "left this list in v1.0.0-rc.78" paragraph modelled on the object-storage one, and keeps the list's 2026-09-26 re-check date (the new paragraph carries its own 2026-09-29 date instead of claiming a full re-check) and its ~76-char wrap. Known lag, deliberately untouched here: the linked answer page landing/answers/does-xerj-support-s3-alerting-plugins.md (and its .html twin) still carries the old "no code exists" wording; it needs its own surgical PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the arc's last milestone: corpus packs go from checksummed to signed, and the rust-vulns pack goes from a local build to a scheduled, signed, rolling GitHub Release.
Stacked on #1047 (M4+M5, merged). The arc's earlier half is #1046 (M1–M3).
xerj corpus keygen --out <prefix>— ed25519 keypair. Writes<prefix>.key(32-byte hex seed, SECRET) +<prefix>.pub(verifying key). Creates the parent dir on first run.xerj corpus sign <pack-dir> --key <prefix>.key— raw 64-byte detached ed25519 over the pack'sSHA256SUMS, written asSHA256SUMS.sig. The sig is deliberately not listed in SUMS (signing a list containing the signature is circular).xerj corpus add <name> --from <pack> --verify-sig <pubkey>— origin check before anything is materialized. A pack whose checksums verify but whose origin does not is refused whole, exit 2, nothing written..github/workflows/pack-publish.yml— daily 04:23 UTC +workflow_dispatch: build → sign (seed fromPACK_SIGNING_SEEDsecret, materialized to$RUNER_TEMP, deleted after) → verify its own output against the committed.pub→ zip + loose SUMS/sig + freshness.json → rolling releasepack-rust-vulns.tools/packs/keys/rust-vulns.pub— the production verifying key, committed. The seed exists only as the CI secret; the local.keywas deleted after the secret was set.docs/CORPUS_PACKS.md— the pack format, trust model, and honest limits, end to end.one-vulnerability-three-names, sitemap, pagedata, fix_heads).ROADMAP.mdrecords-pack half updated: built and signed; what remains is the hub and the pre-indexed half (corpus/index: portable one-file bundles — indexed data that works across machines #1030).Why signatures at all — the attack
Checksums prove integrity, not origin. The attack they cannot stop: take a pack, edit records (flip a
fixed_versionsstring), recompute every checksum, rewriteSHA256SUMShonestly. The pack now verifies perfectly against its own manifest — every checksum agrees, because the checksums were never the thing being attacked.That exact scenario is pinned in a test (
sign.rs): a self-consistent rebuild passes every checksum check and fails the signature. That gap is the entire reason the pack carries one.Prior art (our 2026-09 survey): Trivy DB (OCI), Grype/vunnel feeds, PrimeVul (HF dataset), VulnRAG — none ships a consumer-verifiable signature. The lane is empty.
Trust model
corpus buildemits; it never signs. The seed exists only where signing happens (CI secret)..pubbefore publishing — a rotated key whose.pubforgot to be updated fails the build instead of shipping a pack no consumer can verify. Rotation procedure is written down (tools/packs/keys/README.md) before we need it.ring 0.17 realities (for reviewers)
Ed25519KeyPair::from_seed_unchecked(&[u8])— there is nofrom_pkcs8-only path for a bare seed.public_key()comes from thering::signature::KeyPairtrait, not an inherent method — the trait must be in scope.UnparsedPublicKey::new(&ED25519, bytes).verify(msg, &sig)— the signature is a plain&[u8]; there is noUnparsedSignaturetype at this version.ring::rand::SystemRandom::fill()via theSecureRandomtrait.Proven live before this PR
Production keypair generated with the shipped
keygen, used to sign the real rust-vulns pack (generation 2), then consumed viacorpus add --verify-sigwith the committed.pub: 1,950 records installed (rustsec 1,221 + osv 729). A wrong.pubexits 2 with "nothing materialized (correct)".PACK_SIGNING_SEEDset in CI from the.key; local.keydeleted immediately after.Files
engine/crates/xerj-autoindex/src/harvest/sign.rs(keygen/sign/verify + 5 tests),xc.rs(CLI wiring),.github/workflows/pack-publish.yml,tools/packs/keys/{rust-vulns.pub,README.md},docs/CORPUS_PACKS.md,tools/xerj-code/SKILL.md,CHANGELOG.md,ROADMAP.md, blog + sitemap + pagedata + fix_heads output.Gates
--lib --tests -D warningscleanpack_claims3/3 (README still pinned to the measuredpack-stats.json)