adityamukho · Nov 5, 2026 · Nov 5, 2026
@@ -30,6 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Documentation
 
+- **Regression-history audit** (#840): [scheduled-route evidence and durable archive](docs/benchmarks/840-regression/README.md) verify main/v3 executed/uploaded sets. All 244 native and 26 binding comparison gaps remain; API evidence identifies v3 head replacement when main advances. Separate workflow fixes are proposed in #853/#854. Live notification checks remain open; no runtime or nightly workflow change in this audit.
+
 - **Manual migration sweep** (#838): [frozen fixtures, timings and correctness evidence](docs/benchmarks/838-migration/README.md) for native v7-to-v8 migration through 200k facts and Chromium/Firefox/WebKit through 20k facts. Includes nonmigrating native read-only open, v8 reopen and bounded post-migration cursor churn; larger browser cases and sustained pin scaling remain excluded. No production or nightly-route change.
 
 - **Benchmark exclusions audit** (#836): [pinned enumeration and per-shape dispositions](docs/benchmarks/836-exclusions/README.md) reconcile 156 unselected native IDs, retain 110 manual row-frame cases, and distinguish linear full-prefix UDF accumulation from bounded replay. No new latency measurements or runtime changes.

@@ -10,6 +10,13 @@ See `ROADMAP.md` for planned work and current direction. See `CHANGELOG.md` for
 
 The [#838 manual migration sweep](docs/benchmarks/838-migration/README.md) covers native v7 migration through 200k facts and Chromium/Firefox/WebKit through 20k facts, including bounded post-migration cursor churn. It changes neither nightly coverage nor the Rust test inventory; larger browser cases remain excluded.
 
+The [#840 regression-history audit](docs/benchmarks/840-regression/README.md)
+verifies actual main/v3 scheduled benchmark sets and archives their evidence.
+All 244 native and 26 binding comparison gaps remain: uploads rebase the shared
+v3 Bencher head when main advances. Separate workflow fixes are proposed in
+#853/#854; baseline accumulation and live notification delivery remain open.
+This audit changes no runtime behavior or Rust test counts.
+
 ## Core Philosophy - CRITICAL
 
 **Before implementing ANY feature or change, you MUST assess it against the design philosophy in PHILOSOPHY.md.**

@@ -36,6 +36,11 @@ The unreleased v3.0.0 branch adds bounded sliding windows such as `(avg ?salary
 
 The [manual v7-to-v8 migration sweep](docs/benchmarks/838-migration/README.md) (#838) measures native fixtures through 200k facts and browser fixtures through 20k facts, with separate read-only/reopen controls and bounded cursor churn. These measurements do not expand nightly coverage or establish cold-start and sustained-reader bounds.
 
+The [regression-history audit](docs/benchmarks/840-regression/README.md) (#840)
+verifies scheduled main/v3 benchmark execution and preserves the raw evidence.
+Numeric comparisons for 244 native and 26 binding IDs, and live notification
+delivery checks, remain release work; successful uploads alone do not close them.
+
 On v3.0.0, an ascending value-only query such as `[:find ?v :where [?e :score ?v] :order-by [?v] :limit 10]` can read directly in index order and stop early when all committed `:score` history has one value kind (integer, float, or Boolean) and its source range has no pending writes. A query whose full order cannot be proved uses the existing sort (#641). Built-in `count`, `count-distinct`, `sum`, and `avg` can release completed groups when the pinned source proves their grouping order (#781). `min`/`max` also qualify when their arguments are proved Integer/Float/Boolean/nil; `sum-distinct` requires Integer/nil. A single Integer/Boolean/Float key (including mixed numeric groups) is supported. A longer effective tuple, including hidden `:with` keys, must reduce to one varying Integer/Boolean slot plus repeated occurrences and proved Integer/Boolean/nil constants. Any requested final Sort is retained for `:with`. Small grouped point queries pay proof startup cost; see the [measurements](docs/benchmarks/781-ordered-aggregate.md). Memory follows the active group and batches; distinct members within one group still grow. A limit can stop after enough emitted groups, but dropped min/max groups do not count and a retained Sort drains the input. Callbacks, `collect-set`, aggregate+window queries and unproved arguments keep full-input evaluation. When every built-in window uses the same proved nonconstant partition key, the engine can also finish and release one partition at a time. Its memory follows the largest partition plus bounded batches; one growing partition still grows. Unpartitioned windows, incompatible keys, aggregate queries, callbacks and `collect-set` keep full-input evaluation. A surrounding sort or distinct operation can still require more memory.
 
 On the unreleased v3 branch, selective in-memory ancestor queries can use entity probes instead of reading the whole edge attribute (#723). This is a query-planning optimization; it adds no storage index or configuration.

@@ -130,6 +130,7 @@ Scope — release-gate testing and documentation:
 - Long-running soak test at scale (#392; done on `v3`: `tests/soak_test.rs`, 10M facts and a 5-hour churn weekly, alternating `main` and `v3`)
 - Benchmarks at 100K–10M facts and a published performance envelope (#394; on `v3`: `examples/perf_envelope.rs`, results in `docs/BENCHMARKS.md`; follow-ups #528–#532, with #532 superseded by #551's removal of repeated committed-fact net-assert; [audit and measurements](docs/benchmarks/532-net-assert-audit.md))
 - Acceptance evidence for memory/scaling (#837): named-test and Linux RSS gates, nonlinear/multistratum recursion and custom chunk bounds; [validation status](docs/benchmarks/837-acceptance-gaps.md)
+- Regression-history/reporting evidence (#840): [scheduled-route verification and durable archive](docs/benchmarks/840-regression/README.md). All 244 native and 26 binding comparison gaps remain; the v3 history-rebasing fix awaits separate review (#853/#854), then independent accumulation and live notification checks.
 - Manual migration evidence (#838): native v7-to-v8 through 200k facts; Chromium/Firefox/WebKit through 20k facts, with separate read-only/reopen controls and bounded post-migration cursor churn. [Results, frozen fixtures and exclusions](docs/benchmarks/838-migration/README.md); no new nightly route or sustained-pin bound.
 - Expression keyword/ref equality (#774; on `v3`): resolve literal keywords once per operator, with memory bounded by the expression. [Measurements and regression coverage](docs/benchmarks/774-expression-keywords.md); performance-only, with no v2.x backport.
 - Higher storage/WAL coverage gates and branch-coverage gates (#396; on `v3`: line and branch floors per file in `coverage-thresholds.toml`, checked on every PR)

@@ -18,6 +18,12 @@ The 2 golden-corpus and 7 migration-matrix tests pass, alongside five standalone
 Python evidence-integrity tests. These manual probes add no Rust tests to the
 inventory above and do not establish crash, cold-start or sustained-pin bounds.
 
+**#840 operational benchmark audit**: [offline verification of four Actions runs](benchmarks/840-regression/README.md)
+checks exact selected/executed/uploaded/API sets, hashes and 0.999/30 models.
+Twelve fixture-based notification-policy checks pass, but do not prove live
+delivery. The 244 native and 26 binding baseline gaps remain. No Rust tests are
+added and the inventory above is unchanged.
+
 **#794 verification (2026-10-30)**: one added `storage::keys` unit test checks that a set's element encoding equals its elements' encodings, ordered and deduplicated by a `BTreeSet`, over 2,000 random sets of up to about 60 elements with duplicates, signed zeros, NaNs and strings up to 300 bytes, in shuffled, sorted and reversed order. It passes with both the old stable sort and the new unstable sort. The full default suite passed before merging #795 (2,038 active, 19 ignored). CI-equivalent native, browser and WASI clippy pass on Rust 1.99.1, and Rust 1.89 checks all features. The release binary is 1,260,568 bytes (3,024 smaller), below the unchanged 1,263,616-byte cap.
 
 **#781 verification (2026-10-29)**: 24 active tests and two ignored measurement

@@ -1,5 +1,12 @@
 # v3 nightly benchmark coverage close-out (#776)
 
+Follow-up on 2026-11-05: the [#840 operational audit](benchmarks/840-regression/README.md)
+verifies actual final main/v3 scheduled routes and archives their evidence beyond
+30 days. The same 244 native and 26 binding baselines remain null: native uploads
+replace the shared v3 Bencher head when main advances. Separate fixes #853/#854
+await review; independent baseline accumulation and live notification checks
+remain open. The original validation snapshot below is retained unchanged.
+
 Audit date: 2026-11-03. P01–P17 are merged. This closes out the implementation
 plan and gives every matrix row a disposition; it does **not** close
 [issue #776](https://github.com/project-minigraf/minigraf/issues/776) or certify

@@ -0,0 +1,2 @@
+/evidence/
+__pycache__/
@@ -0,0 +1,158 @@
+# Regression history and scheduled reporting audit
+
+Audit date: 2026-11-05. [Issue #840](https://github.com/project-minigraf/minigraf/issues/840)
+remains open. Actual main/v3 scheduled routes are verified, but the same **244
+native and 26 binding IDs still lack numeric comparisons**. Native uploads
+rebase the shared v3 Bencher head when main advances, preventing those histories
+from accumulating reliably. The proposed fix has separate draft reviews for
+[main #853](https://github.com/project-minigraf/minigraf/pull/853) and
+[v3 #854](https://github.com/project-minigraf/minigraf/pull/854).
+
+This is operational evidence under PHILOSOPHY.md's reliability and documentation
+principles. No benchmark was built, enumerated or timed locally. The observations
+come from existing Actions runs. The proposed workflow fix has not been merged;
+no live notification has been sent by this audit.
+
+## Scheduled routes and exact coverage
+
+| Run | Event | Route | Native suites / IDs | Binding suites / IDs | Numeric native / binding comparisons |
+| --- | --- | --- | --- | --- | --- |
+| [41642356471](https://github.com/project-minigraf/minigraf/actions/runs/41642356471) | workflow_dispatch | v3 | 26 / 427 | 4 / 26 | 183 / 0 |
+| [41711848743](https://github.com/project-minigraf/minigraf/actions/runs/41711848743) | schedule | v3 | 26 / 427 | 4 / 26 | 183 / 0 |
+| [41801278030](https://github.com/project-minigraf/minigraf/actions/runs/41801278030) | schedule | main | 21 / 177 | 0 / 0 | 177 / 0 |
+| [41908407297](https://github.com/project-minigraf/minigraf/actions/runs/41908407297) | schedule | v3 | 26 / 427 | 4 / 26 | 183 / 0 |
+
+All four runs succeeded. Their nonempty native selections are disjoint, and
+selected = raw executed = raw uploaded = API report IDs in every enabled suite.
+The v3 binding jobs cover browser (8), C (6), Node (6) and Python (6), with their
+separate `binding-<runtime>-ubuntu24-v1` testbeds. Native uses `ubuntu-latest`.
+The complete per-ID values, comparison limits, suite, testbed, report UUID,
+head UUID and measured SHA are in [inventory.json](inventory.json).
+[summary.json](summary.json) retains per-suite counts and binding source hashes.
+
+| Run | Measured core SHA | Workflow SHA | Created UTC |
+| --- | --- | --- | --- |
+| 41642356471 | `0394c90ad90e9b95d416f4a0ec4a4c78b16e5b57` | `4d65022ac963a10b582ac762c4ab2e73935c9115` | Nov 3 13:13:01 |
+| 41711848743 | `c5863c79e4039e54c7fa67e0d7a8b8751f9b1599` | `4d65022ac963a10b582ac762c4ab2e73935c9115` | Nov 3 20:21:41 |
+| 41801278030 | `e7923eb710b1e1d8aff517721643c0b0b1f2cb7a` | `e7923eb710b1e1d8aff517721643c0b0b1f2cb7a` | Nov 4 08:40:10 |
+| 41908407297 | `33e5dffadebe0124e6287fc91cede4ff2b7a2b2f` | `e7923eb710b1e1d8aff517721643c0b0b1f2cb7a` | Nov 4 20:37:44 |
+
+The two archived workflow revisions are byte-identical for `bench.yml`. They
+map main to `0 2 * * *` and v3 to `0 14 * * *`, and the event, run title and
+retained measured-ref reports agree. Creation times show delayed execution;
+this verifies the route, not punctual execution at the nominal cron time.
+The Actions REST metadata does not include the original event JSON.
+
+## Why the missing baselines have not accumulated
+
+Every native v3 upload supplies `--start-point main
+--start-point-clone-thresholds`. Bencher creates a new branch head when the
+start point changes. That behavior is documented in
+[Branches and Start Points](https://bencher.dev/docs/explanation/branches/).
+Bindings omit the start-point options, but share branch `v3`: changing its head
+affects their testbeds too.
+
+The retained public report responses establish the change directly:
+
+| v3 head | Main start-point hash | Created UTC | Replaced UTC |
+| --- | --- | --- | --- |
+| `01a18921-537c-719a-9b63-5f2b0506420a` | `4a77092aa8baffee58a8cf57f2d7c8805abf2d9c` | Oct 29 19:27:05 | Nov 3 13:18:17 |
+| `01a1a18f-7bfd-70cf-b3f9-61d5e0a8b037` | `c72d9d1f3a24bcd41d568aa6f06765b23b0f74c2` | Nov 3 13:18:17 | Nov 4 20:42:55 |
+| `01a1a84c-e947-745c-98e7-018e0c2245a3` | `e7923eb710b1e1d8aff517721643c0b0b1f2cb7a` | Nov 4 20:42:55 | active at collection |
+
+The original validation dispatch split 2/28 reports between the first two heads.
+The Nov 4 v3 schedule split 1/29 reports between the last two. Its C report,
+`01a1a84b-b12d-7303-8ee2-da6a1a73a533`, was accepted at 20:41:34, before the head
+changed at 20:42:55. Its six observations remain in the old report/head; they
+are not missing uploads.
+
+The active-head `/perf` API was queried for all 270 affected IDs from epoch zero
+through the recorded collection cutoff, in batches of at most eight benchmark
+UUIDs as required by the [Perf API](https://bencher.dev/docs/api/projects/perf/).
+Results: 244 native and 20 browser/Node/Python series contain exactly one
+observation each, all belonging to the latest v3 run. The six C series have no
+observation on this active head. See
+[active-head-summary.json](active-head-summary.json) and the raw queries/responses
+in the archive. No inherited main observation is counted as an independent v3
+run. The original 270-ID inventory and latest null comparison values are in
+[missing-baselines.json](missing-baselines.json).
+
+All 1,536 observations across the four audited runs use latency t-test models
+with upper probability **0.999** and maximum sample size **30**. A usable
+comparison here requires finite numeric `baseline` and `upper_limit`, not merely
+a stored model. The 0.999 parameter is a statistical probability, not a latency
+percentage budget; 30 is a maximum sample count, not proof of 30 independent
+runs. See [Thresholds and Alerts](https://bencher.dev/docs/explanation/thresholds/).
+
+The proposed fix stops main inheritance for persistent main/v3 histories and
+preserves it for ad hoc comparison branches. Its test executes the actual
+workflow upload shell with a capture-only Bencher function: v3 failed before the
+fix, and main/v3/ad hoc cases pass afterward. Both existing reporting test files
+also pass. This does not restore old heads or prove future accumulation. After
+review and deployment, collect fresh independent runs until every intended ID
+has a finite baseline and upper limit on the correct active head/testbed.
+
+## Notification evidence and remaining live work
+
+All 30 `Report regression` steps were skipped in each inspected run, including
+the disabled v3-only jobs on main. Therefore successful uploads prove neither
+failed-upload notification nor repeated-alert delivery.
+
+[notification-preview.cjs](notification-preview.cjs) executes both archived,
+unmodified notification script bodies offline. Twelve checks cover immediate
+failed-upload reporting, a repeated same-ID alert, different ID/branch/suite
+suppression, and previous-log lookup failure. They pass; the latter warns and
+uses an empty prior-alert set. [notification-preview.json](notification-preview.json)
+contains the exact preview messages and simulated calls. This is policy
+verification with fixtures, **not live end-to-end evidence**.
+
+The [controlled live check protocol](live-checks.md) scopes the remaining test
+history, messages, assertions and cleanup. It requires actual GitHub issue/comment
+delivery and real prior-run log retrieval. Posting those notifications and
+merging the separately reviewed production fixes remain pending authorization.
+
+## Durable evidence and reproduction
+
+[evidence.tar.gz](evidence.tar.gz) preserves all four runs' retained artifact
+contents: native enumeration/selection, raw Criterion logs and Bencher output,
+binding output/metrics/source hashes, report JSON, complete paginated job and
+artifact metadata, run metadata, 111 public Bencher reports, and 35 active-head
+API query responses. It also contains the two executed workflow revisions.
+This Git archive is independent of Actions' 30-day retention. The adjacent
+[SHA256SUMS](SHA256SUMS) checks the archive and analysis files; the archive's
+`evidence/SHA256SUMS` checks every individual evidence file.
+
+From this directory:
+
+```sh
+sha256sum -c SHA256SUMS
+tar -xzf evidence.tar.gz
+(cd evidence && sha256sum -c SHA256SUMS)
+python3 audit.py
+node notification-preview.cjs
+```
+
+`audit.py` independently compares raw execution/upload IDs with selection and
+API records, rejects duplicates and overlapping suites, verifies hashes/models,
+and regenerates all four inventory/summary files. It also verifies the active
+head's one-observation and absent-C findings. It is a frozen-audit verifier;
+its fixed expected results must be reviewed when collecting newer runs.
+
+For a network refresh, download each run with `gh run download RUN --repo
+project-minigraf/minigraf --pattern 'bench-evidence-*' --pattern
+'binding-evidence-*' --pattern bench-enumeration --dir evidence/RUN`, then run
+`python3 collect.py`. The collector uses read-only GitHub and public Bencher
+requests. Preserve the existing archive before refreshing; a newer active head
+or additional observations can legitimately invalidate this frozen audit's
+assertions. Repack and regenerate checksums only after reviewing the new results.
+
+## Acceptance status
+
+| Requirement | Status |
+| --- | --- |
+| Inventory original 244 native / 26 foreign gaps | Complete; exact IDs and API values retained |
+| Usable independent v3 comparisons for every intended ID | **Open**; 270 remain without comparisons; review/deploy the history fix and accumulate fresh runs |
+| Confirm 0.999/30 settings and separate testbeds | Complete for every observation in the four audited runs |
+| Observe final main/v3 scheduled routes and exact sets | Complete for the workflow revisions and measured SHAs above |
+| Live failed-upload and repeated-alert delivery, scoping and cleanup | **Open**; offline previews are not delivery evidence |
+| Preserve complete evidence beyond artifact expiry | Complete for this snapshot in the Git archive; future runs still need collection |
@@ -0,0 +1,13 @@
+c623b8b7d4c1591427747a620c2ec8276212a95a4b9e85a62f83dd17f38ff3b8  .gitignore
+f0e9c92691729f598785e5d6abf05ff509180c62e04ae8c620d1e66571626a81  README.md
+c0883999ce578ada4818280db12e6a8ef80f402aac6c0eda310ccf6b971bd4a0  active-head-summary.json
+f4a21622ccfdb56aa30ecb91c07df14e8a578c23872234ad24b740e104adfb39  audit.py
+9c486d21e2301f47bc25309259916d8bc93803a6b21227fcd4e77ea281f0d400  collect.py
+9151da470a9aaefae263534cd8b2aefac4efff3363841f6e063ef490644b880a  evidence.tar.gz
+3b2e46689dd84b00e52479cda19b773101dd2bcb7f17ce0e53f6932f658eb386  inventory.json
+5aaef5dfc1166053dc09df5db17dd31ac53e4e1a2293306813d68cf5018ed9d8  live-checks.md
+c7026121a1c294999504f4356cd1480a520fa96b329d7417414a03876cdab35f  missing-baselines.json
+2717e5411af7c9fbd139406a311fb1732f4e1b3e2585005c497634078015bb63  notification-preview.cjs
+01e4b1cbd805b0ae24c0b9cae1d5d8a4a859ed65252ebc1828119d2ee868797c  notification-preview.json
+b0f545189c37135338b29ff1e1c509aaaf9b5dda27c296511a6100e1d02fc3bf  summary.json
+59e0e8685d879ba82731f46ef2466cc6da7e466254eaa881105e94bd26862f70  verification.txt