8000 docs: complete regression-history and live reporting verification (#840) by adityamukho · Pull Request #859 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Documentation

- **Regression-history audit** (#840): [scheduled-route evidence and durable archive](docs/benchmarks/840-regression/README.md) verify main/v3 executed/uploaded sets. All 244 native and 26 binding comparison gaps remain; API evidence identifies v3 head replacement when main advances. Separate workflow fixes are proposed in #853/#854. Live notification checks remain open; no runtime or nightly workflow change in this audit.
- **Regression history and reporting verification** (#840): [durable API/run evidence](docs/benchmarks/840-regression/completion.md) establishes numeric comparisons for all 427 native and 26 binding IDs with at least three independent v3 runs each. Separate main/v3 fixes #853/#854 preserve v3 history across measured main advances. Actual scheduled routes, 16 isolated live notification scenarios, four delivered messages and complete synthetic-resource cleanup are verified. This adds no database runtime behavior or Rust tests; further runtime lanes are tracked in #858.

- **Manual migration sweep** (#838): [frozen fixtures, timings and correctness evidence](docs/benchmarks/838-migration/README.md) for native v7-to-v8 migration through 200k facts and Chromium/Firefox/WebKit through 20k facts. Includes nonmigrating native read-only open, v8 reopen and bounded post-migration cursor churn; larger browser cases and sustained pin scaling remain excluded. No production or nightly-route change.

Expand Down
13 changes: 7 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@ See `ROADMAP.md` for planned work and current direction. See `CHANGELOG.md` for

The [#838 manual migration sweep](docs/benchmarks/838-migration/README.md) covers native v7 migration through 200k facts and Chromium/Firefox/WebKit through 20k facts, including bounded post-migration cursor churn. It changes neither nightly coverage nor the Rust test inventory; larger browser cases remain excluded.

The [#840 regression-history audit](docs/benchmarks/840-regression/README.md)
verifies actual main/v3 scheduled benchmark sets and archives their evidence.
All 244 native and 26 binding comparison gaps remain: uploads rebase the shared
v3 Bencher head when main advances. Separate workflow fixes are proposed in
#853/#854; baseline accumulation and live notification delivery remain open.
This audit changes no runtime behavior or Rust test counts.
The [#840 regression-history follow-up](docs/benchmarks/840-regression/completion.md)
verifies numeric comparisons for all 427 native and 26 binding IDs, at least three
independent v3 runs per ID, preserved history after main advances, actual scheduled
routes and isolated live notification delivery/cleanup. Fixes #853/#854 are merged;
checksummed evidence outlives Actions retention. This changes no database runtime
behavior or Rust test counts. Additional JVM/WASI and qualified mobile nightly
lanes are tracked separately in #858.

## Core Philosophy - CRITICAL

Expand Down
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,11 @@ The unreleased v3.0.0 branch adds bounded sliding windows such as `(avg ?salary

The [manual v7-to-v8 migration sweep](docs/benchmarks/838-migration/README.md) (#838) measures native fixtures through 200k facts and browser fixtures through 20k facts, with separate read-only/reopen controls and bounded cursor churn. These measurements do not expand nightly coverage or establish cold-start and sustained-reader bounds.

The [regression-history audit](docs/benchmarks/840-regression/README.md) (#840)
verifies scheduled main/v3 benchmark execution and preserves the raw evidence.
Numeric comparisons for 244 native and 26 binding IDs, and live notification
delivery checks, remain release work; successful uploads alone do not close them.
The [regression-history follow-up](docs/benchmarks/840-regression/completion.md) (#840)
verifies numeric comparisons for all 427 native and 26 binding IDs, independent v3
history, scheduled main/v3 routes and isolated live notification delivery with
cleanup. Raw evidence and offline verifiers are retained in Git. JVM/WASI and
qualified mobile nightly expansion are tracked separately in #858.

On v3.0.0, an ascending value-only query such as `[:find ?v :where [?e :score ?v] :order-by [?v] :limit 10]` can read directly in index order and stop early when all committed `:score` history has one value kind (integer, float, or Boolean) and its source range has no pending writes. A query whose full order cannot be proved uses the existing sort (#641). Built-in `count`, `count-distinct`, `sum`, and `avg` can release completed groups when the pinned source proves their grouping order (#781). `min`/`max` also qualify when their arguments are proved Integer/Float/Boolean/nil; `sum-distinct` requires Integer/nil. A single Integer/Boolean/Float key (including mixed numeric groups) is supported. A longer effective tuple, including hidden `:with` keys, must reduce to one varying Integer/Boolean slot plus repeated occurrences and proved Integer/Boolean/nil constants. Any requested final Sort is retained for `:with`. Small grouped point queries pay proof startup cost; see the [measurements](docs/benchmarks/781-ordered-aggregate.md). Memory follows the active group and batches; distinct members within one group still grow. A limit can stop after enough emitted groups, but dropped min/max groups do not count and a retained Sort drains the input. Callbacks, `collect-set`, aggregate+window queries and unproved arguments keep full-input evaluation. When every built-in window uses the same proved nonconstant partition key, the engine can also finish and release one partition at a time. Its memory follows the largest partition plus bounded batches; one growing partition still grows. Unpartitioned windows, incompatible keys, aggregate queries, callbacks and `collect-set` keep full-input evaluation. A surrounding sort or distinct operation can still require more memory.

Expand Down
3 changes: 2 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,8 @@ Scope — release-gate testing and documentation:
- Long-running soak test at scale (#392; done on `v3`: `tests/soak_test.rs`, 10M facts and a 5-hour churn weekly, alternating `main` and `v3`)
- Benchmarks at 100K–10M facts and a published performance envelope (#394; on `v3`: `examples/perf_envelope.rs`, results in `docs/BENCHMARKS.md`; follow-ups #528–#532, with #532 superseded by #551's removal of repeated committed-fact net-assert; [audit and measurements](docs/benchmarks/532-net-assert-audit.md))
- Acceptance evidence for memory/scaling (#837): named-test and Linux RSS gates, nonlinear/multistratum recursion and custom chunk bounds; [validation status](docs/benchmarks/837-acceptance-gaps.md)
- Regression-history/reporting evidence (#840): [scheduled-route verification and durable archive](docs/benchmarks/840-regression/README.md). All 244 native and 26 binding comparison gaps remain; the v3 history-rebasing fix awaits separate review (#853/#854), then independent accumulation and live notification checks.
- Regression-history/reporting evidence (#840) is complete: [all 453 numeric comparisons, preserved independent v3 history, scheduled routes and live notification/cleanup evidence](docs/benchmarks/840-regression/completion.md). Separate main/v3 fixes #853/#854 are merged; raw evidence is retained beyond artifact expiry.
- Nightly runtime expansion (#858): add Java/JVM and WASI; qualify Swift/iOS simulator and Android emulator timings against physical-device workloads before enabling representative mobile lanes. The platform restriction from #839 applies only to that manual evidence task.
- Manual migration evidence (#838): native v7-to-v8 through 200k facts; Chromium/Firefox/WebKit through 20k facts, with separate read-only/reopen controls and bounded post-migration cursor churn. [Results, frozen fixtures and exclusions](docs/benchmarks/838-migration/README.md); no new nightly route or sustained-pin bound.
- Expression keyword/ref equality (#774; on `v3`): resolve literal keywords once per operator, with memory bounded by the expression. [Measurements and regression coverage](docs/benchmarks/774-expression-keywords.md); performance-only, with no v2.x backport.
- Higher storage/WAL coverage gates and branch-coverage gates (#396; on `v3`: line and branch floors per file in `coverage-thresholds.toml`, checked on every PR)
Expand Down
13 changes: 8 additions & 5 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,14 @@ The 2 golden-corpus and 7 migration-matrix tests pass, alongside five standalone
Python evidence-integrity tests. These manual probes add no Rust tests to the
inventory above and do not establish crash, cold-start or sustained-pin bounds.

**#840 operational benchmark audit**: [offline verification of four Actions runs](benchmarks/840-regression/README.md)
checks exact selected/executed/uploaded/API sets, hashes and 0.999/30 models.
Twelve fixture-based notification-policy checks pass, but do not prove live
delivery. The 244 native and 26 binding baseline gaps remain. No Rust tests are
added and the inventory above is unchanged.
**#840 operational benchmark verification**: [the completed follow-up](benchmarks/840-regression/completion.md)
checks all 453 numeric comparisons, at least three independent v3 runs per ID,
exact selected/executed/uploaded/API sets, source hashes and 0.999/30 models.
The initial four-run schedule archive and five new measurement runs are retained.
Sixteen isolated live notification scenarios verify four actual deliveries and
cleanup of all 46 synthetic resources. Fresh extraction reproduces the evidence
reports, and removing delivery/comparison evidence is rejected. No Rust tests
are added; the inventory above is unchanged. Additional runtime lanes are #858.

**#794 veri 8 fication 10BC8 (2026-10-30)**: one added `storage::keys` unit test checks that a set's element encoding equals its elements' encodings, ordered and deduplicated by a `BTreeSet`, over 2,000 random sets of up to about 60 elements with duplicates, signed zeros, NaNs and strings up to 300 bytes, in shuffled, sorted and reversed order. It passes with both the old stable sort and the new unstable sort. The full default suite passed before merging #795 (2,038 active, 19 ignored). CI-equivalent native, browser and WASI clippy pass on Rust 1.99.1, and Rust 1.89 checks all features. The release binary is 1,260,568 bytes (3,024 smaller), below the unchanged 1,263,616-byte cap.

Expand Down
14 changes: 8 additions & 6 deletions docs/bench-coverage-v3.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
# v3 nightly benchmark coverage close-out (#776)

Follow-up on 2026-11-05: the [#840 operational audit](benchmarks/840-regression/README.md)
verifies actual final main/v3 scheduled routes and archives their evidence beyond
30 days. The same 244 native and 26 binding baselines remain null: native uploads
replace the shared v3 Bencher head when main advances. Separate fixes #853/#854
await review; independent baseline accumulation and live notification checks
remain open. The original validation snapshot below is retained unchanged.
Follow-up on 2026-11-05: [#840's completed verification](benchmarks/840-regression/completion.md)
establishes numeric comparisons for all 427 native and 26 binding IDs with at
least three independent v3 runs each. Merged fixes #853/#854 preserve the v3 head
after a measured main advance. Actual scheduled routes, isolated live notification
delivery and cleanup are verified; complete evidence is archived beyond 30 days.
JVM/WASI and qualified iOS-simulator/Android-emulator expansion are tracked in
#858. The original validation snapshot below is retained unchanged; its null
baselines and X05 open status describe that earlier snapshot.

Audit date: 2026-11-03. P01–P17 are merged. This closes out the implementation
plan and gives every matrix row a disposition; it does **not** close
Expand Down
2 changes: 2 additions & 0 deletions docs/benchmarks/840-regression/.gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
/evidence/
__pycache__/
/live-evidence/
/completion-evidence/
21 changes: 13 additions & 8 deletions docs/benchmarks/840-regression/README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,22 @@
# Regression history and scheduled reporting audit

Audit date: 2026-11-05. [Issue #840](https://github.com/project-minigraf/minigraf/issues/840)
remains open. Actual main/v3 scheduled routes are verified, but the same **244
native and 26 binding IDs still lack numeric comparisons**. Native uploads
rebase the shared v3 Bencher head when main advances, preventing those histories
from accumulating reliably. The proposed fix has separate draft reviews for
Initial snapshot: 2026-11-05, before the history fixes were merged. The
[follow-up report](completion.md) records subsequent real measurements, live
notification delivery and cleanup. The findings and acceptance table below are
retained as the initial state, not a current status report for
[issue #840](https://github.com/project-minigraf/minigraf/issues/840).

At this snapshot, actual main/v3 scheduled routes were verified, but **244
native and 26 binding IDs still lacked numeric comparisons**. Native uploads
rebased the shared v3 Bencher head when main advanced, preventing those histories
from accumulating reliably. Separate fixes were proposed in
[main #853](https://github.com/project-minigraf/minigraf/pull/853) and
[v3 #854](https://github.com/project-minigraf/minigraf/pull/854).

This is operational evidence under PHILOSOPHY.md's reliability and documentation
principles. No benchmark was built, enumerated or timed locally. The observations
come from existing Actions runs. The proposed workflow fix has not been merged;
no live notification has been sent by this audit.
come from existing Actions runs. The fixes and live notifications had not yet
been executed at the time of this initial snapshot.

## Scheduled routes and exact coverage

Expand Down Expand Up @@ -146,7 +151,7 @@ requests. Preserve the existing archive before refreshing; a newer active head
or additional observations can legitimately invalidate this frozen audit's
assertions. Repack and regenerate checksums only after reviewing the new results.

## Acceptance status
## Acceptance status at the initial snapshot

| Requirement | Status |
| --- | --- |
Expand Down
20 changes: 16 additions & 4 deletions docs/benchmarks/840-regression/SHA256SUMS
Original file line number Diff line number Diff line change
@@ -1,13 +1,25 @@
c623b8b7d4c1591427747a620c2ec8276212a95a4b9e85a62f83dd17f38ff3b8 .gitignore
f0e9c92691729f598785e5d6abf05ff509180c62e04ae8c620d1e66571626a81 README.md
18a2d252eba0e733230ca84d2a83e319a362be15eaa06b35a119dc4aeb8f570a .gitignore
1001894e5dc6edbf9ce3b3b0c940e718fcbca33254b0c3d07179e23ff87eb059 README.md
c0883999ce578ada4818280db12e6a8ef80f402aac6c0eda310ccf6b971bd4a0 active-head-summary.json
f4a21622ccfdb56aa30ecb91c07df14e8a578c23872234ad24b740e104adfb39 audit.py
52a31196b90142b1611543b8cd2ad6b5ab7647ed03c4a6f8478048b8963fcf1c archive-evidence.py
9d4db25ce636ab7c5edb7050706cdea4aab0a74257f5e24b9e151f4982d3b882 audit.py
584ae79c5319d545fcedb6e93f3cd1f59b49e1e33d942cd60172f6d9d79ac6bb collect-completion.py
9c486d21e2301f47bc25309259916d8bc93803a6b21227fcd4e77ea281f0d400 collect.py
67b13cb713cf3e7a53ce2ab7f1d502a0be2037fc76dfd41edb4c21056100562d comparison-inventory.json
7d647fca38cbf2449061b5e05bca67cd32696dcce1556c2179785d025ff84134 completion-summary.json
fa78970117547ae11538bcf362453acdda49a95e0ae700a6fbaf4b805c67b518 completion-verification.txt
004b7b68d9b625da260934d4bf384b869d7c9e7bb5f454c68a60f8889f30a0cd completion.md
d35246a3b3783b8cb7522847169299e534c33a3296175e7e87c46dcbc1639168 completion.tar.gz
9151da470a9aaefae263534cd8b2aefac4efff3363841f6e063ef490644b880a evidence.tar.gz
3b2e46689dd84b00e52479cda19b773101dd2bcb7f17ce0e53f6932f658eb386 inventory.json
5aaef5dfc1166053dc09df5db17dd31ac53e4e1a2293306813d68cf5018ed9d8 live-checks.md
24fc5ef735cda3ff2e7c1bdbea4597745ff7cf0ac2148c2afb255c31cf23f31c live-checks.md
3ec9c24bd9023471adc2af0fdb919cf0e18d572699014352b3466a39eeeb8deb live-notifications.tar.gz
2fa1462c2b6b6b064e52d1ba89e97481d698793df3824ae94e8fa2eefd905643 live-summary.json
dd4bdda3955548d891b699c07989efa1c83b2f5fe47e48a6620024349ec32064 live-verification.txt
c7026121a1c294999504f4356cd1480a520fa96b329d7417414a03876cdab35f missing-baselines.json
2717e5411af7c9fbd139406a311fb1732f4e1b3e2585005c497634078015bb63 notification-preview.cjs
01e4b1cbd805b0ae24c0b9cae1d5d8a4a859ed65252ebc1828119d2ee868797c notification-preview.json
b0f545189c37135338b29ff1e1c509aaaf9b5dda27c296511a6100e1d02fc3bf summary.json
59e0e8685d879ba82731f46ef2466cc6da7e466254eaa881105e94bd26862f70 verification.txt
49fdc560d01c4b69c7ee8ecb664e5433cc105146f3f3659b1ce1dcc7f2ea5f5c verify-completion.py
85ff1dd61608d395b02a8f5a1fd095778aaf71f60fa0bfd391684f9dd53482d4 verify-live.py
31 changes: 31 additions & 0 deletions docs/benchmarks/840-regression/archive-evidence.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/usr/bin/env python3
"""Deterministically package one #840 follow-up evidence directory."""
import gzip
import hashlib
from pathlib import Path
import sys
import tarfile

root = Path(__file__).resolve().parent
name = sys.argv[1]
output = {'live-evidence': 'live-notifications.tar.gz', 'completion-evidence': 'completion.tar.gz'}[name]
source = root / name
files = sorted(p for p in source.rglob('*') if p.is_file() and p.name != 'SHA256SUMS')
assert files and all(not p.is_symlink() for p in files)
(source / 'SHA256SUMS').write_text(''.join(
hashlib.sha256(p.read_bytes()).hexdigest() + ' ' + str(p.relative_to(source)) + '\n' for p in files))
with (root / output).open('wb') as raw:
with gzip.GzipFile(filename='', fileobj=raw, mode='wb', mtime=0) as compressed:
with tarfile.open(fileobj=compressed, mode='w') as archive:
for p in sorted(source.rglob('*')):
if p.is_file():
info = archive.gettarinfo(str(p), arcname=str(p.relative_to(root)))
info.uid = info.gid = 0
info.uname = info.gname = ''
info.mtime = 0
with p.open('rb') as stream:
archive.addfile(info, stream)
files = sorted(p for p in root.iterdir() if p.is_file() and p.name != 'SHA256SUMS')
(root / 'SHA256SUMS').write_text(''.join(
hashlib.sha256(p.read_bytes()).hexdigest() + ' ' + p.name + '\n' for p in files))
print(output, (root / output).stat().st_size, 'bytes')
3 changes: 3 additions & 0 deletions docs/benchmarks/840-regression/audit.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@
import re
from urllib.parse import parse_qs, urlparse

if not __debug__:
raise RuntimeError('Evidence verification requires Python assertions; do not use -O')

ROOT = Path(__file__).resolve().parent
RUNS = (41642356471, 41711848743, 41801278030, 41908407297)

Expand Down
Loading
Loading
0