8000 docs(bench): map failure paths to correctness tests (Refs #843) by adityamukho · Pull Request #866 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Documentation

- **Failure-path correctness map** (#843): [X08's correctness-target map](docs/benchmarks/843-failure-correctness/README.md) ties invalid bounds, windows and binds, callback errors, exhaustion, corrupt pages, failed-handle retry, v1–v6 refusal and heavy single-entry WAL replay to 138 named tests at a pinned v3 revision, with their error and data invariants, and limits the C08 repair and C30 rollback claims to their fixtures. Gaps found: heavy WAL replay through `Minigraf::open` (#865) and the recursion round bound (#861). X08 stays excluded. No production, test, public API or nightly-route change.

- **Semantic permutation map** (#842): [X07's correctness-target map](docs/benchmarks/842-semantic-permutations/README.md) ties numeric, bind, cross-kind join and forced-planner permutations to 128 named tests at a pinned v3 revision, states what the representative nightly controls do and do not establish, and records the hash-scan and keyword binding/clone confounders. X07 stays excluded. No production, test, public API or nightly-route change.

- **Manual shape sweep** (#841): [native shape and retention experiments](docs/benchmarks/841-shape-sweeps/README.md) cover sort/window, set and UUID cases, matched recursive controls and sustained pins/FactLogs. The report distinguishes query timings from process RSS and query-meter observations, and retains input rejections, capped attempts and attribution limits. No production, public API or nightly-route change.
Expand Down
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,11 @@ ties the excluded numeric, bind, join and forced-plan permutations to 128
correctness tests and documents the nightly controls' attribution limits. It
changes no code, tests or nightly routes.

The [#843 failure-path map](docs/benchmarks/843-failure-correctness/README.md)
ties the failure paths excluded from nightly timing to 138 correctness tests
with their error and data invariants. It changes no code, tests or nightly
routes; the heavy WAL replay test gap is #865.

The [#841 manual shape sweep](docs/benchmarks/841-shape-sweeps/README.md)
records native sort/window, set, UUID, recursive-control and sustained-reader
experiments, including rejected inputs, capped attempts and attribution limits.
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,11 @@ The [semantic permutation map](docs/benchmarks/842-semantic-permutations/README.
correctness tests and states what the natural-plan nightly controls do not
isolate. It adds no nightly benchmark.

The [failure-path correctness map](docs/benchmarks/843-failure-correctness/README.md)
(#843) ties invalid inputs, callback errors, corruption, failed-handle refusal,
v1–v6 refusal and heavy WAL replay to the correctness tests that check their
error codes and data. It adds no nightly benchmark; a heavy-replay test gap is #865.

The [manual shape sweep](docs/benchmarks/841-shape-sweeps/README.md) (#841)
examines native sort/window, set and UUID shapes, recursive controls, and
sustained reader retention. The report separates query latency, process RSS,
Expand Down
1 change: 1 addition & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
8000 Expand Up @@ -134,6 +134,7 @@ Scope — release-gate testing and documentation:
- Nightly runtime expansion (#858): add Java/JVM and WASI; qualify Swift/iOS simulator and Android emulator timings against physical-device workloads before enabling representative mobile lanes. The platform restriction from #839 applies only to that manual evidence task.
- Manual migration evidence (#838): native v7-to-v8 through 200k facts; Chromium/Firefox/WebKit through 20k facts, with separate read-only/reopen controls and bounded post-migration cursor churn. [Results, frozen fixtures and exclusions](docs/benchmarks/838-migration/README.md); no new nightly route or sustained-pin bound.
- Semantic permutations (#842): [correctness-target map and control limits](docs/benchmarks/842-semantic-permutations/README.md) for X07. The exclusion stands; no nightly route or acceptance gate is added.
- Failure paths (#843): [correctness-target map](docs/benchmarks/843-failure-correctness/README.md) for X08. The exclusion stands; the heavy WAL replay test gap is #865.
- Manual shape evidence (#841): [native shape sweeps, recursive controls and sustained-reader experiments](docs/benchmarks/841-shape-sweeps/README.md), with explicit input, timeout and attribution limits. The small-nightly recommendation is separate from execution; this audit adds no nightly route or release-acceptance gate.
- Expression keyword/ref equality (#774; on `v3`): resolve literal keywords once per operator, with memory bounded by the expression. [Measurements and regression coverage](docs/benchmarks/774-expression-keywords.md); performance-only, with no v2.x backport.
- Higher storage/WAL coverage gates and branch-coverage gates (#396; on `v3`: line and branch floors per file in `coverage-thresholds.toml`, checked on every PR)
Expand Down
9 changes: 9 additions & 0 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

This document records reproducible local benchmark snapshots. Completed release-performance history belongs in the [CHANGELOG](../CHANGELOG.md); continuous CI history is available on [Bencher](https://bencher.dev/perf/minigraf/plots).

## Failure paths (#843, unreleased)

The [X08 map](benchmarks/843-failure-correctness/README.md) assigns invalid
bounds, windows and binds, callback errors, exhaustion, corrupt pages,
failed-handle retry, v1–v6 refusal and heavy single-entry WAL replay to
correctness tests rather than nightly IDs. The C08 rebuild and C30 rollback IDs
time successful work around one fixed failure; they are not failure-latency
measurements.

## Semantic permutations and control limits (#842, unreleased)

The [X07 map](benchmarks/842-semantic-permutations/README.md) assigns
Expand Down
6 changes: 6 additions & 0 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@ cleanup of all 46 synthetic resources. Fresh extraction reproduces the evidence
reports, and removing delivery/comparison evidence is rejected. No Rust tests
are added; the inventory above is unchanged. Additional runtime lanes are #858.

**#843 failure-path map**: [138 curated correctness tests](benchmarks/843-failure-correctness/README.md)
(35 cells; 5 tests exist only in the `test-hooks` build) cover the failure paths excluded from
nightly timing, with their error-code and data invariants. Gaps: no `Minigraf::open` replay of
one heavy WAL entry (#865) and no test of the fixed recursion round bound (planned in #861).
No test is added; the inventory above is unchanged.

**#842 semantic permutation map**: [128 curated correctness tests](benchmarks/842-semantic-permutations/README.md)
(130 targets in 29 cells; 17 tests exist only in the `test-hooks` build) cover the numeric, bind,
cross-kind join and forced-planner families excluded from nightly benchmarks.
Expand Down
6 changes: 3 additions & 3 deletions docs/bench-coverage-v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ selected IDs in Appendix A, not every registered case in a family.
| C05 | WAL append and explicit small transactions | `insert/{single_fact,batch_100,explicit_tx}/…`, `insert_file/{single_fact,batch_100,explicit_tx}/…` | M4/F4 plus pending writes; repeated benchmark triples | insert `^insert/`; insert_file `^insert_file/`; V+B | COVERED; chunked paths in C30 |
| C06 | Checkpointed v8 open and WAL v2 replay | `open/{checkpointed,wal_replay}/…` | F4 checkpointed; W 1k/10k/100k facts in 100-fact statements, read-write open/drop | open `^open/`; V+B | COVERED; million-fact single-entry replay EXCLUDED from latency timing; correctness-only (X08) |
| C07 | Native v7 migration and nonmigrating read-only v7 | `open/storage_v3/{migrate_v7,read_only_v7}/multi_checkpoint`; `storage_v3.rs` | Frozen v7 multi-checkpoint corpus, all manifest queries validated; copies outside timed open | open; V+B | COVERED small native representative; large/browser migration MANUAL (X03) |
| C08 | Integrity verification and index rebuild | `checkpoint/storage_v3/verify/100k_history`, `checkpoint/storage_v3/rebuild_damaged_avet/100k_history` | 100,150 stored/99,950 live facts; rebuild starts from CRC-damaged current AVET root, copied per iteration | checkpoint; V+B | COVERED; arbitrary failure-code latency EXCLUDED (X08) |
| C08 | Integrity verification and index rebuild | `checkpoint/storage_v3/verify/100k_history`, `checkpoint/storage_v3/rebuild_damaged_avet/100k_history` | 100,150 stored/99,950 live facts; rebuild starts from CRC-damaged current AVET root, copied per iteration; damage detection and clean repair checked once at setup ([#843](benchmarks/843-failure-correctness/README.md#c08-and-c30-claims)) | checkpoint; V+B | COVERED; arbitrary failure-code latency EXCLUDED (X08) |
| C09 | Committed net-assert/history, current and as-of | `point_query_chain_depth/{churned_attr,sibling_attr,attr_scan,churned_attr_as_of}/{1,2000}`; `history.rs` | 2,000 filler facts, churned triple and sibling; depth includes initial assertion; checkpointed | history_retract shortlist in Appendix A; V+B | COVERED eight IDs; intermediate depth 500 EXCLUDED as redundant (X01) |
| C10 | Retraction, last statement and window replacement | `retract/{single_fact,batch_100}/10k`, `insert_file/chunked/{retract,write_tx}/{65535,65537}` | M 10k repeated retracts; fresh file chunk controls include later-statement window replacement | history_retract; insert_file; V+B | COVERED; other retract scales EXCLUDED (X01) |
| C11 | Read-only open/query and in-memory WAL replay | `open/storage_v3/read_only_v8/10k`, `open/storage_v3/read_only_wal/10k_plus_1k`, C07; ordered query groups | F 10k with/without 1k WAL facts; source bytes checked unchanged; read-only query fixtures | open; query; V+B | COVERED representative native opens and reads |
Expand All @@ -228,7 +228,7 @@ selected IDs in Appendix A, not every registered case in a family.
| C27 | lag/lead | `window/neighbors/{lag,lead}/{one,hundred}/10k`, `window/neighbors/lag_lead/{one,hundred}/100k` | M 10k/100k rows, one/100 partitions, four-row peer groups and boundary nulls | window `^window/`; V+B | COVERED six IDs |
| C28 | Bounded rows and registered aggregate replay | 18 `window_rows/…` IDs; exact shortlist in Appendix A | M 10k/100k rows, 2n facts; sum zero/two/prefix one/100 partitions; min/max and bounded/full-prefix UDF controls | window_rows exact anchored shortlist in Appendix A; V+B | COVERED 18 IDs; remaining 110 MANUAL, not nightly (X01) |
| C29 | Set storage/queries/binds and collect-set | `{insert_file,checkpoint,query}/set_storage/…`, `query/set_queries/{in,not_in,all_in,any_none,collect}/{pending,committed}/10k` | 10k storage facts, 4-element inline/32-element long repeated/unique sets; 10k entities for pending/committed membership and empty collection; binds in C36 | insert_file; checkpoint; query; V+B | COVERED ten storage + ten query cases; host set binds in C36; cardinality sweeps MANUAL (X06) |
| C30 | Chunked execute/retract/write transactions/rollback | `insert_file/chunked/{execute,retract,write_tx}/{65535,65537}`, `insert_file/chunked/statement_rollback/65537`, C13 staged_read | Default 65,536-fact boundary, fresh file per iteration; two staged statements; late API-011 after first chunk, then valid commit | insert_file; query; V+B | COVERED seven chunk IDs and staged-read control; million-fact/scaling bounds ENVELOPE/acceptance (X02) |
| C30 | Chunked execute/retract/write transactions/rollback | `insert_file/chunked/{execute,retract,write_tx}/{65535,65537}`, `insert_file/chunked/statement_rollback/65537`, C13 staged_read | Default 65,536-fact boundary, fresh file per iteration; two staged statements; late API-011 after first chunk (asserted every iteration), then valid commit | insert_file; query; V+B | COVERED seven chunk IDs and staged-read control; million-fact/scaling bounds ENVELOPE/acceptance (X02) |
| C31 | FactLog storage/tx order, filters and bounded window | `fact_log/…`; `fact_log.rs` | 10k/100k historical records +100 pending; batch 1024, default/4096 window, attribute-only and entity/prefix/tx-range filters | fact_log `^(fact_log\|log_writer)/`; V+B | COVERED six IDs; long-held log pin/churn MANUAL (X06) |
| C32 | LogWriter history/holes/windows and finish | `log_writer/{append,finish}/{10k,100k}`; `fact_log.rs` | Four historical phases, tx holes, long strings/sets; 10k below and 100k above 65,536 writer batch; append and finish isolated | fact_log; V+B | COVERED four IDs; foreign conversions MANUAL (X04) |
| C33 | Fallible UDFs, registry snapshot churn and re-entry | `udf/{aggregate_sum_dispatch,predicate_filter_dispatch}/…`, `udf/v3/…`; `udf_v3.rs` | Matched 10k fallible/infallible successful callbacks; fixed 32-UDF registry with held cursor; 1k callbacks re-enter same database | udf `^udf/`; V+B | COVERED baseline four + six v3 IDs; deliberate callback error timing EXCLUDED (X08) |
Expand Down Expand Up @@ -394,7 +394,7 @@ shortlist. No manual timing or release sign-off is implied by this close-out.
| X05 | Unverified operational follow-ups | Accumulate numeric regression baselines for 244 native and 26 foreign IDs (all have stored v3 observations/model settings, but baseline null in V). Verify live failed-upload and repeated-alert notification delivery; successful uploads and parser/audit unit tests do not exercise it. Cron execution of the final routes is unverified here: V was a dispatch. Preserve/refresh evidence beyond the 30-day artifact retention |
| X06 | MANUAL / ENVELOPE scaling and isolation | Large 10m sorting/window/envelope sweeps, wider set cardinalities, sustained pin/quarantine/log-reader churn and sparse/repeated-ref UUID projection are outside V's fixed shortlists. [#841's report](benchmarks/841-shape-sweeps/README.md) records the finite public shape matrix, a capped-window retry, matched recursive controls with baseline path counters, 544-generation pin/FactLog churn and charged query memory, with every rejected, capped, limit-error and deferred slot; it establishes no universal latency or memory bound. The [protocol](benchmarks/841-shape-sweeps/PROTOCOL.md) distinguishes the finite selection from broader extensions. [#838](benchmarks/838-migration/README.md) measures only eight commits with one old cursor plus a short-lived cursor per commit. Recursive controls are end-to-end comparisons: counters show which baseline paths each case uses, not each path's share of time, and the ancestor alternative's 128-row cutoff remains unmeasured |
| X07 | EXCLUDED exhaustive semantics | All numeric, bind, cross-kind join and forced-planner permutations remain correctness tests. Nightly controls select representative natural plans; broader hash controls include broader scans and keyword dynamic controls include binding/clone work. [#842's map](benchmarks/842-semantic-permutations/README.md) pins 128 correctness tests (130 targets in 29 cells) (v3 5d85c36f, all run in CI's plain or `test-hooks` job), finds no missing answer contract, and records three attribution limits: no join-kind switch, probes indistinguishable by counter, and answer-only preflight outside the six join controls. The exclusion stands; no nightly ID changes |
| X08 | EXCLUDED individual failure latency | Invalid bounds/windows/binds, deliberate callback errors, corrupt-page rejection, v1–v6 refusal, heavy single-entry WAL replay and individual PRS/API/QRY codes belong to correctness testing. C08 measures a successful repair of known damage and C30 measures cross-chunk rollback; neither claims exhaustive failure timing |
| X08 | EXCLUDED individual failure latency | Invalid bounds/windows/binds, deliberate callback errors, corrupt-page rejection, v1–v6 refusal, heavy single-entry WAL replay and individual PRS/API/QRY codes belong to correctness testing. C08 measures a successful repair of known damage and C30 measures cross-chunk rollback; neither claims exhaustive failure timing. [#843's map](benchmarks/843-failure-correctness/README.md) pins 138 correctness tests in 35 cells (v3 a95e51ffe, all run in CI's plain or `test-hooks` job, two ignored ones nightly) with their error and data invariants. Gaps: no test replays one heavy WAL entry through `Minigraf::open` (#865); the fixed recursion round bound is untested (planned in #861). The exclusion stands; no nightly ID changes |
| X09 | EXCLUDED unshipped/non-runtime scope | Public profiler #185, OPFS #441 and rule persistence #241 are open/unshipped at the audited revision. Proof/test/docs infrastructure and #797's study add no shipped latency path; production uuid/dist size uses separate size gates. Re-audit new production features through the release cut |

Issue #776 remains open. Its final release decision still needs the explicit
Expand Down
Loading
Loading
0