adityamukho · Oct 8, 2026 · Oct 8, 2026 · Oct 8, 2026
@@ -1,5 +1,13 @@
 name: SIGKILL crash-safety fuzz (nightly)
 
+# Two branches alternate every 12 hours: main at 03:17 UTC, v3 at 15:17 UTC.
+# Scheduled runs always use this file from the default branch, so the branch
+# under test is chosen from the cron string that fired (as in bench.yml).
+# Keep the cron strings here and in CRASH_REF/run-name identical.
+run-name: >-
+  SIGKILL crash-safety fuzz (${{ github.event_name == 'workflow_dispatch' && inputs.ref
+  || (github.event.schedule == '17 15 * * *' && 'v3' || 'main') }})
+
 # #308: an abrupt kill landing mid-checkpoint had a ~26% chance of leaving
 # the header on disk with a bumped page_count but a stale checksum --
 # permanent corruption with no recovery path. The bug was in the ordering
@@ -8,17 +16,29 @@ name: SIGKILL crash-safety fuzz (nightly)
 # matrix rather than Linux only. tests/crash_kill_test.rs runs a small
 # smoke count in the per-PR suite (rust.yml, already cross-platform); this
 # workflow runs a much larger round count nightly for statistical
-# confidence on each OS.
+# confidence on each OS. Since #384 each round also checks that the reopened
+# file holds exactly the transactions the child confirmed (EAVT, AEVT, AVET,
+# full scan, :as-of; verify on v3). The test prints its base seed; replay a
+# failure with MINIGRAF_CRASH_KILL_SEED.
 
 on:
   schedule:
-    - cron: "17 3 * * *"
+    - cron: "17 3 * * *"   # main
+    - cron: "17 15 * * *"  # v3
   workflow_dispatch:
+    inputs:
+      ref:
+        description: Branch to test
+        required: true
+        default: main
 
 permissions:
   contents: read
   issues: write
 
+env:
+  CRASH_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.ref || (github.event.schedule == '17 15 * * *' && 'v3' || 'main') }}
+
 jobs:
   crash-kill:
     name: SIGKILL fuzz suite
@@ -31,6 +51,8 @@ jobs:
 
     steps:
       - uses: actions/checkout@v4
+        with:
+          ref: ${{ env.CRASH_REF }}
 
       - name: Install Rust stable
         uses: dtolnay/rust-toolchain@stable
@@ -49,16 +71,16 @@ jobs:
         shell: bash
         env:
           GH_TOKEN: ${{ github.token }}
-          ISSUE_TITLE: "Nightly SIGKILL crash-safety failure (${{ matrix.os }})"
+          ISSUE_TITLE: "Nightly SIGKILL crash-safety failure (${{ env.CRASH_REF }}, ${{ matrix.os }})"
         run: |
           report_file="crash-kill-failure.md"
           {
-            echo "The nightly SIGKILL crash-safety fuzz suite failed on ${{ matrix.os }}."
+            echo "The nightly SIGKILL crash-safety fuzz suite failed on ${{ env.CRASH_REF }} (${{ matrix.os }})."
             echo
             echo "- Run: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
-            echo "- Commit: $GITHUB_SHA"
+            echo "- Commit: $(git rev-parse HEAD)"
             echo "- Reproduce: cargo test --test crash_kill_test -- --nocapture"
-            echo "  (set MINIGRAF_CRASH_KILL_ROUNDS to raise the round count)"
+            echo "  (set MINIGRAF_CRASH_KILL_ROUNDS to raise the round count; MINIGRAF_CRASH_KILL_SEED to the printed base seed to replay)"
             echo
             echo "## Test output"
             echo

@@ -49,6 +49,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **Golden-file compatibility corpus (#391).** `tests/golden/` holds v7 files written by minigraf 2.0.3, each with a JSON manifest of its CRC32, `tx_count` and expected query results. They cover one checkpoint, several checkpoints, indexes rebuilt on open, a WAL left by a crash, a WAL whose entries were already checkpointed, and same-transaction multi-values (#371). `tests/golden_corpus_test.rs` opens a copy of each file and checks its manifest after open, after a checkpoint and reopen, and after one more write. Golden files are never regenerated, only added; a new `golden-corpus` job in `policy.yml` fails a pull request that changes one. The generator is a standalone crate pinned to `minigraf = "=2.0.3"` (`tests/golden/gen/v7/`). On `v3`, each v7 file must also carry a v8 meta page after its first open, and seven v8 files are frozen alongside them, written by the `v3` branch at `23aa57c` (`tests/golden/gen/v8/`): one checkpoint, seven copy-on-write checkpoints, a crash-left v2 WAL, an already-checkpointed v2 WAL, same-transaction multi-values, a v7 file migrated in place, and long values (a 4,068-byte string stored twice, the 64/65-byte inline boundary, a 1,000-byte attribute and keyword). All #371 queries now pass.
 - **Model-based storage test (#385).** `tests/model_based_test.rs` runs random sequences of `transact` (multi-valued batches, transaction- and fact-level valid-time windows, a triple repeated with another window), `retract`, write transactions (committed or rolled back, retracting their own assertions), checkpoints, reopens (default, a 4-page cache, auto-checkpoint every 2 writes) and simulated crashes (a copy of the `.graph` file and WAL taken while the handle is open, so the WAL is replayed) against a file in a temp dir. After every step, `current_tx_count`, `verify`, full rows with windows and tx counts, and entity-, entity+attribute-, attribute-, attribute+value-, value-bound and full-scan queries, under `:as-of N` and `:valid-at T`, must match a plain reference model (one valid-time window per triple, #435; API-011 for two windows of one triple in one statement and API-019 for an empty or inverted window, from `execute` and `tx.execute`, #436; the last statement that writes a triple decides it at commit, #477). Reads inside a write transaction are checked against what its commit will hold. Normal CI runs 24 cases; the nightly property-test workflow runs 30,000.
+- **SIGKILL crash test checks committed data (#384).** `tests/crash_kill_test.rs` used to check only that the file reopened and one attribute query ran, so it missed #370 and could not reach #371. The child now runs a seeded workload (multi-valued batches, batched retracts, re-assertions) and logs each transaction after `execute` returns; it checkpoints after every transaction, every third, never (WAL replay), automatically every 2 writes, or every third with `SyncMode::Normal`. After a `SIGKILL` at a random point, the reopened file must hold exactly the model after the last logged transaction or the one in flight, through a full scan, attribute (AEVT), attribute+value (AVET) and entity-bound (EAVT) queries and `:as-of`, with `verify` clean. The checks run again after a second reopen and after a checkpoint plus reopen. Per-PR CI runs 5 rounds (one per mode); the nightly crash-kill workflow runs 100 per OS. `MINIGRAF_CRASH_KILL_SEED` replays a failure.
 
 ## v2.0.3 — 2026-10-06
 

@@ -77,7 +77,7 @@ Scope — query API:
 Scope — release-gate testing and documentation:
 
 - Golden-file compatibility corpus for every format version (#391; done on `v3`: six v7 files from 2.0.3 checked after migration, seven v8 files frozen at `23aa57c`)
-- SIGKILL crash test that checks committed data through EAVT, AEVT and full scan (#384)
+- SIGKILL crash test that checks committed data through EAVT, AEVT and full scan (#384; done on `v3`: `tests/crash_kill_test.rs`, 100 rounds per OS nightly)
 - Model-based test of transact/retract/checkpoint/reopen (#385; done on `v3`: `tests/model_based_test.rs`, 30,000 cases nightly)
 - Wider property tests: file-backed, batched, retracts, temporal, joins (#386)
 - Persistent fuzz corpus and an operation-sequence target (#387)

@@ -0,0 +1,24 @@
+# SIGKILL crash test data check — Plan
+
+Spec: `docs/superpowers/specs/2026-10-08-crash-kill-data-check-design.md` (#384).
+
+1. **Generator and model** in `tests/crash_kill_test.rs`: a xorshift RNG seeded
+   per round; `Model { live: BTreeSet<(e, a, v)>, history: Vec<BTreeSet<…>> }`;
+   `next_tx(&mut rng, &model) -> (edn, new_live)`. The child and the parent
+   both drive it, so they produce the same statements.
+2. **Child entrypoint** reads the env (db path, log path, seed, variant), opens
+   with the variant's options, loops: execute, append `k\n` to the log,
+   checkpoint per variant.
+3. **Parent round**: spawn, poll the log until a random target count, jitter,
+   kill, reap, parse `N`.
+4. **Checks**: tx count is `N` or `N + 1`; full scan, AEVT, EAVT, AVET equal
+   `M(T)`; `:as-of` samples; `verify()`. Run on reopen, second reopen, and
+   checkpoint + reopen.
+5. **Self-check before relying on it**: break the model on purpose (drop the
+   last transaction from `M`) and confirm the test fails; then restore.
+6. Run per-PR rounds and `MINIGRAF_CRASH_KILL_ROUNDS=200` locally; clippy, fmt.
+7. Docs: test count in CLAUDE.md, README.md, docs/TEST_COVERAGE.md (the
+   replaced test keeps the count unless a test is added); comment in
+   `crash-kill.yml`. No CHANGELOG entry beyond the v3.0.0 test-suite line, if
+   one exists.
+8. PR to `v3`, "Refs #384", monitor CI.
@@ -0,0 +1,97 @@
+# SIGKILL crash test checks committed data — Design
+
+**Issue:** #384, tracker #383
+**Milestone:** v3.0.0, branch `v3` (test only; no library change)
+
+## 1. Goal
+
+`tests/crash_kill_test.rs` kills a child process mid-write and then only checks
+that the file opens and that one AEVT query runs. It missed #370 (EAVT broken,
+AEVT intact) and could not reach #371 (one fact per transaction). After this
+change, every round checks that the data on disk is exactly the data the child
+committed.
+
+## 2. Shape
+
+The test keeps its name and its parent/child structure, so `crash-kill.yml`
+(nightly, 100 rounds on Linux, macOS and Windows) and the per-PR suite pick it
+up unchanged. `sigkill_during_checkpoint_never_permanently_corrupts_file` is
+replaced by `sigkill_keeps_exactly_the_committed_transactions`.
+
+Each round:
+
+1. Pick a seed and a workload variant (§4). Spawn the child with the database
+   path, a confirmation-log path, the seed and the variant.
+2. The child runs transaction `k = 1, 2, …` from a deterministic generator
+   (§3). After `execute` returns, it appends `k\n` to the confirmation log with
+   one `write_all`.
+3. The parent waits until the log holds a random target of 1–60 confirmed
+   transactions (poll every 1 ms, 30 s timeout), sleeps 0–20 ms more, then
+   kills the child (`SIGKILL` on Unix, `TerminateProcess` on Windows).
+4. The parent reads the log. `N` is the last complete line (a line with no
+   newline is ignored).
+5. Checks (§5) on: the first reopen; a second reopen; and a reopen after an
+   explicit `checkpoint()`.
+
+The confirmation log is not fsynced. A process kill does not drop the OS page
+cache, so a written line survives the kill the same way the WAL entry does.
+fsync would only slow the child down.
+
+## 3. Workload
+
+A transaction is one `execute`. The generator is a function of the seed and the
+model state, so the parent rebuilds the same model by replaying it.
+
+- Pools: 4 keyword entities `:e0`–`:e3`, 3 attributes (`:a0`, `:a1`,
+  `:tag/丿`), values: integers 0–7 and two strings longer than the 64-byte
+  inline limit.
+- **Multi-valued batch** (`transact`, 1–6 facts, ~60%): several values of one
+  entity and attribute in the same transaction (#371), plus random other facts.
+  Never the same triple twice in one transaction.
+- **Batched retract** (`retract`, ~30%): 1–4 currently live triples. When
+  nothing is live, a batch is generated instead.
+- **Re-assert** (~10%): a batch that asserts a triple retracted earlier.
+
+All facts use the default valid time. Valid-time behaviour is the model-based
+test's job (#385).
+
+## 4. Variants
+
+The round number picks one, so every per-PR run covers them all:
+
+| Variant | Child loop | Kill lands in |
+|---|---|---|
+| `CheckpointEach` | transact, then `checkpoint()` | mostly `save()` |
+| `CheckpointEvery(3)` | `checkpoint()` after every 3rd transaction | WAL append and `save()` |
+| `WalOnly` | never checkpoints (`wal_checkpoint_threshold(usize::MAX)`) | WAL append; replay on reopen |
+| `AutoCheckpoint` | `wal_checkpoint_threshold(2)` | auto-checkpoint inside `execute` |
+| `NormalSync` | as `CheckpointEvery(3)` with `SyncMode::Normal` | same; no WAL fsync |
+
+## 5. Checks
+
+The model gives the live triples after `k` transactions, `M(k)`. On each
+reopened handle:
+
+- `current_tx_count()` is `N` or `N + 1`. Call it `T`. `N + 1` means the
+  in-flight transaction reached the WAL or file before the kill but not the
+  log. Nothing beyond it can exist.
+- Full scan `[?e ?a ?v]`, AEVT `[?e <a> ?v]` per attribute, EAVT `[<e> ?a ?v]`
+  per entity, and AVET `[?e <a> <v>]` per attribute and value: each, rendered
+  without UUIDs, equals the matching part of `M(T)`. The union over entities
+  and attributes must equal the full scan.
+- `:as-of k` full scans for a few `k ≤ T` (1, `T/2`, `T - 1`) equal `M(k)`.
+- `verify()` reports no problems.
+
+Failure messages name the round, seed, variant, `N` and `T`. Setting
+`MINIGRAF_CRASH_KILL_SEED` replays one seed. Assert messages print no `Uuid`
+bearing types (CLAUDE.md).
+
+## 6. Cost
+
+Per-PR: 5 rounds (one per variant), each well under a second. The nightly
+workflow keeps its 100 rounds per OS.
+
+## 7. Out of scope
+
+Kills inside `begin_write`/`commit` (covered by the in-process crash tests in
+`tests/common`), valid-time windows, and fault injection (#390).