adityamukho · Oct 29, 2026 · Oct 29, 2026
@@ -0,0 +1,137 @@
+name: Deterministic storage schedules
+
+# #760: only the storage/snapshot/pin boundary is instrumented. Keep this
+# file identical on main and v3: schedules run from the default branch but
+# explicitly check out v3. Merge the main companion after the suite lands.
+# At the v3 release cut, update the scheduled/dispatch default ref to main.
+on:
+  pull_request:
+    branches: ["v3"]
+    paths:
+      - "src/graph/**"
+      - "src/storage/**"
+      - "src/db.rs"
+      - "src/lib.rs"
+      - "src/error.rs"
+      - "src/temporal.rs"
+      - "Cargo.toml"
+      - "tests/dst-mutations/**"
+      - ".github/workflows/deterministic-schedules.yml"
+  schedule:
+    - cron: "23 19 * * *"
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: Branch or commit containing the storage schedule suite
+        required: true
+        default: v3
+
+permissions:
+  contents: read
+
+concurrency:
+  group: dst-${{ github.event_name }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  schedules:
+    name: Schedules (${{ matrix.os }}, Rust ${{ matrix.toolchain }})
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - os: ubuntu-latest
+            toolchain: stable
+          - os: macos-latest
+            toolchain: stable
+          - os: windows-latest
+            toolchain: stable
+          - os: ubuntu-latest
+            toolchain: "1.89.0"
+    defaults:
+      run:
+        shell: bash
+    env:
+      RUSTFLAGS: --cfg minigraf_shuttle
+      CARGO_TERM_COLOR: never
+      CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS: fallback
+      MINIGRAF_DST_ITERATIONS: ${{ github.event_name == 'pull_request' && '1000' || '10000' }}
+      DST_SEEDS: ${{ github.event_name == 'pull_request' && '760' || '760 761 762' }}
+      DST_TOOLCHAIN: ${{ matrix.toolchain }}
+
+    steps:
+      - uses: actions/checkout@v4
+        if: github.event_name == 'pull_request'
+
+      - uses: actions/checkout@v4
+        if: github.event_name != 'pull_request'
+        with:
+          ref: ${{ github.event_name == 'workflow_dispatch' && inputs.ref || 'v3' }}
+
+      - uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: ${{ matrix.toolchain }}
+
+      - name: Record replay environment
+        run: |
+          mkdir -p dst-artifacts
+          git rev-parse HEAD > dst-artifacts/revision.txt
+          git archive --format=tar.gz --output=dst-artifacts/source.tar.gz HEAD
+          rustc -Vv > dst-artifacts/environment.txt
+          cargo -V >> dst-artifacts/environment.txt
+          printf 'RUSTFLAGS=%s\nMINIGRAF_DST_ITERATIONS=%s\nDST_SEEDS=%s\n' \
+            "$RUSTFLAGS" "$MINIGRAF_DST_ITERATIONS" "$DST_SEEDS" >> dst-artifacts/environment.txt
+          if [ "$DST_TOOLCHAIN" != stable ]; then
+            declared=$(sed -n 's/^rust-version = "\(.*\)"$/\1/p' Cargo.toml)
+            test "$DST_TOOLCHAIN" = "$declared.0"
+          fi
+
+      - name: Resolve compatible dependencies
+        run: |
+          cargo generate-lockfile 2>&1 | tee dst-artifacts/resolve.log
+          cp Cargo.lock dst-artifacts/Cargo.lock
+
+      - uses: Swatinem/rust-cache@v2
+        with:
+          key: dst-${{ matrix.toolchain }}
+
+      - name: Explore bounded schedules
+        run: |
+          # Fail if the cfg/filter ever silently selects zero or fewer tests.
+          cargo test --locked --lib graph::storage::deterministic:: -- --list \
+            2>&1 | tee dst-artifacts/tests.log
+          count=$(grep -c '^graph::storage::deterministic::.*: test$' dst-artifacts/tests.log)
+          test "$count" -eq 3
+          for seed in $DST_SEEDS; do
+            export MINIGRAF_DST_SEED="$seed"
+            cargo test --locked --lib graph::storage::deterministic:: \
+              -- --nocapture --test-threads=1 2>&1 | tee "dst-artifacts/seed-$seed.log"
+            completed=$(grep -cF "DST seed=$seed schedules=$MINIGRAF_DST_ITERATIONS" "dst-artifacts/seed-$seed.log")
+            test "$completed" -eq 3
+          done
+
+      - name: Summarize result
+        if: always()
+        env:
+          RESULT: ${{ job.status }}
+        run: |
+          {
+            echo "### Deterministic storage schedules"
+            echo "- Result: $RESULT"
+            echo "- Rust: $DST_TOOLCHAIN; seeds: $DST_SEEDS"
+            echo "- Executions per workload per seed: $MINIGRAF_DST_ITERATIONS"
+            echo "- Replay artifact: dst-${{ matrix.os }}-${{ matrix.toolchain }}-${{ github.run_attempt }}"
+            echo '- A test failure prints its schedule and test name in the seed log.'
+            echo '- Restore source.tar.gz and Cargo.lock; follow docs/testing/deterministic-schedules.md to replay.'
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Preserve replay inputs and logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: dst-${{ matrix.os }}-${{ matrix.toolchain }}-${{ github.run_attempt }}
+          path: dst-artifacts/
+          if-no-files-found: warn
+          retention-days: 14
@@ -140,6 +140,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Tests
 
+- **Deterministic storage schedule tests (#760).** Three separately enabled native tests explore transaction append, snapshot cutoffs and checkpoint publication with Shuttle, using the real storage methods and immutable committed-reader fixtures. Deliberate fault patches validate failure detection and schedule replay. Focused PR checks cover Linux, macOS, Windows and Linux MSRV; nightly/manual runs explore more schedules and retain replay inputs. [Findings and reproduction](docs/testing/deterministic-schedules.md) describe the limits; ordinary tests and production builds retain `std` synchronization.
 - **Format-v8 scalar key decoders are model checked (#693).** Pure inner decoders let Kani prove round trips for signed and unsigned integers, descending transaction counts, valid-to times, floats and six tagged scalar value kinds, plus no panic and bounded cursor consumption on short malformed inputs. The `Reader` still returns the same INT-049 messages and cursor positions; a test-only copy of its old code checks those against exhaustive 0–2 byte and seeded longer inputs. Strings, sets, packed pages and WAL retain fuzz and property coverage. No key bytes or file format change.
 - **Statement bases and large write transactions (#550).** A `WriteTransaction` whose statements repeat, change, retract and re-assert each other's facts across chunks gives every statement the same outcome, and stores the same records, at chunk sizes 1 to 4 as with one chunk; queries after each statement return what a commit of the statements so far would hold; failed and rolled-back statements and transactions leave the file passing `verify`. The fault matrix covers a large `WriteTransaction` at every write and sync, and with `test-hooks` the crash-kill test runs every third transaction of its large variant as one.
 - **Golden-file compatibility corpus (#391).** `tests/golden/` holds v7 files written by minigraf 2.0.3, each with a JSON manifest of its CRC32, `tx_count` and expected query results. They cover one checkpoint, several checkpoints, indexes rebuilt on open, a WAL left by a crash, a WAL whose entries were already checkpointed, and same-transaction multi-values (#371). `tests/golden_corpus_test.rs` opens a copy of each file and checks its manifest after open, after a checkpoint and reopen, and after one more write. Golden files are never regenerated, only added; a new `golden-corpus` job in `policy.yml` fails a pull request that changes one. The generator is a standalone crate pinned to `minigraf = "=2.0.3"` (`tests/golden/gen/v7/`). On `v3`, each v7 file must also carry a v8 meta page after its first open, and seven v8 files are frozen alongside them, written by the `v3` branch at `23aa57c` (`tests/golden/gen/v8/`): one checkpoint, seven copy-on-write checkpoints, a crash-left v2 WAL, an already-checkpointed v2 WAL, same-transaction multi-values, a v7 file migrated in place, and long values (a 4,068-byte string stored twice, the 64/65-byte inline boundary, a 1,000-byte attribute and keyword). All #371 queries now pass.

@@ -221,6 +221,8 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are
 **2000 tests** (1983 passing, 17 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
 See `docs/TEST_COVERAGE.md` for the full per-file breakdown.
 
+The #760 deterministic storage suite adds three native unit tests in a separate `--cfg minigraf_shuttle` build, outside the default count. Run only `graph::storage::deterministic::` with that cfg; ordinary unit tests do not enter its scheduler. `deterministic-schedules.yml` runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See `docs/testing/deterministic-schedules.md` for scope, controls and findings.
+
 **Testing conventions** — see the Testing Conventions section below before writing any tests.
 
 ## Current Maintainer Context

@@ -63,6 +63,10 @@ pest = "2"
 pest_derive = "2"
 pest_meta = "2"
 
+[target.'cfg(all(minigraf_shuttle, not(target_arch = "wasm32")))'.dev-dependencies]
+# #760 bounded prototype; used only with --cfg minigraf_shuttle in unit tests.
+shuttle = "=0.9.8"
+
 [target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies]
 tempfile = "3"
 criterion = { version = "0.8", features = ["html_reports"] }
@@ -136,7 +140,7 @@ workspace = true
 
 [workspace.lints.rust]
 # Kani proof harnesses (#627): `cargo kani` sets cfg(kani).
-unexpected_cfgs = { level = "warn", check-cfg = ['cfg(kani)'] }
+unexpected_cfgs = { level = "warn", check-cfg = ['cfg(kani)', 'cfg(minigraf_shuttle)'] }
 
 [workspace.lints.clippy]
 # Panic prevention — use Result/? or debug_assert! instead

@@ -289,6 +289,8 @@ This is a solo-maintained project with a long-term vision. Read [PHILOSOPHY.md](
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, code standards, and the PR process.
 
+The [deterministic storage schedule suite](docs/testing/deterministic-schedules.md) (#760) documents the native/MSRV CI checks, replay commands and coverage limits.
+
 ## License
 
 Licensed under either of:

@@ -114,6 +114,7 @@ Scope — features:
 
 Scope — release-gate testing and documentation:
 
+- Deterministic storage schedules (#760; v3.0.0): [suite and findings](docs/testing/deterministic-schedules.md) cover batch visibility, snapshot cutoffs and checkpoint publication. Focused native/MSRV PR checks and larger nightly/manual searches retain replay artifacts; the workflow-only main companion enables the v3 nightly schedule. Whole-database simulation remains out of scope.
 - Golden-file compatibility corpus for every format version (#391; done on `v3`: six v7 files from 2.0.3 checked after migration, seven v8 files frozen at `23aa57c`)
 - SIGKILL crash test that checks committed data through EAVT, AEVT and full scan (#384; done on `v3`: `tests/crash_kill_test.rs`, 100 rounds per OS nightly)
 - Model-based test of transact/retract/checkpoint/reopen (#385; done on `v3`: `tests/model_based_test.rs`, 30,000 cases nightly)

@@ -24,6 +24,8 @@
 
 Not counted above: the browser tests and the WASI run (see [Browser and WASI](#browser-and-wasi)).
 
+Also outside those counts: three native unit tests in the separate #760 `minigraf_shuttle` build. The [bounded suite and findings](testing/deterministic-schedules.md) record 90,000 passing schedule executions and three deliberately injected faults, each replayed 10/10 times. The suite checks real storage synchronization with immutable committed-reader fixtures; it does not simulate file durability. `deterministic-schedules.yml` runs relevant v3 PRs on three native OSes plus Linux MSRV, with larger nightly/manual searches and replay artifacts. Default-feature test counts are unchanged.
+
 **Scope**: the unreleased file-format-v8 work (#287, #371): the index-key value-collision fix, the v7→v8 migration, and the drop of formats v1–v6. Otherwise the public database API, the Datalog engine, storage and recovery paths, v2.0.0's locking and structured-error-code changes, and v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Tests merged from the v2.x line are marked *(v2.x)*.
 
 ## Covered Areas