adityamukho · Oct 10, 2026 · Oct 10, 2026 · Oct 10, 2026 · Oct 10, 2026 · Oct 10, 2026
@@ -0,0 +1,125 @@
+name: Soak test (weekly)
+
+# Two branches alternate weekly: main on Saturday, v3 on Sunday, both 01:00 UTC.
+# Scheduled runs always use this file from the default branch, so the branch
+# under test is chosen from the cron string that fired (as in bench.yml and
+# crash-kill.yml). Keep the cron strings here and in SOAK_REF/run-name
+# identical.
+run-name: >-
+  Soak test (${{ github.event_name == 'workflow_dispatch' && inputs.ref
+  || (github.event.schedule == '0 1 * * 0' && 'v3' || 'main') }})
+
+# #392: grows a file-backed database to 10M+ facts, then churns it (retracts,
+# re-asserts, long hot-entity histories) for the rest of the budget, with
+# thousands of checkpoints and reopens every few minutes. Samples are checked
+# against a reference throughout; the metrics (file size, open and checkpoint
+# time, RSS, p50/p99 query latency) are uploaded for the performance envelope
+# (#394). The test prints its seed; MINIGRAF_SOAK_SEED replays the writes.
+
+on:
+  schedule:
+    - cron: "0 1 * * 6"  # main
+    - cron: "0 1 * * 0"  # v3
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: Branch to test
+        required: true
+        default: main
+      minutes:
+        description: Workload budget in minutes
+        required: true
+        default: "300"
+
+permissions:
+  contents: read
+  issues: write
+
+env:
+  SOAK_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.ref || (github.event.schedule == '0 1 * * 0' && 'v3' || 'main') }}
+
+jobs:
+  soak:
+    name: Soak at 10M facts
+    runs-on: ubuntu-latest
+    timeout-minutes: 360
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ env.SOAK_REF }}
+
+      - name: Install Rust stable
+        uses: dtolnay/rust-toolchain@stable
+
+      # The database grows to several GB on main (format v7); /mnt has the
+      # runner's spare disk.
+      - name: Prepare the data directory
+        run: |
+          sudo mkdir -p /mnt/soak
+          sudo chown "$USER" /mnt/soak
+          df -h / /mnt
+
+      - name: Build
+        run: cargo test --profile bench --test soak_test --no-run
+
+      - name: Run the soak test
+        shell: bash
+        env:
+          MINIGRAF_SOAK_MINUTES: ${{ github.event_name == 'workflow_dispatch' && inputs.minutes || '300' }}
+          MINIGRAF_SOAK_TARGET_FACTS: "10000000"
+          MINIGRAF_SOAK_METRICS: ${{ github.workspace }}/soak-metrics.jsonl
+          MINIGRAF_SOAK_DIR: /mnt/soak
+          RUST_BACKTRACE: 1
+        run: |
+          set -o pipefail
+          cargo test --profile bench --test soak_test soak -- --ignored --exact --nocapture 2>&1 | tee soak-output.txt
+
+      - name: Upload metrics
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: soak-${{ env.SOAK_REF }}-${{ github.run_id }}
+          path: |
+            soak-metrics.jsonl
+            soak-output.txt
+          retention-days: 90
+          if-no-files-found: warn
+
+      - name: Record soak failure
+        if: ${{ failure() }}
+        shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
+          ISSUE_TITLE: "Weekly soak failure (${{ env.SOAK_REF }})"
+        run: |
+          report_file="soak-failure.md"
+          {
+            echo "The weekly soak test failed on ${{ env.SOAK_REF }}."
+            echo
+            echo "- Run: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
+            echo "- Commit: $(git rev-parse HEAD)"
+            echo "- Reproduce: cargo test --profile bench --test soak_test soak -- --ignored --exact --nocapture"
+            echo "  (MINIGRAF_SOAK_SEED to the printed seed replays the writes; see tests/soak_test.rs for the other variables)"
+            echo
+            echo "## Test output"
+            echo
+            echo '```text'
+            tail -n 300 soak-output.txt 2>/dev/null || echo "(no output: the run failed before the test started)"
+            echo '```'
+          } > "$report_file"
+
+          existing_issue="$(gh issue list \
+            --repo "$GITHUB_REPOSITORY" \
+            --state open \
+            --search "$ISSUE_TITLE in:title" \
+            --json number,title \
+            --limit 100 \
+            --jq ".[] | select(.title == \"$ISSUE_TITLE\") | .number" \
+            | head -n 1)"
+
+          if [ -n "$existing_issue" ]; then
+            gh issue comment "$existing_issue" --repo "$GITHUB_REPOSITORY" --body-file "$report_file"
+          else
+            gh issue create --repo "$GITHUB_REPOSITORY" --title "$ISSUE_TITLE" --body-file "$report_file"
+          fi
@@ -57,6 +57,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **SIGKILL crash test checks committed data (#384).** `tests/crash_kill_test.rs` used to check only that the file reopened and one attribute query ran, so it missed #370 and could not reach #371. The child now runs a seeded workload (multi-valued batches, batched retracts, re-assertions) and logs each transaction after `execute` returns; it checkpoints after every transaction, every third, never (WAL replay), automatically every 2 writes, or every third with `SyncMode::Normal`. After a `SIGKILL` at a random point, the reopened file must hold exactly the model after the last logged transaction or the one in flight, through a full scan, attribute (AEVT), attribute+value (AVET) and entity-bound (EAVT) queries and `:as-of`, with `verify` clean. The checks run again after a second reopen and after a checkpoint plus reopen. Per-PR CI runs 5 rounds (one per mode); the nightly crash-kill workflow runs 100 per OS. `MINIGRAF_CRASH_KILL_SEED` replays a failure.
 - **Fuzzing reaches the v8 decoders, and the corpus is kept between runs (#387, #375).** On `v3`, the `btree_page` and `fact_page` targets still wrote the pre-release single-header layout and stopped at `STG-032` on every input, and `file_header` rarely got past the meta CRC. The page targets now start from a real v8 file built through the public API (rewritten with `LogWriter` at fixed tx times so its bytes are the same in every process). They overwrite one page with the input and recompute that page's checksum. `btree_page` fuzzes any leaf or internal node of the five trees, `file_header` fuzzes either meta slot (with or without a matching CRC, with or without the other slot), and `fact_page` fuzzes the fact page of a v7 golden file through migration and the read-only load. Each run executes one input-chosen probe query or `verify()`. The nightly fuzz workflow now runs every target in its own parallel job, on `main` and `v3` on alternating crons, and keeps each target's corpus in the Actions cache. A new weekly workflow (`fuzz-corpus.yml`) minimizes the cached corpus with `cargo fuzz cmin`, keeps every committed `seed_*` file, and proposes the result as a pull request.
 - **Two fuzz targets check results, not just crashes (#387).** `ops_sequence` decodes the input into up to 32 storage operations (transact, retract, a write transaction that commits or rolls back, checkpoint, reopen, and a crash that opens a copy of the file and WAL) and, after each one, compares the full rows now and at an input-chosen `:as-of`, plus one input-chosen query shape, with the reference model of the model-based test. It uses that test's driver and checks, now in `tests/reference_model`, so the proptest test and the fuzzer share one oracle. `corrupt_file` damages a file written over three checkpoints with up to 16 byte flips and an optional truncation, without fixing any checksum, and requires every query to fail or to return exactly the answers of the last checkpoint or the one before it, all from the same one. Its first run found that a file cut to under one page opens as a new, empty database (#506); the target skips those lengths until that is fixed.
+- **Weekly soak test at scale (#392).** `tests/soak_test.rs` grows a file-backed database to 10M live facts in batches of 1–2,000 facts (multi-valued tags and shard values in one transaction, strings over 64 bytes, refs), then churns it for the rest of a 5-hour budget at 50 transactions a second: retracts and re-asserts of random facts, and long retract/assert histories on 32 hot entities. It checkpoints every 20–200 transactions and reopens every 2–10 minutes, half from a checkpointed file and half replaying the WAL. Every 10 minutes, after every reopen, and between churn writes it checks samples against a reference that derives each entity's facts from its number and keeps only what was retracted: EAVT per entity, entity and attribute on the tags, AVET by unique name, AEVT over a sparse attribute, `:as-of` on hot entities, and the transaction counter; a full scan runs in a child process at 2M facts (so its memory does not hide a leak), and `verify()` at the end. It fails if RSS after a reopen grows past 1.25× the baseline plus 256 MiB. Metrics (file size, open and checkpoint time, RSS and peak RSS, p50/p99 per query path) go to a JSON-lines file that `.github/workflows/soak.yml` uploads weekly, alternating `main` and `v3`. `soak_short` runs the same driver at 20K facts in the per-PR suite. Found #518: attribute-and-value patterns scan the whole attribute.
 
 ### Known issues
 

@@ -198,7 +198,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are
 
 ## Test Coverage
 
-**1406 tests** (1397 passing, 9 ignored; unit + integration + doc).
+**1409 tests** (1399 passing, 10 ignored; unit + integration + doc).
 See `docs/TEST_COVERAGE.md` for the full per-file breakdown.
 
 **Testing conventions** — see the Testing Conventions section below before writing any tests.

@@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit
 
 ```bash
 cargo run          # interactive Datalog REPL
-cargo test         # run 1406 tests
+cargo test         # run 1409 tests
 cargo run < demos/demo_recursive.txt   # recursive rules demo
 ```
 

@@ -93,7 +93,7 @@ Scope — release-gate testing and documentation:
 - Wider property tests: file-backed, batched, retracts, temporal, joins (#386)
 - Persistent fuzz corpus and an operation-sequence target (#387; done on `v3`: corpus caching, weekly `cmin`, the v8 page targets, and the `ops_sequence` and `corrupt_file` targets)
 - Fault injection across save/WAL/recovery (#390)
-- Long-running soak test at scale (#392)
+- Long-running soak test at scale (#392; done on `v3`: `tests/soak_test.rs`, 10M facts and a 5-hour churn weekly, alternating `main` and `v3`)
 - Benchmarks at 100K–10M facts and a published performance envelope (#394)
 - Higher storage/WAL coverage gates and branch-coverage gates (#396)
 - Durability, recovery and operations guide (#398)

@@ -0,0 +1,30 @@
+# Soak test at scale — Plan
+
+Spec: `docs/superpowers/specs/2026-10-10-soak-test-design.md` (#392).
+
+## v3 PR (`test/392-soak` → `v3`)
+
+1. `tests/soak_test.rs`
+   - `Config` (from env for `soak`, fixed for `soak_short`), `Every` schedule by
+     time or by step count so the short run is deterministic in a debug build.
+   - `Reference`: `content(n)` slots, `flipped` Vec + index map, hot history as
+     packed `u64`s, `tx`.
+   - `Driver`: op choice by phase weights, checkpoint and reopen schedule,
+     churn rate cap with checked reads in between.
+   - Checks: EAVT, AVET, AEVT shard, hot `:as-of`, tx counter; end: full-scan
+     count and `verify()`.
+   - `Metrics`: latency samples per path, checkpoint times, open times, RSS
+     from `/proc/self/status`; JSON lines via `serde_json`.
+   - Memory bound after the growth phase (spec §8).
+2. Run `soak_short` in debug; run `soak` locally with a reduced target and a
+   few minutes; then a full 10M run under `--profile bench`.
+3. `.github/workflows/soak.yml` (spec §9).
+4. Docs: `docs/TEST_COVERAGE.md`, `CLAUDE.md` test count, `ROADMAP.md` if it
+   lists #392, `CHANGELOG.md` Unreleased.
+5. PR to `v3` without a closing keyword for the tracker; `Closes #392` only
+   in the main PR description if the user wants it closed on merge.
+
+## main PR (`test/392-soak-v2` → `main`)
+
+Same test with `verify()` dropped and the v2 memory rule (spec §8); same
+`soak.yml`. Run `soak_short` and a short `soak`.