adityamukho · Oct 10, 2026 · Oct 10, 2026
@@ -0,0 +1,125 @@
+name: Soak test (weekly)
+
+# Two branches alternate weekly: main on Saturday, v3 on Sunday, both 01:00 UTC.
+# Scheduled runs always use this file from the default branch, so the branch
+# under test is chosen from the cron string that fired (as in bench.yml and
+# crash-kill.yml). Keep the cron strings here and in SOAK_REF/run-name
+# identical.
+run-name: >-
+  Soak test (${{ github.event_name == 'workflow_dispatch' && inputs.ref
+  || (github.event.schedule == '0 1 * * 0' && 'v3' || 'main') }})
+
+# #392: grows a file-backed database to 10M+ facts, then churns it (retracts,
+# re-asserts, long hot-entity histories) for the rest of the budget, with
+# thousands of checkpoints and reopens every few minutes. Samples are checked
+# against a reference throughout; the metrics (file size, open and checkpoint
+# time, RSS, p50/p99 query latency) are uploaded for the performance envelope
+# (#394). The test prints its seed; MINIGRAF_SOAK_SEED replays the writes.
+
+on:
+  schedule:
+    - cron: "0 1 * * 6"  # main
+    - cron: "0 1 * * 0"  # v3
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: Branch to test
+        required: true
+        default: main
+      minutes:
+        description: Workload budget in minutes
+        required: true
+        default: "300"
+
+permissions:
+  contents: read
+  issues: write
+
+env:
+  SOAK_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.ref || (github.event.schedule == '0 1 * * 0' && 'v3' || 'main') }}
+
+jobs:
+  soak:
+    name: Soak at 10M facts
+    runs-on: ubuntu-latest
+    timeout-minutes: 360
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ env.SOAK_REF }}
+
+      - name: Install Rust stable
+        uses: dtolnay/rust-toolchain@stable
+
+      # The database grows to several GB on main (format v7); /mnt has the
+      # runner's spare disk.
+      - name: Prepare the data directory
+        run: |
+          sudo mkdir -p /mnt/soak
+          sudo chown "$USER" /mnt/soak
+          df -h / /mnt
+
+      - name: Build
+        run: cargo test --profile bench --test soak_test --no-run
+
+      - name: Run the soak test
+        shell: bash
+        env:
+          MINIGRAF_SOAK_MINUTES: ${{ github.event_name == 'workflow_dispatch' && inputs.minutes || '300' }}
+          MINIGRAF_SOAK_TARGET_FACTS: "10000000"
+          MINIGRAF_SOAK_METRICS: ${{ github.workspace }}/soak-metrics.jsonl
+          MINIGRAF_SOAK_DIR: /mnt/soak
+          RUST_BACKTRACE: 1
+        run: |
+          set -o pipefail
+          cargo test --profile bench --test soak_test soak -- --ignored --exact --nocapture 2>&1 | tee soak-output.txt
+
+      - name: Upload metrics
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: soak-${{ env.SOAK_REF }}-${{ github.run_id }}
+          path: |
+            soak-metrics.jsonl
+            soak-output.txt
+          retention-days: 90
+          if-no-files-found: warn
+
+      - name: Record soak failure
+        if: ${{ failure() }}
+        shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
+          ISSUE_TITLE: "Weekly soak failure (${{ env.SOAK_REF }})"
+        run: |
+          report_file="soak-failure.md"
+          {
+            echo "The weekly soak test failed on ${{ env.SOAK_REF }}."
+            echo
+            echo "- Run: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
+            echo "- Commit: $(git rev-parse HEAD)"
+            echo "- Reproduce: cargo test --profile bench --test soak_test soak -- --ignored --exact --nocapture"
+            echo "  (MINIGRAF_SOAK_SEED to the printed seed replays the writes; see tests/soak_test.rs for the other variables)"
+            echo
+            echo "## Test output"
+            echo
+            echo '```text'
+            tail -n 300 soak-output.txt 2>/dev/null || echo "(no output: the run failed before the test started)"
+            echo '```'
+          } > "$report_file"
+
+          existing_issue="$(gh issue list \
+            --repo "$GITHUB_REPOSITORY" \
+            --state open \
+            --search "$ISSUE_TITLE in:title" \
+            --json number,title \
+            --limit 100 \
+            --jq ".[] | select(.title == \"$ISSUE_TITLE\") | .number" \
+            | head -n 1)"
+
+          if [ -n "$existing_issue" ]; then
+            gh issue comment "$existing_issue" --repo "$GITHUB_REPOSITORY" --body-file "$report_file"
+          else
+            gh issue create --repo "$GITHUB_REPOSITORY" --title "$ISSUE_TITLE" --body-file "$report_file"
+          fi
@@ -12,6 +12,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **A file shorter than one page is refused at open with `STG-045` unless it is a torn first write (#506).** Open treated any file under 4 KiB as a new database and wrote a new header over it, so a database cut below one page by an interrupted copy, or an unrelated file at the path, was silently replaced. Such a file now opens as a new database only when its bytes are zeros or a prefix of the header a new database writes (what a crash during creation can leave); anything else is `STG-045` and the file is not modified. A zero-length file is still a new database. New code `STG-045`, numbered as in v3.0.0, which also fixes this.
 - **A commit made after a torn WAL entry is no longer lost on recovery (#513).** Recovery replays the WAL up to its first partial or damaged entry, but the writer appended after the end of the file. After a crash during an append, the next session's commits went in after the partial entry, so a second crash before any checkpoint lost them, although each had been fsynced and reported as committed. A failed append (short write, full disk, I/O error) did the same to the next commit through that handle. Opening a database for writing now cuts the WAL back to the end of its last valid entry, and syncs it, before anything is appended. After a failed append, every later write through that handle fails with the new `WAL-007` until the database is reopened or a `checkpoint()` succeeds. The transaction whose append failed is not applied; after a reopen it is there in full or not at all. Affected every release since v0.5.0. Also fixed in v3.0.0.
 
+### Tests
+
+- **Weekly soak test at scale (#392).** `tests/soak_test.rs` grows a file-backed database to 10M live facts in batches of 1–2,000 facts, then churns it for the rest of a 5-hour budget at 50 transactions a second (retracts, re-asserts, long histories on 32 hot entities), with a checkpoint every 20–200 transactions and a reopen every 2–10 minutes. Samples are checked against a reference through EAVT, entity and attribute, AVET, AEVT, `:as-of` and the transaction counter, with a full scan in a child process at 2M facts. RSS after a reopen is bounded relative to the fact versions held. Metrics go to a JSON-lines file that `.github/workflows/soak.yml` uploads weekly, alternating `main` (Saturday) and `v3` (Sunday). On v2.x no transaction writes two values of one entity and attribute, because of the known issue #371; the v3 workload fails here. It also checkpoints every 100–1,000 transactions and samples `:as-of` less, since both cost time in the database size on v2.x. `soak_short` runs the driver at 20K facts in the normal suite.
+
 ## v2.0.4 — 2026-10-08
 
 Patch release on the v2.x line under the [support policy](PHILOSOPHY.md#support-policy): one `BrowserDb` durability hardening fix, plus new compatibility and crash tests. File format is unchanged (v7). No API changes. Native and WASI builds behave exactly as in v2.0.3.

@@ -174,7 +174,7 @@ Auto-migrates v1/v2/v3/v4/v5/v6 → v7 on open/checkpoint.
 
 ## Test Coverage
 
-**1231 tests** (1223 passing, 8 ignored; unit + integration + doc).
+**1234 tests** (1225 passing, 9 ignored; unit + integration + doc).
 See `docs/TEST_COVERAGE.md` for the full per-file breakdown.
 
 **Testing conventions** — see the Testing Conventions section below before writing any tests.

@@ -105,7 +105,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit
 
 ```bash
 cargo run          # interactive Datalog REPL
-cargo test         # run 1231 tests
+cargo test         # run 1234 tests
 cargo run < demos/demo_recursive.txt   # recursive rules demo
 ```
 

@@ -79,7 +79,7 @@ Scope — release-gate testing and documentation:
 - Wider property tests: file-backed, batched, retracts, temporal, joins (#386)
 - Persistent fuzz corpus and an operation-sequence target (#387)
 - Fault injection across save/WAL/recovery (#390)
-- Long-running soak test at scale (#392)
+- Long-running soak test at scale (#392; `tests/soak_test.rs`, 10M facts and a 5-hour churn weekly, alternating `main` and `v3`)
 - Benchmarks at 100K–10M facts and a published performance envelope (#394)
 - Higher storage/WAL coverage gates and branch-coverage gates (#396)
 - Durability, recovery and operations guide (#398)

@@ -4,7 +4,7 @@
 
 **Verified**: 2026-10-10 with `cargo test`
 
-**Result**: 1,223 passing tests and 8 ignored tests (1,231 total), including the golden-file corpus (#391)
+**Result**: 1,225 passing tests and 9 ignored tests (1,234 total), including the golden-file corpus (#391)
 
 Since v2.0.4, `tests/edge_cases_test.rs` checks that a file of 1 to 4,095 bytes is `STG-045`, unchanged and with no WAL, when it is a database cut below one page or foreign bytes, and opens as a new database that takes a write and reads it back after reopen when it is a prefix of the initial header or all zeros (#506).
 
@@ -24,15 +24,16 @@ The v2.0.4 suite covers the public database API, the Datalog engine, storage and
 - **Storage correctness**: packed pages, B+tree indexes and range scans, cache behavior, file-header validation, migration, WAL replay, checksums, corruption handling, and fault injection.
 - **Reliability and concurrency**: concurrent reads and writes, rollback behavior, same-process handle exclusion, cross-process locking, PID namespaces, NFS and container lock behavior, and SIGKILL recovery.
 - **Public diagnostics**: structured parser, query, storage, WAL, API, and internal error codes; the error-code registry is checked against [`ERROR_REFERENCE.md`](ERROR_REFERENCE.md).

8
-- **C
FC81
ompatibility and quality**: golden-file corpus of released file formats, cross-platform format compatibility, property-based tests, long-haul smoke coverage, XTDB/Datomic semantic compatibility, grammar conformance, and rustdoc examples.
+- **Compatibility and quality**: golden-file corpus of released file formats, cross-platform format compatibility, property-based tests, long-haul smoke coverage, a weekly soak at 10M facts with a 5-hour churn (#392), XTDB/Datomic semantic compatibility, grammar conformance, and rustdoc examples.
 
 ### Ignored Tests
 
-The eight ignored tests are intentionally excluded from a normal local run:
+The nine ignored tests are intentionally excluded from a normal local run:
 
 - Six rustdoc examples that reference internal types and cannot compile as standalone examples.
 - One high-contention concurrency stress test intended for scheduled runs.
 - One long-haul smoke test intended for scheduled runs.
+- One soak test at 10M facts (`soak`), run weekly by `.github/workflows/soak.yml` (#392); `soak_short` runs the same driver at 20K facts in the normal suite.
 
 ### Reproducing
 

@@ -0,0 +1,30 @@
+# Soak test at scale — Plan
+
+Spec: `docs/superpowers/specs/2026-10-10-soak-test-design.md` (#392).
+
+## v3 PR (`test/392-soak` → `v3`)
+
+1. `tests/soak_test.rs`
+   - `Config` (from env for `soak`, fixed for `soak_short`), `Every` schedule by
+     time or by step count so the short run is deterministic in a debug build.
+   - `Reference`: `content(n)` slots, `flipped` Vec + index map, hot history as
+     packed `u64`s, `tx`.
+   - `Driver`: op choice by phase weights, checkpoint and reopen schedule,
+     churn rate cap with checked reads in between.
+   - Checks: EAVT, AVET, AEVT shard, hot `:as-of`, tx counter; end: full-scan
+     count and `verify()`.
+   - `Metrics`: latency samples per path, checkpoint times, open times, RSS
+     from `/proc/self/status`; JSON lines via `serde_json`.
+   - Memory bound after the growth phase (spec §8).
+2. Run `soak_short` in debug; run `soak` locally with a reduced target and a
+   few minutes; then a full 10M run under `--profile bench`.
+3. `.github/workflows/soak.yml` (spec §9).
+4. Docs: `docs/TEST_COVERAGE.md`, `CLAUDE.md` test count, `ROADMAP.md` if it
+   lists #392, `CHANGELOG.md` Unreleased.
+5. PR to `v3` without a closing keyword for the tracker; `Closes #392` only
+   in the main PR description if the user wants it closed on merge.
+
+## main PR (`test/392-soak-v2` → `main`)
+
+Same test with `verify()` dropped and the v2 memory rule (spec §8); same
+`soak.yml`. Run `soak_short` and a short `soak`.