8000 feat: large single statements commit as a checkpoint (#550 PR 2) by adityamukho · Pull Request #685 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]
Skip to content

feat: large single statements commit as a checkpoint (#550 PR 2) - #685

Merged
adityamukho merged 9 commits into
v3from
feat/550-large-statements
Oct 22, 2026
Merged

feat: large single statements commit as a checkpoint (#550 PR 2)#685
adityamukho merged 9 commits into
v3from
feat/550-large-statements

Conversation

@adityamukho
@adityamukho adityamukho commented Oct 22, 2026
Copy link
Copy Markdown
Collaborator

Part of #550 (streaming engine 8/9, #473): PR 2 of 3, large single statements (spec docs/superpowers/specs/2026-10-21-streaming-transactions-design.md §4; plan PR 2 section in docs/superpowers/plans/2026-10-21-550-streaming-transactions.md, committed here).

What changes

Minigraf::execute of a transact/retract with more than TX_CHUNK_FACTS facts on a writable file database becomes large:

  1. WAL guard: the WAL exists (header only, base g) before any page is written (§4.5 rule 1).
  2. Cache generation bound raised to g + 1; next tx_count reserved; tx_id fixed; allocator from_chain(.., g + 1) with the quarantine and id tracking.
  3. Each chunk is checked (API-019 at the fixed tx_id, API-011, WAL-003), stamped, encoded by a per-chunk Encoder over the working DICT (no SharedDictCache), checked against earlier chunks (a repeated fact stored once, another window of its triple API-011), and applied with cow_update. Replaced pages the transaction wrote are released (PageAllocator::release), g's go to freed.
  4. Commit as a checkpoint: the pending store joins the working trees, free list, sync, meta g + 1 (the commit point), tx_count allocated, publish, WAL deleted.
  5. Rollback / failures (§4.4): statement errors roll back (bound back to g, written pages evicted, handle dirty); I/O failures also mark the handle failed (STG-046).
  6. Open-time dirty rule (§4.5 rule 2): a WAL found at open marks the handle dirty, so the first checkpoint writes a meta before deleting the WAL.

Statement::chunks yields one Chunk at a time. test-oracle gets minigraf::__tx_chunk_facts and __statement_facts_high_water.

Deviations from the spec (listed in the plan)

  1. WriteTransaction keeps the staged path until PR 3 (it needs §5 deletes and §7 reads); spec §9 had first-statement-large WriteTransactions in PR 2.
  2. tx_count is allocated at the commit point, not when the transaction becomes large, so a rolled-back large transaction takes none.
  3. Rollback evicts every id handed out, not only those still written.
  4. Errors of different chunks come in chunk order.
  5. A record with a default valid_from and one whose explicit valid_from equals the fixed tx_id, in different chunks, are kept once instead of being API-011.
  6. The acceptance count covers Facts only.
  7. The model-based test runs at chunk size 2 (spec: 3), so its retractions are large too.

Tests

  • large_tx.rs (7): same records as the one-chunk path at chunk sizes 1–4 over a workload (repeats, windows, long values, refs, retractions, pending WAL entries), returned tx_id and :as-of, rejected large statements (API-011, WAL-003) write nothing, take no tx_count and keep the WAL until a checkpoint writes a meta, the open-time dirty rule, the §4.5 reproduction (a new file reopens through the guard and is STG-033 without it), and cursors and fact logs opened before or between chunks keep their snapshot.
  • Fault matrix: large transactions on one and two valid meta slots, with and without a prior WAL, every fault at every call.
  • crash_kill_test.rs: a test-oracle variant where most transactions are large and a failing large statement rolls back after writing pages (120 rounds locally). crash-kill.yml passes --features test-oracle for v3 only; the file has to land on main too (follow-up PR), because scheduled runs use main's copy.
  • model_based_test.rs under test-oracle at chunk size 2 (300 cases locally).
  • tests/large_transaction_acceptance_test.rs (ignored, nightly in smoke.yml): 1M-fact transact held at most 16,384 facts, peak RSS 37 MiB, 24 s including the commit. Today's path on the same input: 1,000,000 facts held, 1028 MiB, 10.2 s for the WAL entry alone, with no checkpoint.
  • Not committed: the whole test-oracle suite with the default chunk size forced to 3 passes (1752/0).

Counts: 1752 tests (1738 passing, 14 ignored).

No CHANGELOG.md or docs-site change: PR 3 documents them.

🤖 Generated with Claude Code

https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT

adityamukho and others added 9 commits October 22, 2026 08:44
@adityamukho @claude
docs(plan): #550 PR 2 in detail
ddcbebb 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
feat(storage): allocator tracking and release, cache evict, cursor pe…
b893164 
…ek (#550)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
feat(parser): statement chunks and test-oracle chunk hooks (#550)
242a088 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
feat(storage): large single statements commit as a checkpoint (#550)
45440f4 
Minigraf::execute of a statement over TX_CHUNK_FACTS facts on a file
database writes it a chunk at a time into copy-on-write working trees
behind a header-only WAL guard and commits it as a checkpoint. A WAL
found at open, or a rolled-back large transaction, marks the handle
dirty so the next checkpoint writes a meta before deleting the WAL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
test: large transactions in the fault matrix, crash-kill, model and 1…
c8ef120 
…M acceptance (#550)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
docs: #550 PR 2 module notes, test counts, plan notes
fbdcc2a 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
test: run native-only large-transaction unit tests off wasm32 only (#550
25e2abd 
)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
test: allocator tracking asserts without short-circuit branches (#550)
cce0072 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@adityamukho @claude
test: read the active generation through the handle, not the locked f…
8d43c1d 
…ile (#550)

Windows refuses reading a file another handle holds an exclusive lock on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
@codecov
codecov Bot commented Oct 22, 2026
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.00000% with 33 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/storage/persistent_facts/large_tx.rs 84.89% 29 Missing ⚠️
src/db.rs 92.50% 3 Missing ⚠️
src/query/datalog/parser/statement.rs 97.50% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@adityamukho
adityamukho merged commit 9ecc02f into v3 Oct 22, 2026
24 of 25 checks passed
@adityamukho
adityamukho deleted the feat/550-large-statements branch October 22, 2026 05:10
This was referenced Oct 22, 2026
Closed
Closed
Closed
Merged
adityamukho added a commit that referenced this pull request Oct 23, 2026
@adityamukho @claude
ci(crash-kill): run v3's nightly rounds with test-oracle (#687) (#695)
45ac8c5 
* ci(crash-kill): run v3's nightly rounds with test-oracle (#687)

Scheduled workflows run the copy on the default branch, so v3's
--features test-oracle change (#685) never reached the nightly v3 run
and its Large variant (#550) did not run. Land v3's crash-kill.yml on
main unchanged; main has no test-oracle feature and keeps its command.

Closes #687

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT

* ci(crash-kill): pass test-oracle wherever Cargo.toml defines it (#687)

A dispatch on a main-based ref other than main would have passed a
feature that does not exist there; a v3-based branch keeps the variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@adityamukho
0