Repository navigation
feat(storage): statement bases and large write transactions (#550 PR 3) - #692
Merged
Merged
Conversation
Decisions 25-36, deviations, steps 16-23 and the chunk-size measurement results, checked against v3 at 75a3fac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
A WriteTransaction on a file database becomes large when its staged facts would pass TX_CHUNK_FACTS: the staged statements are checked at its tx_id, then written as one chunk, and every later statement writes its chunks under a statement base (roots, counters, freed length, ids handed out since). A rejected statement returns to its base and the transaction stays usable. An earlier statement's keys of a triple a statement writes are deleted, and the triple joins the statement's claims tree, so a repeat of an earlier fact does not hide a second window of it (plan decision 30). A failed statement truncates freed. The pending store is written when a transaction becomes large, so queries inside it read the working trees alone. Every WriteTransaction query now reads a Snapshot (staged facts merged with the pending ones by index order) and runs on the engine; merged_query_facts and the executor's facts_override are gone. TX_CHUNK_FACTS stays 16,384 after measurement. Fault matrix and crash-kill cover large WriteTransactions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The chunk size of large transactions is an open option, default 65,536 for speed: at 1M facts with random entities 16,384 took up to twice as long for 40-55 MiB less. Kept between 1 and 1,000,000, the most facts WAL replay reads from one entry, since a small transaction is one entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#550 PR 3 of 3 (part of #473): statement bases, large
WriteTransactions, snapshot reads, the chunk-size measurement, docs. Plan: PR 3 section ofdocs/superpowers/plans/2026-10-21-550-streaming-transactions.md(decisions 25–36, deviations 1–8, results).What changes
WriteTransactions. On a file database, the statement that takes a transaction's staged facts pastTX_CHUNK_FACTS(or a larger statement) makes it large. The staged statements are reduced bykeep_last_statement, checked at the transaction'stx_idfirst (a failure rejects the statement and leaves the transaction small, nothing to undo), then written as one chunk. Commit / rollback / drop go throughcommit_large/rollback_large.fact_count,freedlength and the ids handed out since (PageAllocator::begin_base). A rejected statement returns to its base; the transaction stays usable. An earlier statement's keys of a triple the statement writes are deleted from all four index trees (cow_updatedeletes).API-011. Each statement keeps a copy-on-write tree of the triples whose base keys it deleted; its pages are released when the statement ends.freed(deviation 4). Pages ofgit replaced are referenced again by the restored roots.WriteTransactionquery reads aSnapshot: staged facts are merged with the pending ones by index order (PendingView), or a large transaction's working-treeOnDiskReader.merged_query_facts,facts_overrideandread_now_floorare gone; these queries run on the engine (oracle under test) and count towardquery_memory_limit.tx_count(deviation 7, found by the model-based test).Measurement →
OpenOptions::transaction_chunk_facts, default 65,5361M facts, idle bench machine,
opt-level = 3, v3 before #550 (d24fb585) → 4,096 / 16,384 / 65,536. Full table in the plan.execute, sequential, shortexecute, random, shortRSS is the rise above the input text's peak. The chunk size is now an open option,
OpenOptions::transaction_chunk_facts(kept between 1 and 1,000,000, the WAL entry limit), defaulting to 65,536 for speed. Two costs: random entities are slower than before (spec §8.2), and a multi-statement transaction over random entities can leave the file up to 1.8× larger (held base pages). The space is reused by later checkpoints. 16,384 holds 40–55 MiB less and takes up to twice as long with random entities.Tests
1767 tests (1753 passing, 14 ignored).
test-oracle: 1767 passing, 0 failed. New: 9large_txtests (incl. the open option) (WriteTx differential at chunk sizes 1–4 withAPI-011cases incl. the claims case, in-tx reads against the committed result of every statement prefix,g-page restore, rollback, empty commit, step-5API-019,STG-046at the switch and at commit), 4 fault-matrix ops, base-id andtree_pagesunit tests. Crash-killLargevariant runs every third transaction as a 3-statementWriteTransaction(180 rounds locally). Mutating the claims check, thefreedtruncation, or the held/claims release fails tests. Codex review: no findings.Docs:
CHANGELOG.md,docs/TEST_COVERAGE.md,CLAUDE.md; docs site PR to follow.Not here: #686, #687, #551. No closing keywords: #550 is closed by hand after merge; #473 stays open.
🤖 Generated with Claude Code
https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT