Repository navigation
streaming engine 1/9: snapshots and page quarantine (#473) #543
Description
Part 1 of 9 of #473. Design: docs/superpowers/specs/2026-10-11-streaming-engine-design.md (branch docs/473-streaming-engine-spec), §4.
At open, a cursor takes the active generation's committed reader, a pin on that generation, an Arc of the pending store with its length at open, and the valid-time "now". A checkpoint replaces the pending store instead of clearing it, so the cursor's pending positions stay readable. PageAllocator does not hand out an id freed by the checkpoint that produced generation g while a pin on a generation below g exists. Pins and quarantine are in memory only: no format change, and nothing is pinned after a crash or reopen.
- Pins and quarantine in
PageAllocator(file, memory and browser backends) - Checkpoint swaps in a new pending store (
facts,pending_keys,pending_indexes) instead ofpost_checkpoint_clear - Cursor snapshot: committed reader, pin, pending
Arc+ length, valid-time now; no lock held acrossnext_batch - Whole snapshots:
WriteTransaction::commitand WAL replay append a transaction in one batch; a checkpoint publishescommittedand the new pending store under oneFactDatawrite lock (§4.1) - Quarantine also applies to
rebuild_indexes' page pool and toBrowserDb's release of freed pages (take_released, BrowserDb: bound page growth via v8 free-list reuse; avoid v7→v8 migration bloat #440) -
fact_logmoves to the pin and opens without the write lock; checkpoints proceed while a log is open;API-013deprecated (never raised, stays registered) - Tests: answer unchanged by writes and checkpoints that free pages a cursor reads; quarantined ids are reused after the pin drops; crash with an open cursor
Fixes the v3 side of #554 and #555 (data-integrity: those issues stay open until a release ships the fix; no closing keywords). Un-ignore their reproducers in tests/snapshot_atomicity_test.rs (branch test/snapshot-atomicity-repro).
PRs for this part target v3 and must not use a closing keyword for #473.
Activity
adityamukho commentedon Oct 13, 2026
PR 1 of 3 merged on
v3: #578 (a727e08). Step 1 of the plan: a checkpoint publishes the new committed reader, its generation and a new pending store under oneFactDatawrite lock (spec §4.1), and the pending store is replaced, not cleared. Checklist item 4 (whole snapshots) is now done for the checkpoint side.Remaining: PR 2 (pins, page quarantine in
PageAllocatorandrebuild_indexes, fact log on the pin,API-013deprecated) and PR 3 (query and cursor snapshots,BrowserDbrelease, crash test with an open cursor, docs).Merge pu 9423 ll request #579 from project-minigraf/feat/543-pins
Merge pull request #580 from project-minigraf/feat/543-snapshots-read
Merge pull request #587 from project-minigraf/perf/543-all-facts-copy
adityamukho commentedon Oct 14, 2026
Done on v3: PR #578 (PendingStore + one-lock publish), PR #579 (pins, allocator quarantine, fact log on the pin, API-013 deprecated), PR #580 (query/cursor snapshots, BrowserDb, crash test, docs); recursive-query regression from #580 fixed by #587. Docs site: project-minigraf/minigraf-docs#14.