8000 docs(spec): streaming query engine design (#473) by adityamukho · Pull Request #552 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content

docs(spec): streaming query engine design (#473) - #552

Merged
adityamukho merged 4 commits into
v3from
docs/473-streaming-engine-spec
Oct 11, 2026
Merged

adityamukho merged 4 commits into
v3from
docs/473-streaming-engine-spec

Conversation

@adityamukho
@adityamukho adityamukho commented Oct 11, 2026 •
Copy link
Copy Markdown
Collaborator

Design spec for the streaming query engine, #473. No code.

  • §1 Terms: pending store, generation, checkpoint, statement, chunk.
  • §4 Snapshots: a cursor holds the active generation's reader and a pin, plus an Arc of the pending store and its length at open. Nothing is copied. A checkpoint swaps the pending store instead of clearing it. Freed pages are quarantined in memory while an older generation is pinned, with no format change. The fact log uses the same pin, and API-013 is deprecated.
  • §5 Rows: columnar batches of DICT or provisional ids until Project (perf: committed scans look up every entity id in the DICT tree #530). The matcher's cross-kind rules and pseudo-attributes are kept.
  • §6 Operators and planner: Scan over all four indexes, folding in perf: attribute+value patterns scan the whole attribute; AVET index is never read #518.
  • §7 Memory: no default limit and no spilling. Byte accounting, try_reserve, and an optional query_memory_limit that raises QRY-012.
  • §8 Transactions: one with more than TX_CHUNK_FACTS staged facts writes copy-on-write working trees and commits as a checkpoint. Statement bases keep "last statement wins" and API-011 exact across chunks.
  • §9 Migration: the existing executor is a test-only oracle, deleted before v3.0.0.
  • §11 lists the changes from the summary approved in the brainstorm.

Sub-issues #543–#551, in delivery order.

Refs #473

🤖 Generated with Claude Code

https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h

Snapshots with page quarantine, columnar id batches, the operator set,
memory accounting with an optional cap (QRY-012), large transactions
committed as checkpoints, oracle-based migration and the sub-issue plan.

Refs #473

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h
@codecov
codecov Bot commented Oct 11, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

adityamukho and others added 3 commits October 11, 2026 22:11
…nsaction statement rules (#473)

Checked every claim against the code: pending store and checkpoint terms,
in-memory databases, pseudo-attributes, max_results, float join keys,
last-statement-wins across chunks.

Refs #473

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h
Whole snapshots (single-append commit, one-step checkpoint publish),
quarantine in rebuild_indexes and BrowserDb page release, fact log
without the write lock, executor behaviours made explicit, large
transaction commit order, recovery and statement rollback.

Refs #473

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h
Refs #473

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h
@adityamukho
adityamukho merged commit f3ee8d3 into v3 Oct 11, 2026
22 checks passed
@adityamukho
adityamukho deleted the docs/473-streaming-engine-spec branch October 11, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0