Repository navigation
feat(query): lazy cursor over the streaming engine (#544, PR 3) - #591
Merged
Merged
Conversation
Cursor pulls one internal batch at a time from the engine's operators (Operator trait: next_batch, close, counters); close and drop stop all work. Rows gathered before a read error are returned first, the error by the next call. DatalogExecutor::open gives cursors an Answer: a whole result, or an engine Stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
IdSpace kept every committed UUID and ident it projected, so a cursor's memory grew with the rows it returned (189 MiB for an attribute scan over 10M facts). DICT ids never change, so their lookups are now dropped after each pull; provisional ids stay for the whole query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#544 PR 3 of 3 (plan steps 10–12 in
docs/superpowers/plans/2026-10-14-544-operator-core.md). Part of #473; no closing keywords.What changes
Cursorholds the engine's operator tree (engine::Stream, owning itsSnapshot) and pulls one internal batch (≤ 1,024 rows) when its buffered rows run out.closeand drop stop all work.DatalogExecutor::openreturns anAnswer: the stream for an engine query, a whole result for everything else.executedrains the same stream.Operatortrait (next_batch,close,counters);ScanandApply(Filter/Bind/pseudo step) implement it.Counters(rows in/out, batches) per operator, shown in the cursor'sDebug; the profiler (post-1.0: build query profiler #185) reads them later.None. The iterator does the same.IdSpacekept every committed UUID it projected, so cursor memory grew with the answer (189 MiB at 10M in the first commit). DICT lookups are now cached for one pull; provisional ids stay for the whole query.INT-040), not for the whole query.Tests (+8; 1,479 passing, 10 ignored)
cursor::tests: one pull reads one internal batch (scan counters), buffered rows need no pull, a closed stream reads nothing more (spec §10), filter counters chain, the answer is unchanged by writes and by checkpoints that free leaves the cursor has yet to read (spec §10, file-backed), and by later writes (in memory).query::engine::tests: read error after gathered rows (failing test operator); DICT cache empty after each pull.query::engine::ids: provisional ids survive a pull.cargo test --features test-oraclepasses.tests/cursor_test.rsand binding suites unchanged.Measurement (#394 envelope, A/B, two swapped rounds per side, NUMA-pinned on an idle server)
attribute_scanp50, v3 → PRattribute_scanpeak RSS, v3 → PR:as-of/:valid-at/ hot entityExecutor-only shapes (joins, aggregates, rules) unchanged. Details in
docs/BENCHMARKS.md.Codex review
max_resultsnot applied on the engine path: not a bug, it limits rule derivation only (bug: a rule query fails with INT-020 on any database with 1,000,000 or more live facts #528) and the engine runs no rules.Docs
CHANGELOG, CLAUDE.md, ROADMAP.md, README, TEST_COVERAGE, BENCHMARKS. Docs site: project-minigraf/minigraf-docs PR (streaming-engine section under
@since v3.0.0).🤖 Generated with Claude Code
https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR