Repository navigation
feat(query): streaming engine core, Scan and oracle (#544, PR 1 of 3) - #588
Merged
Merged
Conversation
Three PRs: key-level reads, Scan, Project, planner v1 and the oracle; Filter/Bind, pseudo-attributes and cross-kind constants; the lazy cursor, measurement and docs. Limit/Offset move to #306. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Part 2 of the streaming engine (#473), first PR. Queries with one pattern and plain :find variables run on a new engine in src/query/engine/: - CommittedReader::key_source gives key-level reads: live_keys walks an index range with net-assert on keys and resumes after a whole triple, so a read holds no lock between batches; DICT accessors. - Scan reads the committed range (EAVT, AEVT or AVET by the bound positions; [?e :a C] reads the AVET range of C, #518), leaving out the triples a pending record decides, then the pending range by whole (entity, attribute) runs. Cells hold ids until Project (#530); pending entities and idents get provisional ids. - Every other query, and reads inside a write transaction, stay on the executor. The executor is the oracle: under cfg(test) and the test-oracle feature each engine query also runs there and the answers must agree; CI runs the whole suite with the feature. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of the streaming engine (#473), first of three PRs for #544. Plan:
docs/superpowers/plans/2026-10-14-544-operator-core.md(steps 1–6).What changes
Queries whose
:whereis one pattern with a stored attribute and whose:findholds plain variables run on a new engine insrc/query/engine/. Everything else runs on the existing executor as before.CommittedReader::key_source()→KeySource: DICT accessors andlive_keys, which walks an index range with net-assert on keys (perf: compute net-assert on v8 index keys before resolving facts (O(live) point queries) #379) and resumes after a whole triple. A read holds no lock between batches.Scan. Picks EAVT, AEVT or AVET from the bound positions;[?e :a C]reads the AVET range ofC(perf: attribute+value patterns scan the whole attribute; AVET index is never read #518). It reads the committed range, leaving out triples that a pending record decides, then reads the pending range by whole(entity, attribute)runs (Snapshot::pending_live).Project, so only returned rows are translated (perf: committed scans look up every entity id in the DICT tree #530, for these queries). Pending entities and idents get provisional ids per query.cfg(test)and the newtest-oraclefeature, every query the engine runs is also run by the executor on the same snapshot, and the answers must agree as bags. The neworaclejob inrust.ymlruns the whole suite with the feature, so the model-based, property, golden-corpus and integration suites check the engine.The cursor is still eager in this PR: the engine drains into a
QueryResult. Laziness,closepropagation and counters come in PR 3. Filter/Bind, pseudo-attributes and keyword/UUID/float constants come in PR 2.Deviations from the spec
They are listed in the plan. Briefly:
Metavariant;Scanreturns its committed rows, then its pending rows, rather than one index order (merging the two is streaming engine 5/9: Aggregate, Sort and Window (#473) #547's);Limit/Offsetmove to No working result-limiting/pagination mechanism for plain (non-recursive) :find queries #306;PreparedQueryplans on each execution.Measurements
Measured with the #394 envelope at 100K and 1M facts on an idle machine. Base
e2bddeband this branch ran in parallel on separate sockets, with the nodes swapped for a second round. Values are p50.attribute_value[?e :name "p…"]attribute_scan(166,666 rows)point_entityas_of/valid_atQueries that stay on the executor (joins, recursion,
count_grouped,hot_*) are within ±3%.Checks
cargo test: 1,467 passed, 10 ignored.cargo test --features test-oracle: same result.Project, and the engine tests fail when pending records no longer hide the committed records they replace.clippy -D warnings: native,--all-features,wasm, wasm32browser, andwasm32-wasip1.#518 is fixed by this PR and will be closed by hand after merge. #530 remains open until joins move over (#545). No closing keyword for #473, #544 or #530.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR