Repository navigation
streaming engine 2/9: operator core, Scan and oracle harness (#473) #544
Description
Part 2 of 9 of #473. Design: docs/superpowers/specs/2026-10-11-streaming-engine-design.md (branch docs/473-streaming-engine-spec), §5, §6.1, §6.3, §9.
- Internal
Batch(1,024 rows, columnar, plan-time variable slots) andCell(Eid/Iid/Val), provisional ids for pending entities and idents -
Scanover EAVT/AEVT/AVET/VAET with net-assert on keys and temporal pushdown, merging committed and pending facts; folds in perf: attribute+value patterns scan the whole attribute; AVET index is never read #518 (rebaseperf/518-avet-lookup) -
Filter/Expr,Project(id→value only for rows returned; folds in perf: committed scans look up every entity id in the DICT tree #530),Limit/Offsetwithclo 8000 sepropagation - Planner v1 producing operator trees; single-pattern queries run on the new engine
- Old executor kept as a test-only oracle; differential runs over the model-based, property, golden-corpus and integration suites
- Cross-kind matching rules (§5.4) pinned by oracle tests
- Pseudo-attributes (
:db/valid-from,:db/valid-to,:db/tx-count,:db/tx-id) asMetacolumns ofScan(§5.5) - Provisional ids assigned lazily per cursor; counted as query state
PRs for this part target v3 and must not use a closing keyword for #473.
Activity
docs: engine module and test counts (#544)
test(query): float constants stay on the executor (#544)
docs: CHANGELOG for the engine's first PR (#544)
Merge pull request #588 from project-minigraf/feat/544-operator-core
adityamukho commentedon Oct 14, 2026
PR 1 of 3 merged (#588): key-level reads, Scan, Project, planner v1 for single plain patterns, oracle + CI job. Plan: docs/superpowers/plans/2026-10-14-544-operator-core.md. Limit/Offset move to #306 (no syntax reaches them before then); close propagation stays here (PR 3).
adityamukho commentedon Oct 14, 2026
For PR 2 (plan step 9, cross-kind constants): reuse from the parked branch
perf/518-avet-lookup(fe545dc) rather than rewriting:keys::attribute_kind_prefix(withValueKind) andkeys::attribute_string_prefix;tests/live_reads_test.rs(keyword↔ref, UUID↔keyword, floats,:as-of, retractions, reopen), ported as engine tests.Not its executor changes (
selective_fact_fetch,Scan::AttributeValue/AttributeKind): the executor stays unchanged as the oracle (plan decision 8).git diff a16f385d fe545dc1 -- src/storage/keys.rs src/storage/mod.rs tests/live_reads_test.rs.Delete
perf/518-avet-lookup(local and origin) once #544 is closed.test(storage): one assertion for the kind prefix test (#544)
Merge pull request #590 from project-minigraf/feat/544-exprs
docs: lazy cursor, engine operators, test count (#544)
fix(query): cache DICT lookups for one pull only (#544)
docs: streaming cursor measurements (#544)
Merge pull request #591 from project-minigraf/feat/544-lazy-cursor
adityamukho commentedon Oct 14, 2026
Done on v3 in three PRs: #588 (operator core, Scan, Project, oracle; #518 folded in), #590 (Filter/Bind, pseudo-attributes, cross-kind constants), #591 (lazy cursor, close and counters, envelope measurement). Docs site: project-minigraf/minigraf-docs#16. Limit/Offset moved to #306. Attribute scan at 10M facts: 458 → 7 MiB peak RSS, 3.12 → 2.59 s; point queries unchanged. Next: #545.