Repository navigation
feat(engine): rule queries on the streaming engine (#548 PR 2) - #670
Merged
Merged
Conversation
Recursive derives each stratum's relations by semi-naive rounds over id rows: round 1 reads every relation in full; later rounds run each rule once per top-level invocation of its stratum (that one Delta, those before Old, after Full); a rule reading its stratum otherwise runs in full every round. Scans read derived rows after the committed and pending ranges, with nil metadata. Rule bodies and the query body plan with the existing Builder, rule invocations read as patterns. Rule queries route to the engine unless the executor must run them (write transaction, :as-of timestamp, a body the engine does not plan). Binary-size gate raised to 1,260 KiB until #551. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
#548) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
This was referenced Oct 20, 2026
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 6 of #473, second of three PRs for #548 (plan steps 8–15 of
docs/superpowers/plans/2026-10-20-548-recursive.md; specdocs/superpowers/specs/2026-10-20-recursive-engine-design.md§5).What
Rule queries run on the streaming engine.
engine/relation.rs): per evaluated predicate, derived rows of two cells (entity, value), a set byencode_valueidentity (kinds distinct, keyword ≠ ref,-0.0≠0.0, one NaN), seeded with the predicate's stored facts. Rows are appended when a round ends, soOld/Delta/Fullare ranges of oneVec; an entity index serves probes. Magic seeds are rows.engine/scan.rs): after the committed and pending ranges, the rows of the relations the pattern names (:p→p; a variable or_attribute → every relation).Deltareads derived rows only. Derived rows havenilmetadata;:db/valid-atbinds the query's valid time.engine/plan.rs):plan_rulescollects the evaluated predicates and strata asStratifiedEvaluatordoes, plans every rule body with the existingBuilder(rule calls read as patterns), and plans the query body the same way. For rounds after the first, a rule gets one run per top-level call of its own stratum (that callDelta, earlier onesOld, later onesFull, the delta pattern first). A rule that reads its stratum another way (in anorbranch, a:p/ variable-attribute pattern,:db/valid-at) is naive: allFullevery round.Recursive(engine/recursive.rs): strata in order, rounds to a fixpoint; heads checked against the relation,QRY-013/INT-022as on the executor;INT-020after 1,000 rounds ormax_derived_factsderived facts.DatalogExecutor::open_rule_query): a rule query runs on the engine when its body and every rule body plan, outside a write transaction, without an:as-oftimestamp; the executor otherwise. Errors in the executor's order. The oracle compares both.Size gate
This PR goes over the temporary 1,230 KiB gate, by about 15 KiB on CI's toolchain (+34 KiB locally: 1,247 → 1,281 KiB; v3 was ~1,211 KiB on CI). The growth is the new rule planner,
Recursiveand the derived scan source; the executor's rule path stays until #551. Approved: gate raised to 1,260 KiB until #551 (binary-size.yml, PHILOSOPHY.md §4).Measured
#394 envelope at 100K facts, bench machine, one NUMA node, base
origin/v3vs this branch, two alternating runs:recursive_ancestorsEvery other shape within run-to-run noise in both orders. 1M/10M and the 5M-derived acceptance test come in PR 3.
Tests
tests/recursive_engine_test.rs(10): transitive closure (linear, non-linear), mutual recursion,not/orstrata recursing (Recursion through a rule with not / not-join / or returns too few rows #633's cases), naive rules, merged visibility (magic attribute included), identity,nilmetadata, heads, bothINT-020limits, rule queries with aggregates/not-join/or/windows; in memory, pending and committed, oracle on.relation.rsidentity and ranges. Two routing tests updated: rule calls innot/ornow run on the engine.test-oracle1,705 passing.Codex review: no findings.
Deviations from the plan
Docs site: project-minigraf/minigraf-docs (branch
v3/548-recursive), architecture page's streaming-engine section.Part of #473 and #548 (both closed by hand after PR 3).
🤖 Generated with Claude Code
https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT