Repository navigation
feat(engine): query memory accounting and query_memory_limit, QRY-012 (#549 PR 1) - #676
Merged
Merged
Conversation
…#549) Every engine holder of state counts a per-entry byte estimate on one Meter per query (spec §3): HashJoin build sides, AntiJoin and Union tables, Union's buffered branch rows, Distinct, Aggregate (groups, count-distinct and sum-distinct members, finished rows), Window, rule Relations (derived rows and stored-fact keys) and their Recursive temporaries, and IdSpace (provisional ids and the DICT cache). Containers grow with try_reserve. OpenOptions::query_memory_limit (default usize::MAX) and a refused allocation both raise QRY-012; Stream::close drops the operator tree so the meter returns to 0. A keyless HashJoin compares its inner pattern with the product of the outer patterns' estimates (spec §6.1), so the #575 shape builds on each small inner pattern. The oracle accepts an engine QRY-012. Prepared queries now get the database's limits; they ran rule queries with the default max_derived_facts. Plan: docs/superpowers/plans/2026-10-21-549-memory-limit.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
Codex review: end_round grew by_entity uncounted and without try_reserve. add now indexes each row's entity at its position after the round, charging a new entity's entry and reserving both containers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Oct 21, 2026
adityamukho
added a commit
that referenced
this pull request
Oct 21, 2026
…#682) Backport of the v3 fix in #676. PreparedQuery::execute built its DatalogExecutor without set_limits, so a prepared rule query ran with the default max_derived_facts and max_results whatever OpenOptions set: a lower limit did not stop it, and a higher one did not help it. prepare now hands the database's (max_derived_facts, max_results) to the PreparedQuery, which sets them on every execute, as every other read path in db.rs does. Claude-Session: https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 1 of #549 (streaming engine 7/9). Spec:
docs/superpowers/specs/2026-10-21-query-memory-limit-design.md(#674). Plan:docs/superpowers/plans/2026-10-21-549-memory-limit.md(in this PR).What
meter.rs: oneMeterper query (limit, used, peak), each holder counting through aHeldthat charges before insert, takes back on a refusedtry_reserve, and releases onrewind/close/drop.HashJoinbuild side,AntiJoinandUnionhashed tables,Union's buffered branch rows,Distinct,Aggregate(groups,count-distinct/sum-distinctmembers, finished rows),Window(cells, values, finished rows),Relations(derived rows, stored-fact keys, entity index),Recursivetemporaries,IdSpace(provisional ids, DICT cache). Every container they grow goes throughtry_reserve.QRY-012, in two forms (limit, allocator).ERROR_REFERENCErow and section; the "reserved" note is gone.OpenOptions::query_memory_limit(pub field and builder, defaultusize::MAX). It applies toexecute,query, prepared and rule queries.Stream::closedrops the operator tree and keeps its counters, so the meter is 0 after a drain, an early close, an error or a dropped cursor. Hashednot/ortables stay charged across a correlated parent's close, as they stay held.QRY-012.HashJoinbuild side (spec §6.1):optimizer::product_costof the outer patterns. The Nightly fuzz failure (main) #575 shape builds on each 3-row inner pattern from the second join on.Changes from the spec (approved before implementation)
Relations::know(stored facts' identity keys) andsum-distinctset members are counted too.min/max's one value per group, and the transient columnsaggregate::windowsreturns.Found along the way
OpenOptionslimits.PreparedQuerybuilt its executor withoutset_limits, so prepared rule queries ran with the defaultmax_derived_facts. Fixed here, with a test that fails without the fix.mainhas the same gap; a backport will be a separate issue.WriteTransactionis counted until the transaction stages a write. After that it runs on the executor, uncounted until streaming engine 8/9: streaming transactions (#473) #550/streaming engine 9/9: remove the old executor, measure, docs (#473) #551. This is documented.Tests
meter.rs(4), engine unit tests (7: charge/release per holder,QRY-012naming each holder, the allocator form through a test hook, cursor rows then error,rewind/closerelease, keyless build side),tests/memory_limit_test.rs(7, public API).test-oraclesuites green locally. Clippy is clean (--all-features, wasm32browser).end_round), fixed in the second commit.Binary size
+13 KiB on the same local toolchain (
origin/v31,281 → 1,294 KiB; this toolchain builds about 36 KiB larger than CI). Expect about 1,258 of the 1,260 KiB gate on CI.Follow-ups
PR 2 (
feat/549-acceptance: 1M-group and #575 k=13 acceptance, smoke job, measured figures), then the binding PRs. A docs-site PR follows. #549 closes by hand after PR 2.🤖 Generated with Claude Code
https://claude.ai/code/session_019kY61Y11gcRD91Y6b7jbGT