Repository navigation
streaming engine 7/9: memory accounting and optional cap (QRY-012) (#473) #549
Description
Part 7 of 9 of #473. Design: docs/superpowers/specs/2026-10-11-streaming-engine-design.md (branch docs/473-streaming-engine-spec), §7.
- Byte accounting in every stateful operator; growth through
try_reserve -
OpenOptions::query_memory_limit, off by default (usize::MAX) -
QRY-012query memory limit exceeded, naming the operator and the limit;docs/ERROR_REFERENCE.md - Bindings: the option on every
OpenOptionsrecord (UniFFI mapsusize::MAXtoi64::MAX) and the new code - Acceptance: grouped
countwith 1M groups completes; with the cap set below that, fails withQRY-012 - Acceptance (Nightly fuzz failure (main) #575): a join of k patterns with no shared variables and a small
:findprojection (e.g. k = 13 over 3 facts, 1.6M intermediate rows) drained through a cursor (Minigraf::query;:findreturns a bag, soexecuteholds every row by design) runs in memory bounded by the hash-join build sides, not by the row count (today it peaks >5.7 GB); withquery_memory_limitset below that, it fails withQRY-012instead of aborting. Requires the build-side choice to use the product estimate for cartesian joins
PRs for this part target v3 and must not use a closing keyword for #473.
Activity
fuzz(datalog_eval): skip inputs whose patterns join to a cross product (
docs(spec): address review — DICT cache, Union buffer, rule temporari…
adityamukho commentedon Oct 21, 2026
Spec: PR #674 (
docs/superpowers/specs/2026-10-21-query-memory-limit-design.md). Acceptance text corrected::findis a bag, so the #575 test drains a cursor; there is no distinct set.Merge pull request #674 from project-minigraf/docs/549-memory-spec
Merge pull request #676 from project-minigraf/feat/549-memory-limit
docs: memory limit acceptance figures and test counts (#549)
fix(query): keep sum-distinct members charged through Aggregate::fini…
Merge pull request #681 from project-minigraf/feat/549-acceptance
adityamukho commentedon Oct 21, 2026
Done on v3: PR 1 #676 (meter, accounting, QRY-012, OpenOptions::query_memory_limit, keyless build side) and PR 2 #681 (acceptance tests at 1M groups and the #575 product, smoke.yml job, measured figures in docs/BENCHMARKS.md). Binding PRs for the option and QRY-012 follow.