8000 perf(query): stream certified aggregate groups (#781) by adityamukho · Pull Request #793 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Performance

- **Release certified aggregate groups** (#781). Ordered aggregation now includes ordinary `sum`/`avg`, fixed-scalar `min`/`max`, and Integer/nil `sum-distinct`. Effective grouping tuples may include hidden `:with` keys, constants and repetitions when they reduce to one ascending Integer/Boolean dimension; existing single numeric/Boolean keys remain supported. Arguments are certified from constants, metadata or raw attribute history using the pinned reader, with pending/staged data and unsupported origins retaining full-input evaluation. Every `:with` Sort is retained. Limits count emitted groups after min/max drops; a retained Sort still drains its input. Completed groups are released, but one growing distinct group remains linear. Small selective grouped queries pay proof startup cost. Common result-buffer memory failures can precede later unread source failures, as on the existing ordered-count path; registered finalizers retain full-drain behavior. No aggregate semantics, syntax, public API, dependency or storage format changes.

- **Resolve expression keyword literals once per operator** (#774). `=` and `!=` reuse a literal keyword's entity UUID across input rows, including nested expressions, rule bodies and prepared slots. The cache contains only literals from that expression and is discarded with the operator; dynamic operands retain the existing comparison semantics. No API, dependency or file-format change. See the [measurements](docs/benchmarks/774-expression-keywords.md).

- **Avoid whole-attribute reads for selective in-memory ancestors** (#723). Eligible three-pattern recursive delta runs can start from a small derived relation and probe stored facts by entity, preserving the same semi-naive ranges and answers. The choice is reconsidered each round using bounded pending-index counts. Populated committed generations retain AVET value probes; general value joins retain their existing behavior. No persistent index, dependency, API or format change. The approved v3 binary-size budget increases from 1,230 to 1,234 KiB for this change.
Expand Down
8 changes: 4 additions & 4 deletions CLAUDE.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The unreleased v3.0.0 branch supports ordinary `(avg ?score)` (#777). It skips n

The unreleased v3.0.0 branch adds bounded sliding windows such as `(avg ?salary :over (:partition-by ?dept :order-by ?hire-date :rows 2 preceding))` (#183).

On v3.0.0, an ascending value-only query such as `[:find ?v :where [?e :score ?v] :order-by [?v] :limit 10]` can read directly in index order and stop early when all committed `:score` history has one value kind (integer, float, or Boolean) and its source range has no pending writes. A query whose full order cannot be proved uses the existing sort (#641). Built-in `count` and `count-distinct` can also release completed groups when one committed integer/Boolean/float key is proved ordered (including mixed integer/float groups), with no `:with`, windows, callbacks or `collect-set`. Memory then follows the active group and batches; distinct members within one group still grow. Other aggregates keep full-input validation. When every built-in window uses the same proved nonconstant partition key, the engine can also finish and release one partition at a time. Its memory follows the largest partition plus bounded batches; one growing partition still grows. Unpartitioned windows, incompatible keys, aggregate queries, callbacks and `collect-set` keep full-input evaluation. A surrounding sort or distinct operation can still require more memory.
On v3.0.0, an ascending value-only query such as `[:find ?v :where [?e :score ?v] :order-by [?v] :limit 10]` can read directly in index order and stop early when all committed `:score` history has one value kind (integer, float, or Boolean) and its source range has no pending writes. A query whose full order cannot be proved uses the existing sort (#641). Built-in `count`, `count-distinct`, `sum`, and `avg` can release completed groups when the pinned source proves their grouping order (#781). `min`/`max` also qualify when their arguments are proved Integer/Float/Boolean/nil; `sum-distinct` requires Integer/nil. A single Integer/Boolean/Float key (including mixed numeric groups) is supported. A longer effective tuple, including hidden `:with` keys, must reduce to one varying Integer/Boolean slot plus repeated occurrences and proved Integer/Boolean/nil constants. Any requested final Sort is retained for `:with`. Small grouped point queries pay proof startup cost; see the [measurements](docs/benchmarks/781-ordered-aggregate.md). Memory follows the active group and batches; distinct members within one group still grow. A limit can stop after enough emitted groups, but dropped min/max groups do not count and a retained Sort drains the input. Callbacks, `collect-set`, aggregate+window queries and unproved arguments keep full-input evaluation. When every built-in window uses the same proved nonconstant partition key, the engine can also finish and release one partition at a time. Its memory follows the largest partition plus bounded batches; one growing partition still grows. Unpartitioned windows, incompatible keys, aggregate queries, callbacks and `collect-set` keep full-input evaluation. A surrounding sort or distinct operation can still require more memory.

On the unreleased v3 branch, selective in-memory ancestor queries can use entity probes instead of reading the whole edge attribute (#723). This is a query-planning optimization; it adds no storage index or configuration.

Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ Scope — query API:
- Bounded sliding row frames, `:rows N preceding`, for window aggregates (#183; v3.0.0, implemented on `v3`; builds on #182)
- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`). The binding cursor objects follow. Behind the cursor, the streaming engine (#473) runs every query and produces rows as they are pulled: patterns, expressions and pseudo-attributes (#544), joins and `:db/valid-at` (#545), `not` and `or` (#546), aggregates and windows (#547), rules (#548), and transaction reads (#550); the executor's query path is gone (#551; done on `v3`). Operators do not spill: memory follows operator state, with an opt-in per-query limit (`QRY-012`, #549).
- Projected UUID lookup batches walk dense DICT ranges (#530; implemented on `v3`), keeping temporary memory bounded to a batch and query-cache charges in projection order.
- Native `:limit N`, `:order-by` and `:offset`, with a top-k `Sort` whose memory follows `N` (#306; done on `v3`, with the 10M-fact acceptance tests moved from #473; spec `docs/superpowers/specs/2026-10-24-limit-order-by-design.md`). #641 targets v3.0.0 in three slices: snapshot-bound order proofs and Sort omission for committed scalar ranges, then one-group count/count-distinct and partition-at-a-time windows. All three slices are implemented in the dependent #641 PRs for `v3`. Streaming groups require a proved scalar grouping key and no `:with`, windows, callbacks or `collect-set`; other aggregates retain full-input validation. Compatible built-in windows with one proved nonconstant partition key hold one partition and bounded batches; unpartitioned/incompatible windows, aggregate queries, callbacks and `collect-set` retain full-input evaluation. See `docs/superpowers/specs/2026-10-28-641-input-order-design.md`.
- Native `:limit N`, `:order-by` and `:offset`, with a top-k `Sort` whose memory follows `N` (#306; done on `v3`, with the 10M-fact acceptance tests moved from #473; spec `docs/superpowers/specs/2026-10-24-limit-order-by-design.md`). #641 targets v3.0.0 in three slices: snapshot-bound order proofs and Sort omission for committed scalar ranges, then one-group count/count-distinct and partition-at-a-time windows. All three slices are implemented in the dependent #641 PRs for `v3`. #781 extends ordered grouping to sum/avg, fixed-scalar min/max, Integer/nil sum-distinct, and effective tuples reducible to one ordered Integer/Boolean slot plus scalar constants/repetitions, including hidden `:with` keys. Every `:with` Sort is retained. Unproved keys/arguments, callbacks, `collect-set`, rules and aggregate+window queries retain full-input evaluation. Compatible built-in windows with one proved nonconstant partition key hold one partition and bounded batches; unpartitioned/incompatible windows, aggregate queries, callbacks and `collect-set` retain full-input evaluation. See `docs/superpowers/specs/2026-10-28-641-input-order-design.md` and `docs/superpowers/specs/2026-10-29-781-ordered-aggregate-design.md`.
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log pins its generation: checkpoints run but do not reuse its pages (#543; `API-013` deprecated). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462.
- Read-only open: `OpenOptions::read_only` takes a shared lock, so several readers can hold a file at once, writes nothing to the file or its WAL (a WAL is applied in memory, a v7 file is read without migrating), and refuses writes with `API-014` (#429; Rust API done on `v3`). The bindings' `open_with_options` follows.
- Raw log writer: `LogWriter` builds a new database from `FactRecord`s with their `tx_count`, `tx_id`, valid-time windows and assertion flags kept, gaps allowed (a purged transaction stays a hole for `:as-of`), at `<path>.partial` renamed into place by `finish` (#431; Rust API done on `v3`). Completes extract (#429, #430), transform, load. The binding objects follow.
Expand Down
17 changes: 16 additions & 1 deletion benches/minigraf_bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,16 @@ fn bench_ordered_groups(c: &mut Criterion) {
("distinct", "count-distinct", false),
("count_limit", "count", true),
("distinct_limit", "count-distinct", true),
("avg", "avg", false),
("sum", "sum", false),
("min", "min", false),
("max", "max", false),
("sum_distinct", "sum-distinct", false),
("avg_limit", "avg", true),
("sum_limit", "sum", true),
("min_limit", "min", true),
("max_limit", "max", true),
("sum_distinct_limit", "sum-distinct", true),
] {
group.bench_with_input(BenchmarkId::new(mode, label), &n, |b, &n| {
let dir = tempfile::tempdir().unwrap();
Expand All @@ -258,8 +268,13 @@ fn bench_ordered_groups(c: &mut Criterion) {
.path(&path)
.open()
.unwrap();
let arg = if matches!(func, "count" | "count-distinct") {
"?e"
} else {
"?v"
};
let query = format!(
"(query [:find ?v ({func} ?e) :where [?e :val ?v] :order-by [?v] {}])",
"(query [:find ?v ({func} {arg}) :where [?e :val ?v] :order-by [?v] {}])",
if limited { ":limit 10" } else { "" }
);
b.iter(|| {
Expand Down
11 changes: 11 additions & 0 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

This document records reproducible local benchmark snapshots. Completed release-performance history belongs in the [CHANGELOG](../CHANGELOG.md); continuous CI history is available on [Bencher](https://bencher.dev/perf/minigraf/plots).

## Certified ordered aggregates (#781, unreleased)

[Measurements and reproduction](benchmarks/781-ordered-aggregate.md) cover
sum/avg, scalar extrema, integer distinct sums and safe effective grouping
tuples after #789. Fixed-group Aggregate peaks remain 480–1,200 bytes from
100K to 10M facts, compared with 285–755 MB for forced full-input evaluation.
Separate growing distinct controls retain linear state. At 10M facts, the
bounded full drains improve by 17.4–31.2% and limit-10 queries take 0.60–0.81 ms.
Small grouped point queries instead pay about 176–188 µs of proof startup work.
The release binary remains below the unchanged 1,263,616-byte cap.

## Literal keyword/ref expression equality (#774, unreleased)

On 2026-10-29, caching expression literal resolutions reduced the measured
Expand Down
Loading
Loading
0