8000 feat(query): add ordinary avg with window-compatible value handling by adityamukho · Pull Request #791 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- **Ordinary average** (#777): `(avg ?v)` no longer raises PRS-027. Grouped and ungrouped queries use the same numeric population and arithmetic as window `avg`: skip nil and nonnumeric values, sum exactly, round the total once to Float, then divide by the numeric count. Integer totals are not saturated. Results are Float, or nil for an existing group without numbers; no matching rows yields no rows. The total can overflow before division, and division rounds separately. Existing window behavior is unchanged. Retained exact-sum allocations for ordinary `sum` and `avg` are now charged to the query memory limit.

### Documentation

- Retire tracker #383 in favor of the v3.0.0 milestone and its release gates. Other ordinary/window value-handling differences are tracked in #789.

### Performance

- **Resolve expression keyword literals once per operator** (#774). `=` and `!=` reuse a literal keyword's entity UUID across input rows, including nested expressions, rule bodies and prepared slots. The cache contains only literals from that expression and is discarded with the operator; dynamic operands retain the existing comparison semantics. No API, dependency or file-format change. See the [measurements](docs/benchmarks/774-expression-keywords.md).
Expand Down
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,13 +218,15 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**2006 tests** (1989 passing, 17 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**2018 tests** (2001 passing, 17 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

The #760 deterministic storage suite adds three native unit tests in a separate `--cfg minigraf_shuttle` build, outside the default count. Run only `graph::storage::deterministic::` with that cfg; ordinary unit tests do not enter its scheduler. `deterministic-schedules.yml` runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See `docs/testing/deterministic-schedules.md` for scope, controls and findings.

**Testing conventions** — see the Testing Conventions section below before writing any tests.

On v3, ordinary `avg` skips nil/nonnumeric values and divides the exact rounded total by the numeric count, matching window `avg` (#777). It returns Float, or nil for an existing group with no numbers; no matching bindings yields no rows. Other aggregate consistency work is #789; ordered-avg optimization remains #781.

## Current Maintainer Context

On `v3`, expression `=` / `!=` resolves literal keywords once per operator (#774), including nested expressions and substituted prepared slots. Dynamic values do not populate the cache. This is a performance-only change, with no v2.x backport; see `docs/benchmarks/774-expression-keywords.md`.
Expand All @@ -234,7 +236,7 @@ On `v3`, expression `=` / `!=` resolves literal keywords once per operator (#774
Current plan (see `ROADMAP.md` and the milestones):
- **In-memory ancestors (#723)**: eligible positive three-pattern delta runs retain an entity-first alternative with the same `Old`/`Delta`/`Full` ranges and separately planned head slots. Each round can select it when the committed dictionary is empty, the source relation has at most 128 rows and no stored records, and a capped AEVT count favors entity probes over building an uncached pending value map. Populated committed generations keep AVET probes. No additional write-time index.
- **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release.
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, optional `:distinct` output #493, `lag`/`lead` #182, sliding row frames #183, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, optional `:distinct` output #493, `lag`/`lead` #182, sliding row frames #183, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1)) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **Input order (#641)** targets v3.0.0 in three slices: committed scalar Sort omission, one-group count/count-distinct and partition-at-a-time windows are implemented in the dependent PRs for `v3`. The reviewed design, fallback scope and acceptance gates are in `docs/superpowers/specs/2026-10-28-641-input-order-design.md`.
- **Scalar key decoder proofs (#693)** target v3.0.0 before the release cut. The pure decoder refactor preserves key bytes and INT-049 behavior; Kani covers round trips and bounded malformed inputs. The approved design and CI limits are in `docs/superpowers/specs/2026-10-29-693-decoder-proofs.md`.
- **v3.next** (the release after v3.0.0) holds new features (rule persistence #241; OPFS `BrowserDb` #441).
Expand Down
2 changes: 1 addition & 1 deletion PHILOSOPHY.md
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,7 @@ When evaluating a feature or design choice, ask:
These are goals, not a description of today. You'll know Minigraf has succeeded when:

1. **Ubiquity**: Developers say "just use Minigraf" for embedded graph storage
2. **Trust**: Known for never losing data, crash-safe, reliable. Not yet met: v2.x has a known data-integrity bug (#371), and the work to meet this goal is tracked in #383. See the [known issues](https://github.com/project-minigraf/minigraf/issues?q=is%3Aissue+is%3Aopen+label%3Aknown-issue).
2. **Trust**: Known for never losing data, crash-safe, reliable. Not yet met: v2.x has a known data-integrity bug (#371), and the work to meet this goal is tracked in the [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1). See the [known issues](https://github.com/project-minigraf/minigraf/issues?q=is%3Aissue+is%3Aopen+label%3Aknown-issue).
3. **Simplicity**: New users are productive in under 5 minutes
4. **Size**: Core binary within its ~1.2MB budget, minimal dependencies
5. **Portability**: Runs everywhere from Raspberry Pi to browsers
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ Minigraf is a **single-file embedded graph database** that lets you:

For example, `(lag ?salary :over (:partition-by ?dept :order-by ?hire-date))` reads the preceding employee's salary within each department; the first row gets `nil`. `lead` reads the following row.

The unreleased v3.0.0 branch supports ordinary `(avg ?score)` (#777). It skips nil and nonnumeric values, counts only numbers, and returns a Float average or nil for an existing group without numbers. No matching rows produces no result row. It uses the same exact-total-then-divide arithmetic as window `avg`; a finite total can overflow before division. V2 requires `:over` for `avg`.

The unreleased v3.0.0 branch adds bounded sliding windows such as `(avg ?salary :over (:partition-by ?dept :order-by ?hire-date :rows 2 preceding))` (#183).

On v3.0.0, an ascending value-only query such as `[:find ?v :where [?e :score ?v] :order-by [?v] :limit 10]` can read directly in index order and stop early when all committed `:score` history has one value kind (integer, float, or Boolean) and its source range has no pending writes. A query whose full order cannot be proved uses the existing sort (#641). Built-in `count` and `count-distinct` can also release completed groups when one committed integer/Boolean/float key is proved ordered (including mixed integer/float groups), with no `:with`, windows, callbacks or `collect-set`. Memory then follows the active group and batches; distinct members within one group still grow. Other aggregates keep full-input validation. When every built-in window uses the same proved nonconstant partition key, the engine can also finish and release one partition at a time. Its memory follows the largest partition plus bounded batches; one growing partition still grows. Unpartitioned windows, incompatible keys, aggregate queries, callbacks and `collect-set` keep full-input evaluation. A surrounding sort or distinct operation can still require more memory.
Expand Down
8 changes: 5 additions & 3 deletions ROADMAP.md
8BD0
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# Minigraf Roadmap

> Planned work for Minigraf, a bi-temporal Datalog database. v2.x has known data-integrity issues ([pinned list](https://github.com/project-minigraf/minigraf/issues/421)); v3.0.0 fixes them (tracker #383).
> Planned work for Minigraf, a bi-temporal Datalog database. v2.x has known data-integrity issues ([pinned list](https://github.com/project-minigraf/minigraf/issues/421)); v3.0.0 fixes them ([v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1)).

**Philosophy**: Embedded graph memory for agents, mobile, and the browser — built on the SQLite approach: be boring, be reliable, be embeddable.

Ordinary `avg` is implemented for v3.0.0 (#777), matching window numeric filtering. Consistent handling for the other ordinary aggregates remains a v3.0.0 release requirement (#789).

Completed releases and their implementation details live in the [CHANGELOG](CHANGELOG.md). This document tracks planned, exploratory, and other future work.

---
Expand Down Expand Up @@ -47,7 +49,7 @@ Further v2.x releases are data-integrity and security patches only.

## v2.0.2 — Final Planned v2.x Release (released 2026-09-27)

**The v2.x line (`main`) ends with a patch.** File format stays v7 and the public API does not change. All new feature work has moved to v3.x. Tracker: #383. Details are in the [CHANGELOG](CHANGELOG.md).
**The v2.x line (`main`) ends with a patch.** File format stays v7 and the public API does not change. All new feature work has moved to v3.x. Work list: [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1). Details are in the [CHANGELOG](CHANGELOG.md).

Shipped:

Expand Down Expand Up @@ -222,7 +224,7 @@ When evaluating features, ask:

- v2.x maintenance on `main`: data-integrity and security fixes only
- v3.0.0 (format v8 and data integrity) on the `v3` branch, then v3.next features
- Production readiness for high-stakes deployments, tracked in #383
- Production readiness for high-stakes deployments, tracked in the [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1)
- Ecosystem work tracked in [`minigraf-examples`](https://github.com/project-minigraf/minigraf-examples)
- Developer tools tracked in `minigraf-inspector` and `minigraf-visualizer`

Expand Down
9 changes: 5 additions & 4 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -736,16 +736,17 @@ See the [Datalog Reference](https://project-minigraf.github.io/minigraf-docs/lat

**Error text**: `'{}' is a window function and requires an ':over (...)' clause`

**Cause**: Window functions (`rank`, `row-number`, `dense-rank`, `ntile`, `percent-rank`, `cume-dist`) must be accompanied by an `:over` clause specifying ordering and/or partitioning.
**Cause**: Window functions `rank` and `row-number` must be accompanied by an `:over` clause specifying ordering and/or partitioning.

**Resolution**:
- Add an `:over` clause: `(rank :over (:order-by ?score))`.
- Since v3.0.0, `avg` also works as an ordinary aggregate without `:over` (#777); v2 requires the window form.
- See the [Datalog Reference — Window Functions](https://project-minigraf.github.io/minigraf-docs/latest/datalog-reference/#window-functions).

**Example**:
```datalog
(query {:find [?e (rank)]
:where [[?e :score ?score]]})
(query [:find ?e (rank ?score)
:where [?e :score ?score]])
```
*Missing `:over`; use `(rank :over (:order-by ?score))`*

Expand Down Expand Up @@ -4046,7 +4047,7 @@ these two arities.
**Error text**: `unknown aggregate function: '{}'`

**Cause**: A `:find` or `:with` clause referenced an aggregate function
name that is neither a built-in (`count`, `sum`, `avg`, `min`, `max`, etc.)
name that is neither a built-in (`count`, `count-distinct`, `sum`, `sum-distinct`, `avg`, `min`, `max`)
nor a user-defined function registered via `db.register_aggregate()`.

**Resolution**:
Expand Down
8 changes: 6 additions & 2 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@

| | Default features | With `--features test-hooks` |
|---|---|---|
| **Passing** | 1,989 | 2,022 |
| **Passing** | 2,001 | 2,035 |
| **Ignored** | 17 | 18 |
| **Total** | 2,006 | 2,040 |
| **Total** | 2,018 | 2,053 |

#777 adds 12 default-feature tests (three accumulator tests, one engine memory test and eight integration tests) and one additional `test-hooks` plan matrix. The focused 13-test run passes. Strict native/browser/WASI clippy passes, and the rebased release binary is 1,263,144 bytes against the unchanged 1,263,616-byte cap.

#774 adds one default-feature unit test (also present with `test-hooks`). On
2026-10-29, after rebasing onto #723, its full default suite passed 1,989 tests
Expand Down Expand Up @@ -43,6 +45,8 @@ Also outside those counts: three native unit tests in the separate #760 `minigra

## Covered Areas

- **Ordinary average** (#777; `tests/ordinary_avg_test.rs`, `aggregate/tests.rs`, `engine/tests.rs`): grouped and ungrouped averages, every nonnumeric value kind skipped, nil versus no matching rows, Float results, exact unsaturated totals, signed zero/subnormal underflow, infinities/NaNs, input permutations, prepared execution/cursors, batches, recursive rules, transaction reads, checkpoint/reopen and alternative plans. Default and bounded windows preserve skipped rows. Sum/avg exact-sum heap state is charged once before allocation, transferred on finish, and released on errors, rewind and close. Existing neighboring aggregate errors and dropped groups are preserved.

- **Expression keyword literals** (#774): a test-only conversion counter verifies that increasing input from 8 to 2,048 rows does not increase literal resolution work, across `=` / `!=`, both operand orders, nested expressions, bindings, negation, disjunction and rule bodies. Existing join-equality tests also cover changing prepared keyword slots across executions and unchanged keyword output, in memory, after checkpoint and after reopen. The envelope harness checks all eight new expression/control shapes on regular and irregular fixtures.

- **In-memory ancestor join ordering** (#723; four tests): a 1,999-edge tree returns three ancestors within a 128 KiB query budget, also after staged edge replacement; cursor close releases its metered state. Reordered and text-order plans return identical bags across keyword/ref edges, cycles, multi-valued edges, retractions, transaction/valid-time filters, empty and unbound queries, in memory and with checkpointed facts. A direct comparison against the original delta plan protects keyword/ref output representation in rule heads. The bounded cost count excludes writes after a captured snapshot, includes staged records, respects the cap, and excludes prefix-neighbor attributes.
Expand Down
Loading
Loading
0