8000 feat: add lag and lead window functions (#182) by adityamukho · Pull Request #763 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **`lag` and `lead` window functions** (#182). `(lag ?value :over (:partition-by ?group :order-by ?key))` reads the previous row's value; `lead` reads the next row's. Both use the established total partition order, including named-variable tie breaks, and return `nil` at partition edges. `:desc` reverses the primary key, so the previous row has a higher or equal key. They run over grouped rows after aggregation and before `:distinct`, sorting and pagination. An optional offset or explicit default is not part of this release. The plain `(lag ?v)` and `(lead ?v)` forms remain available to registered user aggregates. `PRS-030` is deprecated and no longer emitted.
- **Optional distinct query results** (#493). A bare `:distinct` flag removes repeated final `:find` tuples after aggregates and windows and before `:order-by`, `:offset` and `:limit`. Queries still return bags by default; `:with` keeps its aggregate grouping behavior. Equality follows `count-distinct` (including one NaN and one floating zero), with canonical `-0.0`/NaN output, also inside sets. The distinct keys count toward `query_memory_limit`; ordered top-k queries still retain a key for every unique tuple they encounter.
- **Fallible and re-entrant user functions** (#180, PR 1). `Minigraf::try_register_predicate` and `Minigraf::try_register_aggregate` take closures that return `Result<_, UdfError>` (`UdfError` is exported at the crate root). A call that returns `Err` fails the query with the new `QRY-014`, ``user function `<name>` failed: <message>``, at the first failing call: it is never read as `false` or `nil`, and no later call of any user function is made for that query. `execute` and `PreparedQuery::execute` return it; a cursor treats it as any other read error, returning the rows of the batches finished before the failure, then the error, then `None`. `register_predicate` and `register_aggregate` are unchanged. A query now reads its functions from a snapshot of the registry taken when it opens, and holds no database lock while a user function runs: a function registered later is not seen by a query or cursor that is already open, and a user function may call back into its own database (query, prepare, read the fact log, write, define rules, register functions, checkpoint, verify, rebuild indexes). Its writes are not seen by the query that called it. Inside a `WriteTransaction` those calls return `INT-001` on that thread, except registration, which runs. Before, a user function that registered a function deadlocked. This is the core of UDF registration in the language bindings, which follows binding by binding.
- **User functions in `BrowserDb`** (#180, PR 7). `db.registerPredicate(name, fn)` and `db.registerAggregate(name, { init, step, finish })`, both synchronous. A function receives each value as a `{ type, value }` object, as `@minigraf/node`'s `FactValue` (an integer's `value` is a `bigint`, a ref's its UUID string, a set's an array of these); a predicate returns a boolean; `init()` returns an accumulator, `step(acc, value)` the next one and `finish(acc, count)` the result, as a plain value or a typed form, as a prepared query's binding takes it. A function that throws fails the query with `QRY-014` and the error's `message` (else `String(e)`), and so does a predicate returning a non-boolean or a `finish` result that is not a value. A UDF may call `query`, `prepare` and the `register*` methods synchronously; `execute`, `checkpoint` and `importGraph` started from one run after the outer query returns. The registry holds no JavaScript value, only ids into a table local to the registering thread, so the crate stays free of `unsafe`.
Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1874 tests** (1860 passing, 14 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**1880 tests** (1866 passing, 14 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand All @@ -227,8 +227,8 @@ See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

Current plan (see `ROADMAP.md` and the milestones):
- **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release.
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, optional `:distinct` output #493, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **v3.next** (the release after v3.0.0) holds new features (lag/lead #182, sliding row frames #183, rule persistence #241, OPFS `BrowserDb` #441).
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, optional `:distinct` output #493, `lag`/`lead` #182, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **v3.next** (the release after v3.0.0) holds new features (sliding row frames #183, rule persistence #241, OPFS `BrowserDb` #441).
- Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0. All v2.x known issues are listed in pinned issue #421 (`known-issue` label); `data-integrity`/`corruption`/`durability` issues fixed in a v2.x patch stay open until that patch is published; issues fixed on the unreleased `v3` branch close on merge (closing keywords do not fire on `v3`, so close them with `gh issue close`) and stay listed in #421 (CONTRIBUTING.md).
- Support policy and tiers: PHILOSOPHY.md §7 and §10. v2.x gets data-integrity and security fixes for 12 months after v3.0.0. Tier 1 = Rust + Python + browser WASM (from v3.0.0); other bindings are Tier 2.

Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,14 @@ A tiny, self-contained graph database with **Datalog queries** and **bi-temporal
Minigraf is a **single-file embedded graph database** that lets you:
- ✅ **Query relationships with Datalog** - Recursive rules, natural graph traversal
- ✅ **Time travel through history** - Bi-temporal queries (transaction time + valid time)
- ✅ **Window functions** - `sum/count/min/max/avg/rank/row-number :over (partition-by … :order-by …)` in `:find` clauses
- ✅ **Window functions** - `sum/count/min/max/avg/rank/row-number/lag/lead :over (partition-by … :order-by …)` in `:find` clauses
- ✅ **Bounded and ordered results** - `:limit N` returns at most N rows and stops the work once it has them; `:order-by [?score :desc] :limit 10` keeps only the top ten as it reads, and `:offset` pages through an ordered answer
- ✅ **Prepared statements** - Parse + plan once with `$slot` bind tokens, execute thousands of times
- ✅ **Embed anywhere** - Native, WASM, mobile, IoT - one `.graph` file
- ✅ **Zero configuration** - Just `Minigraf::open("data.graph")` and you're done

For example, `(lag ?salary :over (:partition-by ?dept :order-by ?hire-date))` reads the preceding employee's salary within each department; the first row gets `nil`. `lead` reads the following row.

**Status**: See [ROADMAP.md](ROADMAP.md) for planned work and current direction.

## Why Datalog?
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ Scope — format and storage integrity:
Scope — query API:

- Optional `:distinct` result tuples, preserving bag output by default (#493; done on `v3`)
- `lag` and `lead` window functions for the previous/next row within an ordered partition (#182; v3.0.0)
- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`). The binding cursor objects follow. Behind the cursor, the streaming engine (#473) runs every query and produces rows as they are pulled: patterns, expressions and pseudo-attributes (#544), joins and `:db/valid-at` (#545), `not` and `or` (#546), aggregates and windows (#547), rules (#548), and transaction reads (#550); the executor's query path is gone (#551; done on `v3`). Operators do not spill: memory follows operator state, with an opt-in per-query limit (`QRY-012`, #549).
- Native `:limit N`, `:order-by` and `:offset`, with a top-k `Sort` whose memory follows `N` (#306; done on `v3`, with the 10M-fact acceptance tests moved from #473; spec `docs/superpowers/specs/2026-10-24-limit-order-by-design.md`). Skipping the sort when an index already returns the rows in order is #641 (v3.next).
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log pins its generation: checkpoints run but do not reuse its pages (#543; `API-013` deprecated). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462.
Expand Down Expand Up @@ -128,7 +129,6 @@ Scope — release-gate testing and documentation:
Additive features, kept out of v3.0.0 so the #371 fix is not held up by them:

- Rule persistence — design first (#241)
- `lag`/`lead` window functions (#182)
- Sliding row frames — `:rows N preceding` (#183; builds on #182)
- OPFS-backed persistent mode for `BrowserDb`: a sync access handle in a dedicated Worker, no in-memory buffer; IndexedDB stays the main-thread backend. Starts at Tier 2 inside the Tier 1 browser binding until it has shipped (#441)
- Temporal graph-traversal research (#273; may result in documentation or a blog post rather than code)
Expand Down
15 changes: 8 additions & 7 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ with no `CodedError` anywhere in its chain.
| PRS-027 | Window function requires :over clause | Parser |
| PRS-028 | Window expression cannot be empty | Parser |
| PRS-029 | Window function name must be a symbol | Parser |
| PRS-030 | lag/lead not supported in this version | Parser |
| PRS-030 | lag/lead not supported (deprecated) | Parser |
| PRS-031 | Function is not window-compatible | Parser |
| PRS-032 | Function requires variable before :over | Parser |
| PRS-033 | Function requires :over after variable | Parser |
Expand Down Expand Up @@ -295,7 +295,7 @@ with no `CodedError` anywhere in its chain.
| INT-055 | Rule predicate disappeared during rollback | Internal |
| INT-056 | Write to a read-only storage backend | Internal |

Deprecated codes: STG-003 and STG-004 (pre-v7 header sizes) are deprecated as of v3.0.0 and are no longer emitted, because formats v1–v6 are rejected with STG-028 before any header-size check. API-013 (checkpoint while a fact log is open) is deprecated as of v3.0.0 and is no longer emitted, because checkpoints run while fact logs are open. INT-033, INT-034 and INT-035 (bind-value mistakes) are deprecated as of v3.0.0 and are no longer emitted: the same mistakes are API-022, API-023 and API-024, since they are the caller's. They stay registered and documented, and error codes are never removed or reused.
Deprecated codes: STG-003 and STG-004 (pre-v7 header sizes) are deprecated as of v3.0.0 and are no longer emitted, because formats v1–v6 are rejected with STG-028 before any header-size check. API-013 (checkpoint while a fact log is open) is deprecated as of v3.0.0 and is no longer emitted, because checkpoints run while fact logs are open. PRS-030 (lag/lead unsupported) is deprecated as of v3.0.0 because those functions are supported. INT-033, INT-034 and INT-035 (bind-value mistakes) are deprecated as of v3.0.0 and are no longer emitted: the same mistakes are API-022, API-023 and API-024, since they are the caller's. They stay registered and documented, and error codes are never removed or reused.

---

Expand Down Expand Up @@ -781,22 +781,23 @@ See the [Datalog Reference](https://project-minigraf.github.io/minigraf-docs/lat
```
*Use `(rank :over ...)` not `(:rank :over ...)`*

### PRS-030 lag/lead not supported in this version
### PRS-030 lag/lead not supported (deprecated)

> **Deprecated in v3.0.0.** No longer emitted: `lag` and `lead` work as window functions. Kept for reference; this code will not be reused.

**Error text**: `'{}' is not supported in this version; lag/lead are planned for a future release`

**Cause**: The `lag` and `lead` window functions are not yet implemented in this version.
**Cause**: A v2.x query used `lag` or `lead` in a window expression; v2.x did not implement them.

**Resolution**:
- Remove `lag`/`lead` from the query.
- Follow [#182](https://github.com/project-minigraf/minigraf/issues/182) for progress on `lag`/`lead` implementation.
- Use v3.0.0 or later for `lag`/`lead` window expressions.

**Example**:
```datalog
(query {:find [?e (lag ?score :over (:order-by ?ts))]
:where [[?e :score ?score] [?e :ts ?ts]]})
```
*`lag` is not yet available; restructure the query without it*
*On v2.x this returned `PRS-030`; on v3.0.0 and later it is accepted.*

### PRS-031 Function is not window-compatible

Expand Down
1 change: 1 addition & 0 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,7 @@ These tests compare the database against an independent answer rather than a sin
- `INT-041` from a failing group.
- `src/query/datalog/aggregate.rs` (15 unit tests) checks every accumulator over every order of small inputs, and that `value_order` is a total order.
- `tests/window_functions_test.rs` checks `PRS-082` and a window over the groups of an aggregate query.
- `tests/window_functions_test.rs` (#182) also checks `lag`/`lead` on whole and partitioned results, descending order and ties, singleton and mixed-type partitions, `nil` edges, long strings, keywords, refs and sets, grouped rows, distinct output, pagination, prepared queries, cursors and preserved user aggregates of those names. Parser tests cover malformed shapes and `tests/plan_independence_test.rs` checks both directions under reordered plans.
- `Aggregate` on the engine (#547 PR 2, `src/query/engine/tests.rs`, 3 tests): every built-in aggregate grouped by keywords, refs, booleans, floats and long strings, over `nil` values, keywords and refs, with `:with` and `or`, at six temporal settings in memory, after a checkpoint and after reopen, against their frozen answers. Also: an ungrouped aggregate reads every row and returns one batch of one row; the zero row and empty answers; groups in value order over several batches; `INT-041` from a later group; a group dropped by `min` of no value.
- `tests/aggregation_test.rs` checks that a group of `0.0` and `-0.0` returns `-0.0` in either insertion order.
- **Query validation** (#406, #414)
Expand Down
8 changes: 8 additions & 0 deletions docs/superpowers/plans/2026-10-26-lag-lead-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Implementation plan: lag and lead (#182)

1. Confirm the `origin/v3` worktree's window suite is green. Review this plan and its design with Claude; resolve findings before implementation.
2. Replace the old `PRS-030` tests in `tests/window_functions_test.rs`, `tests/error_codes_prs_test.rs`, and the parser unit test. Add end-to-end cases for unpartitioned/partitioned `lag`/`lead`, singleton and `nil` boundaries, descending order, peer ties, grouped input, value types (long strings, keyword, ref, set), pagination and distinct, prepared queries and cursors. Add parser tests for missing/extra arguments, missing order key, unbound/grouped variables and the plain registered aggregate form. Add tied and descending cases to `tests/plan_independence_test.rs`. Run each relevant new test first and observe the expected failure.
3. Add `WindowFunc::{Lag, Lead}` and map names in `types.rs`. Teach the parser to accept their window form; leave the no-`:over` user-aggregate path intact. Exempt both names from the registry lookup in `executor::check_find_functions` and from `Running::new` inside `aggregate::windows`. In `aggregate::windows`, use the already sorted partition row indices to fetch each neighbor's argument, defaulting to `Value::Null` at the edge. Keep running-aggregate peer semantics unchanged. Run focused tests to green.
4. Mark `PRS-030` deprecated in `docs/ERROR_REFERENCE.md` while leaving its code registered. Update `CLAUDE.md`, `ROADMAP.md`, `README.md`, `CHANGELOG.md`, `docs/TEST_COVERAGE.md`, Datalog examples/reference, and directly affected docs for v3.0.0. The pest grammar already admits the form: update its rank/row-number comment and run the verifier. Update the separate docs site with `@since v3.0.0` text and open its PR per CLAUDE.md.
5. Run `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`, the grammar/plan-independence suites, and the v3 binary-size gate. Inspect diff and confirm no storage or dependency changes. Ask Claude for a final review, verify and address every finding, then run required checks again.
6. Commit and push the implementation branch, and open a PR against `v3` with a concise description and validation. Monitor CI and fix failures until green. Do not merge.
Loading
Loading
0