8000 feat: Phase 7.7b — User-Defined Functions (v0.17.0) by adityamukho · Pull Request #70 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


8000
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,26 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## v0.17.0 — Phase 7.7b: User-Defined Functions (2026-04-02)

### Added
- `Minigraf::register_aggregate(name, init, step, finalise)` — register a custom aggregate
function usable in both `:find` grouping and `:over` (window) clauses
- `Minigraf::register_predicate(name, f)` — register a single-argument filter predicate
usable in `[(name? ?var)]` `:where` clauses
- `FunctionRegistry::register_aggregate_desc` / `register_predicate_desc` (internal API)
- `WindowFunc::Udf(String)` and `UnaryOp::Udf(String)` AST variants for runtime-resolved functions
- `UdfOps`, `AggImpl`, `PredicateDesc` types in `functions.rs`

### Changed
- `AggregateDesc` now uses `AggImpl` discriminator instead of `window_compatible`+`window_ops`
- `apply_expr_clauses` now returns `Result<Vec<Binding>>` and accepts `&FunctionRegistry`
- `eval_expr` accepts `Option<&FunctionRegistry>` for UDF predicate resolution
- `WindowSpec::func_name()` now returns `String` instead of `&'static str`
- Parser emits `Udf` variants for unknown names instead of erroring (runtime validation)

### Test count: 727 tests

## v0.16.0 — Phase 7.7a: Window Functions (2026-04-02)

### Added
10BC8 Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ Auto-migrates v1/v2/v3/v4/v5 → v6 on open/checkpoint.

## Test Coverage

**707 tests passing** (unit + integration + doc).
**727 tests passing** (unit + integration + doc).
See `TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "minigraf"
version = "0.16.0"
version = "0.17.0"
edition = "2024"
description = "Zero-config, single-file, embedded graph database with bi-temporal Datalog queries"
license = "MIT OR Apache-2.0"
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ db.execute(r#"(rule [(reachable ?a ?b) [?a :friend ?b]])

```bash
cargo run # interactive Datalog REPL
cargo test # run 705 tests
cargo test # run 727 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
38 changes: 24 additions & 14 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,7 @@ Current v5 stores index data as paged blobs (page type `0x11`). v6 introduces pr
- **7.5** ✅ Tests + Error Coverage (617 tests; executor.rs 85.71%, evaluator.rs 89.29% branch coverage; CI coverage gate + nightly llvm-cov)
- **7.6** ✅ Temporal Metadata Bindings + Range Queries (`:db/valid-from`, `:db/valid-to`, `:db/tx-count` as queryable pseudo-attributes; unlocks Time Interval, Time-Point Lookup, Time-Interval Lookup query classes)
- **7.7a** Window Functions — `sum`, `count`, `min`, `max`, `avg`, `rank`, `row-number` with unbounded-preceding frame; `FunctionRegistry` introduced (built-ins only); `lag`/`lead` and sliding frames deferred to post-1.0 backlog
- **7.7b** User-Defined Functions (UDFs) — public `register_aggregate`/`register_predicate` API on the `FunctionRegistry` introduced in 7.7a
- **7.7b** ✅ User-Defined Functions (UDFs) — public `register_aggregate`/`register_predicate` API on the `FunctionRegistry` introduced in 7.7a
- **7.8** Prepared Statements (parse + plan once, execute many times, temporal bind slots; implemented after full clause set including predicate-argument positions)
- **7.9** Publish Prep (crates.io — API cleanup, rustdoc, clippy, `unwrap` audit, CI matrix)

Expand Down Expand Up @@ -890,7 +890,15 @@ UDFs are the natural generalisation: if the engine can call built-in aggregates

---

#### 7.7b User-Defined Functions (UDFs)
#### 7.7b User-Defined Functions (UDFs) ✅ COMPLETE

**Status**: ✅ Complete (v0.17.0, 2026-04-02)

**Summary**: `register_aggregate` and `register_predicate` public API added to `Minigraf`.
UDFs plug into grouping aggregation, window computation, and `:where` predicate filtering
via type-erased `Box<dyn Any + Send>` accumulators and `Arc<dyn Fn>` closures.
`FunctionRegistry` extended with `AggImpl` discriminator and `PredicateDesc`.
727 tests.

**Goal**: Allow embedders to extend the query engine with custom aggregate functions and filter predicates registered at runtime, using the same `FunctionRegistry` that built-in aggregates and window functions use.

Expand Down Expand Up @@ -1575,7 +1583,8 @@ When evaluating features, ask:
- ✅ Phase 7.5: Complete (March 2026) - Cross-feature tests, error-path coverage, ~86-89% branch coverage, 617 tests
- ✅ Phase 7.6: Complete (April 2026) - Temporal metadata bindings (`:db/valid-from`, `:db/valid-to`, `:db/tx-count`, `:db/tx-id`, `:db/valid-at`), 647 tests
- ✅ Phase 7.7a: Complete (April 2026) - Window functions (`sum/count/min/max/avg/rank/row-number :over`), `FunctionRegistry`, 705 tests
- 🎯 Phase 7.7b–7.8: UDFs, prepared statements; ≥90% branch coverage - **NEXT**
- ✅ Phase 7.7b: Complete (April 2026) - User-Defined Functions (`register_aggregate`/`register_predicate`), 727 tests
- 🎯 Phase 7.8: Prepared statements; ≥90% branch coverage - **NEXT**
- 🎯 Phase 8: 3-4 months (Cross-platform — WASM, mobile, language bindings)
- 🎯 Phase 9: Ongoing (Ecosystem — integration examples, cookbook, GraphRAG/LangChain examples)
- 🎯 **v1.0.0: 9-12 months**
Expand All @@ -1586,19 +1595,20 @@ When evaluating features, ask:

## Current Focus

**Right Now**: Phase 7.7a Complete — Phase 7.7b Next (UDFs)
**Right Now**: Phase 7.7b Complete — Phase 7.8 Next (Prepared Statements)

**Phase 7.7a Achievements**:
1. ✅ `FunctionRegistry` in `src/query/datalog/functions.rs` — string-keyed registry; all built-in aggregates migrated; `window_ops` (init/step/finalise)
2. ✅ `WindowFunc`, `Order`, `WindowSpec`, `FindSpec::Window` types in `types.rs`
3. ✅ `parse_window_expr` in `parser.rs` — `(func ?v :over (:partition-by ?p :order-by ?o :desc))` syntax; `lag`/`lead` rejected with "not supported"
4. ✅ `apply_post_processing` + `apply_window_functions` + `compute_aggregation` in `executor.rs`
5. ✅ `FunctionRegistry` wired through `db.rs` (`Minigraf::Inner`)
6. ✅ `tests/window_functions_test.rs` — 12 integration tests covering all window functions, partition-by, desc order, mixed aggregate+window, edge cases
7. ✅ 705 tests passing; version bumped to v0.16.0
**Phase 7.7b Achievements**:
1. ✅ `WindowFunc::Udf(String)` and `UnaryOp::Udf(String)` AST variants in `types.rs`
2. ✅ `UdfOps`, `AggImpl`, `PredicateDesc` types in `functions.rs`; `UdfStepFn`/`UdfFinaliseFn` type aliases
3. ✅ Parser emits `Udf` variants for unknown function names (runtime validation)
4. ✅ `apply_expr_clauses` returns `Result<Vec<Binding>>` and pre-validates UDF predicate names
5. ✅ `eval_expr` accepts `Option<&FunctionRegistry>` for UDF predicate resolution
6. ✅ `Minigraf::register_aggregate` and `Minigraf::register_predicate` public API methods
7. ✅ `tests/udf_test.rs` — 9 integration tests (custom aggregate, predicate, window UDF, name collisions, edge cases)
8. ✅ 727 tests passing; version bumped to v0.17.0

**Immediate Next Steps (Phase 7.7b)**:
1. UDFs — embedder-registered aggregate and predicate functions via `FunctionRegistry`
**Immediate Next Steps (Phase 7.8)**:
1. Prepared Statements — parse + plan once, execute many times, temporal bind slots

**Key Decisions Made**:
- ✅ Datalog query language (simpler, better for temporal)
Expand Down
67 changes: 55 additions & 12 deletions TEST_COVERAGE.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Minigraf Test Coverage Report

**Last Updated**: Phase 7.7a COMPLETE - Window Functions ✅
**Last Updated**: Phase 7.7b COMPLETE - User-Defined Functions (UDFs) ✅

## Test Summary

**Total Tests**: 707 ✅
- ✅ 478 unit tests (lib)
**Total Tests**: 727 ✅
- ✅ 487 unit tests (lib)
- ✅ 10 bi-temporal tests (integration)
- ✅ 10 complex query tests (integration)
- ✅ 9 recursive rules tests (integration)
Expand All @@ -25,9 +25,22 @@
- ✅ 8 error handling tests (integration, Phase 7.5 — error-path coverage; 1 ignored: or+neg-cycle bug)
- ✅ 16 temporal metadata tests (integration, Phase 7.6 — `:db/valid-from`, `:db/valid-to`, `:db/tx-count`, `:db/tx-id`, `:db/valid-at`)
- ✅ 12 window function tests (integration, Phase 7.7a — cumulative sum/count/min/avg, rank with ties, row-number, partition-by, desc ordering, mixed aggregate+window, edge cases, lag/lead parse rejection)
- ✅ 9 UDF tests (integration, Phase 7.7b — custom aggregates, custom predicates, UDF as window function, name collision guards, runtime errors, thread safety)
- ✅ 6 doc tests

**Status**: ✅ **All 707 tests passing** (1 ignored: confirmed or+neg-cycle stratification bug)
**Status**: ✅ **All 727 tests passing** (1 ignored: confirmed or+neg-cycle stratification bug)

## Phase 7.7b Completion Status: ✅ COMPLETE

**Phase 7.7b Features** (current, complete):
- ✅ `UdfOps` and `PredicateDesc` types in `src/query/datalog/functions.rs` — register custom aggregates (init/step/finalise closures) and custom predicates (filter closure)
- ✅ `FunctionRegistry::register_aggregate` and `register_predicate` methods; collision guards reject re-registration of built-in names or duplicate UDFs
- ✅ `FindSpec::Udf` and `WhereClause::UdfPredicate` variants in `types.rs`; UDF aggregates usable in `:find` and `:over` window specs; UDF predicates usable in `:where`
- ✅ Parser extended: UDF aggregate invocations in `:find` / `:over`; UDF predicate invocations in `:where`; unknown function names deferred to runtime, not rejected at parse time
- ✅ Executor routes UDF aggregates through `FunctionRegistry` at query time; UDF predicates evaluated per binding row
- ✅ `Minigraf::register_aggregate` and `register_predicate` on the public API (`db.rs`)
- ✅ `tests/udf_test.rs` — 9 integration tests
- ✅ 727 tests passing (unit + integration + doc)

## Phase 7.7a Completion Status: ✅ COMPLETE

Expand All @@ -38,7 +51,7 @@
- ✅ `apply_post_processing`, `compute_aggregation`, `apply_window_functions`, `project_find_specs` in `executor.rs` — replaces `apply_aggregation`/`apply_agg_func`
- ✅ `FunctionRegistry` wired through `db.rs` (`Minigraf::Inner` gains `Arc<RwLock<FunctionRegistry>>`)
- ✅ `tests/window_functions_test.rs` — 12 integration tests (cumulative sum, running count/min/avg, rank with ties, row-number, partition-by, desc ordering, mixed aggregate+window, single-row and empty-result edge cases, lag/lead parse rejection)
- ✅ 707 tests passing (unit + integration + doc)
- ✅ 718 tests passing (unit + integration + doc)

## Phase 7.6 Completion Status: ✅ COMPLETE

Expand Down Expand Up @@ -451,6 +464,32 @@
- ✅ Negative cycle via `not-join` at rule registration → `Err`, rule not registered
- ✅ `not` and `not-join` coexist in the same query

### Window Functions (`tests/window_functions_test.rs`) - ✅ 12 tests (Phase 7.7a)

- ✅ Cumulative sum over ordered partition
- ✅ Running count and running min
- ✅ Running average
- ✅ Rank with ties (equal values share rank)
- ✅ Row-number (unique sequential position regardless of ties)
- ✅ Partition-by — window resets per group
- ✅ Descending order in window spec
- ✅ Mixed aggregate + window in same `:find`
- ✅ Single-row result (window function on one row)
- ✅ Empty-result edge case (no matching facts)
- ✅ `lag` / `lead` rejected at parse time

### User-Defined Functions (`tests/udf_test.rs`) - ✅ 9 tests (Phase 7.7b)

- ✅ `custom_aggregate_geometric_mean` — UDF aggregate registered and used in `:find`
- ✅ `custom_aggregate_empty_result` — UDF aggregate on empty result set returns correct identity
- ✅ `custom_predicate_filter` — UDF predicate in `:where` filters binding rows
- ✅ `udf_as_window_function` — UDF aggregate used inside `:over` window spec
- ✅ `name_collision_builtin_aggregate` — registering a UDF with a built-in name returns `Err`
- ✅ `name_collision_udf_on_udf` — registering a second UDF with the same name returns `Err`
- ✅ `unknown_function_runtime_error` — invoking an unregistered aggregate name at query time returns `Err`
- ✅ `unknown_predicate_runtime_error` — invoking an unregistered predicate name at query time returns `Err`
- ✅ `thread_safety` — concurrent UDF registration and query execution from multiple threads

---

## Coverage Metrics
Expand Down Expand Up @@ -600,6 +639,8 @@ cargo test --test negation_test # stratified not (10)
cargo test --test not_join_test # not-join (14)
cargo test --test aggregation_test # aggregation (24)
cargo test --test predicate_expr_test # arithmetic & predicate expr (28)
cargo test --test window_functions_test # window functions (12)
cargo test --test udf_test # user-defined functions (9)

# Run with output
cargo test -- --nocapture
Expand All @@ -609,9 +650,9 @@ cargo test -- --nocapture

## Conclusion

**Phase 7.2b Status**: ✅ **COMPLETE**
**Phase 7.7b Status**: ✅ **COMPLETE**

**Test Quality**: ✅ **Excellent** — High confidence in all Phase 3-7.2b features
**Test Quality**: ✅ **Excellent** — High confidence in all Phase 3-7.7b features

**Strengths**:
- WAL crash safety verified with real `mem::forget` simulation
Expand All @@ -629,14 +670,16 @@ cargo test -- --nocapture
- Aggregation verified: all 6 aggregate functions, `:with` grouping, bi-temporal + aggregate, rule + aggregate (Phase 7.2a)
- Arithmetic & predicate expressions verified: all operators, silent-drop semantics, int/float promotion, regex validation, expr in not/rule body, bi-temporal + expr (Phase 7.2b)
- Disjunction (`or` / `or-join`) verified: flat queries, rule bodies, nested or/not/expr, or-join with private variables, dependency graph (Phase 7.3)
- 562 tests covering all Phase 3-7.3 features
- Window functions verified: cumulative aggregates, rank/row-number, partition-by, desc ordering, mixed aggregate+window (Phase 7.7a)
- User-defined functions verified: custom aggregates, custom predicates, UDF as window function, name collision guards, runtime error handling, thread safety (Phase 7.7b)
- 727 tests covering all Phase 3-7.7b features

**Confidence Level**: ✅ **Production-ready for Phase 7.3 scope**
**Confidence Level**: ✅ **Production-ready for Phase 7.7b scope**

**Readiness for Phase 7.4**: ✅ **Ready to proceed**
**Readiness for Phase 7.8**: ✅ **Ready to proceed**

The disjunction + aggregation + arithmetic/predicate expression capable, stratified-negation-capable, on-disk B+tree indexed, packed, cached bi-temporal Datalog engine is **solid, well-tested, and benchmarked**.
The UDF-capable, window-function-capable, disjunction + aggregation + arithmetic/predicate expression capable, stratified-negation-capable, on-disk B+tree indexed, packed, cached bi-temporal Datalog engine is **solid, well-tested, and benchmarked**.

---

**Next Steps**: Begin Phase 7.3 (Disjunction — `or` / `or-join`) 🚀
**Next Steps**: Begin Phase 7.8 (Query Planner Improvements / Prepared Statements)
102 changes: 101 additions & 1 deletion src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,12 @@ use crate::graph::types::{Fact, VALID_TIME_FOREVER};
/// epoch (1970-01-01T00:00:00Z), which is a legitimate `valid_from` value.
const VALID_FROM_USE_TX_TIME: i64 = i64::MIN;
use crate::graph::FactStorage;
use crate::graph::types::Value;
use crate::query::datalog::executor::DatalogExecutor;
use crate::query::datalog::executor::QueryResult;
use crate::query::datalog::functions::FunctionRegistry;
use crate::query::datalog::functions::{
AggImpl, AggregateDesc, FunctionRegistry, PredicateDesc, UdfFinaliseFn, UdfOps, UdfStepFn,
};
use crate::query::datalog::parser::parse_datalog_command;
use crate::query::datalog::rules::RuleRegistry;
use crate::query::datalog::types::{AttributeSpec, DatalogCommand, Transaction};
Expand All @@ -27,6 +30,7 @@ use crate::storage::backend::file::FileBackend;
use crate::storage::persistent_facts::PersistentFactStorage;
use crate::wal::WalWriter;
use anyhow::{Result, bail};
use std::any::Any;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex, MutexGuard, RwLock};

Expand Down Expand Up @@ -631,6 +635,102 @@ impl Minigraf {

Ok(facts)
}

// ── UDF registration ─────────────────────────────────────────────────────

/// Register a custom aggregate function.
///
/// `Acc` is any `Send + 'static` type that serves as the accumulator.
/// It is type-erased internally. The function is usable in both `:find`
/// grouping position and `:over` (window) position.
///
/// Returns `Err` if `name` is already registered (built-in or UDF).
///
/// # Example
/// ```
/// # use minigraf::db::Minigraf;
/// # use minigraf::graph::types::Value;
/// let db = Minigraf::in_memory().unwrap();
/// db.register_aggregate(
/// "mysum",
/// || 0i64,
/// |acc: &mut i64, v: &Value| { if let Value::Integer(i) = v { *acc += i; } },
/// |acc: &i64, _n: usize| Value::Integer(*acc),
/// ).unwrap();
/// ```
pub fn register_aggregate<Acc>(
&self,
name: &str,
init: impl Fn() -> Acc + Send + Sync + 'static,
step: impl Fn(&mut Acc, &Value) + Send + Sync + 'static,
finalise: impl Fn(&Acc, usize) -> Value + Send + Sync + 'static,
) -> Result<()>
where
Acc: Any + Send + 'static,
{
let init_boxed: Arc<dyn Fn() -> Box<dyn Any + Send> + Send + Sync> =
Arc::new(move || Box::new(init()) as Box<dyn Any + Send>);
let step_boxed: UdfStepFn = Arc::new(move |acc, v| {
// SAFETY: `init_boxed` always creates `Box<Acc>`, so downcast is infallible.
step(
acc.downcast_mut::<Acc>()
.expect("UDF accumulator type mismatch"),
v,
);
});
let finalise_boxed: UdfFinaliseFn = Arc::new(move |acc, n| {
finalise(
acc.downcast_ref::<Acc>()
.expect("UDF accumulator type mismatch"),
n,
)
});

let desc = AggregateDesc {
impl_: AggImpl::Udf(UdfOps {
init: init_boxed,
step: step_boxed,
finalise: finalise_boxed,
}),
is_builtin: false,
};
self.inner
.functions
.write()
.map_err(|e| anyhow::anyhow!("function registry lock poisoned: {}", e))?
.register_aggregate_desc(name.to_string(), desc)
}

/// Register a custom single-argument filter predicate.
///
/// The predicate is usable in `[(name? ?var)]` `:where` clauses.
/// Returns `Err` if `name` is already registered (built-in or UDF).
///
/// # Example
/// ```
/// # use minigraf::db::Minigraf;
/// # use minigraf::graph::types::Value;
/// let db = Minigraf::in_memory().unwrap();
/// db.register_predicate(
/// "email?",
/// |v: &Value| matches!(v, Value::String(s) if s.contains('@')),
/// ).unwrap();
/// ```
pub fn register_predicate(
&self,
name: &str,
f: impl Fn(&Value) -> bool + Send + Sync + 'static,
) -> Result<()> {
let desc = PredicateDesc {
f: Arc::new(f),
is_builtin: false,
};
self.inner
.functions
.write()
.map_err(|e| anyhow::anyhow!("function registry lock poisoned: {}", e))?
.register_predicate_desc(name.to_string(), desc)
}
}

// ─── WriteTransaction ─────────────────────────────────────────────────────────
Expand Down
Loading
Loading
0