8000 feat(udf): fallible UDFs, QRY-014, registry snapshot and re-entry (#180 PR 1) by adityamukho · Pull Request #746 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1ac61e0
feat(udf): UdfError and QRY-014 (#180)
adityamukho Oct 24, 2026
c5e7235
feat(udf): fallible UDF descriptors and try_register_* (#180)
adityamukho Oct 24, 2026
36bbc3d
feat(udf): a failing predicate fails the query with QRY-014 (#180)
adityamukho Oct 24, 2026
ee81596
feat(udf): a failing aggregate or window fails the query with QRY-014…
adityamukho Oct 24, 2026
b94c367
feat(udf): queries read functions from a registry snapshot (#180)
adityamukho Oct 24, 2026
c3183d7
test(udf): re-entry from inside a user function, both columns of spec…
adityamukho Oct 24, 2026
dcf4fbb
docs(udf): try_register_*, QRY-014, registry snapshot and re-entry (#…
adityamukho Oct 24, 2026
efae0d1
docs(udf): API-008/QRY-008 causes, cursor rows on QRY-014 (#180)
adityamukho Oct 24, 2026
08ae493
Merge remote-tracking branch 'origin/v3' into feat/180-fallible-udfs
adityamukho Oct 24, 2026
1337511
docs: test counts for #180 PR 1
adityamukho Oct 24, 2026
d513680
docs: WASI test count for #180 PR 1
adityamukho Oct 24, 2026
2f6505b
Merge remote-tracking branch 'origin/v3' into feat/180-fallible-udfs
adityamukho Oct 24, 2026
78944a4
Merge remote-tracking branch 'origin/v3' into feat/180-fallible-udfs
adityamukho Oct 24, 2026
27cba53
docs: test counts after merging v3
adityamukho Oct 24, 2026
49d2e61
Merge remote-tracking branch 'origin/v3' into feat/180-fallible-udfs
adityamukho Oct 24, 2026
2a815d0
docs: test counts after merging v3 (#745)
adityamukho Oct 24, 2026
7adf04d
docs: ignored-test count in TEST_COVERAGE prose
adityamukho Oct 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **Fallible and re-entrant user functions** (#180, PR 1). `Minigraf::try_register_predicate` and `Minigraf::try_register_aggregate` take closures that return `Result<_, UdfError>` (`UdfError` is exported at the crate root). A call that returns `Err` fails the query with the new `QRY-014`, ``user function `<name>` failed: <message>``, at the first failing call: it is never read as `false` or `nil`, and no later call of any user function is made for that query. `execute` and `PreparedQuery::execute` return it; a cursor treats it as any other read error, returning the rows of the batches finished before the failure, then the error, then `None`. `register_predicate` and `register_aggregate` are unchanged. A query now reads its functions from a snapshot of the registry taken when it opens, and holds no database lock while a user function runs: a function registered later is not seen by a query or cursor that is already open, and a user function may call back into its own database (query, prepare, read the fact log, write, define rules, register functions, checkpoint, verify, rebuild indexes). Its writes are not seen by the query that called it. Inside a `WriteTransaction` those calls return `INT-001` on that thread, except registration, which runs. Before, a user function that registered a function deadlocked. This is the core of UDF registration in the language bindings, which follows binding by binding.
- **`:limit N` bounds the rows a query returns** (#306, PR 1 of 2). `[:find … :where … :limit 10]` returns at most 10 rows of the answer, and the engine stops reading once it has them: over 10M facts `:limit 10` reads one internal batch. Without `:order-by` (PR 2) which rows is unspecified, and may differ between plans; they are always rows of the unlimited answer. `:limit 0` returns no rows. It applies after aggregates and windows, to rule queries and to cursors, and a prepared query takes `:limit $n`, bound with `BindValue::Val(Value::Integer(n))`. A value that is not an integer `>= 0` or a slot fails with the new `PRS-115`; a repeated `:limit` with `INT-012` and a missing value with `INT-013`, as for the other query options. `:max-results` and `:max-derived-facts` still limit only the facts rules derive.
- **Set values: `#{...}`** (#316, PR 1 of 3). A fact's value can be a finite set of scalars, `[:alice :user/scopes #{:scope/read :scope/write}]`, written in query text, returned as `Value::Set(ValueSet)`, transacted, retracted and queried by exact value (`[?e :user/scopes #{:scope/write :scope/read}]`), joined on and grouped by. Duplicates are removed and element order does not matter. Two elements are the same when `Value`'s order says so: `0.0` and `-0.0` are one element (`-0.0` is kept), every NaN is one element, a keyword never equals a ref and `1` never equals `1.0`. A set cannot hold a set (`ValueSet::new` fails with the new `API-021`; in query text, `PRS-119`); its elements are literals only (`PRS-120`), and an unclosed `#{` is `PRS-121`. In format v8 a set has value tag `0x08`, after refs; a set whose encoded elements take at most 64 bytes is stored in its index keys, a longer one once in a value page, as a long string is. On a file database a stored set may take up to 4,068 bytes, counting each keyword element as 6 bytes and each ref as 10 (`WAL-003`). The REPL prints sets as `#{...}` and `BrowserDb` returns them as JSON arrays, in `Value`'s order.
- **Set-membership predicates** (#316, PR 2 of 3). `[(in? ?role #{:role/admin :role/owner})]` keeps rows whose value is in a set, and `not-in?`, `any-in?` (the two sets share an element), `all-in?` (every element of the first is in the second) and `none-in?` (they share none) complete the five. Each works as a filter or binds a Boolean, `[(any-in? ?scopes #{:scope/write}) ?can-write]`, and takes literal sets, set-valued attributes and bind slots (`BindValue::Val(Value::Set(..))`). Membership is set equality, as for set values: `0.0` is in `#{-0.0}`, NaN is in a set holding NaN, a keyword is not a ref. `in?` of an empty set is false and `not-in?` true; `all-in?` and `none-in?` of an empty first set are true and `any-in?` false. A set where a scalar is needed, or a scalar where a set is needed, is a type mismatch: the filter drops the row and the binding binds nothing, so `not-in?` with a non-set is never true. `contains?` stays string-only. The five names cannot be registered as functions.
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ cargo run < demos/demo_negation.txt
- `executor.rs`: `DatalogExecutor` — `transact`/`retract`/`rule` commands, and every query on the engine (`open_query`, `open_rule_query`; planner errors after the pseudo-attribute check, then every predicate the query and its rules name, `QRY-007`); its own query path is gone (#551)
- `matcher.rs`: `edn_to_value`, `edn_to_entity_id`
- `aggregate.rs`: `Agg` (one aggregate of one group) and `Running` (a window's running value), used by the engine; exact sums (`i128` integers saturated once, fixed-point floats rounded once), `value_order` (one total order of values for window sorts, peers, `rank` and `min`/`max`), `named_vars` (a window's tie-break variables), `windows` (each window's value per row: partitions, sort, peers) (#547)
- `functions.rs`: `FunctionRegistry` (built-ins and UDFs; `Clone` shares the closures), `Functions` — the registry a database shares, `RwLock<Arc<FunctionRegistry>>`: a query takes a `snapshot()` when it opens and holds no lock while a UDF runs, so UDFs may call back into the database; registration copies, inserts and swaps (#180). `UdfError`: a `try_register_*` closure's failure, `QRY-014`; `eval_expr`'s `EvalError` is `NoValue` (row dropped) or `Udf` (query fails)
- `evaluator.rs`: `rule_invocation_to_pattern`, the rule limits and their errors
- `stratification.rs`: `DependencyGraph`, `stratify()` — negative edges + cycle detection; a positive edge to a rule with `not`/`not-join`/`or`/`or-join` also raises the caller, off a cycle (#628)
- `rules.rs`: `RuleRegistry` — thread-safe rule management
Expand Down Expand Up @@ -215,7 +216,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1810 tests** (1796 passing, 14 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**1838 tests** (1824 passing, 14 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Scope — features:
- Query profiler (#185)
- Set values `#{...}`, the set predicates `in?`/`not-in?`/`any-in?`/`all-in?`/`none-in?` and `collect-set` (#316; done on `v3`: core, `BrowserDb` set binds, and set values in Python, Java, Android, Swift, Node.js, C and the browser/WASI packages on their `v3` branches)
- Prepared queries in every binding (#181; done on `v3`: core, `BrowserDb.prepare`, and Python, Java, Android, Swift, Node.js and C on their `v3` branches)
- UDF registration in every binding: the UniFFI ones, C, Node and the browser `BrowserDb` (#180)
- UDF registration in every binding: the UniFFI ones, C, Node and the browser `BrowserDb` (#180; core done on `v3`: fallible UDFs, `QRY-014`, registry snapshot and re-entry; the bindings follow)
- `OpenOptions` exposed over UniFFI through `open_with_options`, so embedders can suppress the close-time checkpoint (#322; done on `v3` in every binding with #465)

Scope — release-gate testing and documentation:
Expand Down
35 changes: 27 additions & 8 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ with no `CodedError` anywhere in its chain.
| QRY-011 | Unknown window function | Query Execution |
| QRY-012 | Query memory limit exceeded | Query Execution |
| QRY-013 | Rule head entity is not an entity | Query Execution |
| QRY-014 | User function failed | Query Execution |
| QRY-015 | collect-set found a set value | Query Execution |
| STG-001 | Invalid header: too short | Storage |
| STG-002 | Invalid magic number: not a .graph file | Storage |
Expand Down Expand Up @@ -2363,14 +2364,12 @@ predicate evaluation, or fact transacting.

**Error text**: `functions lock poisoned`

**Cause**: An internal Rust mutex guarding the custom function registry was poisoned — a previous operation panicked while holding the lock.
**Cause**: The lock guarding the custom function registry was poisoned: a thread panicked while it held the lock. A query takes the lock only to copy the registry's handle when it opens. Since v3.0.0 no user code runs under it, so a custom function that panics does not poison it.

**Resolution**:
- Restart the process.
- If panics recur when calling custom registered functions, investigate the function implementations for panics.
- If this occurs without any custom functions, file a bug.
- Restart the process. This indicates a bug in Minigraf; file an issue.

**Scenario**: `db.register_predicate()` or a query invoking a custom predicate panics; subsequent calls to any query return this error.
**Scenario**: A panic while a function was being registered left the registry lock poisoned; every later query returns this error.

### QRY-009 Rules lock poisoned

Expand Down Expand Up @@ -2458,6 +2457,26 @@ db.execute("(query [:find ?g (count ?e) :where [?e :g ?g]])")?;
```
*`?n` binds a string, so rule `named` gives a String as its entity*

### QRY-014 User function failed

**Error text**: `user function `{}` failed: {}`

**Cause**: A user-defined predicate or aggregate registered with `try_register_predicate` or `try_register_aggregate` returned `Err(UdfError)`, or a language binding's callback threw. The first argument is the function's name and the second is the `UdfError`'s message. The query stops at the failing call: no later call of any user function is made for it. A cursor treats it as any other read error: it returns the rows of the batches the query finished before the failure first, then this error, then no more rows. Rows of the batch the failure happens in are not returned. A query inside a `WriteTransaction` that fails this way leaves the transaction open. Functions registered with `register_predicate` or `register_aggregate` never fail this way.

**Resolution**:
- Read the message: it is the one the function gave.
- Fix the input the function rejected, or make the function handle it (return `Ok(false)` for a predicate that should not match).

**Example**:
```rust
db.try_register_predicate("email?", |v| match v {
Value::String(s) => Ok(s.contains('@')),
_ => Err(UdfError::new("not a string")),
})?;
db.execute("(query [:find ?e :where [?e :mail ?m] [(email? ?m)]])")?;
```
*An entity's `:mail` is an integer, so `email?` fails with `not a string`*

---

### QRY-015 collect-set found a set value
Expand Down Expand Up @@ -3291,12 +3310,12 @@ db.execute("(rule [(ancestor ?x ?y) [?x :parent ?y]])")?;

**Error text**: `function registry lock poisoned: PoisonError { .. }`

**Cause**: An internal Rust mutex guarding the custom function/predicate registry was poisoned — a previous operation panicked while registering or invoking a custom function.
**Cause**: The lock guarding the custom function/predicate registry was poisoned: a thread panicked while it held the lock. The lock is held only while a query copies the registry's handle at open and while a function is registered. No user code runs under it: since v3.0.0 a query holds no registry lock while a custom function runs, so a function that panics does not poison it.

**Resolution**:
- Restart the process. If panics recur during `db.register_predicate()` or `db.register_aggregate()` calls, investigate the closure implementations for panics.
- Restart the process. This indicates a bug in Minigraf; file an issue.

**Scenario**: A custom predicate closure registered via `db.register_predicate()` panics during a query, poisoning the function registry mutex.
**Scenario**: A panic inside `db.register_predicate()`, `db.register_aggregate()` or their `try_` forms while the registry lock is held; the next registration returns this error.

### API-009 WAL not initialized

Expand Down
15 changes: 11 additions & 4 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@

| | Default features | With `--features test-hooks` |
|---|---|---|
| **Passing** | 1,796 | 1,819 |
| **Passing** | 1,824 | 1,847 |
| **Ignored** | 14 | 15 |
| **Total** | 1,810 | 1,834 |
| **Total** | 1,838 | 1,862 |

**Verified**: 2026-10-24 with `cargo test --quiet` (and `--features test-hooks`).

Expand Down Expand Up @@ -289,6 +289,13 @@ These tests compare the database against an independent answer rather than a sin
- `tests/write_tx_guard_test.rs` (#734): `checkpoint`, `verify`, `rebuild_indexes`, `prepare` and `PreparedQuery::execute`/`query` on a thread holding a `WriteTransaction` return `INT-001` (each under a 30 s timeout, so a deadlock fails instead of hanging), and the transaction then commits; outside a transaction they run.
- `PreparedQuery::query` matches `PreparedQuery::execute`.
- A cursor is `Send` and outlives its database handle.
- **User functions** (#180, `tests/udf_test.rs`, `tests/udf_reentry_test.rs`)
- Every UDF test runs once through `register_*` and once through `try_register_*` with closures that never fail, and gets the same rows.
- A `try_` predicate failing on its fifth call fails the query with `QRY-014` (name and message in the text) in `:where`, a rule body, a `not` body and an `or` branch, and is not called again; through a prepared query too. Through a cursor, the rows of the first batch come first, then the error, then `None`.
- A `try_` aggregate failing in `init`, `step` or `finalise`, as a `:find` aggregate and as a partitioned `:over` window: `QRY-014`, and no call after the failing one.
- A predicate, an aggregate and a rule's predicate that register a function from inside the call return (they deadlocked before; each under a 30 s timeout), and a later query sees the new function. Registering between a cursor's pulls neither blocks it nor changes its rows; an unknown predicate is `QRY-007` at open.
- Re-entry, one test per row of spec §5's table, on a file database with committed and pending facts, as a plain read and inside a `WriteTransaction`: `execute` of a query, `query` drained, `prepare` + `execute`, `PreparedQuery::execute` and `query`, `fact_log`, `transact`, `retract`, `rule`, `begin_write` + commit, `checkpoint`, `verify`, `rebuild_indexes`, both kinds of registration both ways, and a nested query whose UDF queries again. In a plain read each runs, the outer query's rows are the database's before the call, a later query sees the write, and `verify` is clean; in a transaction each returns `INT-001` except registration.
- Unit tests: `UdfError`'s message and `QRY-014`'s code, category and text.
- **Fact log** (#430, `tests/fact_log_test.rs`)
- `fact_log()` returns exactly the records written (assertions and retractions, with `tx_count` and `tx_id`), in non-decreasing `tx_count` in Tx order. Checked in memory, with everything pending, half checkpointed, fully checkpointed and after reopen, in both orders, with windows of 1 to 1,000 records and batch sizes of 1 to 1,000.
- Every layout gives identical records, including valid-time windows and long string values.
Expand Down Expand Up @@ -476,11 +483,11 @@ None of these are counted in the summary. The wasm32 targets run in CI (`.github
- `importGraph()` deletes the replaced database's IndexedDB keys.
- Two buffer tests check that releasing a page drops its bytes and any unflushed write, and that export writes a released page as zeros.
- A native storage test checks that a file whose free pages are all zeros (as the browser exports them), both just migrated from v7 and after 20 saves, opens, verifies clean and keeps committing. The free-list test checks that a checkpoint reports exactly the ids it newly lists.
- **On `wasm32-wasip1` under Wasmtime**: the unit and integration tests that do not use a file run, 956 passing; ten that spawn threads or a child process are ignored on WASI.
- **On `wasm32-wasip1` under Wasmtime**: the unit and integration tests that do not use a file run, 1,121 passing (`wasm.yml`, #746); ten that spawn threads or a child process are ignored on WASI.

## Ignored Tests

The thirteen ignored tests (fourteen with `test-hooks`) are intentionally excluded from a normal local run:
The fourteen ignored tests (fifteen with `test-hooks`) are intentionally excluded from a normal local run:

- Four rustdoc examples that reference internal types and cannot compile as standalone examples.
- One high-contention concurrency stress test intended for scheduled runs.
Expand Down
8 changes: 4 additions & 4 deletions src/browser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ use crate::browser::buffer::BrowserBufferBackend;
use crate::browser::indexeddb::IndexedDbBackend;
use crate::graph::FactStorage;
use crate::query::datalog::executor::{DatalogExecutor, QueryResult};
use crate::query::datalog::functions::FunctionRegistry;
use crate::query::datalog::functions::Functions;
use crate::query::datalog::parser::parse_datalog_command;
use crate::query::datalog::prepared::{BindValue, PreparedQuery};
use crate::query::datalog::rules::RuleRegistry;
Expand All @@ -30,7 +30,7 @@ use wasm_bindgen::prelude::*;
struct BrowserDbInner {
fact_storage: FactStorage,
rules: Arc<RwLock<RuleRegistry>>,
functions: Arc<RwLock<FunctionRegistry>>,
functions: Arc<Functions>,
pfs: PersistentFactStorage<BrowserBufferBackend>,
/// `None` for in-memory databases (no IDB backing).
idb: Option<IndexedDbBackend>,
Expand Down Expand Up @@ -67,7 +67,7 @@ impl BrowserDb {
inner: Rc::new(RefCell::new(BrowserDbInner {
fact_storage,
rules: Arc::new(RwLock::new(RuleRegistry::new())),
functions: Arc::new(RwLock::new(FunctionRegistry::with_builtins())),
functions: Arc::new(Functions::with_builtins()),
pfs,
idb: None,
withheld: Vec::new(),
Expand Down Expand Up @@ -101,7 +101,7 @@ impl BrowserDb {
inner: Rc::new(RefCell::new(BrowserDbInner {
fact_storage,
rules: Arc::new(RwLock::new(RuleRegistry::new())),
functions: Arc::new(RwLock::new(FunctionRegistry::with_builtins())),
functions: Arc::new(Functions::with_builtins()),
pfs,
idb: Some(idb),
withheld: Vec::new(),
Expand Down
Loading
Loading
0