8000 feat(browser): BrowserDb.registerPredicate / registerAggregate (#180, PR 7) by adityamukho · Pull Request #757 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- **Fallible and re-entrant user functions** (#180, PR 1). `Minigraf::try_register_predicate` and `Minigraf::try_register_aggregate` take closures that return `Result<_, UdfError>` (`UdfError` is exported at the crate root). A call that returns `Err` fails the query with the new `QRY-014`, ``user function `<name>` failed: <message>``, at the first failing call: it is never read as `false` or `nil`, and no later call of any user function is made for that query. `execute` and `PreparedQuery::execute` return it; a cursor treats it as any other read error, returning the rows of the batches finished before the failure, then the error, then `None`. `register_predicate` and `register_aggregate` are unchanged. A query now reads its functions from a snapshot of the registry taken when it opens, and holds no database lock while a user function runs: a function registered later is not seen by a query or cursor that is already open, and a user function may call back into its own database (query, prepare, read the fact log, write, define rules, register functions, checkpoint, verify, rebuild indexes). Its writes are not seen by the query that called it. Inside a `WriteTransaction` those calls return `INT-001` on that thread, except registration, which runs. Before, a user function that registered a function deadlocked. This is the core of UDF registration in the language bindings, which follows binding by binding.
- **User functions in `BrowserDb`** (#180, PR 7). `db.registerPredicate(name, fn)` and `db.registerAggregate(name, { init, step, finish })`, both synchronous. A function receives each value as a `{ type, value }` object, as `@minigraf/node`'s `FactValue` (an integer's `value` is a `bigint`, a ref's its UUID string, a set's an array of these); a predicate returns a boolean; `init()` returns an accumulator, `step(acc, value)` the next one and `finish(acc, count)` the result, as a plain value or a typed form, as a prepared query's binding takes it. A function that throws fails the query with `QRY-014` and the error's `message` (else `String(e)`), and so does a predicate returning a non-boolean or a `finish` result that is not a value. A UDF may call `query`, `prepare` and the `register*` methods synchronously; `execute`, `checkpoint` and `importGraph` started from one run after the outer query returns. The registry holds no JavaScript value, only ids into a table local to the registering thread, so the crate stays free of `unsafe`.
- **`:limit N` bounds the rows a query returns** (#306). `[:find … :where … :limit 10]` returns at most 10 rows of the answer, and the engine stops reading once it has them: over 10M facts `:limit 10` reads one internal batch. Without `:order-by` which rows is unspecified, and may differ between plans; they are always rows of the unlimited answer. `:limit 0` returns no rows. It applies after aggregates and windows, to rule queries and to cursors, and a prepared query takes `:limit $n`, bound with `BindValue::Val(Value::Integer(n))`. A value that is not an integer `>= 0` or a slot fails with the new `PRS-115`; a repeated `:limit` with `INT-012` and a missing value with `INT-013`, as for the other query options. `:max-results` and `:max-derived-facts` still limit only the facts rules derive.
- **`:order-by` and `:offset`** (#306). `:order-by [(count ?e) :desc ?name]` orders the rows a query returns by `:find` elements (plain variables, or aggregates and windows written as in `:find`), each `:asc` (the default) or `:desc`; ties are broken by the other `:find` columns ascending, so every position of an ordered answer is decided and every plan returns the same list. Values compare as windows and `min`/`max` compare them: numbers by value (an integer before an equal float, `-0.0` before `0.0`, NaN after every number), then strings, booleans, refs, keywords, `nil` and sets. `:offset N` skips the first `N` rows of the ordered answer, before `:limit`, so `:offset 20 :limit 10` is the third page of ten. A new `Sort` operator reads its whole input; under a `:limit` it keeps only the best `offset + limit` rows, so `:order-by [?v :desc] :limit 10` over 10M facts holds ten rows, not ten million. It applies after aggregates and windows, to rule queries, cursors and prepared queries (`:offset $o`, bound like `:limit`), and counts toward `query_memory_limit`. New codes: `PRS-116` (`:offset` without `:order-by`: without an order, pages could overlap or miss rows), `PRS-117` (an entry that is not a `:find` element) and `PRS-118` (`:order-by` is not a non-empty vector, names an element twice, or has a misplaced or unknown keyword). `PRS-115` now also covers `:offset`. Inside a window's `:over`, `:order-by` is unchanged.
- **Set values: `#{...}`** (#316, PR 1 of 3). A fact's value can be a finite set of scalars, `[:alice :user/scopes #{:scope/read :scope/write}]`, written in query text, returned as `Value::Set(ValueSet)`, transacted, retracted and queried by exact value (`[?e :user/scopes #{:scope/write :scope/read}]`), joined on and grouped by. Duplicates are removed and element order does not matter. Two elements are the same when `Value`'s order says so: `0.0` and `-0.0` are one element (`-0.0` is kept), every NaN is one element, a keyword never equals a ref and `1` never equals `1.0`. A set cannot hold a set (`ValueSet::new` fails with the new `API-021`; in query text, `PRS-119`); its elements are literals only (`PRS-120`), and an unclosed `#{` is `PRS-121`. In format v8 a set has value tag `0x08`, after refs; a set whose encoded elements take at most 64 bytes is stored in its index keys, a longer one once in a value page, as a long string is. On a file database a stored set may take up to 4,068 bytes, counting each keyword element as 6 bytes and each ref as 10 (`WAL-003`). The REPL prints sets as `#{...}` and `BrowserDb` returns them as JSON arrays, in `Value`'s order.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ cargo run < demos/demo_negation.txt

8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`. `MinigrafError::invalid_argument` (API-017) and `MinigrafError::closed` (API-018) exist for language bindings

9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object, read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`
9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object, read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316); `udfs.rs`: `registerPredicate`/`registerAggregate` (#180; values as `{type, value}` objects, a throw is `QRY-014`; JS values held in a thread-local table, the registry keeps ids). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`

### Data Model

Expand Down
4 changes: 3 additions & 1 deletion docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -472,10 +472,12 @@ These tests compare the database against an independent answer rather than a sin

None of these are counted in the summary. The wasm32 targets run in CI (`.github/workflows/wasm.yml`).

- **In a browser** (`wasm-pack test --headless`, Firefox and Chrome): 43 `wasm_bindgen_test` tests.
- **In a browser** (`wasm-pack test --headless`, Firefox and Chrome): 58 `wasm_bindgen_test` tests.
- Twenty-five for `BrowserDb`, including IndexedDB persistence, the cursor, and the flush returning every dirty page and failing on one it cannot read (#470).
- Eleven for `BrowserBufferBackend` (which used `#[test]` and never ran before).
- Two checking that IndexedDB page keys accept only whole numbers from 0 to 2^53.
- Fifteen for `BrowserDb` user functions (#180), below.
- **`BrowserDb.registerPredicate` / `registerAggregate`** (#180): a predicate filters rows; every value kind reaches JS as its `{type, value}` object (an integer as a `bigint`, a ref as its UUID, a set's elements in canonical order); an aggregate in `:find` and with `:over`; a plain `finish` result read as a bind value; a throw (an `Error`, a string), a predicate returning a non-boolean, a failing `init`/`step`/`finish` and a `finish` result that is a bad ref or not a value each fail with `QRY-014` (`execute`, cursor and prepared query; aggregates also as windows); `TypeError` for a non-function and `INT-040` for a taken or built-in name; from inside a UDF, a nested `query` running a UDF, a registration (not seen by the outer query, seen by the next), and `execute`/`checkpoint`/`importGraph` that resolve after the outer query with both answers right; a value missing from the thread's table fails the call (`called off the JavaScript thread`), and a dropped function frees its entry.
- **`BrowserDb.prepare`** (#181): one prepared query run with three bindings; every plain JS value (boolean, whole and fractional `number`, `bigint`, string, `null`) and every `{type, value}` form; a prepared cursor gives the rows of `execute`; `TypeError` for a non-bind value or non-object bindings, `API-017` for a bad UUID, a negative tx count or a value on `null`/`anyValidTime`, `API-022`, `API-023`, `API-024` (both `execute` and `query`) and `API-005`; a later write is seen by the next run.
- **`BrowserDb` cursor**: `query()` batches of 1, 7 and 1,000 give the rows of `execute()`; a cursor keeps the answer from when it opened and returns the end after `close()`; a `transact` is `API-012`.
- **Page reuse** (#411, #440; five `BrowserDb` tests)
Expand Down
2 changes: 1 addition & 1 deletion docs/superpowers/specs/2026-10-24-udf-uniffi-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ int minigraf_register_aggregate(MiniGrafDb *db, const char *name, minigraf_acc_n
- JS can run only on the thread that owns its `napi_env`. Every binding method is synchronous on that thread, so every UDF call happens there. The binding stores the function as a napi reference wrapped in a type marked `Send + Sync`, and records the owning thread's id at registration. A call from any other thread is a failing call, ``called off the JavaScript thread``, and never touches the env. No current code path reaches that check: it guards future async APIs. It is tested by calling the wrapper from a spawned Rust thread in the binding's Rust tests.
- Worker threads each have their own env, and a database object belongs to the env that opened it. A function registered on one is called only on that env's thread.

**Browser (`BrowserDb`, this repo)**: `registerPredicate(name, fn)` and `registerAggregate(name, { init, step, finish })`, with the same JS shapes as Node and `js_sys::Function`. wasm32 without atomics has one thread, so the wrapper is `unsafe impl Send + Sync` under `cfg(all(target_arch = "wasm32", not(target_feature = "atomics")))`, and a build with atomics does not compile it. `BrowserDb.execute` is `async` only for the IndexedDB flush, and the query and its UDF calls run synchronously before any `await`. Re-entry differs from §5 here. A query holds `inner.borrow()` (`browser/mod.rs` ~125) while it runs. Every method that takes `borrow_mut` (`checkpoint`, `importGraph`, `apply_write` under `execute`) is `async`, and wasm-bindgen-futures runs its body from a microtask, after the outer query has returned. So from inside a UDF, `query` runs synchronously (it takes only `borrow()`), and `registerPredicate`/`registerAggregate` run synchronously (snapshot rule). `execute`, `checkpoint` and `importGraph` return a Promise that starts only after the outer query, and the UDF cannot wait for it. The plan checks that no synchronous method takes `borrow_mut`, with a browser test per method.
**Browser (`BrowserDb`, this repo)**: `registerPredicate(name, fn)` and `registerAggregate(name, { init, step, finish })`, with the same JS shapes as Node and `js_sys::Function`. The registry needs `Send + Sync` and a `JsValue` is neither. The crate is `#![forbid(unsafe_code)]`, so the registry holds no 8000 JavaScript value: each function and live accumulator sits in a `thread_local!` table keyed by an id unique across threads, and the registered closures and the accumulator hold only that id, removing the entry on drop. A call on a thread whose table lacks the id (only a build with atomics could make one) is a failing call, ``called off the JavaScript thread``, as in Node. (An earlier draft of this spec had an `unsafe impl Send + Sync` wrapper under `cfg(not(target_feature = "atomics"))`; the crate's `forbid(unsafe_code)` rules it out, found in PR 7.) `BrowserDb.execute` is `async` only for the IndexedDB flush, and the query and its UDF calls run synchronously before any `await`. Re-entry differs from §5 here. A query holds `inner.borrow()` (`browser/mod.rs` ~125) while it runs. Every method that takes `borrow_mut` (`checkpoint`, `importGraph`, `apply_write` under `execute`) is `async`, and wasm-bindgen-futures runs its body from a microtask, after the outer query has returned. So from inside a UDF, `query` runs synchronously (it takes only `borrow()`), and `registerPredicate`/`registerAggregate` run synchronously (snapshot rule). `execute`, `checkpoint` and `importGraph` return a Promise that starts only after the outer query, and the UDF cannot wait for it. The plan checks that no synchronous method takes `borrow_mut`, with a browser test per method.
- Values reach JS as `{type, value}` objects, built with `js_sys::Object` (no new dependency). `BrowserDb`'s registry (`browser/mod.rs:30`) moves to `Functions` with the core's in PR 1.

## 7. Tests
Expand Down
39 changes: 25 additions & 14 deletions src/browser/binds.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,23 @@ pub(super) fn read_bindings(bindings: &JsValue) -> Result<Vec<(String, BindValue
.map(|entry| {
let pair: Array = entry.unchecked_into();
let name = pair.get(0).as_string().unwrap_or_default();
let value = read_bind_value(&name, &pair.get(1))?;
let value = read_bind_value(&format!("slot ${name}"), &pair.get(1))?;
Ok((name, value))
})
.collect()
}

fn read_bind_value(name: &str, v: &JsValue) -> Result<BindValue, JsValue> {
/// The value `v`, in a bind value's plain or typed form; `what` names it in
/// errors. Used for a user function's result (#180).
pub(super) fn read_value(what: &str, v: &JsValue) -> Result<Value, JsValue> {
match read_bind_value(what, v)? {
BindValue::Val(value) => Ok(value),
_ => Err(invalid(&format!("{what} is not a value"))),
}
}

/// The bind value `v`; `what` names it in errors (`slot $name`).
fn read_bind_value(what: &str, v: &JsValue) -> Result<BindValue, JsValue> {
if let Some(b) = v.as_bool() {
return Ok(BindValue::Val(Value::Boolean(b)));
}
Expand All @@ -69,29 +79,30 @@ fn read_bind_value(name: &str, v: &JsValue) -> Result<BindValue, JsValue> {
return Ok(BindValue::Val(Value::String(s)));
}
if v.is_bigint() || v.as_f64().is_some() {
return Ok(BindValue::Val(number(name, v)?));
return Ok(BindValue::Val(number(what, v)?));
}
if let Some(set) = v.dyn_ref::<js_sys::Set>() {
let mut elements = Vec::new();
set.for_each(&mut |e, _, _| elements.push(e));
return value_set(name, &format!("slot ${name}"), elements);
return value_set(what, what, elements);
}
if v.is_object()
&& !Array::is_array(v)
&& let Some(kind) = Reflect::get(v, &JsValue::from_str("type"))?.as_string()
&& let Some(bind) = read_typed(name, &kind, v)?
&& let Some(bind) = read_typed(what, &kind, v)?
{
return Ok(bind);
}
Err(type_error(&format!(
"slot ${name}: a bind value is a boolean, number, bigint, string, null, Set or {{type, value}} object"
"{what}: a bind value is a boolean, number, bigint, string, null, Set or {{type, value}} object"
)))
}

/// A typed form, or `None` if `kind` names no type.
fn read_typed(name: &str, kind: &str, v: &JsValue) -> Result<Option<BindValue>, JsValue> {
fn read_typed(what: &str, kind: &str, v: &JsValue) -> Result<Option<BindValue>, JsValue> {
let value = Reflect::get(v, &JsValue::from_str("value"))?;
let what = || format!("slot ${name} of type {kind:?}");
let label = what;
let what = || format!("{label} of type {kind:?}");
let string = || value.as_string().ok_or_else(|| mismatch(&what()));
Ok(Some(match kind {
"string" => BindValue::Val(Value::String(string()?)),
Expand All @@ -115,7 +126,7 @@ fn read_typed(name: &str, kind: &str, v: &JsValue) -> R 8BD0 esult<Option<BindValue>,
"timestamp" => BindValue::Timestamp(integer(&what(), &value)?),
"anyValidTime" if value.is_undefined() => BindValue::AnyValidTime,
"set" => match value.dyn_ref::<Array>() {
Some(elements) => value_set(name, &what(), elements.iter().collect())?,
Some(elements) => value_set(label, &what(), elements.iter().collect())?,
None => return Err(mismatch(&what())),
},
// These two take no value.
Expand All @@ -125,13 +136,13 @@ fn read_typed(name: &str, kind: &str, v: &JsValue) -> Result<Option<BindValue>,
}

/// The set of `elements`, each read as a bind value that must be a value.
/// Element `i` is named `$name[i]` in errors.
fn value_set(name: &str, what: &str, elements: Vec<JsValue>) -> Result<BindValue, JsValue> {
/// Element `i` is named `{label}[i]` in errors.
fn value_set(label: &str, what: &str, elements: Vec<JsValue>) -> Result<BindValue, JsValue> {
let values = elements
.iter()
.enumerate()
.map(
|(i, e)| match read_bind_value(&format!("{name}[{i}]"), e)? {
|(i, e)| match read_bind_value(&format!("{label}[{i}]"), e)? {
BindValue::Val(v) => Ok(v),
_ => Err(invalid(&format!("{what}: element {i} is not a value"))),
},
Expand All @@ -143,9 +154,9 @@ fn value_set(name: &str, what: &str, elements: Vec<JsValue>) -> Result<BindValue
}

/// A plain `number` or `bigint`: an integer when it is whole, else a float.
fn number(name: &str, v: &JsValue) -> Result<Value, JsValue> {
fn number(what: &str, v: &JsValue) -> Result<Value, JsValue> {
if v.is_bigint() || Number::is_integer(v) {
integer(&format!("slot ${name}"), v).map(Value::Integer)
integer(what, v).map(Value::Integer)
} else {
Ok(Value::Float(v.as_f64().unwrap_or(f64::NAN)))
}
Expand Down
2 changes: 2 additions & 0 deletions src/browser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ mod binds;
pub mod buffer;
/// Async IndexedDB backend for browser WASM persistence.
pub mod indexeddb;
/// User-defined functions from JavaScript (#180).
mod udfs;

use crate::browser::buffer::BrowserBufferBackend;
use crate::browser::indexeddb::IndexedDbBackend;
Expand Down
Loading
Loading
0