8000 feat: prepared-query core for the bindings — API-022/023/024, Entity≡Val(Ref), BrowserDb.prepare (#181) by adityamukho · Pull Request #738 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A fact a rule derives has no metadata** (#548). `:db/valid-from`, `:db/valid-to`, `:db/tx-count` and `:db/tx-id` read from a derived fact (`(reach ?x ?y) [?x :reach ?y] [?x :db/tx-count ?c]`) now bind `nil`, and a constant (`[?x :db/tx-count 1]`) never matches one; a stored fact with the same attribute keeps its own. They used to bind the rule evaluator's working values: the time the query ran, forever, and a counter of its own. `:db/valid-at` still binds the query's valid time.
- **A `not` or nested `or` inside an `or` branch sees a variable an expression of the branch binds** (#551). The executor applied a branch's patterns, then its nested `or`s, then its `not`s, then its expressions, so a `not` or nested `or` reading a variable that only an expression of the branch binds saw it unbound. A branch now runs in the order a `:where` does (#625), and the `not` sees the binding. For example, over `[:x :n 1] [:y :n 2] [:z :n 5]`, `[:find ?e :where [?e :n ?v] (or-join [?e ?v] (and [(+ ?v 1) ?w] (not [?f :n ?w])) [?e :n 5])]` returned `:z` (the `not` matched any fact, so the first branch kept no row) and now returns `:y` and `:z`. Only such branches change.
- **An unknown predicate is an error when the query opens, wherever it is** (#551). Every predicate a query names, in an `or` branch (of the query or of a rule it evaluates) or a `not` / `not-join` body, is now looked up when the query opens, as top-level predicates and `:find` functions already were (#414): an unknown one fails with `QRY-007` whatever the data. Before, the executor looked it up only when a row reached it. In an `or` branch, a query whose rows never reached the predicate returned an answer, and now fails. In a `not` / `not-join` body the failure was swallowed: the body had no row, so the `not` kept every outer row and the query returned a wrong answer instead of an error.
- **Bind-value mistakes are `API` errors** (#181). A missing bind value, a bind slot in an attribute position and a bind value of the wrong kind for its slot were `INT-033`, `INT-034` and `INT-035`, codes for internal faults, although each is the caller's mistake. They are now `API-022`, `API-023` and `API-024`, with the same messages, from `prepare`, `PreparedQuery::execute` and `PreparedQuery::query`. A caller that matched the `INT` codes must match the new ones. `INT-033`–`INT-035` are deprecated (no longer emitted, still registered, never reused).

### Added

Expand All @@ -58,6 +59,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Read-only open: `OpenOptions::read_only`** (#429). Opens a database so that nothing done through the handle writes to the `.graph` file or its WAL. The file is locked in shared mode: any number of read-only handles, in this process or others, can hold it at once, and they exclude a read-write open (and it excludes them) with the existing STG-025/STG-026. The handle shows the same data as a read-write open: a WAL left by an earlier session is applied in memory but not checkpointed or deleted, and a format v7 file is read into memory instead of migrated. A missing file is the new `STG-042` (nothing is created). `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` fail with the new `API-014`; queries, cursors, `fact_log`, `verify` and rule registration work, and dropping the handle does not checkpoint. `page_cache_size` applies as usual, so a tool can keep the whole file resident. Rust API only for now; the bindings follow.
- **Cursor API: `Minigraf::query()` and `PreparedQuery::query()`** (#432). Both return a `Cursor` that hands a query's rows back in batches: `vars()`, `next_batch(max_rows) -> Result<Option<Batch>, _>`, `close()`, and an iterator over rows. The answer is fixed when the cursor opens, so writes that commit while it is open do not change it. A cursor borrows nothing from the database handle: it can outlive it and move to another thread. For now the answer is still computed when the cursor opens, under the same limits as `execute()`; the API is shaped so the engine can later produce rows as they are pulled without breaking callers. `query()` rejects `transact`, `retract` and `rule` with the new `API-012`, and a query with bind slots with `API-010`. `execute()` is unchanged. Like a fact log, a cursor pins the committed generation it read (#543): checkpoints and `rebuild_indexes()` run while it is open but do not reuse that generation's pages, so on a database taking many writes the file grows by the pages checkpoints replace until the cursor is closed or dropped. `BrowserDb.query()` cursors do the same: the pages they hold keep their bytes in memory and IndexedDB until a later commit after the cursor closes. Rust API only for now; the bindings follow.
- **`Minigraf::verify()` and `Minigraf::rebuild_indexes()`** (#373). Page checksums turn a torn or rotted page into an error, but they cannot see a page that is valid and says the wrong thing. `verify()` walks every committed page and returns an `IntegrityReport` with one structured error per finding: an index that holds different facts than EAVT or a different count than the meta page (new `STG-038`), keys out of order or a page reached twice (new `STG-039`), dictionary maps that disagree (new `STG-040`), ids with no dictionary entry (`STG-036`), and leaked, doubly used or wrongly free pages (`STG-035`). It keeps no per-fact state: indexes are compared by an order-independent digest. `rebuild_indexes()` rebuilds EAVT, AEVT, AVET and VAET from an intact index that agrees with another one (or the only intact one), derives the free list again, and commits like a checkpoint, including uncheckpointed writes; a crash at any point leaves the previous checkpoint. If no index can serve as the source, or the dictionary is damaged, it fails with new `STG-041` and writes nothing. Both are Rust API only for now; the bindings follow.
- **`BrowserDb.prepare`** (#181). The browser binding gets prepared queries: `db.prepare(text)` parses and validates a `(query ...)` once and returns a `BrowserPreparedQuery`, whose `execute(bindings)` returns the JSON `execute` returns and whose `query(bindings)` returns a `BrowserCursor`. Bindings are an object of slot name to a plain value (a boolean; a `number`, an integer when `Number.isInteger` accepts it and a float otherwise; a `bigint`; a string; `null`) or a typed `{ type, value }` form for the other kinds (`float` with no fractional part, `keyword`, `ref`, `entity`, `txCount`, `timestamp`, `anyValidTime`). A value that is not a bind value is a `TypeError`, and a typed value that does not fit its type (a bad UUID, a negative tx count) is `API-017`. The other bindings follow.
- **`BindValue::Entity` and `BindValue::Val(Value::Ref(..))` fit each other's slots** (#181). An entity slot (`[$e :attr ?v]`) takes `Val(Value::Ref(uuid))`, and a value, expression or rule-argument slot takes `Entity(uuid)` as the ref. Before, each was `INT-035` in the other's position, and a rule argument needed `Val(Value::Ref(..))`. A binding can now pass a host UUID without knowing where its slot is.

### Fixed

Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ cargo run < demos/demo_negation.txt
- `rules.rs`: `RuleRegistry` — thread-safe rule management
- `types.rs`: `EdnValue`, `Pattern`, `DatalogQuery`, `AsOf`, `ValidAt`, `WhereClause` (incl. `Not`, `NotJoin`); `PseudoAttr` enum, `AttributeSpec` wrapper
- `optimizer.rs`: Selectivity-based join reordering on every build (the `wasm` feature is a deprecated no-op). `reorder()` gates every reordering site; tests turn it off per thread (`set_reorder`, `minigraf::__reorder` under `test-hooks`, #596); `plan_seeded` orders a body with its imported variables counted as bound, `plan_after` orders a rule body's clauses after its delta, each joining on a variable bound before it where one can (#694), and `subplans()` lets tests force the engine's negation bodies and `or` branches hashed or correlated (`set_subplans`, `minigraf::__subplans`, #546)
- `prepared.rs`: `BindValue`, `PreparedQuery` — parse-once/execute-many with named `$slot` bind slots; `prepare_query`, `substitute`
- `prepared.rs`: `BindValue`, `PreparedQuery` — parse-once/execute-many with named `$slot` bind slots; `prepare_query`, `substitute`; `Entity` and `Val(Ref)` fit each other's slots, bind mistakes are `API-022`/`023`/`024` (#181)
- `magic_sets.rs`: Magic-sets rewriting for demand-driven recursive rule evaluation (not applied to rules with `not`/`not-join`)

3a. **`src/query/engine/`** — Streaming query engine (#473), runs every query (#551): `plan.rs` (the operator tree of every query, or the executor's error code for it: patterns, expression clauses, `not`/`not-join` and `or`/`or-join`, and rule invocations, read as patterns of their relations (`plan_rules`, #548; a magic-set head constant is `HeadArg::Const`), in `optimizer::plan` order, then the `or`s in text order, then the deferred `not`s; an `or` branch or a `not` body is planned as a `:where` is; a clause list with no pattern first starts from `Seed`; a constant no cell equals is `Term::Never`, a scan that reads nothing; `:as-of` resolved when the query opens (`Snapshot::tx_bound`: a counter bound when `tx_id` rises with `tx_count`, else `TxBound::Timestamp`: committed `tx_count` ranges and pending `tx_id <= t`); a per-fact pseudo-attribute pattern reads the metadata column its source's scan fills (the hidden variable the parser named, #594), and `:db/valid-at` joins as `[e _ _]`; plain `:find` variables, aggregates with `:with`, and window functions, over an aggregate's groups too), `scan.rs` (one pattern over committed key ranges via `CommittedReader::key_source`/`live_keys`, AVET ranges for cross-kind constants, then the pending range by whole runs; one entity per probe for a join), `join.rs` (`IndexNestedLoopJoin` when the pattern's entity is bound, or, under a constant attribute and entity not constant, its value (#721: per outer row the AVET ranges of the values that join-equal it and the attribute's pending facts with its key, read once per query in `IdSpace::probes`; an entity cell needs the keyword naming it, known only where the query made the entity from a keyword (rule heads, magic seeds), else the join becomes the planner's `HashJoin`); `HashJoin` otherwise, and on a relation; `JoinKey`: floats by IEEE `==`, a keyword equals a ref to the entity it names), `filter.rs` (`Filter`/`Bind` and pseudo-attribute steps over batches; a `Bind` on a bound slot keeps the rows where the result is join-equal, #596), `anti.rs` (`AntiJoin`, #546: a body with the same rows for every outer row runs once into a hash set of its imported-variable keys; any other body, or one whose first pattern's entity is imported, runs per outer row from `seed.rs`'s one-row `Seed`, via `Operator::rewind`; never deduplicates), `aggregate.rs` (`Aggregate`, #547: a hash table of groups keyed on cells as `Value`'s `Ord` compares values, one entry when ungrouped; `count`/`count-distinct` on cells, other aggregates on values via `datalog::aggregate::Agg`; groups sorted by value and finished in that order; a float group shows its least value by `value_order`), `window.rs` (`Window`, #547: holds each input row's passed-on cells and its named variables' values, then computes every window with `datalog::aggregate::windows`, the executor's code, and returns the rows in input order), `union.rs`/`distinct.rs` (`Union` and `Distinct`, #546: each `or` branch planned as a `not` body is, hashed into a table or run per outer row; then one row per distinct binding, as the executor's `Value`s compare), `cell.rs`/`batch.rs` (id cells in columnar batches of 1,024), `ids.rs` (DICT and provisional ids, `keyword_eid`), `project.rs` (ids → values for returned rows only), `relation.rs`/`recursive.rs` (#548: rule queries; a relation per evaluated predicate holds its derived rows, a set by `encode_value` identity, appended per round so `Old`/`Delta`/`Full` are ranges; `Recursive` runs each stratum's semi-naive rounds, one run per same-stratum top-level invocation, a rule reading its stratum otherwise run in full every round; scans read derived rows after committed and pending ones, with `nil` metadata), `meter.rs` (#549: one `Meter` per query, shared by every holder of state through a `Held` — hash join build sides, `not`/`or` tables, distinct sets, aggregate groups, window rows, rule relations and their temporaries, `IdSpace`; each entry charged by a per-entry estimate before it is inserted, containers grown with `try_reserve`, released on `rewind`/`close`/drop; over `OpenOptions::query_memory_limit`, or a refused allocation, is `QRY-012`; `Stream::close` drops the operator tree, keeping its counters). `mod.rs`: `Operator` (`next_batch`/`rewind`/`close`/`counters`), `Stream` (the operator tree over an owned `Snapshot`, pulled one batch at a time). `DatalogExecutor::open` / `execute_query` run every query on it: a cursor gets the `Stream` (`Answer::Stream`), `execute` drains it.
Expand All @@ -162,7 +162,7 @@ cargo run < demos/demo_negation.txt

8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`. `MinigrafError::invalid_argument` (API-017) and `MinigrafError::closed` (API-018) exist for language bindings

9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`
9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object, read by `binds.rs`: plain JS values or `{type, value}` forms, #181). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`

### Data Model

Expand Down Expand Up @@ -212,7 +212,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1716 tests** (1703 passing, 13 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**1720 tests** (1707 passing, 13 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
74A0
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ Scope — features:

- Query profiler (#185)
- Set membership predicates (#316)
- `PreparedQuery` over UniFFI (#181)
- Prepared queries in every binding (#181; core and `BrowserDb.prepare` done on `v3`, the other bindings follow)
- UDF registration in every binding: the UniFFI ones, C, Node and the browser `BrowserDb` (#180)
- `OpenOptions` exposed over UniFFI through `open_with_options`, so embedders can suppress the close-time checkpoint (#322; done on `v3` in every binding with #465)

Expand Down
Loading
Loading
0