8000 feat: binding support — API-017/API-018 and BrowserDb cursor (#462, #465, #467) by adityamukho · Pull Request #468 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **Binding support for cursors, open options, the fact log and the log writer** (#462, #465, #467). Two new codes let language bindings report what Rust's types rule out: `API-017` (an argument the binding cannot convert, such as an entity id that is not a UUID string), from `MinigrafError::invalid_argument`, and `API-018` (a log writer used after `finish` or `close`), from `MinigrafError::closed`. `BrowserDb.query(datalog)` returns a `BrowserCursor` with `vars()`, `nextBatch(maxRows)` (rows as JSON, as in `execute()`, or `undefined` at the end) and `close()`. The shared binding shape is in `docs/superpowers/specs/2026-10-07-binding-etl-surface-design.md`. The wasm32 test build compiles again (a duplicate `page_cache_capacity` and `tempfile` tests on wasm32).
- **Fact log: `Minigraf::fact_log()`** (#430). Streams every fact version in the database as `FactRecord`s (entity, attribute, value, `tx_count`, `tx_id`, valid-time bounds, and whether it is an assertion or a retraction), without Datalog. `FactFilter` selects attributes (exact or by prefix), entities and a `tx_count` range, checked on index keys so excluded records are never decoded. `FactOrder::Tx` (the default) returns records in ascending `tx_count`; format v8 has no index in transaction order, so it holds at most `FactFilter::window` records (default 262,144) and scans the selected keys once per window. `FactOrder::Storage` reads in one pass. The log reads the database as it was when it opened; while one is open, checkpoints are deferred (writes stay in the WAL, automatic checkpoints wait, and `checkpoint()` or `rebuild_indexes()` with work to do fail with the new `API-013`). Close, drop or finish the log to release it. Like `Cursor`, a `FactLog` is owned and `Send`. Rust API only for now; the bindings follow.
- **Raw log writer: `LogWriter`** (#431). Builds a new database from explicit `FactRecord`s, the load step of an offline migration after `fact_log()` on a read-only source. Each record keeps its `tx_count`, `tx_id`, valid-time window and assertion flag. Records come in transaction order, gaps allowed: leaving a transaction out leaves a hole, and `:as-of` on it shows the state before. `advance_tx_count` keeps the source's counter after trailing purged transactions. The checks are structural only: order (new `API-015`), one `tx_id` per transaction (new `API-016`), one valid-time window per fact per transaction (`API-011`) and size (`WAL-003`); a rejected record changes nothing. There is no WAL: records are committed in batches with the normal copy-on-write checkpoint, never splitting a transaction. The file is built at `<path>.partial` and renamed into place by `finish`, so a crash or an abandoned build never leaves part of a log at `path`; an existing target or WAL is the new `STG-043`. The page-range copy the issue proposed is not provided, since v8 pages share one dictionary and are not self-contained. The bindings follow.
- **Read-only open: `OpenOptions::read_only`** (#429). Opens a database so that nothing done through the handle writes to the `.graph` file or its WAL. The file is locked in shared mode: any number of read-only handles, in this process or others, can hold it at once, and they exclude a read-write open (and it excludes them) with the existing STG-025/STG-026. The handle shows the same data as a read-write open: a WAL left by an earlier session is applied in memory but not checkpointed or deleted, and a format v7 file is read into memory instead of migrated. A missing file is the new `STG-042` (nothing is created). `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` fail with the new `API-014`; queries, cursors, `fact_log`, `verify` and rule registration work, and dropping the handle does not checkpoint. `page_cache_size` applies as usual, so a tool can keep the whole file resident. Rust API only for now; the bindings follow.
Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,9 +145,9 @@ cargo run < demos/demo_negation.txt

7. **`src/wal.rs`** — Fact-level sidecar WAL, CRC32-protected entries, crash recovery

8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`
8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`. `MinigrafError::invalid_argument` (API-017) and `MinigrafError::closed` (API-018) exist for language bindings

9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking), `indexeddb.rs` (IndexedDB persistence)
9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches)

### Data Model

Expand Down Expand Up @@ -197,7 +197,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1345 tests** (1336 passing, 9 ignored; unit + integration + doc).
**1346 tests** (1337 passing, 9 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1345 tests
cargo test # run 1346 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
26 changes: 26 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,8 @@ with no `CodedError` anywhere in its chain.
| API-014 | Write on a read-only handle | Database API |
| API-015 | Log writer record out of transaction order | Database API |
| API-016 | Log writer record with a second tx_id for one transaction | Database API |
| API-017 | Invalid argument passed through a language binding | Database API |
| API-018 | Call on a closed binding object | Database API |
| INT-000 | Unclassified internal error | Internal |
| INT-001 | WriteTransaction already in progress on this thread | Internal |
| INT-002 | Invalid entity (API layer) | Internal |
Expand Down Expand Up @@ -2578,6 +2580,30 @@ let mut log = src.fact_log(&FactFilter::new())?;

**Scenario**: A transform sets `tx_id` from a per-record field instead of from the transaction.

### API-017 Invalid argument passed through a language binding

**Error text**: `invalid argument: {}`

**Cause**: A language binding (Python, Java, Swift, Node.js, C and the others) received an argument it could not convert to the Rust type, such as an entity id or `Ref` value that is not a UUID string, or a negative or out-of-range number. The `{}` names the argument and the problem. The Rust API takes typed arguments and never returns this code. Nothing was changed.

**Resolution**:
- Pass entity ids and `Ref` values as the UUID strings that `fact_log` records and query results carry.
- Check the binding's documentation for the argument's type and range.

**Scenario**: A migration script builds a `FactRecord` with `entity="alice"` instead of the entity's UUID.

### API-018 Call on a closed binding object

**Error text**: `{} is closed`

**Cause**: A language binding object was used after it was closed or finished, for example `append` or `finish` on a log writer after `finish` or `close`. Rust consumes these objects, so the Rust API never returns this code. Bindings cannot consume an object a caller still holds, so they return this error instead. A cursor or fact log reports the end (no more batches) after `close`, not this error. The `{}` names the object.

**Resolution**:
- Create a new object instead of reusing a finished one.
- In Python, use the writer as a context manager so it is finished once, on exit.

**Scenario**: A loader calls `finish()` in a `finally` block after it already called it on success.

---

## INT — Internal Errors
Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

121 changes: 121 additions & 0 deletions docs/superpowers/specs/2026-10-07-binding-etl-surface-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Binding surface for cursors, open options, the fact log and the log writer — design

**Issues:** #462 (binding cursors), #465 (`open_with_options`), #467 (`LogWriter` and
`FactRecord`). **Core:** #432 (PR #461), #430 (PR #463), #429 (PR #464), #431 (PR #466).
**Branch:** `v3` in core and in every binding repo.

## Goal

Every binding can do what the offline migration in temporal_reasoning#378 needs:
open a source read-only, stream its query answers and its full fact log, and write a
new file from the records. The shape is the same in every binding, with each
language's idiom on top.

## Branching

The bindings pin minigraf from crates.io, but these APIs exist only on the
unreleased `v3` branch. Each binding repo gets a `v3` branch that depends on core by
git revision:

```toml
minigraf = { git = "https://github.com/project-minigraf/minigraf", rev = "<v3 sha>" }
```

Feature PRs target `v3`, and each repo's CI runs on PRs to `v3`. `main` keeps serving
v2.x patch releases. At the v3.0.0 release the dependency switches to
`minigraf = "=3.0.0"` and `v3` becomes `main`, as in core.

## Core additions (this PR)

- **`API-017`** `invalid argument: {}`, from `MinigrafError::invalid_argument(detail)`.
A binding returns it for an argument it cannot convert, such as an entity id that is
not a UUID string or a number out of range. The Rust API takes typed arguments and
never returns it.
- **`API-018`** `{} is closed`, from `MinigrafError::closed(object)`. A binding
cannot consume an object its caller still holds, so a log writer used after `finish`
or `close` returns it. Rust consumes these objects and never returns it.
- **`BrowserDb.query(datalog)`** returns a `BrowserCursor` (`vars()`,
`nextBatch(maxRows)` with JSON rows or `undefined`, `close()`). `transact`, `retract`
and `rule` are `API-012`, and bind slots are `API-010`, as in `Minigraf::query`.

Bindings build these errors with the core constructors, so every binding reports the
same code and text, and `docs/ERROR_REFERENCE.md` stays the one registry.

## Shared shape

| Concept | UniFFI (Python, Java, Android, Swift) | Node (napi) | C | Browser |
|---|---|---|---|---|
| Open options | `OpenOptions` record, every field optional | options object | JSON object string | not applicable |
| Open | `MiniGrafDb.open_with_options(path, options)` | `MiniGrafDb.openWithOptions(path, options)` | `minigraf_open_with_options(path, json)` | — |
| Cursor | `db.query(datalog) -> MiniGrafCursor` | `db.query(datalog) -> Cursor` | `minigraf_query` | `BrowserDb.query` |
| Cursor rows | JSON string of rows, as in `execute()` | JSON string of rows | JSON string of rows | JSON string of rows |
| Fact log | `db.fact_log(filter) -> MiniGrafFactLog` | `db.factLog(filter)` | `minigraf_fact_log(db, json)` | not exposed |
| Fact record value | `MiniGrafValue` enum | `{ type, value }` object | `{"type", "value"}` JSON | — |
| Log writer | `MiniGrafLogWriter.create(path, options)` | `LogWriter.create(path, options)` | `minigraf_log_writer_create` | not built on wasm32 |

### Decisions

1. **Cursor rows stay JSON.** A batch is a JSON array of rows encoded exactly like the
`results` of `execute()`, so a cursor and `execute()` give the same rows and every
binding reuses its existing decoding. `max_rows == 0` counts as 1, a batch is never
empty, and the end is `None`/`null`/`undefined`. After `close()` a cursor returns
the end, not an error.
2. **Fact records are lossless.** A migration must write back exactly what it read, so
`FactRecord.value` keeps the value's type, and a `Ref` and a string with the same
text stay different. UniFFI bindings get an enum with the variants `Text`, `Int64`,
`Float64`, `Bool`, `Ref`, `Keyword` and `Null`; these names avoid `String`, `Int`,
`Float` and `Boolean`, which would shadow built-in types in Kotlin and Swift. JSON
and Node use `{ "type": "string" | "integer" | "float" | "boolean" | "ref" |
"keyword" | "null", "value": ... }`. Entity ids and `Ref` values are UUID strings;
anything else is `API-017`.
3. **64-bit fields stay exact.** `valid_to` is `i64::MAX` for forever. Node uses
`BigInt` for `txCount`, `txId`, `validFrom` and `validTo`, because a JS number
cannot hold `i64::MAX`. JSON writes them as integers.
4. **Fact fil 8000 ter.** A record with optional `attributes`, `attribute_prefixes`,
`entities`, `tx_from`, `tx_to` (inclusive), `order` (`tx` or `storage`) and
`window`. An absent field does not filter.
5. **`append_batch(records)`** appends in order and stops at the first rejected
record. The error is that record's coded error with ` (batch index N)` appended,
and the records before it stay appended, as if appended one by one.
6. **Writer lifetime.** `finish()` consumes the core writer behind a mutex. `close()`
drops it unfinished, which deletes `<path>.partial`, and is a no-op when already
finished or closed. Every other call after `finish` or `close` is `API-018`.
Python's context manager finishes on a clean exit and closes on an exception; Java's
`close()` (`AutoCloseable`) frees the writer, which abandons an unfinished build.
7. **Options map onto `OpenOptions` builders.** In the UniFFI shim every count
(options, `tx_count`, `tx_id`, batch sizes, filter bounds) is a signed integer
(`i64`, `i32` for batch sizes): Kotlin's `ULong`/`UInt` are name-mangled and cannot
be called from Java. A negative or oversized value is `API-017`. C and JSON use
unsigned integers and `size_t`. An absent field keeps the Rust default.
8. **Kotlin renames `close`.** UniFFI's Kotlin objects are `AutoCloseable`, and
`close()` frees the Rust object, which releases a cursor or fact log and abandons an
unfinished writer. The shim's own `close` methods clash with it, so the Kotlin
bindings rename them in `uniffi.toml`: `release()` (cursor, fact log) and `abandon()`
(writer).
9. **Errors keep their code in the message.** Every binding already surfaces
`[CODE] message` (the `Display` of `MinigrafError`), so API-014, API-015, API-017,
API-018, STG-042 and STG-043 reach callers like every other code. (UniFFI wraps
it: Python's `e.msg`, Kotlin's `Minigraf.errorMessage(e)`, Swift's `e.message`.)

### Not applicable

- **Browser (`BrowserDb`, minigraf-wasm):** `read_only`, `allow_unlocked`,
`page_cache_size`, the WAL options and `LogWriter` are file-backed only. BrowserDb
keeps its pages in memory, has no WAL and no file lock, and `LogWriter` is not built
on wasm32. The fact log is not exposed in the browser; the cursor is.
- **WASI package:** it ships the REPL binary, not a library API, so there is nothing to
expose.
- **`PreparedQuery.query`** follows #181 (v3.1.0).

## Tests

Each binding tests, in its own language:

- cursor batches of 1, 7 and 1,000 equal `execute()`, `close()` after the first batch,
a cursor opened before a `transact` keeps the earlier answer, and a non-query is
`API-012`;
- a read-only open with queries working, a write giving `API-014`, a missing file
`STG-042`, and two read-only handles on one file;
- a round trip through `fact_log` and the log writer that gives the same records and
`current_tx_count`, a hole from a skipped transaction, `API-015` and `STG-043`, and a
writer closed without `finish` leaving no file.
Loading
Loading
0