8000 fix(browser): fail the IndexedDB handle after a failed flush and settle every flush (#894) by adityamukho · Pull Request #904 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed

- **IndexedDB flushes are atomic, settle on abort, and refuse later writes after a failure (#894)**, backported from v3 for the next v2.x patch. A synchronous exception while queueing pages aborts the whole batch; an abort with no request error rejects instead of hanging, and repeated request errors cannot throw from a consumed callback. The failed operation reports `STG-048` naming the IndexedDB write and browser exception (page-preparation errors retain their own code). Subsequent transact/retract, checkpoint and import calls report `STG-046` without changing memory. Overlapping flushes run in order; a checkpoint waits for earlier writes even when it has no dirty pages. Queries, rules and export remain available, including rejected work already applied in memory; export is not a durability acknowledgement. Discard the failed handle and reopen to recover the successful prefix before the failure. A failed import leaves IndexedDB unchanged. Format v7, the IndexedDB schema, public methods, healthy write volume and read-only open behavior are unchanged. Native v2.x behavior is unchanged; the two error codes retain their v3 meanings.

### Documentation

- Fix the Rust Quick Start (#884): register recursive rule clauses in separate `execute()` calls and define the keyword entity IDs used by prepared queries. Clarify the entity ID in the prepared-query error example.
Expand Down
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ Auto-migrates v1/v2/v3/v4/v5/v6 → v7 on open/checkpoint.

## Test Coverage

**1335 tests** (1326 passing, 9 ignored; unit + integration + doc).
**1335 native tests** (1326 passing, 9 ignored; unit + integration + doc), plus **26 browser wasm-bindgen tests** (14 added for #894).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand All @@ -188,6 +188,8 @@ Ordinary aggregates on v2 are `count`, `count-distinct`, `sum`, `sum-distinct`,

## Current Maintainer Context

**Unreleased, next v2.x patch** — the #894 IndexedDB backport makes flushes atomic and ordered, rejects abort-only failures, and refuses later durable work on a failed browser handle with STG-046. STG-048 names the failed IndexedDB write and DOM exception. Format v7 and the public API are unchanged.

**v2.0.11 released** (2026-10-26) — expression `=` / `!=` equates a keyword with a ref to its entity (#744), float-literal rule calls return `-0.0` rows in every run (#750), and `checkpoint` inside a `WriteTransaction` returns `INT-001` instead of deadlocking (#734). **v2.0.10 released** (2026-10-23) — WAL entries of more than 1,000,000 facts replay (#686), float constants match equal floats only (#706), prepared queries use the database's rule limits (#679); Kani proofs of `encode_value` backported (#627). **v2.0.9** (2026-10-21) — a pseudo-attribute in a rule body reads the stored fact (#661), a fact of more than four elements is rejected with PRS-108 (#649), `BrowserDb` rejections carry their error code (#673); every binding package is consumer-tested before and after publishing. **v2.0.8** (2026-10-20) — query data-integrity patch: `:db/valid-at` in `not-join` bodies and rule definitions (#615, #617), `or` branches with outer-variable expressions, `or`-bound expression variables and keyword/entity joins (#620, #625, #637), rules calling or recursing through a `not`/`or` rule (#628, #633). **v2.0.7** (2026-10-16) — prepared queries keep a literal `:as-of`/`:valid-at` (#603), rule calls with a literal argument return every row (#608), an `or` sharing no variable returns its cross product (#610). **v2.0.6** (2026-10-14) — concurrent-query data-integrity patch: queries see whole transactions (#554), one state per query (#555), no half-written checkpoint reads (#558). **v2.0.5** (2026-10-11) — short-file open refused with STG-045 (#506), commit after a torn WAL entry no longer lost, WAL-007 (#513); backported rule-query `max_results` (#528), all-checkpointed WAL removal (#457) and `usize::MAX` close-time save (#476); weekly soak test (#392), real coverage gates (#396). **v2.0.4** (2026-10-08) — `BrowserDb` IndexedDB flush returns an error for an unreadable dirty page (#470); golden-file v7 corpus (#391) and SIGKILL committed-data crash test (#384). **v2.0.3** (2026-10-06) — data-integrity patch under the support policy: indexed queries with >65,535 uncheckpointed facts (#445), tx counter rewound on reopen next to an already-checkpointed WAL (#447). **v2.0.2** (2026-09-27) — last planned v2.x feature/fix release: query fixes (#297, #405, #407, #410), directory fsync (#389), point-query/attribute-scan/checkpoint perf mitigations (#323, #381, #315), CI policy checks (#395), support policy and tiers (#397, #399, #400). **v2.0.1** (2026-09-25) fixed index rebuild on open (#370). **v2.0.0** introduced kernel file locking (#317, #304), structured runtime error codes (#277), `OpenOptions` `#[non_exhaustive]`, and an MSRV of Rust 1.89. See `CHANGELOG.md` for the full rationale and release history.

Current plan (see `ROADMAP.md` and the milestones):
Expand Down
1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ web-sys = { version = "0.3", optional = true, features = [
"IdbOpenDbRequest",
"IdbRequest",
"IdbTransaction",
"DomException",
"IdbTransactionMode",
"Window",
] }
Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,13 @@ See the [Mobile Integration](https://project-minigraf.github.io/minigraf-docs/la

### For WASM / Browser

**Unreleased, next v2.x patch (#894):** after an IndexedDB flush fails, later
writes, checkpoints and imports reject with `STG-046`. Discard the handle and
reopen to recover the successful prefix before the failure. Queries, rules and
export remain usable; export includes rejected work still held in memory and
does not acknowledge durability. See the [changelog](CHANGELOG.md#unreleased)
and [error reference](docs/ERROR_REFERENCE.md#stg-046-earlier-checkpoint-or-indexeddb-flush-failed-on-this-handle).

Published as [`@minigraf/browser`](https://www.npmjs.com/package/@minigraf/browser) on npm (IndexedDB-backed, `wasm-pack`). WASI build (`wasm32-wasip1`) available as [`@minigraf/wasi`](https://www.npmjs.com/package/@minigraf/wasi) on npm and as a GitHub Releases artifact (Wasmtime / Wasmer). See the [Use Cases](https://project-minigraf.github.io/minigraf-docs/latest/use-cases/). The [playground](https://minigraf-playground.vercel.app/) and the [time travel visualizer](https://project-minigraf.github.io/minigraf-visualizer/) are both built on `@minigraf/browser`.

### For Python / Node.js / Java / C
Expand Down
11 changes: 11 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@ Completed releases and their implementation details live in the [CHANGELOG](CHAN

---

## Next v2.x Patch — IndexedDB Durability (unreleased)

Backport #894 from v3: atomic, always-settling IndexedDB flushes, ordered
persistence and checkpoint boundaries, and STG-046 refusal of later durable
work after a failed browser flush. The failure reports STG-048 (preparation
errors retain their engine code). Reopen recovers the successful prefix before
the failure. Format v7, IndexedDB schema, public methods and open behavior stay
unchanged. See [Unreleased](CHANGELOG.md#unreleased).

---

## v2.0.11 — Query and Write-Transaction Patch (released 2026-10-26)

File format stays v7 and the public API does not change. Details are in the [CHANGELOG](CHANGELOG.md).
Expand Down
52 changes: 52 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,8 @@ with no `CodedError` anywhere in its chain.
| STG-026 | Database locked by another process | Storage |
| STG-027 | Filesystem does not support file locking | Storage |
| STG-045 | File shorter than one page and not a new database | Storage |
| STG-046 | Earlier checkpoint or IndexedDB flush failed on this handle | Storage |
| STG-048 | Browser storage operation failed | Storage |
| WAL-001 | Invalid WAL magic number | WAL |
| WAL-002 | Unsupported WAL version | WAL |
| WAL-003 | Fact serialised size exceeds maximum | WAL |
Expand Down Expand Up @@ -2027,6 +2029,56 @@ See the [file format section in README](../README.md#file-format) for version hi

---

### STG-046 Earlier checkpoint or IndexedDB flush failed on this handle

**Error text**: `an earlier checkpoint or IndexedDB flush failed on this handle and is not retried; reopen the database`

**Cause**: In v2.x this code applies to `BrowserDb` backed by IndexedDB. A failed
flush rejects with STG-048, naming the IndexedDB write and DOM exception; a
failure preparing its pages keeps its engine code. The handle then refuses
newly admitted transact/retract, checkpoint and import operations with STG-046,
without changing facts, transaction counters or the live store. Native v2.x
handles do not use this failed-handle rule.

Reads, rules and `exportGraph()` still work. Memory includes the failed operation
and overlapping work applied before the failure became known. Export reflects
that memory, not an acknowledgement of durability. Later flushes wait for earlier
ones and send nothing if an earlier flush failed. A checkpoint also waits for
writes applied before it, even when it has no dirty pages of its own.

**Resolution**:
- Address the browser storage failure, discard the failed handle and open the
same name again. A fresh open loads the successful prefix before the failed
flush; a rejected IndexedDB write is never stored. There is no automatic retry.
- A failed import remains visible in memory, but leaves every byte in IndexedDB
unchanged. Use the fresh handle for subsequent writes.

**Scenario**: A multi-page write exceeds the origin quota. It remains queryable
in memory, but every later durable operation is refused until the caller reopens.

---

### STG-048 Browser storage operation failed

**Error text**: `browser storage operation failed: {} ({})`

**Cause**: An IndexedDB write failed during transaction creation, request
queueing, error or abort. The message names the operation (`IndexedDB write`)
and the browser exception (for example `QuotaExceededError` or `AbortError`).
None of the rejected flush's changes reach IndexedDB; even a synchronous
exception after earlier puts were queued aborts the whole transaction.

**Resolution**:
- Check the operation and exception name, available origin quota and browser permissions.
- Later durable operations on this handle fail with STG-046. Discard it and reopen
to recover the successful prefix before the failed flush. Reads and export on
the failed handle may still include rejected work held only in memory.

**Scenario**: Origin quota is exhausted while writing a data page; `execute()`
rejects with STG-048 naming the IndexedDB write and `QuotaExceededError`.

---

## WAL — Write-Ahead Log Errors

WAL errors relate to the sidecar `.wal` file written alongside the `.graph` file.
Expand Down
45 changes: 42 addit 8 ions &am 10BC8 p; 3 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

#777 corrects documentation only on v2; query behavior and test counts are unchanged. Ordinary `avg` still raises PRS-027 without `:over`.

## v2.0.11 Coverage Summary
## Unreleased v2.x Coverage Summary

| | |
|---|---|
| **Verified** | 2026-10-26 with `cargo test` |
| **Verified** | 2026-11-18 with `cargo test` |
| **Passing** | 1,326 |
| **Ignored** | 9 (see [Ignored Tests](#ignored-tests)) |
| **Total** | 1,335, including the golden-file corpus (#391) |
Expand Down Expand Up @@ -155,12 +155,51 @@ These tests compare the database against an independent answer rather than a sin

### Browser

The `BrowserDb` tests run only on wasm32 with `wasm-bindgen-test` in a browser and are not in the counts above.
The **26 `BrowserDb` wasm-bindgen tests** (12 existing, 14 added for #894) run
only on wasm32 in a browser and are not in the native counts above. All 26 passed
in headless Firefox on 2026-11-18.

- The IndexedDB flush returns every page a write dirtied.
- A dirty page that cannot be read is an error rather than left out.
- A parse, query and transact error reject with their codes (`PRS-011`, `QRY-007`, `PRS-046`), as on the native API (#673).

The #894 tests in `src/browser/flush_tests.rs` use real IndexedDB transactions
with test-only JavaScript interception. They cover abort-only settlement,
recoverable abort, multiple bubbling request errors, queue-time exceptions,
closed connections, a 400-fact failed write, checkpoint and smaller imports,
repeated STG-046 refusal without state changes, and INT-052 preparation failures.
Overlap cases cover write/write, write/import, import/write and write/checkpoint;
controls cover successful mixed operations, checkpoint boundaries excluding later
writes, earlier success before preparation failure, validation failures and
release of settled barriers. Queries, rules and export stay usable. Fresh open
performs no writeback and accepts subsequent writes.

The quiescent oracle compares durable page bytes and recovered logical rows,
checks that every v7 header page reference is present, and exports the recovered
store. The verification run opened and queried **16 recovered exports with the
native CLI** using `scripts/check_browser_exports.py`. Export transport is opt-in:

```bash
# Put matching wasm-bindgen-test-runner and geckodriver executables on PATH.
# The native binary and all scratch files remain under target/.
MINIGRAF_EXPORT_RECOVERY=1 WASM_BINDGEN_USE_BROWSER=1 MOZ_HEADLESS=1 \
CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=wasm-bindgen-test-runner \
cargo test --target wasm32-unknown-unknown --features browser --lib -- --nocapture > target/browser-exports.log 2>&1
cargo build
python3 scripts/check_browser_exports.py target/browser-exports.log target/debug/minigraf
```

`wasm-pack test --headless --firefox --features browser` currently also compiles
native-only integration tests that call `Minigraf::open`, unavailable on wasm.
The direct library runner above exercises the browser suite without those tests.

Before the production fix, the abort-only regression failed at its 1.5-second
deadline (`deadline: flush did not settle`), and the multi-page regression failed
because durable bytes had changed after the queue-time exception. Both pass with
the backport. v2 tests do not include v3 transactions, close, OPFS or open-time
migration flushes. Empty-file import cases use a real v7 header: importing zero
bytes on v2 sends no writes and cannot exercise an adapter failure.

## Ignored Tests

The nine ignored tests are intentionally excluded from a normal local run:
Expand Down
54 changes: 54 additions & 0 deletions scripts/check_browser_exports.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/usr/bin/env python3
"""Open recovered browser exports with the native CLI and compare their rows.

Capture a successful browser test run with MINIGRAF_EXPORT_RECOVERY=1 and
--nocapture, then pass its log and the native minigraf binary to this script.
Only the Python standard library is needed. Scratch files stay under target/.
"""

import argparse
import base64
import json
from pathlib import Path
import subprocess
import tempfile


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("log", type=Path)
parser.add_argument("binary", type=Path)
args = parser.parse_args()
binary = args.binary.resolve()
target = Path(__file__).resolve().parents[1] / "target"
target.mkdir(exist_ok=True)
count = 0
with tempfile.TemporaryDirectory(prefix="browser-exports-", dir=target) as scratch:
for line in args.log.read_text().splitlines():
if not line.startswith("MINIGRAF_RECOVERED_EXPORT "):
continue
record = json.loads(line.split(" ", 1)[1])
path = Path(scratch) / f"recovered-{count}.graph"
path.write_bytes(base64.b64decode(record["bytes"], validate=True))
result = subprocess.run(
[str(binary), "--file", str(path)],
input="(query [:find ?n :where [?e :n ?n]])\n",
text=True,
capture_output=True,
check=True,
)
if result.stderr:
raise RuntimeError(f"native open/query failed for export {count}: {result.stderr}")
rows = sorted(
int(row) for row in result.stdout.splitlines() if row.isdecimal()
)
if rows != record["expected"]:
raise RuntimeError(f"native rows differ from the successful prefix in export {count}")
count += 1
if not count:
raise RuntimeError("no exports found; enable MINIGRAF_EXPORT_RECOVERY=1 and --nocapture")
print(f"Native open and query passed for {count} recovered browser exports.")


if __name__ == "__main__":
main()
Loading
Loading
0