8000 fix(browser): fail the IndexedDB handle after a failed flush and settle every flush (#894) by adityamukho · Pull Request #903 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **IndexedDB flush recovery** (#894, v3.0.0). Page puts and deletes now land atomically, including when queueing throws; abort-only failures and repeated request errors settle once with STG-048 naming the IndexedDB write and DOM exception. Flushes run in application order, and checkpoint waits for earlier writes even with no pages of its own. After failure, new writes, commits, checkpoints and imports reject with STG-046 without changing memory; already applied work stays readable and exportable. Close and reopen recovers the successful prefix before failure. Unstarted flushes reject with API-018 after close. No new dependencies, file-format, schema or public-method changes. The next v2.x patch backport is separate; #894 stays open until release.

- **Window partition scratch obeys `query_memory_limit`** (#765). Each window now charges its partition row indices while they coexist with input rows and earlier result columns, returns `QRY-012` if reserving them fails, and releases the charge after the window or on error. An in-place total sort removes the stable sort's uncharged merge buffer; partition and row order are unchanged.
- **An expression's `=` and `!=` equate a keyword with a ref to the entity it names** (#744). Patterns, joins, `not` bodies and `or` branches treat the keyword `:bob` as equal to a ref to the entity `:bob` names, but `[(= ?g :bob)]` compared the two values as they are, so with `?g` bound to that ref it kept no row and `[(!= ?g :bob)]` kept the row: the same question answered differently by how the query was written, with no error. `=` and `!=` now compare as joins do, in queries, rule bodies, `not` clauses and binding expressions, against a keyword constant, a keyword value or a bound `Val(Ref)`. A keyword still equals only the same keyword, and `<`, `>`, `<=` and `>=` on a keyword or a ref are still a type mismatch (no row).
- **`checkpoint`, `verify` and `rebuild_indexes` inside a `WriteTransaction` return `INT-001` instead of deadlocking** (#734). Called on a thread that holds a `WriteTransaction`, they waited on the write lock that the transaction already held, so the thread hung. They now return `INT-001`, as `execute`, `query`, `fact_log` and `begin_write` already did, and the transaction stays usable. `prepare` and `PreparedQuery::execute`/`query` make the same check; they could not deadlock, but they ran inside the transaction where `execute`/`query` refused. `checkpoint` deadlocked in every release since `WriteTransaction` was added; the fix for it and for the prepared calls ships in v2.x too.
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ cargo run < demos/demo_negation.txt

8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`. `MinigrafError::invalid_argument` (API-017) and `MinigrafError::closed` (API-018) exist for language bindings; browser OPFS adds API-026 (unavailable context), API-027 (busy) and STG-048 (failed browser storage operation)

9. **`src/browser/`** — Browser WASM backend (`browser` feature): `backend.rs` (buffer/OPFS dispatch), `opfs.rs` (`OpfsBackend` over an exclusive dedicated-worker sync access handle, one `<name>.graph` file and a 256-page cache; no WAL; sole-valid-meta recovery; crash-atomic import stages `<name>.graph-import`), `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence on the main thread or in workers); `openOpfs(name)` selects OPFS explicitly (Tier 2 in v3.0.0); all modes have synchronous `close()`, invalidating dependents and releasing the OPFS lock, with no save; `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object; `withMaxDerivedFacts`/`withQueryMemoryLimit`/`withMaxIterations` return a new one, #872; read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316); `BrowserDb.beginWrite` returns a `BrowserWriteTransaction` (#824: sync `execute` staging through `db::Staged`, `commit()` a `Promise` resolved after the IndexedDB flush, `rollback()`; `BrowserDbInner::tx_open` makes every other read/write call `INT-001`; finished = `API-018`; tests in `tx_tests.rs`); `udfs.rs`: `registerPredicate`/`registerAggregate` (#180; values as `{type, value}` objects, a throw is `QRY-014`; JS values held in a thread-local table, the registry keeps ids). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`
9. **`src/browser/`** — Browser WASM backend (`browser` feature): `backend.rs` (buffer/OPFS dispatch), `opfs.rs` (`OpfsBackend` over an exclusive dedicated-worker sync access handle, one `<name>.graph` file and a 256-page cache; no WAL; sole-valid-meta recovery; crash-atomic import stages `<name>.graph-import`), `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence on the main thread or in workers; atomic puts/deletes, abort/error settlement, STG-048 on failure, ordered flushes and persistent per-handle STG-046 refusal after failure, #894; checkpoint waits for earlier applied writes, close discards unstarted flushes, reads/export retain already-applied memory); `openOpfs(name)` selects OPFS explicitly (Tier 2 in v3.0.0); all modes have synchronous `close()`, invalidating dependents and releasing the OPFS lock, with no save; `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object; `withMaxDerivedFacts`/`withQueryMemoryLimit`/`withMaxIterations` return a new one, #872; read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316); `BrowserDb.beginWrite` returns a `BrowserWriteTransaction` (#824: sync `execute` staging through `db::Staged`, `commit()` a `Promise` resolved after the IndexedDB flush, `rollback()`; `BrowserDbInner::tx_open` makes every other read/write call `INT-001`; finished = `API-018`; tests in `tx_tests.rs`); `udfs.rs`: `registerPredicate`/`registerAggregate` (#180; values as `{type, value}` objects, a throw is `QRY-014`; JS values held in a thread-local table, the registry keeps ids). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`

### Data Model

Expand Down Expand Up @@ -250,7 +250,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**2172 tests** (2149 active, 23 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**2172 tests** (2149 active, 23 ignored; unit + integration + doc). Browser-only inventory is separate: #894 adds 14 wasm tests (95 main-thread / 128 dedicated-worker tests in total); see `docs/testing/894-idb-flush-recovery.md` for execution results. Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

The #760 deterministic storage suite adds three native unit tests in a separate `--cfg minigraf_shuttle` build, outside the default count. Run only `graph::storage::deterministic::` with that cfg; ordinary unit tests do not enter its scheduler. `deterministic-schedules.yml` runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See `docs/testing/deterministic-schedules.md` for scope, controls and findings.
Expand Down
1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ web-sys = { version = "0.3", optional = true, features = [
"IdbOpenDbRequest",
"IdbRequest",
"IdbTransaction",
"DomException",
"IdbTransactionMode",
"StorageManager",
"Window",
Expand Down
24 changes: 19 additions & 5 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3151,9 +3151,9 @@ let mut out = LogWriter::create("memory.v2.graph", OpenOptions::new())?;

---

### STG-046 Earlier checkpoint failed on this handle
### STG-046 Earlier checkpoint or IndexedDB flush failed on this handle

**Error text**: `an earlier checkpoint failed on this handle and is not retried; reopen the database`
**Error text**: `an earlier checkpoint or IndexedDB flush failed on this handle and is not retried; reopen the database`

**Cause**: A checkpoint (automatic, `checkpoint()`, `rebuild_indexes()`, or the one on close) failed with an I/O error after it may have written pages: a failed write, a full disk, or a failed `fsync`. After a failed `fsync` the operating system may have dropped the written pages while marking them clean, so a second `fsync` can report success for data that never reached the disk (the PostgreSQL "fsyncgate" problem). The checkpoint's meta page may also have reached the disk although the call failed; a retry would then overwrite pages that page references. So the handle never retries: every later write, checkpoint and index rebuild through it fails with STG-046.

Expand All @@ -3165,6 +3165,19 @@ working. There is no WAL. Close and reopen to select the complete previous or
new generation; the rejected transaction may be wholly present. Closing does
not retry the failed commit.

In IndexedDB mode, a failed flush rejects with STG-048 (naming the IndexedDB
write and DOM exception); failure to prepare its pages keeps its engine code.
The handle then refuses newly admitted transact/retract, commit (even empty),
checkpoint and import with STG-046, changing nothing. Reads, cursors, prepared
queries, rules, transaction staging, `exportGraph()` and `close()` still work.
Memory includes the failed operation and overlapping work applied before the
failure became known. Export reflects that memory, not an acknowledgement of
durability. Later flushes wait for earlier ones and send nothing if an earlier
flush failed. Close and reopen recovers the state before the failed flush;
a rejected IndexedDB write is never stored. A failed import stays visible in
memory but its puts and deletes leave IndexedDB unchanged. Closing does not
retry anything.

**Resolution**:
- Fix the cause (free disk space, check the device and the kernel log), then drop every clone of the handle and open the database again. Opening picks the newest complete checkpoint and replays the WAL.
- Do not copy the `.graph` file without its `.wal` while the handle is in this state.
Expand All @@ -3191,12 +3204,13 @@ not retry the failed commit.

**Error text**: `browser storage operation failed: {} ({})`

**Cause**: An OPFS operation failed: opening an entry, acquiring its sync access handle, reading, writing, flushing, checking size, truncating, or staging an import. The message names the operation and the browser exception (for example `QuotaExceededError`), or a short read/write. Short transfers are never accepted as complete pages. Lock contention uses STG-026; an unavailable OPFS context uses API-026.
**Cause**: An IndexedDB write (transaction creation, request queueing, error or abort), or an OPFS operation failed: opening an entry, acquiring its sync access handle, reading, writing, flushing, checking size, truncating, or staging an import. The message names the operation and the browser exception (for example `QuotaExceededError`), or a short read/write. Short transfers are never accepted as complete pages. Lock contention uses STG-026; an unavailable OPFS context uses API-026.

**Resolution**:
- Check the operation and exception name, available origin quota and browser permissions.
- After a commit may have written, later writes fail with STG-046. Call `close()` and reopen. Reads may still include the unconfirmed transaction; a rejected commit does not prove it is absent on disk.
- An import failure at publication closes the database; reopen to recover the complete old or new image.
- In OPFS mode, after a commit may have written, later writes fail with STG-046. Call `close()` and reopen. Reads may still include the unconfirmed transaction; a rejected commit does not prove it is absent on disk.
- An OPFS import failure at publication closes the database; reopen to recover the complete old or new image.
- After an IndexedDB flush fails, later durable operations fail with STG-046. Close and reopen to recover the successful prefix before the failed flush. The rejected write, including all its page deletes, is absent from IndexedDB.

**Scenario**: Origin quota is exhausted while a worker writes a data page; `execute()` rejects with STG-048 naming the write and `QuotaExceededError`.

Expand Down
19 changes: 18 additions & 1 deletion docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,21 @@

## Coverage Summary

**#894 evaluator follow-up**: the nonempty smaller-import abort regression raises
the flush suite to 14 browser-only tests (95 main-thread / 128 dedicated-worker
tests in the inventory). Native counts are unchanged. The test checks failed-handle
refusal, the unchanged durable page map, and the 400-row reopen integrity oracle.
Firefox main-thread verification passes all 95 tests; the worker count is inventory
only for this follow-up (the earlier worker run is recorded below).

**#894 verification (2026-11-17)**: native default tests pass (2,149 active,
23 ignored); strict native all-target and browser-target Clippy, build and
formatting pass. Firefox passes **94 main-thread** and **127 dedicated-worker**
tests. The distribution binary is **1,270,880 bytes** (+16 bytes), below the
1,280,000-byte gate. The pre-fix multi-page durable-map and abort-only deadline
failures, one-line existing crash-harness Clippy correction, and unverified lanes
are documented in the [PR 1 evidence](testing/894-idb-flush-recovery.md).

| | Default features | With `--features test-hooks` |
|---|---|---|
| **Active** | 2,149 | 2,187 |
Expand Down Expand Up @@ -728,6 +743,8 @@ These are not `#[test]`s; run them with `cargo kani` or `.github/workflows/kani.

None of these are counted in the summary. The wasm32 targets run in CI (`.github/workflows/wasm.yml`).

- **IndexedDB flush recovery** (#894; `src/browser/flush_tests.rs`, 13 new browser tests, plus the updated closed-connection case in `tx_tests.rs`): atomic puts/deletes after queue-time throws; abort-only settlement with a deadline; multiple request errors without callback exceptions; failed-handle refusal without counter/fact changes; execute/commit/checkpoint/smaller-import failures; missing dirty-page preparation errors; ordered overlaps and checkpoint boundaries; close/reopen with queued work; migration failure and intact-page-map recovery. Reopened exports pass the shared native image integrity walk, allowing absent free pages. Healthy operation retains no settled flush history. See [PR 1 evidence and scope](testing/894-idb-flush-recovery.md). Native test inventory is unchanged.

- **OPFS worker mode** (#441; `src/browser/opfs_tests.rs`, 34 worker tests plus one main-thread test):
- AC1/AC4: `acknowledged_writes_survive_close_and_reopen`, `the_database_is_one_file_and_no_wal`.
- AC2/AC8: `invalid_names_are_api_017_before_anything_is_created`, `a_second_open_is_stg_026_until_close`, `open_opfs_on_the_main_thread_is_api_026_and_creates_nothing`.
Expand All @@ -738,7 +755,7 @@ None of these are counted in the summary. The wasm32 targets run in CI (`.github
- AC13/AC14: `opfs_open_read_bound`, `opfs_cache_residency_bound`, `indexeddb_mode_works_in_a_worker`.
- AC3 parity: the worker build runs the existing transaction, prepared-query and UDF suites against OPFS as well as IndexedDB. The main-thread build preserves the existing IndexedDB tests. Worker CI sets `RUSTFLAGS=--cfg minigraf_browser_worker` for Firefox and Chrome.

- **In a browser** (`wasm-pack test --headless`, Firefox and Chrome): 81 `wasm_bindgen_test` tests (inherited Firefox main-thread result, 2026-11-14, after #441). The final #641 snapshot passed in headless Firefox on the bench; Chrome remains a CI run.
- **In a browser** (`wasm-pack test --headless`, Firefox and Chrome): 94 `wasm_bindgen_test` tests after #894 (81 inherited plus 13 flush-recovery tests); see the [#894 verification record](testing/894-idb-flush-recovery.md) for actual runs. The final #641 snapshot passed in headless Firefox on the bench; Chrome remains a CI run.
- Three committed-memory-backend input-order fixtures: scalar Sort omission, streaming counts with Boolean/float representatives and mixed numeric group order, and partition windows preserving float edges and numeric group classes (#641).
- Twenty-six for `BrowserDb`, including IndexedDB persistence, the default `allow_overrides` list (#869), the cursor, and the flush returning every dirty page and failing on one it cannot read (#470).
- Eleven for `BrowserBufferBackend` (which used `#[test]` and never ran before).
Expand Down
Loading
Loading
0