Repository navigation
BrowserDb: OPFS-backed persistent mode (sync access handle, no in-memory buffer) #441
Description
Motivation
The persistent BrowserDb keeps a full in-memory copy of the database. StorageBackend is synchronous and IndexedDB is async, so BrowserDb.open() loads every page into BrowserBufferBackend and every transact writes the dirty pages through to IndexedDB. Heap use and open time are O(database size), which is where the ~20K-fact guideline comes from. #440 bounds growth, but the database still has to fit in memory.
The Origin Private File System (OPFS) offers FileSystemSyncAccessHandle, a synchronous file API (Baseline since March 2023: Chrome 102+, Firefox 111+, Safari 15.2+). It lets the browser use a real .graph file through the normal sync StorageBackend, with no buffer.
Proposal
Add an OPFS-backed persistent mode to BrowserDb, for use inside a dedicated Web Worker. Keep the IndexedDB + buffer mode as the main-thread fallback.
Backend mapping
StorageBackend |
FileSystemSyncAccessHandle |
|---|---|
read_page(id) |
read(buf, {at: id * PAGE_SIZE}) |
write_page(id, data) |
write(buf, {at}) |
sync() |
flush() |
page_count() |
getSize() / PAGE_SIZE |
close() |
close() (releases the lock) |
- Only open is async (
navigator.storage.getDirectory()→getFileHandle({create})→createSyncAccessHandle()). Every operation after that is synchronous. - No buffer. The regular
PageCacheapplies, so memory is O(cache) and open does not load every page. - The OPFS file is a byte-for-byte
.graphfile.exportGraph()/importGraph()read and write through the handle, becausegetFile()is blocked while a sync handle is open. - web-sys:
FileSystemSyncAccessHandle,FileSystemFileHandle::create_sync_access_handle,FileSystemGetFileOptions,StorageManagerandWorkerNavigatorare stable. Do not usecreate_sync_access_handle_with_options(mode): it is behindweb_sys_unstable_apisand supported only in Chrome 121+. No new crates.
Locking
The default readwrite mode takes an exclusive lock. A second tab or worker that opens the same database gets NoModificationAllowedError. Map it to the existing file-lock error code, which matches native kernel-lock semantics (single writer, fail fast). Multi-tab shared access is out of scope, because the shared modes are Chrome-only.
Durability
The WHATWG spec gives no explicit durability guarantee for flush() and says nothing about torn writes on sync handles. The OPFS backend must therefore:
- go through the native recovery path (v8 page CRCs storage: add per-page checksums to fact pages and B+tree pages (file format v8) #388, dual meta pages and copy-on-write save() is not crash-atomic: new fact pages overwrite old index pages before header commit #374), not the browser path that assumes IndexedDB has no torn writes (v8 spec §4.1.1);
- document
sync()as "durable as far as the browser'sflush()guarantees"; - get a crash or kill test on each engine in CI where practical.
No WAL
Keep checkpoint-per-transaction like BrowserDb today. With v8 O(change) checkpoints (#434) that is cheap, and it keeps the WAL file-backend only and the database a single file.
Send + Sync
StorageBackend: Send + Sync, but the handle is a JsValue. Wrap it in a newtype with unsafe impl Send/Sync, gated on single-threaded wasm (cfg(not(target_feature = "atomics"))) with a safety comment, or an equivalent wrapper.
API surface
BrowserDb.openOpfs(name)(name TBD), or a mode option onopen. It fails with a clear error when called outside a dedicated worker or when OPFS is unavailable.- Consider shipping a thin worker wrapper with a promise-based main-thread API (as SQLite-WASM's Worker1/Promiser does), so apps don't hand-roll
postMessage. - Related fix:
IndexedDbBackend::openusesweb_sys::window(), so the existing IndexedDB mode fails inside workers. Use the global scope (js_sys::global()) instead, so both modes work in a worker.
Open questions
- What OPFS does in private browsing on each engine (availability and quota).
- Eviction: OPFS shares the best-effort storage bucket with IndexedDB. Document
navigator.storage.persist(). - Whether
executecan become synchronous in OPFS mode, or should stayPromise-returning for API parity.
Acceptance criteria
- Worker-based OPFS
BrowserDbpasses the existing browser test suite. - Heap use stays roughly flat as the database grows (bounded by the page cache), and open time does not scale with database size.
- A second open of a locked database fails with the file-lock error code.
- A file exported from OPFS opens natively with
Minigraf::open(), and a native.graphfile imports and opens. - Docs: wiki
Use-Cases.md/Architecture.md, README browser section,docs/ERROR_REFERENCE.mdif a new code is needed.
Dependencies
Needs v8 (#374, #434, #388), which provides copy-on-write, page CRCs and O(change) checkpoints. Related: #440.
Philosophy
Aligned: embedded, single .graph file, zero configuration, no new crates. Cost: a second browser persistence path to test, and a worker-only API.
Activity
adityamukho commentedon Oct 7, 2026
One more reason to move away from IndexedDB, found while working on #411/#440 (PR #474):
If an IndexedDB flush fails (quota exceeded, transaction aborted),
BrowserDbhas already drained the buffer's dirty set viatake_dirty()before the asyncwrite_pagescall, and it does not restore it on error. The in-memory state has already activated the new meta. So the next successful flush writes only its dirty pages, and IndexedDB can end up with a meta page that references pages that never reached the store. This affectsapply_write,checkpoint()andimportGraph()insrc/browser/mod.rs.The cause is the split between a sync
StorageBackend(the buffer) and async IndexedDB persistence: the commit point in memory and the commit point in storage are two separate steps. With an OPFSFileSystemSyncAccessHandlebackend,save()writes and syncs through the backend directly, so the native copy-on-write commit protocol (§8.3 of the v8 spec) applies unchanged and there is no separate flush to fail.Until then, the IndexedDB fallback still needs a fix: on flush error, put the pages back into the dirty set (or roll back the in-memory meta) so a later flush can't skip them.
adityamukho commentedon Oct 14, 2026
Moved to v3.1.0: a new persistence path and worker-only API, done after the streaming engine (#473). The browser pins and quarantine from #543 are already on
v3, and an OPFS backend picks them up throughPageAllocatorlike the file backend. Expect a rebase against #550 (large transactions commit as a checkpoint) insrc/browser/mod.rs.docs(roadmap): list the OPFS BrowserDb backend (#441) under v3.1.0