adityamukho · Oct 7, 2026 · Oct 7, 2026
@@ -30,6 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 
 - **The wasm32 builds are tested again, and the browser layer no longer panics or drops pages.** The repo split (#231) moved the browser and WASI CI jobs to minigraf-wasm, which only builds the package, so core's own wasm tests stopped running and the wasm32 test build broke unnoticed. A new `wasm.yml` workflow runs the `wasm-bindgen` tests in headless Firefox and Chrome, runs the unit and integration tests on `wasm32-wasip1` under Wasmtime, and lints both targets with `clippy -D warnings`. Fixes found on the way: `BrowserDb` skipped any dirty page it could not read when flushing to IndexedDB and still reported success, and now returns the error. The IndexedDB upgrade callback, the IndexedDB backend handle and the page-buffer lock no longer `unwrap()` (on wasm32 a panic aborts the whole instance). IndexedDB keys that are not whole page numbers are rejected instead of truncated. The nine `BrowserBufferBackend` tests used `#[test]`, which the wasm test runner never runs, and now run. File-only modules (`wal`, `storage::verify`, `storage::dir_sync`) are no longer compiled for wasm32, so the wasm32 builds have no warnings.
+- **`BrowserDb` stores upgraded pages and keeps only live pages** (#411, #440). `BrowserDb.open()` now writes the pages an open produced to IndexedDB, so a v7 store is upgraded to v8 once instead of in memory on every open. Pages on the free list no longer keep their bytes in memory or their keys in IndexedDB: open drops them (after a v7 upgrade, the whole old v7 region), and each `transact`, `retract`, `checkpoint()` and `importGraph()` drops the pages its commit freed, in the same IndexedDB transaction as the commit's pages and meta page. Memory and IndexedDB size now follow the live data rather than the write history, and opening no longer reads dead pages back. `importGraph()` also deletes the keys of the replaced database's pages, which a reopen used to load next to the imported ones. `exportGraph()` writes free pages as zeros and is as long as the file's meta page says; the blob opens natively as before.
 - Two values of the same attribute written in one transaction are both returned by every query path. Previously only one came back, and entity-bound and attribute-bound queries could return different ones (#371, #287).
 - A retract that removes several values of one attribute in one call now hides all of them (#371).
 - Queries no longer lose or duplicate facts when more than 65,535 facts are waiting for a checkpoint (#445; also fixed in v2.0.3).

@@ -197,7 +197,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are
 
 ## Test Coverage
 
-**1346 tests** (1337 passing, 9 ignored; unit + integration + doc).
+**1347 tests** (1338 passing, 9 ignored; unit + integration + doc).
 See `docs/TEST_COVERAGE.md` for the full per-file breakdown.
 
 **Testing conventions** — see the Testing Conventions section below before writing any tests.

@@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit
 
 ```bash
 cargo run          # interactive Datalog REPL
-cargo test         # run 1346 tests
+cargo test         # run 1347 tests
 cargo run < demos/demo_recursive.txt   # recursive rules demo
 ```
 

@@ -0,0 +1,90 @@
+# BrowserDb: write back migrated pages, keep only live pages — Design
+
+Issues: #411 (migrated pages are not written back to IndexedDB), #440 (bound page
+growth via v8 free-list reuse; avoid v7→v8 migration bloat). Milestone v3.0.0, branch
+`v3`.
+
+## 1. Problem
+
+- `BrowserDb::open()` runs `PersistentFactStorage::new` on the pages loaded from
+  IndexedDB. A v7 store is migrated to v8 in the buffer, but nothing is flushed, so the
+  stored copy stays v7 and every open migrates again (#411).
+- v8 checkpoints reuse freed pages (spec 2026-10-05 §8), so the *live* page set is
+  bounded. But a free page keeps its old bytes in `BrowserBufferBackend` and its key in
+  IndexedDB. After a v7→v8 migration the whole v7 region is free: one extra copy of the
+  database in the WASM heap and in IndexedDB, for good (#440 §2).
+
+## 2. Decisions
+
+D1. **Free pages hold no bytes in the browser.** After open and after each commit,
+`BrowserDb` drops the bytes of every page on the active free list from the buffer and
+deletes its IndexedDB key. Chosen over "migrate into a fresh object store" because it
+also covers ordinary churn, needs no IndexedDB schema version bump, and has one code
+path. Safe because:
+  - A free page is never read before it is written: the allocator hands out ids and the
+    writer fills them; only free-list *chain* pages are read, and those are referenced
+    by the meta, so never dropped.
+  - The single-valid-slot probe (§4.1.1) already skips an unreadable page.
+  - A read of an absent page is still `INT-052`, free or not.
+
+D2. **One IndexedDB transaction per flush.** `IndexedDbBackend::write_pages` takes the
+puts *and* the deletes. The meta page is in the dirty set of the same commit, so data
+pages, meta and deletes commit or abort together (v8 spec §12, verified).
+
+D3. **O(change) per commit.** `PersistentFactStorage::save` records the ids it lists in
+the new free-list head pages (freed pages, chain pages read, ids read but not handed
+out). The untouched tail was already released by an earlier commit or by open.
+`take_released()` hands that list to the caller. Open releases the whole free list once
+(`free_page_ids()`).
+
+D4. **Open flushes (#411).** After `PersistentFactStorage::new`, `open()` releases the
+free list and flushes dirty pages and deletes, like `import_graph()`. A normal open of a
+v8 store with nothing to release does no IndexedDB write.
+
+D5. **Export pads free pages with zeros.** `exportGraph()` writes
+`max(meta.page_count, buffer high-water)` pages. A page the buffer released is written
+as zeros; any other absent page is an error. Natively a zero page on the free list is
+never read (allocator writes first; probe and verify do not decode free pages), so the
+blob opens and verifies with `Minigraf::open()`.
+
+D6. **Buffer bookkeeping.** `BrowserBufferBackend` gains `release(ids) -> Vec<u64>`
+(drops bytes, clears dirty, returns ids that had bytes, i.e. the IndexedDB keys to
+delete) and a `released` id set used only by export. A later `write_page` to a released
+id removes it from the set.
+
+## 3. Failure model
+
+IndexedDB transactions are atomic, so a crash leaves the store at either the previous
+or the new commit. A save that reaches the buffer but never the store (tab closed
+before the flush) reopens at the previous commit, whose pages were never deleted (the
+deletes are in the unflushed transaction).
+
+Out of scope (pre-existing): if a flush *fails*, the drained dirty set is not restored,
+so a later flush can miss pages. Tracked separately.
+
+## 4. Tests
+
+wasm (`wasm-pack test --headless --firefox --features browser`):
+1. v7 fixture pages written straight into IndexedDB → `open()` + query only → page 0
+   in IndexedDB is a v8 meta; IndexedDB key count equals live pages; no v7 bytes left
+   (#411, #440 §2).
+2. N transacts on an IndexedDB store → buffer page count and IndexedDB key count equal
+   `meta.page_count − freelist_count` and stay below a bound independent of N.
+3. A save applied to the buffer but not flushed, then reopen → previous commit, free
+   list consistent, a further transact succeeds.
+4. Export after churn → page 0 meta valid, length `meta.page_count` pages, imports back.
+5. Buffer unit tests for `release`.
+
+Native:
+6. `PersistentFactStorage`: zeroing every free page after save, then reopen, verify and
+   more saves → no findings, facts intact (what an exported blob looks like).
+7. `take_released()` after save lists exactly the ids newly put on the free list.
+
+## 5. Docs
+
+CHANGELOG (v3.0.0, browser), TEST_COVERAGE, CLAUDE.md test count. No new error codes.
+
+## 6. Philosophy
+
+Aligned: reliability (no lost migrations), self-contained (no new deps), single store,
+no format change.
@@ -8,9 +8,16 @@ use std::collections::{HashMap, HashSet};
 /// Implements `StorageBackend` so it can be used with `PersistentFactStorage`.
 /// After `PersistentFactStorage::save()` writes updated pages here, call
 /// `take_dirty()` to retrieve the page IDs that must be flushed to IndexedDB.
+///
+/// A page on the free list holds no bytes (#440): [`release`](Self::release)
+/// drops them. Reading a released page is an error like any absent page; the
+/// allocator always writes a free page before anything reads it.
 pub struct BrowserBufferBackend {
     pages: HashMap<u64, Vec<u8>>,
     dirty: HashSet<u64>,
+    /// Free pages whose bytes were dropped. Only export reads this, to write
+    /// them as zeros.
+    released: HashSet<u64>,
 }
 
 impl BrowserBufferBackend {
@@ -19,6 +26,7 @@ impl BrowserBufferBackend {
         Self {
             pages: HashMap::new(),
             dirty: HashSet::new(),
+            released: HashSet::new(),
         }
     }
 
@@ -28,14 +36,19 @@ impl BrowserBufferBackend {
         Self {
             pages,
             dirty: HashSet::new(),
+            released: HashSet::new(),
         }
     }
 
     /// Load pages and mark every page dirty.
     /// Used during `BrowserDb::import_graph()` so all pages are flushed to IDB.
     pub fn load_pages_all_dirty(pages: HashMap<u64, Vec<u8>>) -> Self {
         let dirty: HashSet<u64> = pages.keys().copied().collect();
-        Self { pages, dirty }
+        Self {
+            pages,
+            dirty,
+            released: HashSet::new(),
+        }
     }
 
     /// Drain and return the set of page IDs written since the last call.
@@ -44,6 +57,40 @@ impl BrowserBufferBackend {
         std::mem::take(&mut self.dirty)
     }
 
+    /// Drop the bytes of the free pages `ids` and forget any unflushed write to
+    /// them. Returns the ids that held bytes: the IndexedDB keys to delete in
+    /// the same transaction as the commit's meta page.
+    pub fn release(&mut self, ids: impl IntoIterator<Item = u64>) -> Vec<u64> {
+        let mut dropped = Vec::new();
+        for id in ids {
+            self.dirty.remove(&id);
+            if self.pages.remove(&id).is_some() {
+                dropped.push(id);
+            }
+            self.released.insert(id);
+        }
+        dropped
+    }
+
+    /// Page `page_id` for export: a released page is all zeros, any other
+    /// absent page is an error.
+    pub fn export_page(&self, page_id: u64) -> Result<Vec<u8>> {
+        if !self.pages.contains_key(&page_id) && self.released.contains(&page_id) {
+            return Ok(vec![0u8; PAGE_SIZE]);
+        }
+        self.read_page(page_id)
+    }
+
+    /// The ids of the pages holding bytes.
+    pub fn page_ids(&self) -> impl Iterator<Item = u64> + '_ {
+        self.pages.keys().copied()
+    }
+
+    /// The number of pages holding bytes.
+    pub fn stored_page_count(&self) -> usize {
+        self.pages.len()
+    }
+
     /// Mark `page_id` dirty without writing it, for tests of the flush path.
     #[cfg(test)]
     pub(crate) fn mark_dirty_for_test(&mut self, page_id: u64) {
@@ -76,6 +123,7 @@ impl StorageBackend for BrowserBufferBackend {
         }
         self.pages.insert(page_id, data.to_vec());
         self.dirty.insert(page_id);
+        self.released.remove(&page_id);
         Ok(())
     }
 
@@ -178,6 +226,43 @@ mod tests {
         assert!(buf.write_page(0, &[0u8; 100]).is_err());
     }
 
+    #[wasm_bindgen_test]
+    fn release_drops_bytes_and_dirty_and_reports_stored_ids() {
+        let mut buf = BrowserBufferBackend::load_pages(HashMap::from([(2u64, page(2))]));
+        buf.write_page(3, &page(3)).unwrap();
+        let mut dropped = buf.release([2, 3, 4]);
+        dropped.sort_unstable();
+        assert_eq!(dropped, vec![2, 3], "page 4 held no bytes");
+        assert!(
+            buf.take_dirty().is_empty(),
+            "a released write is not flushed"
+        );
+        assert!(buf.read_page(2).is_err(), "a released page reads as absent");
+        assert_eq!(buf.stored_page_count(), 0);
+        assert!(
+            buf.release([2]).is_empty(),
+            "releasing twice deletes nothing"
+        );
+    }
+
+    #[wasm_bindgen_test]
+    fn export_page_zeroes_released_pages_only() {
+        let mut buf = BrowserBufferBackend::new();
+        buf.write_page(2, &page(9)).unwrap();
+        let _ = buf.release([2]);
+        assert_eq!(buf.export_page(2).unwrap(), page(0));
+        assert!(
+            buf.export_page(5).is_err(),
+            "an absent page that is not free"
+        );
+        buf.write_page(2, &page(7)).unwrap();
+        assert_eq!(
+            buf.export_page(2).unwrap(),
+            page(7),
+            "rewritten after reuse"
+        );
+    }
+
     #[wasm_bindgen_test]
     fn read_missing_page_errors() {
         let buf = BrowserBufferBackend::new();

@@ -142,31 +142,41 @@ impl IndexedDbBackend {
         }
     }
 
-    /// Write a batch of pages to IndexedDB in a single `readwrite` transaction.
+    /// Write a batch of pages and delete a batch of page keys in a single
+    /// `readwrite` transaction.
     ///
-    /// All `put` operations are queued synchronously on the store, then we wait
-    /// for the transaction's `oncomplete` event. If any put fails, the transaction
-    /// is aborted and an error is returned.
+    /// All requests are queued synchronously on the store, then we wait for the
+    /// transaction's `oncomplete` event. If any request fails, the transaction
+    /// is aborted and an error is returned, so the pages of a commit, its meta
+    /// page and the keys of the pages it freed land together or not at all.
     ///
-    /// `pages` is a list of `(page_id, page_bytes)` pairs. Empty input is a no-op.
-    pub async fn write_pages(&self, pages: Vec<(u64, Vec<u8>)>) -> Result<(), JsValue> {
-        if pages.is_empty() {
+    /// `pages` is a list of `(page_id, page_bytes)` pairs; `deletes` is a list
+    /// of page ids. Both empty is a no-op.
+    pub async fn write_pages(
+        &self,
+        pages: Vec<(u64, Vec<u8>)>,
+        deletes: Vec<u64>,
+    ) -> Result<(), JsValue> {
+        if pages.is_empty() && deletes.is_empty() {
             return Ok(());
         }
         let tx = self
             .db
             .transaction_with_str_and_mode(&self.store_name, IdbTransactionMode::Readwrite)?;
         let store = tx.object_store(&self.store_name)?;
 
+        for page_id in &deletes {
+            store.delete(&JsValue::from_f64(*page_id as f64))?;
+        }
         for (page_id, data) in &pages {
             let key = JsValue::from_f64(*page_id as f64);
             let arr = Uint8Array::from(data.as_slice());
             store.put_with_key(&arr, &key)?;
         }
 
         // Wait for the transaction to commit. The IDB transaction commits
-        // automatically once all put requests have been processed and no
-        // new requests are made. We wait here to ensure durability before
+        // automatically once all requests have been processed and no new
+        // requests are made. We wait here to ensure durability before
         // returning to the caller.
         JsFuture::from(transaction_to_promise(&tx)).await?;
         Ok(())