8000 feat(browser): OPFS mode package tests, README and core pin (#441) by adityamukho · Pull Request #26 · project-minigraf/minigraf-wasm · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,18 @@ jobs:
# and the consumer contract: @minigraf/browser in headless Chromium,
# @minigraf/wasi under node:wasi on every OS.
consumer:
name: Consumer test (${{ matrix.package }}, ${{ matrix.os }})
name: Consumer test (${{ matrix.package }}, ${{ matrix.os }}, ${{ matrix.browser }})
needs: package
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- { package: browser, os: ubuntu-24.04 }
- { package: browser, os: ubuntu-24.04, browser: chromium }
- { package: browser, os: ubuntu-24.04, browser: firefox }
- { package: browser, os: ubuntu-24.04, browser: webkit }
# macOS WebKit (Cocoa) has OPFS in workers; the Linux build does not.
- { package: browser, os: macos-15, browser: webkit }
- { package: wasi, os: ubuntu-latest }
- { package: wasi, os: macos-15 }
- { package: wasi, os: windows-latest }
Expand All @@ -63,6 +67,7 @@ jobs:
shell: bash
env:
PLAYWRIGHT_DEPS: '1'
BROWSER: ${{ matrix.browser }}
run: bash consumer-test/run.sh ${{ matrix.package }} local dist-${{ matrix.package }} ${{ inputs.version || '0.0.0-ci' }}

readme-sync:
Expand All @@ -74,5 +79,6 @@ jobs:
ref: ${{ inputs.ref }}
- run: |
python3 -I consumer-test/readme_sync.py README.md consumer-test/browser/quickstart.js '## Browser (`@minigraf/browser`)'
python3 -I consumer-test/readme_sync.py README.md consumer-test/browser/opfs-example-worker.js '### OPFS worker'
python3 -I consumer-test/readme_sync.py README.md consumer-test/wasi/repl.mjs '## WASI (`@minigraf/wasi`)'
python3 -I consumer-test/readme_sync.py wasi-pkg/README.md consumer-test/wasi/repl.mjs
2 changes: 1 addition & 1 deletion .github/workflows/package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
const pkg = JSON.parse(fs.readFileSync(p, 'utf8'));
pkg.name = '@minigraf/browser';
pkg.version = process.env.VERSION;
pkg.description = 'Minigraf for the browser: a bi-temporal graph database with Datalog queries, persisted in IndexedDB';
pkg.description = 'Minigraf for the browser: a bi-temporal graph database with Datalog queries, persisted in IndexedDB or OPFS';
pkg.files.push('LICENSE-MIT', 'LICENSE-APACHE');
fs.writeFileSync(p, JSON.stringify(pkg, null, 2) + '\n');
"
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ crate-type = ["cdylib", "rlib"]

[dependencies]
# Unreleased v3 core (format v8). Switches to version = "=3.0.0" at the v3.0.0 release.
minigraf = { git = "https://github.com/project-minigraf/minigraf", rev = "32d71ce013a5af0748185fa69403d9151ba94453", features = ["browser"] }
minigraf = { git = "https://github.com/project-minigraf/minigraf", rev = "df7e8170d4cf3bedcdbf91e85d81f7b7ea1ca959", features = ["browser"] }

[features]
browser = ["minigraf/browser"]
Expand Down
92 changes: 87 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,87 @@ const result = JSON.parse(await db.execute('(query [:find ?n :where [?e :name ?n
console.log(result.results[0][0]) // "Alice"
```

### Choosing a storage mode

From core v3.0.0, `BrowserDb.openOpfs(name)` stores one real `<name>.graph` file
in the Origin Private File System (OPFS). It requires a dedicated module worker
in a secure context (HTTPS or localhost). Pages are read on demand through a
256-page cache; commits write directly to the file, with no WAL. OPFS starts at
Tier 2 support. `BrowserDb.open(name)` uses IndexedDB on the main thread or in a
worker and keeps the database's pages in wasm memory. `openInMemory()` is transient.

The same name in the two persistent modes names two independent databases. There
is no automatic fallback or migration. Invalid OPFS names (empty, `.`, `..`, a
slash/backslash, or NUL) throw `[API-017]`; names are not
normalised. Database `<name>` maps to `<name>.graph`; the import staging entry
is `<name>.graph-import`, which no database name maps to. No suffix is reserved.
The staging entry is removed after publication or recovery. A second worker/tab opening an in-use OPFS file gets
`[STG-026]`. Call `close()` to release its handle and lock immediately; terminating
the worker also releases it. Main-thread or unsupported OPFS opens give `[API-026]`.

`close()` is synchronous and idempotent in all modes. It performs no save and
discards an open transaction. Earlier cursors and prepared queries fail with
`[API-018]` after close. Reentrant close inside a UDF, or any call admitted during
an OPFS import, gets `[API-027]`. Freeing a dependent, cursor `close()` and
transaction `rollback()` remain permitted. OPFS import admits an empty blob;
a nonempty v8 image with a selectable meta and complete, verified contents
(including an empty v8 image); or a supported v7 image that migrates completely.
It publishes the complete old or new database across interruption, and invalidates
earlier cursors and prepared queries on success. A failure at publication closes
the database; reopen to recover.

A write resolves after both page and meta flushes. Browser `flush()` does not
promise power-loss durability. A rejected write means **unconfirmed**, and may be
wholly present after reopen; later writes fail with `[STG-046]` until close/reopen.
If one meta page is damaged, OPFS reopens at the other valid page, which can lose
an acknowledged commit if its newest meta later rots. Origin quota and eviction
are browser-managed, as with IndexedDB. The application may request
`navigator.storage.persist()`; Minigraf never does so itself. See the core
[durability contract](https://github.com/project-minigraf/minigraf/blob/v3/docs/DURABILITY.md).

### OPFS worker

Serve these files alongside the installed package (the worker's import URL below
assumes `/node_modules` is served; a bundler may resolve the package instead).
Save this as `opfs-example-worker.js`:

```js
import init, { BrowserDb } from '/node_modules/@minigraf/browser/minigraf_wasm.js'

const ready = init()
self. ({ data: { name } }) => {
let db
try {
await ready
db = await BrowserDb.openOpfs(name)
await db.execute('(transact [[:alice :name "Alice"]])')
const names = JSON.parse(await db.execute('(query [:find ?n :where [?e :name ?n]])')).results
self.postMessage({ names })
} catch (error) {
self.postMessage({ error: String(error) })
} finally {
db?.close()
}
}
```

From a page module:

```js
const worker = new Worker('./opfs-example-worker.js', { type: 'module' })
worker. data }) => {
if (data.error) console.error(data.error)
else console.log(data.names) // [["Alice"]]
worker.terminate()
}
worker.postMessage({ name: 'agent-memory' })
```

To move a database between modes, export its v8 bytes with `source.exportGraph()`
and pass them to `await target.importGraph(bytes)` in the other mode. With a
worker, send the `Uint8Array` by `postMessage`; after importing, close both handles
when finished. Migration is explicit and replaces the target's contents.

### Query cursors

`db.query(datalog)` returns a `BrowserCursor` whose answer is fixed when it opens.
Expand Down Expand Up @@ -141,8 +222,7 @@ try {
stages nothing and leaves the transaction open. No cursors or prepared queries inside
a transaction.
- `await tx.commit()` applies everything as one transaction at once (other code sees the
facts as soon as `commit()` is called) and resolves after the IndexedDB write: the
commit is durable when the promise resolves. There is no WAL. A failed IndexedDB write
facts as soon as `commit()` is called) and resolves after the IndexedDB write or OPFS flushes. There is no WAL. A failed storage operation
rejects it, as a failed `execute` write does. `tx.rollback()` discards the
transaction, and so does freeing it while open (`tx.free()`).
- After a commit (successful or not) or a rollback the transaction is finished:
Expand All @@ -157,9 +237,11 @@ try {

### Not available in the browser

`BrowserDb` keeps its pages in memory and writes them to IndexedDB; it has no `.graph`
file, WAL or file lock. So the file-backed options of the other bindings
(`read_only`, `allow_unlocked`, `page_cache_size`, the WAL options) do not apply.
`BrowserDb.open()` keeps its pages in memory and writes them to IndexedDB; OPFS
mode uses a `.graph` file inside the origin. Neither mode exposes a native
filesystem path or WAL. OPFS uses its exclusive sync handle as a browser lock.
The file-backed options of the other bindings (`read_only`, `allow_unlocked`,
`page_cache_size`, the WAL options) cannot be configured here.
`BrowserDb` takes no open options at all: the override list (`allow_overrides`), the
query limits and `transaction_chunk_facts` keep their defaults, and a prepared query's
limits are set with its setters. The raw log writer (`LogWriter`) is not built for
Expand Down
47 changes: 41 additions & 6 deletions consumer-test/browser/driver.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@
// quick start and the contract pages in headless Chromium.
// usage: node driver.mjs <expected version>
import { createServer } from "node:http";
import { readFile } from "node:fs/promises";
import { mkdtemp, readFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { extname, join, normalize } from "node:path";
import { chromium } from "playwright";
import { chromium, firefox, webkit } from "playwright";

const expected = process.argv[2];
const root = process.cwd();
Expand All @@ -28,17 +29,32 @@ const server = createServer(async (req, res) => {
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const origin = `http://127.0.0.1:${server.address().port}`;

const browser = await chromium.launch();
const engine = process.env.BROWSER ?? "chromium";
const engines = { chromium, firefox, webkit };
if (!engines[engine]) throw new Error(`unknown browser ${engine}`);
// WebKit treats Playwright's default (ephemeral) context like a private
// window and denies OPFS there, so WebKit runs in a persistent context with
// a fresh profile directory.
let browser, context;
if (engine === "webkit") {
const profile = await mkdtemp(join(tmpdir(), "minigraf-webkit-"));
context = await webkit.launchPersistentContext(profile);
} else {
browser = await engines[engine].launch();
context = await browser.newContext();
}
const version = browser ? browser.version() : context.browser()?.version() ?? "(persistent context)";
console.log(`browser: ${engine} ${version}`);
let failed = false;
try {
const page = await browser.newPage();
const page = await context.newPage();
const errors = [];
page.on("pageerror", (e) => errors.push(String(e)));
page.on("console", (m) => console.log(` [page] ${m.text()}`));

// Waits for the page's module to finish and returns window.__result.
const result = async () => {
await page.waitForFunction(() => window.__result !== undefined, null, { timeout: 60_000 })
await page.waitForFunction(() => window.__result !== undefined, null, { timeout: 180_000 })
.catch(() => {
throw new Error(`the page did not finish: ${errors.join("; ") || "no error reported"}`);
});
Expand All @@ -58,11 +74,30 @@ try {
const second = await result();
if (second.error || second.phase !== "read") throw new Error(second.error ?? `phase ${second.phase}`);
for (const c of second.checks) console.log(`ok: ${c}`);

await page.goto(`${origin}/opfs.html`);
const opfs = await result();
if (opfs.error) throw new Error(opfs.error);
for (const c of opfs.checks) console.log(`ok: ${c}`);
if (opfs.unavailable) {
// Playwright's Linux WebKit has no OPFS in workers, so openOpfs must
// reject with API-026 there. Anywhere else (macOS WebKit included) OPFS
// is required.
if (engine !== "webkit" || process.platform !== "linux") {
throw new Error(`OPFS unavailable on ${engine} (${process.platform}): ${opfs.unavailable}`);
}
// Only the known missing capability; any other API-026 on WebKit fails.
if (!opfs.unavailable.includes("no navigator.storage.getDirectory()")) {
throw new Error(`unexpected OPFS rejection on ${engine}: ${opfs.unavailable}`);
}
console.log(`limitation: ${engine} ${version} has no OPFS in a dedicated worker; openOpfs rejected with ${opfs.unavailable}`);
}
} catch (e) {
console.error(`::error::${e.message}`);
failed = true;
} finally {
await browser.close();
await context.close();
if (browser) await browser.close();
server.close();
}
process.exit(failed ? 1 : 0);
17 changes: 17 additions & 0 deletions consumer-test/browser/opfs-example-worker.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import init, { BrowserDb } from '/node_modules/@minigraf/browser/minigraf_wasm.js'

const ready = init()
self. ({ data: { name } }) => {
let db
try {
await ready
db = await BrowserDb.openOpfs(name)
await db.execute('(transact [[:alice :name "Alice"]])')
const names = JSON.parse(await db.execute('(query [:find ?n :where [?e :name ?n]])')).results
self.postMessage({ names })
} catch (error) {
self.postMessage({ error: String(error) })
} finally {
db?.close()
}
}
93 changes: 93 additions & 0 deletions consumer-test/browser/opfs-worker.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
// A real module worker consuming the installed package, without an import map.
import init, { BrowserDb } from '/node_modules/@minigraf/browser/minigraf_wasm.js'

await init()
let db
const rows = (query) => JSON.parse(db.prepare(query).execute({})).results

self. ({ data: { id, op, name, bytes } }) => {
try {
let value
switch (op) {
case 'open':
db = await BrowserDb.openOpfs(name)
value = true
break
case 'write':
await db.execute('(transact [[:alice :name "Alice"]])')
value = true
break
case 'names':
value = rows('(query [:find ?n :where [?e :name ?n]])')
break
case 'close':
db.close()
value = true
break
case 'drop-all': {
const before = db.exportGraph()
const cursor = db.query('(query [:find ?n :where [?e :name ?n]])')
const prepared = db.prepare('(query [:find ?n :where [?e :name ?n]])')
const tx = db.beginWrite()
tx.execute('(transact [[:unsaved :name "unsaved"]])')
db.free()
let lockError
try { await BrowserDb.openOpfs(name) } catch (e) { lockError = String(e) }
cursor.free()
prepared.free()
tx.free()
db = await BrowserDb.openOpfs(name)
const after = db.exportGraph()
value = {
heldByDependents: lockError?.startsWith('[STG-026]') === true,
unchanged: before.length === after.length && before.every((b, i) => b === after[i]),
rows: rows('(query [:find ?n :where [?e :name ?n]])'),
}
break
}
case 'stream':
for (let sequence = 1; sequence <= 10000; sequence++) {
await db.execute(`(transact [[:commit-${sequence} :sequence ${sequence}] [:commit-${sequence} :complete true]])`)
self.postMessage({ event: 'ack', sequence })
}
value = true
break
case 'prefix':
value = {
rows: rows('(query [:find ?s ?ok :where [?e :sequence ?s] [?e :complete ?ok] :order-by [?s :asc]])'),
sequences: rows('(query [:find ?s :where [?e :sequence ?s]])').length,
markers: rows('(query [:find ?ok :where [?e :complete ?ok]])').length,
}
break
case 'indexeddb': {
const indexed = await BrowserDb.open(name)
await indexed.execute('(transact [[:alice :name "Alice"]])')
await indexed.checkpoint()
indexed.close()
const reopened = await BrowserDb.open(name)
value = JSON.parse(await reopened.execute('(query [:find ?n :where [?e :name ?n]])')).results
reopened.close()
break
}
case 'replacement': {
const source = BrowserDb.openInMemory()
const facts = Array.from({ length: 1000 }, (_, i) => `[:new-${i} :name "new-${i}"]`).join(' ')
await source.execute(`(transact [${facts}])`)
value = source.exportGraph()
source.close()
break
}
case 'import':
self.postMessage({ event: 'import-start' })
await db.importGraph(new Uint8Array(bytes))
value = true
break
default:
throw new Error(`unknown operation ${op}`)
}
self.postMessage({ id, value })
} catch (error) {
self.postMessage({ id, error: String(error) })
}
}
self.postMessage({ event: 'ready' })
7 changes: 7 additions & 0 deletions consumer-test/browser/opfs.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
<!doctype html>
<meta charset="utf-8">
<title>OPFS package consumer</title>
<script type="importmap">
{"imports":{"@minigraf/browser":"/node_modules/@minigraf/browser/minigraf_wasm.js"}}
</script>
<script type="module" src="opfs.js"></script>
Loading
Loading
0