8000 Merge main into v3 (v2.0.4 release) by adityamukho · Pull Request #488 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **The wasm32 builds are tested again, and the browser layer no longer panics or drops pages.** The repo split (#231) moved the browser and WASI CI jobs to minigraf-wasm, which only builds the package, so core's own wasm tests stopped running and the wasm32 test build broke unnoticed. A new `wasm.yml` workflow runs the `wasm-bindgen` tests in headless Firefox and Chrome, runs the unit and integration tests on `wasm32-wasip1` under Wasmtime, and lints both targets with `clippy -D warnings`. Fixes found on the way: `BrowserDb` skipped any dirty page it could not read when flushing to IndexedDB and still reported success, and now returns the error. The IndexedDB upgrade callback, the IndexedDB backend handle and the page-buffer lock no longer `unwrap()` (on wasm32 a panic aborts the whole instance). IndexedDB keys that are not whole page numbers are rejected instead of truncated. The nine `BrowserBufferBackend` tests used `#[test]`, which the wasm test runner never runs, and now run. File-only modules (`wal`, `storage::verify`, `storage::dir_sync`) are no longer compiled for wasm32, so the wasm32 builds have no warnings.
- **The wasm32 builds are tested again, and the browser layer no longer panics or drops pages.** The repo split (#231) moved the browser and WASI CI jobs to minigraf-wasm, which only builds the package, so core's own wasm tests stopped running and the wasm32 test build broke unnoticed. A new `wasm.yml` workflow runs the `wasm-bindgen` tests in headless Firefox and Chrome, runs the unit and integration tests on `wasm32-wasip1` under Wasmtime, and lints both targets with `clippy -D warnings`. Fixes found on the way: `BrowserDb` skipped any dirty page it could not read when flushing to IndexedDB and still reported success, and now returns the error (also fixed in v2.0.4, #470). The IndexedDB upgrade callback, the IndexedDB backend handle and the page-buffer lock no longer `unwrap()` (on wasm32 a panic aborts the whole instance). IndexedDB keys that are not whole page numbers are rejected instead of truncated. The nine `BrowserBufferBackend` tests used `#[test]`, which the wasm test runner never runs, and now run. File-only modules (`wal`, `storage::verify`, `storage::dir_sync`) are no longer compiled for wasm32, so the wasm32 builds have no warnings.
- **`BrowserDb` stores upgraded pages and keeps only live pages** (#411, #440). `BrowserDb.open()` now writes the pages an open produced to IndexedDB, so a v7 store is upgraded to v8 once instead of in memory on every open. Pages on the free list no longer keep their bytes in memory or their keys in IndexedDB: open drops them (after a v7 upgrade, the whole old v7 region), and each `transact`, `retract`, `checkpoint()` and `importGraph()` drops the pages its commit freed, in the same IndexedDB transaction as the commit's pages and meta page. Memory and IndexedDB size now follow the live data rather than the write history, and opening no longer reads dead pages back. `importGraph()` also deletes the keys of the replaced database's pages, which a reopen used to load next to the imported ones. `exportGraph()` writes free pages as zeros and is as long as the file's meta page says; the blob opens natively as before.
- Two values of the same attribute written in one transaction are both returned by every query path. Previously only one came back, and entity-bound and attribute-bound queries could return different ones (#371, #287).
- A retract that removes several values of one attribute in one call now hides all of them (#371).
Expand All @@ -51,6 +51,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Model-based storage test (#385).** `tests/model_based_test.rs` runs random sequences of `transact` (multi-valued batches, transaction- and fact-level valid-time windows, a triple repeated with another window), `retract`, write transactions (committed or rolled back, retracting their own assertions), checkpoints, reopens (default, a 4-page cache, auto-checkpoint every 2 writes) and simulated crashes (a copy of the `.graph` file and WAL taken while the handle is open, so the WAL is replayed) against a file in a temp dir. After every step, `current_tx_count`, `verify`, full rows with windows and tx counts, and entity-, entity+attribute-, attribute-, attribute+value-, value-bound and full-scan queries, under `:as-of N` and `:valid-at T`, must match a plain reference model (one valid-time window per triple, #435; API-011 for two windows of one triple in one statement and API-019 for an empty or inverted window, from `execute` and `tx.execute`, #436; the last statement that writes a triple decides it at commit, #477). Reads inside a write transaction are checked against what its commit will hold. Normal CI runs 24 cases; the nightly property-test workflow runs 30,000.
- **SIGKILL crash test checks committed data (#384).** `tests/crash_kill_test.rs` used to check only that the file reopened and one attribute query ran, so it missed #370 and could not reach #371. The child now runs a seeded workload (multi-valued batches, batched retracts, re-assertions) and logs each transaction after `execute` returns; it checkpoints after every transaction, every third, never (WAL replay), automatically every 2 writes, or every third with `SyncMode::Normal`. After a `SIGKILL` at a random point, the reopened file must hold exactly the model after the last logged transaction or the one in flight, through a full scan, attribute (AEVT), attribute+value (AVET) and entity-bound (EAVT) queries and `:as-of`, with `verify` clean. The checks run again after a second reopen and after a checkpoint plus reopen. Per-PR CI runs 5 rounds (one per mode); the nightly crash-kill workflow runs 100 per OS. `MINIGRAF_CRASH_KILL_SEED` replays a failure.

### Known issues

- Same-transaction multi-valued facts can read back as one value (#371, #287), a fact retracted and re-asserted in one `WriteTransaction` is lost at commit (#477), a later assertion does not replace an earlier valid-time window (#435), and `save()` is not crash-atomic (#374); all are fixed in v3.0.0. All v2.x known issues, with affected versions and workarounds, are listed in the pinned issue #421.

## v2.0.4 — 2026-10-08

Patch release on the v2.x line under the [support policy](PHILOSOPHY.md#support-policy): one `BrowserDb` durability hardening fix, plus new compatibility and crash tests. File format is unchanged (v7). No API changes. Native and WASI builds behave exactly as in v2.0.3.

### Fixed

- **`BrowserDb` no longer drops a dirty page it cannot read from the IndexedDB flush (#470).** `checkpoint()`, `importGraph()` and every write collected dirty pages with `read_page_raw(id).ok()`, so a page that failed to read was left out of the flush while the call still reported success. The read error is now returned. No current code path leaves a dirty id without a page, so this guards against a future bug rather than fixing observed data loss. Affected releases: v0.20.0 through v2.0.3, `BrowserDb` only. The browser unit tests compile again on wasm32 (a duplicate `page_cache_capacity` and `tempfile`-based `dir_sync` tests).

### Tests

- **Golden-file compatibility corpus (#391).** `tests/golden/` holds v7 files written by minigraf 2.0.3, each with a JSON manifest of its CRC32, `tx_count` and expected query results. They cover one checkpoint, several checkpoints, indexes rebuilt on open, a WAL left by a crash, a WAL whose entries were already checkpointed, and same-transaction multi-values (#371). `tests/golden_corpus_test.rs` opens a copy of each file and checks its manifest after open, after a checkpoint and reopen, and after one more write. Golden files are never regenerated, only added; a new `golden-corpus` job in `policy.yml` fails a pull request that changes one. The generator is a standalone crate pinned to `minigraf = "=2.0.3"` (`tests/golden/gen/v7/`).
- **SIGKILL crash test checks committed data (#384).** `tests/crash_kill_test.rs` used to check only that the file reopened and one attribute query ran, so it missed #370. The child now runs a seeded workload (batches over several entities and attributes, batched retracts, re-assertions after a retract) and logs each transaction after `execute` returns; it checkpoints after every transaction, every third, never (WAL replay), automatically every 2 writes, or every third with `SyncMode::Normal`. After a `SIGKILL` at a random point, the reopened file must hold exactly the model after the last logged transaction or the one in flight, through a full scan, attribute (AEVT), attribute+value (AVET) and entity-bound (EAVT) queries and `:as-of`; again after a second reopen and after a checkpoint plus reopen. On v2.x the workload avoids the known issues #371 (two values of one attribute in one transaction) and #435 (asserting a triple that is already live returns one row per assertion); the v3 version covers both. Per-PR CI runs 5 rounds; the nightly crash-kill workflow runs 100 per OS. `MINIGRAF_CRASH_KILL_SEED` replays a failure.

### Known issues

- Same-transaction multi-valued facts can read back as one value (#371, #287), a fact retracted and re-asserted in one `WriteTransaction` is lost at commit (#477), a later assertion does not replace an earlier valid-time window (#435), and `save()` is not crash-atomic (#374); all are fixed in v3.0.0. All v2.x known issues, with affected versions and workarounds, are listed in the pinned issue #421.

## v2.0.3 — 2026-10-06

Patch release on the v2.x line with two data-integrity fixes, under the [support policy](PHILOSOPHY.md#support-policy). File format is unchanged (v7). No API changes. Upgrading is recommended for every v2.x user.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

## Current Maintainer Context

**v2.0.3 released** (2026-10-06) — data-integrity patch under the support policy: indexed queries with >65,535 uncheckpointed facts (#445), tx counter rewound on reopen next to an already-checkpointed WAL (#447). **v2.0.2** (2026-09-27) — last planned v2.x feature/fix release: query fixes (#297, #405, #407, #410), directory fsync (#389), point-query/attribute-scan/checkpoint perf mitigations (#323, #381, #315), CI policy checks (#395), support policy and tiers (#397, #399, #400). **v2.0.1** (2026-09-25) fixed index rebuild on open (#370). **v2.0.0** introduced kernel file locking (#317, #304), structured runtime error codes (#277), `OpenOptions` `#[non_exhaustive]`, and an MSRV of Rust 1.89. See `CHANGELOG.md` for the full rationale and release history.
**v2.0.4 released** (2026-10-08) — `BrowserDb` IndexedDB flush returns an error for an unreadable dirty page (#470); golden-file v7 corpus (#391) and SIGKILL committed-data crash test (#384). **v2.0.3** (2026-10-06) — data-integrity patch under the support policy: indexed queries with >65,535 uncheckpointed facts (#445), tx counter rewound on reopen next to an already-checkpointed WAL (#447). **v2.0.2** (2026-09-27) — last planned v2.x feature/fix release: query fixes (#297, #405, #407, #410), directory fsync (#389), point-query/attribute-scan/checkpoint perf mitigations (#323, #381, #315), CI policy checks (#395), support policy and tiers (#397, #399, #400). **v2.0.1** (2026-09-25) fixed index rebuild on open (#370). **v2.0.0** introduced kernel file locking (#317, #304), structured runtime error codes (#277), `OpenOptions` `#[non_exhaustive]`, and an MSRV of Rust 1.89. See `CHANGELOG.md` for the full rationale and release history.

Current plan (see `ROADMAP.md` and the milestones):
- **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release.
Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ resolver = "2"

[package]
name = "minigraf"
version = "2.0.3"
version = "2.0.4"
edition = "2024"
rust-version = "1.89"
description = "Zero-config, single-file, embedded graph database with bi-temporal Datalog queries"
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Minigraf is a **single-file embedded graph database** that lets you:

```toml
[dependencies]
minigraf = "2.0.3"
minigraf = "2.0.4"
```

Or via cargo:
Expand Down
13 changes: 12 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@ Completed releases and their implementation details live in the [CHANGELOG](CHAN

---

## v2.0.4 — Browser Durability Patch (released 2026-10-08)

File format stays v7 and the public API does not change. Details are in the [CHANGELOG](CHANGELOG.md).

Shipped:

- `BrowserDb` returns an error instead of leaving an unreadable dirty page out of the IndexedDB flush (#470)
- Golden-file compatibility corpus for format v7 (#391) and a SIGKILL crash test that checks the committed data (#384)

---

## v2.0.3 — Data-Integrity Patch (released 2026-10-06)

**First v2.x patch under the [support policy](PHILOSOPHY.md#support-policy).** File format stays v7 and the public API does not change. Details are in the [CHANGELOG](CHANGELOG.md).
Expand Down Expand Up @@ -195,4 +206,4 @@ See [GitHub Issues](https://github.com/project-minigraf/minigraf/issues) for spe

---

**Last updated**: October 2026 — v2.0.3, a data-integrity patch on v2.x, is the current release; v3.0.0 work, including the file-format v8 fix for #287/#371, is developed on the long-lived `v3` branch and merges into `main` at the v3.0.0 cut.
**Last updated**: October 2026 — v2.0.4, a browser durability patch on v2.x, is the current release; v3.0.0 work, including the file-format v8 fix for #287/#371, is developed on the long-lived `v3` branch and merges into `main` at the v3.0.0 cut.
42 changes: 0 additions & 42 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

Loading
0