8000 perf: WAL fsync-per-write batching/relaxed-durability escape hatch (#302) by adityamukho · Pull Request #353 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`OpenOptions` `#[non_exhaustive]` would immunise every future field
addition, but is itself a breaking change, so it is a decision to make at
the same time as the version number rather than after it.
- **`OpenOptions` gains a second post-1.0 field: `synchronous`** (#302). Same semver consequence as `allow_unlocked` above — a struct-literal `OpenOptions { .. }` built without `..Default::default()` no longer compiles. One in-tree literal (`src/db.rs`, a test) needed updating; every other in-tree construction already used the spread form or the chainable builder.
- **Minimum supported Rust version is now 1.89** (was effectively 1.85, implied
by `edition = "2024"`). Required for `std::fs::File::try_lock`, stabilised in
1.89. Declared as `rust-version` in `Cargo.toml`, so cargo reports a clear
Expand Down Expand Up @@ -117,6 +118,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- **`or`/`or-join` branches can now short-circuit** (#250): branches are still sorted cheapest-first (landed earlier), but once the branches evaluated so far already produce a match for every incoming binding, remaining (costlier) branches are skipped instead of always being evaluated. This is always safe for `or-join` — branches are projected down to already-bound outer variables before merging, so a skipped branch could only have produced a row identical to one already kept. For plain `or`, it's only safe when the branches evaluated so far introduce no variable beyond what's already bound in the incoming scope (a pure filter) — a branch that binds a genuinely new variable can legitimately contribute a distinct row for an already-"covered" key, so that case still evaluates every branch, exactly as before. Benchmark: `disjunction/or_short_circuit` and `disjunction/or_join_short_circuit` (10k facts, both branches fully covering) show ~11% and ~27% improvement respectively.

- **WAL writes no longer pay a full `fsync()`, and can skip the per-write flush entirely** (#302): `WalWriter::append_entry` called `File::sync_all()` (fsync) after every write, flushing inode metadata the WAL — a pure-append file — never needed alongside the data. It now calls `File::sync_data()` (fdatasync) instead: same durability guarantee, one fewer metadata flush, for every existing caller, no other change required. A new `OpenOptions::synchronous` field (`SyncMode::Full`, the default, matches this fdatasync behavior; `SyncMode::Normal` skips the per-write flush altogether) lets bulk loaders/migrations that can safely re-run from a checkpoint watermark trade per-write durability for throughput — `checkpoint()` still fsyncs the main file unconditionally in both modes, so it remains the hard durability boundary regardless of `synchronous`. Separately: `begin_write()` + N `execute()` + one `commit()` already collapsed N facts into a single WAL write / single fsync before this change; that existing batching pattern is now documented (README, wiki Performance Tuning page) since it was easy to miss.

## v1.2.3 — 2026-08-10

No changes to the core crate. Released so the language bindings have a version
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,8 @@ Benchmarks on Intel Core i7-1065G7 @ 1.30GHz, 16 GB RAM, Rust 1.92.0. See [BENCH

File-backed databases enforce a maximum fact size of **4 080 serialised bytes** per fact. In-memory databases have no limit.

**Durability tuning:** every write is `fsync`'d immediately by default (`SyncMode::Full`). Bulk loaders/migrations that can safely re-run from a checkpoint watermark on failure can trade that for throughput with `OpenOptions::new().synchronous(SyncMode::Normal)` — `checkpoint()` still fsyncs unconditionally in both modes. See the [Performance Tuning wiki page](https://github.com/project-minigraf/minigraf/wiki/Performance-Tuning#configuration-knobs) for the full tradeoff and the write-batching pattern that pairs with it.

## Contributing

This is a solo-maintained project with a long-term vision. Read [PHILOSOPHY.md](PHILOSOPHY.md) and [ROADMAP.md](ROADMAP.md) before proposing features.
Expand Down
Loading
Loading
0