8000 fix(wal): never append after a torn WAL entry (#513, v2.x backport) by adityamukho · Pull Request #516 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
8000
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed

- **A file shorter than one page is refused at open with `STG-045` unless it is a torn first write (#506).** Open treated any file under 4 KiB as a new database and wrote a new header over it, so a database cut below one page by an interrupted copy, or an unrelated file at the path, was silently replaced. Such a file now opens as a new database only when its bytes are zeros or a prefix of the header a new database writes (what a crash during creation can leave); anything else is `STG-045` and the file is not modified. A zero-length file is still a new database. New code `STG-045`, numbered as in v3.0.0, which also fixes this.
- **A commit made after a torn WAL entry is no longer lost on recovery (#513).** Recovery replays the WAL up to its first partial or damaged entry, but the writer appended after the end of the file. After a crash during an append, the next session's commits went in after the partial entry, so a second crash before any checkpoint lost them, although each had been fsynced and reported as committed. A failed append (short write, full disk, I/O error) did the same to the next commit through that handle. Opening a database for writing now cuts the WAL back to the end of its last valid entry, and syncs it, before anything is appended. After a failed append, every later write through that handle fails with the new `WAL-007` until the database is reopened or a `checkpoint()` succeeds. The transaction whose append failed is not applied; after a reopen it 8000 is there in full or not at all. Affected every release since v0.5.0. Also fixed in v3.0.0.

## v2.0.4 — 2026-10-08

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ Auto-migrates v1/v2/v3/v4/v5/v6 → v7 on open/checkpoint.

## Test Coverage

**1224 tests** (1216 passing, 8 ignored; unit + integration + doc).
**1231 tests** (1223 passing, 8 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1224 tests
cargo test # run 1231 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
18 changes: 18 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ with no `CodedError` anywhere in its chain.
| WAL-004 | Fact serialised size exceeds u32 range | WAL |
| WAL-005 | WAL num_facts exceeds platform usize | WAL |
| WAL-006 | Failed to delete WAL file | WAL |
| WAL-007 | Earlier WAL write failed on this handle | WAL |
| API-001 | Write lock poisoned | Database API |
| API-002 | Unexpected command variant in write path | Database API |
| API-003 | Attribute must be a keyword | Database API |
Expand Down Expand Up @@ -2089,6 +2090,23 @@ The WAL is replayed on open and deleted on checkpoint.

---

### WAL-007 Earlier WAL write failed on this handle

**Error text**: `an earlier write to the WAL failed on this handle; reopen the database or call checkpoint() before writing again`

**Cause**: A write or fsync of the `.wal` file failed earlier on this handle (for example a full disk or an I/O error), and that write returned its own error. The WAL may now end in a partial entry. Recovery stops at the first partial entry, so anything appended after it would be lost after a crash. Every later write through this WAL fails with WAL-007 instead.

The transaction whose write failed was not applied in memory. Whether it is durable is unknown: if its entry reached the disk whole, it is replayed when the database is next opened; otherwise it is lost. It is never applied in part.

**Resolution**:
- Fix the cause (free disk space, check the device), then reopen the database. Opening cuts off a partial entry at the end of the WAL.
- Or call `checkpoint()`. A successful checkpoint writes the committed facts to the `.graph` file and deletes the WAL; the next write starts a new one. The failed transaction is then gone for good.
- If the failed transaction matters, check after reopening whether it is there, and write it again if it is not.

**Scenario**: The disk fills during `db.execute("(transact ...)")`, which fails with an I/O error. The next `transact` fails with WAL-007 until the database is reopened or `checkpoint()` succeeds.

---

## API — Database API Errors

API errors indicate a violated contract in how the public `Minigraf` or
Expand Down
4 changes: 3 additions & 1 deletion docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@

**Verified**: 2026-10-10 with `cargo test`

**Result**: 1,216 passing tests and 8 ignored tests (1,224 total), including the golden-file corpus (#391)
**Result**: 1,223 passing tests and 8 ignored tests (1,231 total), including the golden-file corpus (#391)

Since v2.0.4, `tests/edge_cases_test.rs` checks that a file of 1 to 4,095 bytes is `STG-045`, unchanged and with no WAL, when it is a database cut below one page or foreign bytes, and opens as a new database that takes a write and reads it back after reopen when it is a prefix of the initial header or all zeros (#506).

Since v2.0.4, torn-WAL-tail tests (#513) check that opening a WAL writer cuts off a partial last entry (cut in the checksum, the header or the payload) or an entry with a bad checksum and everything after it, then appends where replay reaches, and leaves an intact WAL byte for byte; that an append that fails part-way makes every later append `WAL-007` while a rejected oversize fact (`WAL-003`, nothing written) does not; through `Minigraf`, that the failed transaction is not applied, later `transact` and `commit` are `WAL-007`, and a `checkpoint()` makes writes work again; and, in `tests/wal_test.rs`, that a commit made after reopening a crash-left WAL with each kind of torn tail survives a second crash.

`tests/property_test.rs` writes random histories (batched transacts, retracts, write transactions, checkpoints) in memory and to a file, and compares random queries (one to three joined patterns, `not` or `not-join`, a comparison predicate, `:as-of`, `:valid-at` or `:any-valid-time`) with a reference evaluator, on the open file and after a reopen; a query error fails the test (#386). The histories leave out the v2.x known issues #371, #435 and #477.

`tests/crash_kill_test.rs` kills a child process running a seeded workload under five checkpoint modes at a random point and checks that the reopened file holds exactly the transactions the child logged as committed, or those plus the one in flight, through full-scan, AEVT, AVET, EAVT and `:as-of` queries, again after a second reopen and after a checkpoint plus reopen (#384). The workload l 10000 eaves out the v2.x known issues #371 and #435.
Expand Down
50 changes: 48 additions & 2 deletions src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -531,8 +531,11 @@ impl Minigraf {
/// replay the facts on next open.
///
/// If the WAL write fails, an error is returned and the in-memory store is
/// left unchanged. The database remains consistent for subsequent in-process
/// operations.
/// left unchanged. The WAL may then end in a partial entry, so later writes
/// through this handle fail with WAL-007 until the database is reopened or
/// a [`checkpoint`](Self::checkpoint) succeeds; reads keep working. Whether
/// the failed transaction is durable is unknown: after a reopen it is there
/// in full or not at all.
///
/// Returns `Err` if called from the same thread that holds an active
/// `WriteTransaction` (use `tx.execute()` instead).
Expand Down Expand Up @@ -2005,6 +2008,49 @@ mod tests {
);
}

// ── #513: a failed WAL append ─────────────────────────────────────────────

/// A WAL append that fails part-way leaves its transaction unapplied and
/// refuses later writes with WAL-007, so nothing lands after the partial
/// entry. A checkpoint deletes the WAL and writes work again.
#[test]
fn test_failed_wal_append_refuses_writes_until_checkpoint() {
fn names(db: &Minigraf) -> usize {
match db
.execute("(query [:find ?n :where [?e :name ?n]])")
.unwrap()
{
QueryResult::QueryResults { results, .. } => results.len(),
_ => 0,
}
}
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("test.graph");
let db = Minigraf::open(&db_path).unwrap();
db.execute("(transact [[:a :name \"A\"]])").unwrap();

crate::wal::tear_next_append(9);
db.execute("(transact [[:b :name \"B\"]])")
.expect_err("the torn append fails");
assert_eq!(names(&db), 1, "the failed transaction is not applied");

let err = db
.execute("(transact [[:c :name \"C\"]])")
.expect_err("later writes are refused");
assert_eq!(err.code(), "WAL-007");
let mut tx = db.begin_write().unwrap();
tx.execute("(transact [[:c :name \"C\"]])").unwrap();
assert_eq!(tx.commit().unwrap_err().code(), "WAL-007");
assert_eq!(names(&db), 1);

db.checkpoint().unwrap();
db.execute("(transact [[:d :name \"D\"]])").unwrap();
assert_eq!(names(&db), 2);
drop(db);
let db = Minigraf::open(&db_path).unwrap();
assert_eq!(names(&db), 2, "A and D survive reopen");
}

// ── file-backed: checkpoint deletes WAL and updates main file ─────────────

#[test]
Expand Down
9 changes: 9 additions & 0 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ pub(crate) enum ErrorCode {
Wal004,
Wal005,
Wal006,
Wal007,
Api001,
Api002,
Api003,
Expand Down Expand Up @@ -980,6 +981,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[
"failed to delete WAL file {}: {}",
ErrorCategory::Wal,
),
(
ErrorCode::Wal007,
"WAL-007",
"an earlier write to the WAL failed on this handle; reopen the database or call checkpoint() before writing again",
ErrorCategory::Wal,
),
(
ErrorCode::Api001,
"API-001",
Expand Down Expand Up @@ -1765,6 +1772,7 @@ mod tests {
| ErrorCode::Wal004
| ErrorCode::Wal005
| ErrorCode::Wal006
| ErrorCode::Wal007
| ErrorCode::Api001
| ErrorCode::Api002
| ErrorCode::Api003
Expand Down Expand Up @@ -1957,6 +1965,7 @@ mod tests {
ErrorCode::Wal004,
ErrorCode::Wal005,
ErrorCode::Wal006,
ErrorCode::Wal007,
ErrorCode::Api001,
ErrorCode::Api002,
ErrorCode::Api003,
Expand Down
Loading
Loading
0