8000 fix(db): checkpoint removes a WAL whose entries are all checkpointed (#457) by adityamukho · Pull Request #525 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A commit made after a torn WAL entry is no longer lost on recovery** (#513). Recovery replays the WAL up to its first partial or damaged entry, but the writer appended after the end of the file. After a crash during an append, the next session's commits went in after the partial entry, so a second crash before any checkpoint lost them, although each had been fsynced and reported as committed. A failed append (short write, full disk, I/O error) did the same to the next commit through that handle. Opening a database for writing now cuts the WAL back to the end of its last valid entry, and syncs it, before anything is appended. After a failed append, every later write through that handle fails with the new `WAL-007` until the database is reopened or a `checkpoint()` succeeds. The transaction whose append failed is not applied; after a reopen it is either there in full (its entry reached the disk) or not at all. Affected every release since v0.5.0.
- **A v7 fact page with a damaged type byte fails the upgrade with `STG-014`** (#496). Opening a v7 file skipped, without an error, any page in its fact-page range whose type byte was not the fact-page type. The read-write open then upgraded the file to v8 and freed the old v7 pages, so that page's facts were lost for good; the read-only open returned the same answer with them missing. A v2.x writer fills that range with fact pages only, so any other type there is damage: open now fails with `STG-014`, read-write and read-only, and leaves the v7 file as it was. v7 fact pages have no checksum of their own, so damage elsewhere in a page can still decode to wrong facts or fail with another code; keep a copy of a v7 file until it has opened under v3.0.0.
- Reopening next to a WAL whose entries were all already checkpointed no longer rewinds the transaction counter, so later transactions no longer reuse `tx_count` values and `:as-of` history stays correct (#447; also fixed in v2.0.3).
- `checkpoint()` removes a WAL whose entries are all already in the file (#457). When it had no facts to flush, it returned early and left the WAL until a later write was checkpointed. That WAL is left by a crash between a checkpoint's commit and its WAL delete, or by a failed delete (`WAL-006`). Its contents were always skipped on replay, so no data was affected; the sidecar file just stayed. Not backported to v2.x: it is not a data-integrity fix.

### Internal

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1423 tests** (1413 passing, 10 ignored; unit + integration + doc).
**1424 tests** (1414 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1423 tests
cargo test # run 1424 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

8 changes: 7 additions & 1 deletion src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1076,8 +1076,14 @@ impl Minigraf {
// second handle in this one — #304/#314). The guard still matters where
// the filesystem cannot lock (e.g. NFSv3 without lockd, some FUSE
// mounts) and the caller set `allow_unlocked`.
//
// A WAL can still exist here with every entry already in the file
// (a crash between a checkpoint's commit and its WAL delete, or a
// failed delete): replay counts only the entries it applies. Each
// of those entries is at or below the committed tx count, so
// removing the file loses nothing (#457).
if *wal_entry_count == 0 && !pfs.is_dirty() {
return Ok(());
return Self::delete_wal(wal, db_path, wal_entry_count);
}
// `force_dirty` is needed for the WAL-replay case: facts were loaded
// into memory during `replay_wal` but `pfs.dirty` was not set because
Expand Down
6 changes: 3 additions & 3 deletions tests/golden/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ Manifests for a newer format than the reader supports are skipped.
behavior, with a recipe in a generator crate.
- **Expected rows are written by hand from the recipe**, not recorded from a read.
A recorded read would freeze a bug as expected output.
- `"stale_wal": true` marks a WAL whose entries are all already in the file. A
checkpoint then has nothing to do, so the WAL stays until the next write is
checkpointed; the test does not require it to be gone.
- `"stale_wal": true` marks a WAL whose entries are all already in the file. It
describes the shape only: a checkpoint has no facts to flush but still removes
the WAL (#457), and the test requires it to be gone like any other.
- A query that a release is known to get wrong carries `"min_reader"` (the first
major version that gets it right) and `"known_issue"`.

Expand Down
8 changes: 3 additions & 5 deletions tests/golden_corpus_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -219,11 +219,9 @@ fn check_manifest(m: &Manifest, failures: &mut Vec<String>) {
db.checkpoint().unwrap();
}

// 4. Persist: reopen after the checkpoint, which removes the WAL. A WAL whose
// entries are all checkpointed already (`"stale_wal": true`) gives the
// checkpoint nothing to do, so it stays until the next write is checkpointed.
let stale_wal = m.json["stale_wal"].as_bool().unwrap_or(false);
if !stale_wal && wal_path(&path).exists() {
// 4. Persist: reopen after the checkpoint, which removes the WAL, including
// one whose entries are all checkpointed already (`"stale_wal": true`, #457).
if wal_path(&path).exists() {
failures.push(format!("{}: WAL still present after checkpoint", m.name));
}
{
Expand Down
47 changes: 47 additions & 0 deletions tests/wal_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,53 @@ fn test_reopen_with_checkpointed_wal_does_not_reuse_tx_counts() {
}
}

/// #457: a checkpoint with nothing to flush still removes a WAL whose entries
/// are all checkpointed already. Before the fix it stayed until a later write
/// was checkpointed.
#[test]
fn test_checkpoint_removes_wal_whose_entries_are_all_checkpointed() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("stale.graph");
let wal_path = wal_path_for(&db_path);
{
// usize::MAX: no close-time checkpoint, so the restored WAL survives drop.
let db = OpenOptions::default()
.wal_checkpoint_threshold(usize::MAX)
.path(db_path.to_str().unwrap())
.open()
.unwrap();
db.execute("(transact [[:alice :age 30]])").unwrap();
let stale = std::fs::read(&wal_path).unwrap();
db.checkpoint().unwrap();
// Crash between the checkpoint's commit and the WAL delete.
std::fs::write(&wal_path, &stale).unwrap();
}
{
let db = OpenOptions::default()
.wal_checkpoint_threshold(usize::MAX)
.path(db_path.to_str().unwrap())
.open()
.unwrap();
assert!(wal_path.exists(), "stale WAL present after open");
db.checkpoint().unwrap();
assert!(!wal_path.exists(), "stale WAL removed by checkpoint");
// A second checkpoint with no WAL at all is still a no-op.
db.checkpoint().unwrap();
}
let db = Minigraf::open(&db_path).unwrap();
let n = count_results(
db.execute("(query [:find ?a :where [:alice :age ?a]])")
.unwrap(),
);
assert_eq!(n, 1, "fact kept after the stale WAL is removed");
db.execute("(transact [[:alice :age 31]])").unwrap();
let n = count_results(
db.execute("(query [:find ?a :as-of 1 :where [:alice :age ?a]])")
.unwrap(),
);
assert_eq!(n, 1, "the next write does not reuse tx 1");
}

// ── 4. Partial WAL entry is discarded; earlier entries intact ─────────────────

/// Write 1 fact, crash (no checkpoint), then append garbage bytes to the WAL
Expand Down
Loading
0