8000 fix(durability): fsync parent directory after creating .graph/WAL and deleting WAL (#389) by adityamukho · Pull Request #404 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **A power loss could lose a new database or WAL file, or bring back a deleted WAL (#389).** On Linux and other POSIX systems, a created or deleted file survives a power loss only once its parent directory is fsynced; Minigraf fsynced file contents but never a directory. Creating the `.graph` file or the `<db>.wal` sidecar, and deleting the WAL after a checkpoint, now fsync the parent directory after the file operation. A resurrected WAL was already harmless, since replay skips entries at or below the last checkpointed transaction, but a lost `.graph` or WAL lost committed data. Windows needs no directory sync and is unchanged. Process kills were never affected, because they leave the OS page cache intact.
- **Bound-entity queries could drop rows when one `WriteTransaction` wrote the same attribute in several valid-time windows (#323).** Both facts share a `tx_count`, and the selective lookup path de-duplicated on `(entity, attribute, tx_count, asserted)`, so one window's value was silently lost, while the same query via a full scan returned both. That de-duplication has been removed, and bound-entity queries now always match a full scan.

### Notes
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ cargo run < demos/demo_negation.txt
- `btree_v6.rs`: On-disk B+tree (`build_btree`, `rebuild_btree_incremental`, `OnDiskIndexReader`, `MutexStorageBackend`)
- `btree.rs`: Legacy v5 B+tree (migration only)
- `cache.rs`: LRU page cache (`PageCache`, default 256 pages)
- `dir_sync.rs`: `sync_parent_dir` — fsyncs the parent directory after creating the `.graph`/WAL or deleting the WAL (no-op off Unix)
- `packed_pages.rs`: Packed fact pages (~25 facts/4KB page), `MAX_FACT_BYTES`
- `persistent_facts.rs`: `PersistentFactStorage` — v7 save/load, auto-migration v1–v6→v7

Expand Down Expand Up @@ -167,7 +168,7 @@ Auto-migrates v1/v2/v3/v4/v5/v6 → v7 on open/checkpoint.

## Test Coverage

**1190 tests passing** (1182 passing, 8 ignored; unit + integration + doc).
**1200 tests passing** (1192 passing, 8 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1190 tests
cargo test # run 1200 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@

**Verified**: 2026-09-26 with `cargo test`

**Result**: 1,182 passing tests and 8 ignored tests (1,190 total)
**Result**: 1,192 passing tests and 8 ignored tests (1,200 total)

The v2.0.1 suite covers the public database API, the Datalog engine, storage and recovery paths, and the v2.0.0 locking and structured-error-code changes, plus v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Since v2.0.1, `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a cyclic leaf chain or damaged slot directory is rejected rather than copied (#315).
The v2.0.1 suite covers the public database API, the Datalog engine, storage and recovery paths, and the v2.0.0 locking and structured-error-code changes, plus v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Since v2.0.1, `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a cyclic leaf chain or damaged slot directory is rejected rather than copied (#315). Directory-sync unit tests check that creating the `.graph` file or the WAL, and deleting the WAL, fsync the parent directory after the file operation, and that reopening an existing file does not (#389).

### Covered Areas

Expand Down
54 changes: 53 additions & 1 deletion src/storage/backend/file.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/// File-based storage backend for native platforms.
use crate::error::{ErrorCode, bail_coded, err_coded};
use crate::storage::dir_sync::sync_parent_dir;
use crate::storage::{FileHeader, PAGE_SIZE, StorageBackend};
use anyhow::Result;
use std::fs::{File, OpenOptions};
Expand Down Expand Up @@ -285,9 +286,14 @@ impl FileBackend {
}
}
} else {
// New file or empty file: write initial header
// New file or empty file: write initial header, then make the
// file's directory entry durable. Without the directory sync a
// power loss can lose the whole file despite the header fsync
// (#389). An empty file left by a crash before this point takes
// this branch again on the next open, so the sync is repeated.
let header = FileHeader::new();
Self::write_header(&mut file, &header)?;
sync_parent_dir(&path)?;
header
};

Expand Down Expand Up @@ -855,4 +861,50 @@ mod tests {
"exactly one concurrent claimant should be admitted"
);
}

// ── #389: parent-directory fsync ────────────────────────────────────────

#[test]
fn test_create_syncs_parent_dir_after_header_write() {
use crate::storage::dir_sync::take_sync_log;
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("new.graph");
take_sync_log();

let backend = FileBackend::open(&path).unwrap();

let log = take_sync_log();
assert_eq!(
log.len(),
1,
"creating the file must sync its directory once"
);
assert_eq!(
log[0].dir,
dir.path(),
"must sync the file's parent directory"
);
assert!(
log[0].child_existed,
"directory sync must follow the create"
);
drop(backend);
}

#[test]
fn test_reopen_existing_does_not_sync_parent_dir() {
use crate::storage::dir_sync::take_sync_log;
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("existing.graph");
drop(FileBackend::open(&path).unwrap());
take_sync_log();

let backend = FileBackend::open(&path).unwrap();

assert!(
take_sync_log().is_empty(),
"reopening an initialized file creates nothing, so needs no directory sync"
);
drop(backend);
}
}
126 changes: 126 additions & 0 deletions src/storage/dir_sync.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
//! Parent-directory fsync (#389).
//!
//! On POSIX systems a newly created, deleted or renamed file survives a power
//! loss only once its parent directory has been fsynced: `fsync` on the file
//! itself makes the file's *contents* durable, not the directory entry that
//! names it. Without this, a power loss (not a process kill -- a kill leaves
//! the OS page cache intact) can lose a freshly created `.graph` or `<db>.wal`,
//! or bring back a WAL that `checkpoint()` deleted.
//!
//! Durability order:
//!
//! - Create: create file → write + `sync_all` its first bytes → sync parent dir.
//! - Delete: remove file → sync parent dir.
//!
//! Windows needs no directory sync (NTFS journals directory metadata), and
//! cannot open a directory as a `File` without extra flags, so this is a no-op
//! there and on every other non-Unix target.

use std::io;
use std::path::Path;

/// Fsync the directory that contains `child`, making `child`'s creation or
/// removal durable.
///
/// `child` is the path of the file that was just created or removed, not the
/// directory itself. A bare file name (no parent component) refers to the
/// current directory.
pub(crate) fn sync_parent_dir(child: &Path) -> io::Result<()> {
let dir = match child.parent() {
Some(p) if !p.as_os_str().is_empty() => p,
_ => Path::new("."),
};

#[cfg(test)]
record(dir, child);

#[cfg(unix)]
std::fs::File::open(dir)?.sync_all()?;
#[cfg(not(unix))]
let _ = dir;
Ok(())
}

// ─── Test-only call log ─────────────────────────────────────────────────────
//
// A power loss cannot be simulated in a test, so tests instead check that the
// sync happens at all and in the right order relative to the file operation:
// each call records the directory and whether `child` existed at that moment.
// After a create it must exist; after a delete it must not. Thread-local, so
// concurrently running tests do not see each other's calls.

#[cfg(test)]
thread_local! {
static SYNC_LOG: std::cell::RefCell<Vec<DirSync>> =
const { std::cell::RefCell::new(Vec::new()) };
}

/// One recorded `sync_parent_dir` call.
#[cfg(test)]
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct DirSync {
/// The directory that was synced.
pub dir: std::path::PathBuf,
/// The file the sync was made on behalf of.
pub child: std::path::PathBuf,
/// Whether `child` existed when the directory was synced.
pub child_existed: bool,
}

#[cfg(test)]
fn record(dir: &Path, child: &Path) {
SYNC_LOG.with(|log| {
log.borrow_mut().push(DirSync {
dir: dir.to_path_buf(),
child: child.to_path_buf(),
child_existed: child.exists(),
})
});
}

/// Return and clear this thread's recorded directory syncs.
#[cfg(test)]
pub(crate) fn take_sync_log() -> Vec<DirSync> {
SYNC_LOG.with(|log| std::mem::take(&mut *log.borrow_mut()))
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn test_syncs_parent_of_child() {
let dir = tempfile::tempdir().unwrap();
let child = dir.path().join("a.graph");
std::fs::write(&child, b"x").unwrap();
take_sync_log();

sync_parent_dir(&child).unwrap();

let log = take_sync_log();
assert_eq!(log.len(), 1, "expected exactly one directory sync");
assert_eq!(log[0].dir, dir.path(), "must sync the child's parent");
assert!(log[0].child_existed, "child existed at sync time");
}

#[test]
fn test_bare_file_name_syncs_current_dir() {
take_sync_log();
sync_parent_dir(Path::new("no-such-file.graph")).unwrap();
let log = take_sync_log();
assert_eq!(log.len(), 1, "expected exactly one directory sync");
assert_eq!(log[0].dir, Path::new("."), "bare name means current dir");
assert!(!log[0].child_existed, "child does not exist");
}

#[cfg(unix)]
#[test]
fn test_missing_parent_dir_is_an_error() {
let dir = tempfile::tempdir().unwrap();
let child = dir.path().join("gone").join("a.graph");
assert!(
sync_parent_dir(&child).is_err(),
"a directory that cannot be opened must surface as an error"
);
}
}
1 change: 1 addition & 0 deletions src/storage/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ pub mod backend;
pub mod btree;
pub mod btree_v6;
pub mod cache;
pub(crate) mod dir_sync;
pub mod index;
pub mod packed_pages;
pub mod persistent_facts;
Expand Down
Loading
Loading
0