8000 fix(storage): refuse a sub-page file that is not a torn first write (#506, v2.x backport) by adityamukho · Pull Request #509 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Unreleased

### Fixed

- **A file shorter than one page is refused at open with `STG-045` unless it is a torn first write (#506).** Open treated any file under 4 KiB as a new database and wrote a new header over it, so a database cut below one page by an interrupted copy, or an unrelated file at the path, was silently replaced. Such a file now opens as a new database only when its bytes are zeros or a prefix of the header a new database writes (what a crash during creation can leave); anything else is `STG-045` and the file is not modified. A zero-length file is still a new database. New code `STG-045`, numbered as in v3.0.0, which also fixes this.

## v2.0.4 — 2026-10-08

Patch release on the v2.x line under the [support policy](PHILOSOPHY.md#support-policy): one `BrowserDb` durability hardening fix, plus new compatibility and crash tests. File format is unchanged (v7). No API changes. Native and WASI builds behave exactly as in v2.0.3.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ Auto-migrates v1/v2/v3/v4/v5/v6 → v7 on open/checkpoint.

## Test Coverage

**1223 tests** (1215 passing, 8 ignored; unit + integration + doc).
**1224 tests** (1216 passing, 8 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1223 tests
cargo test # run 1224 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
14 changes: 14 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ with no `CodedError` anywhere in its chain.
| STG-025 | Database already open in this process | Storage |
| STG-026 | Database locked by another process | Storage |
| STG-027 | Filesystem does not support file locking | Storage |
| STG-045 | File shorter than one page and not a new database | Storage |
| WAL-001 | Invalid WAL magic number | WAL |
| WAL-002 | Unsupported WAL version | WAL |
| WAL-003 | Fact serialised size exceeds maximum | WAL |
Expand Down Expand Up @@ -1987,6 +1988,19 @@ See the [file format section in README](../README.md#file-format) for version hi

**NFSv3 `nolock` is not covered by this error — it is a silent, undetectable gap (#334).** An export explicitly mounted with `-o nolock` behaves differently from plain "no `lockd`": the Linux NFS client serves `flock`/OFD locks out of its own local, in-kernel lock table instead of going over NLM to the server. `try_lock` sees an ordinary local success, `classify` takes the same branch it would on a filesystem where locking genuinely works, and this error is never raised — `allow_unlocked` is never consulted because the code never learns the mount can't really lock. Two separate client hosts writing the same `nolock` export each get a false `Ok(())` from their own kernel with zero cross-host coordination, and can silently corrupt the file. There is no way to detect this from inside `try_lock`'s result, so `nolock` NFSv3 exports are an unsupported deployment for multi-writer use, the same way running mixed Minigraf versions against one file is unsupported (see the kernel-locking CHANGELOG entry) — avoid `nolock` exports rather than relying on this check to catch them.

### STG-045 File shorter than one page and not a new database

**Error text**: `File is {} bytes, shorter than one page, and is not a new Minigraf database`

**Cause**: The file is longer than zero bytes but shorter than one 4 KiB page, and its bytes are not a torn first write of a new database. Creating a database writes its header page in one go, so a crash during that write can leave only a prefix of that page (or zeros); such a file is opened as a new, empty database. Any other short file is something else: a different file at the path, a database cut short by an interrupted copy, or a damaged file. Earlier versions opened it as a new database and overwrote it. The file is not modified. (Codes STG-028 to STG-044 belong to v3.0.0; this code has the same number and meaning there.)

**Resolution**:
- Check that the path points at the intended `.graph` file.
- Copy the database again from its source, or restore it from backup.
- If the file is not needed, delete it and open the path again to create a new database.

**Scenario**: A copy of a `.graph` file stops after its first kilobyte, and the copy is opened.

---

8000 ## WAL — Write-Ahead Log Errors
Expand Down
6 changes: 4 additions & 2 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@

## v2.0.4 Coverage Summary

**Verified**: 2026-10-08 with `cargo test`
**Verified**: 2026-10-10 with `cargo test`

**Result**: 1,215 passing tests and 8 ignored tests (1,223 total), including the golden-file corpus (#391)
**Result**: 1,216 passing tests and 8 ignored tests (1,224 total), including the golden-file corpus (#391)

Since v2.0.4, `tests/edge_cases_test.rs` checks that a file of 1 to 4,095 bytes is `STG-045`, unchanged and with no WAL, when it is a database cut below one page or foreign bytes, and opens as a new database that takes a write and reads it back after reopen when it is a prefix of the initial header or all zeros (#506).

`tests/property_test.rs` writes random histories (batched transacts, retracts, write transactions, checkpoints) in memory and to a file, and compares random queries (one to three joined patterns, `not` or `not-join`, a comparison predicate, `:as-of`, `:valid-at` or `:any-valid-time`) with a reference evaluator, on the open file and after a reopen; a query error fails the test (#386). The histories leave out the v2.x known issues #371, #435 and #477.

Expand Down
9 changes: 9 additions & 0 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ pub(crate) enum ErrorCode {
Stg025,
Stg026,
Stg027,
Stg045,
Wal001,
Wal002,
Wal003,
Expand Down Expand Up @@ -937,6 +938,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[
"Failed to lock database at {}: {}. This filesystem does not support file locking (common on NFSv3 without lockd, and on some FUSE mounts). Set `allow_unlocked` in `OpenOptions` to open anyway — that accepts the risk that concurrent writers corrupt the file.",
ErrorCategory::Storage,
),
(
ErrorCode::Stg045,
"STG-045",
"File is {} bytes, shorter than one page, and is not a new Minigraf database",
ErrorCategory::Storage,
),
(
ErrorCode::Wal001,
"WAL-001",
Expand Down Expand Up @@ -1751,6 +1758,7 @@ mod tests {
| ErrorCode::Stg025
| ErrorCode::Stg026
| ErrorCode::Stg027
| ErrorCode::Stg045
| ErrorCode::Wal001
| ErrorCode::Wal002
| ErrorCode::Wal003
Expand Down Expand Up @@ -1942,6 +1950,7 @@ mod tests {
ErrorCode::Stg025,
ErrorCode::Stg026,
ErrorCode::Stg027,
ErrorCode::Stg045,
ErrorCode::Wal001,
ErrorCode::Wal002,
ErrorCode::Wal003,
Expand Down
22 changes: 22 additions & 0 deletions src/storage/backend/file.rs
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,17 @@ impl FileBackend {
}
}
} else {
// A file shorter than one page is new only if it holds a torn
// write of the initial header below; anything else is not ours
// to overwrite (#506).
if file_len > 0 {
let mut tail = Vec::new();
file.seek(SeekFrom::Start(0))?;
(&mut file).take(PAGE_SIZE as u64).read_to_end(&mut tail)?;
if !is_initial_header_prefix(&tail) {
bail_coded!(ErrorCode::Stg045, tail.len());
}
}
// New file or empty file: write initial header, then make the
// file's directory entry durable. Without the directory sync a
// power loss can lose the whole file despite the header fsync
Expand Down Expand Up @@ -343,6 +354,17 @@ impl FileBackend {
}
}

/// True if `bytes` are a prefix of the initial header `open` writes to a new
/// file, then zeros: what a torn write of that header can leave.
fn is_initial_header_prefix(bytes: &[u8]) -> bool {
let expected = FileHeader::new().to_bytes();
let written = bytes
.iter()
.rposition(|&b| b != 0)
.map_or(0, |i| i.saturating_add(1));
bytes.get(..written) == expected.get(..written)
}

impl StorageBackend for FileBackend {
fn write_page(&mut self, page_id: u64, data: &[u8]) -> Result<()> {
if data.len() != PAGE_SIZE {
Expand Down
56 changes: 56 additions & 0 deletions tests/edge_cases_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
//! - Oversized fact accepted in-memory (no page size constraint)
//! - Stale WAL after checkpoint is replayed idempotently (no duplicate facts)
//! - More than 65,535 uncheckpointed facts query correctly (#445)
//! - A file shorter than one page is refused unless a torn first write (#506)
#![cfg(not(target_arch = "wasm32"))]

use minigraf::{Minigraf, OpenOptions, QueryResult};
Expand Down Expand Up @@ -203,3 +204,58 @@ fn test_more_than_u16_pending_facts_file_backed() {
db.checkpoint().unwrap();
assert_all_facts_visible(&db, "after checkpoint");
}

// ── File shorter than one page (#506) ─────────────────────────────────────────

fn open_code(path: &std::path::Path) -> String {
match Minigraf::open(path) {
Ok(_) => "opened".to_string(),
Err(e) => e.code().to_string(),
}
}

/// A file shorter than one page opens as a new database only when it is a
/// torn first write: zeros, or a prefix of the initial header. Any other short
/// file (a database cut to under a page, or foreign bytes) is refused with
/// STG-045 and left as it was; it used to be overwritten (#506).
#[test]
fn test_sub_page_file_refused_unless_torn_first_write() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("short.graph");
{
let db = Minigraf::open(&path).unwrap();
db.execute("(transact [[:a :b 1] [:a :c 2]])").unwrap();
db.checkpoint().unwrap();
}
let saved = std::fs::read(&path).unwrap();
std::fs::remove_file(&path).unwrap();
let foreign = vec![0x5Au8; 100];
for bytes in [&saved[..2222], &saved[..20], &foreign[..]] {
std::fs::write(&path, bytes).unwrap();
assert_eq!(open_code(&path), "STG-045");
assert_eq!(std::fs::read(&path).unwrap(), bytes, "file unchanged");
assert!(
!dir.path().join("short.graph.wal").exists(),
"no WAL created"
);
}

// A torn first write: the initial header cut short, or zeros.
let fresh = dir.path().join("fresh.graph");
drop(Minigraf::open(&fresh).unwrap());
let initial = std::fs::read(&fresh).unwrap();
for bytes in [&initial[..4], &initial[..10], &[0u8; 300][..]] {
std::fs::write(&path, bytes).unwrap();
let db = Minigraf::open(&path).unwrap();
db.execute("(transact [[:a :b 1]])").unwrap();
db.checkpoint().unwrap();
drop(db);
let db = Minigraf::open(&path).unwrap();
match db.execute("(query [:find ?v :where [:a :b ?v]])").unwrap() {
QueryResult::QueryResults { results, .. } => assert_eq!(results.len(), 1),
_ => panic!("expected query results"),
}
drop(db);
std::fs::remove_file(&path).unwrap();
}
}
Loading
0