Repository navigation
SIGKILL during an active transact has a ~26% chance of permanently corrupting the entire database (header checksum, no recovery path) #308
Description
Summary
Killing a writer process (SIGKILL) while it is actively transacting has a substantial, repeatable chance of leaving the entire graph file permanently and totally corrupted — not just losing the in-flight write, but making the whole database unopenable forever afterward:
msg='Header checksum mismatch: possible file corruption. Database may be damaged.'
There is no recovery path via the public API. A .wal sidecar file exists (present in every corrupted case observed), but removing it entirely and retrying MiniGrafDb.open() does not help — the header checksum check fails identically with or without the WAL present, meaning either the WAL isn't used for header recovery at all, or whatever recovery logic exists never gets a chance to run because the checksum check aborts first.
This is a very realistic trigger: OOM-killer, a crashed/stopped container, a host reboot, or an operator running kill -9 on a stuck process (which is exactly what we did to a real, live temporal_reasoning MCP server process during unrelated troubleshooting, with no idea at the time that it carried this risk).
Reproduction (fully isolated, scratch files only, deterministic harness)
Driven directly against minigraf.MiniGrafDb (bypassing temporal_reasoning entirely, to isolate this to the core storage engine):
- Spawn a writer subprocess that opens a fresh
.graphfile and loops:transacta ~3000-byte fact,checkpoint(), repeat — no other process touches the file. - After a short randomized delay (20ms–200ms into the run), send
SIGKILLto the writer. - Immediately probe the file from a fresh process:
MiniGrafDb.open(path)+ one trivial query. - Repeat with a fresh graph file every round (so one corruption can't poison later rounds' results).
Result over 50 independent rounds:
| Outcome | Count |
|---|---|
| Clean recovery (open + query succeeded) | 37 |
Permanent corruption (Header checksum mismatch) |
13 (26%) |
| Other error | 0 |
Corrupted rounds and the file size at the moment of the kill (all at small scale — tens to hundreds of KB, so this is not a large-file-only phenomenon):
round 4: size_before_kill=53248 wal_size_before_kill=3189
round 5: size_before_kill=81920 wal_size_before_kill=3189
round 6: size_before_kill=114688 wal_size_before_kill=3189
round 7: size_before_kill=131072 wal_size_before_kill=3189
round 14: size_before_kill=40960 wal_size_before_kill=3189
round 15: size_before_kill=81920 wal_size_before_kill=3189
round 16: size_before_kill=98304 wal_size_before_kill=3189
round 19: size_before_kill=196608 wal_size_before_kill=3189
round 24: size_before_kill=45056 wal_size_before_kill=3189
round 25: size_before_kill=86016 wal_size_before_kill=3189
round 26: size_before_kill=102400 wal_size_before_kill=3189
round 44: size_before_kill=57344 wal_size_before_kill=3189
round 45: size_before_kill=77824 wal_size_before_kill=3189
Note the .wal file size is identical (3189 bytes) across every single case, both corrupted and clean — it never grows during the writer's tight loop. That's suspicious on its own: either the WAL isn't actually being used for the kind of protection you'd expect from a write-ahead log, or it's specifically not covering whatever the header update path is doing.
Evidence pointing at the root cause: non-atomic header update
Compared the raw header bytes (first 64 bytes) of a clean file vs. two independently-corrupted files:
clean (kill2_0, 4096 bytes, 1 page):
4d47 5246 0700 0000 0100 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
...
corrupted (kill2_4, 53248 bytes, 13 pages):
4d47 5246 0700 0000 0d00 0000 0000 0000
0400 0000 0000 0000 0200 0000 0000 0000
0300 0000 0000 0000 0400 0000 0000 0000
0a00 0000 0000 0000 0b00 0000 0000 0000
corrupted (kill2_14, 40960 bytes, 10 pages):
4d47 5246 0700 0000 0a00 0000 0000 0000
0200 0000 0000 0000 0100 0000 0000 0000
0200 0000 0000 0000 0300 0000 0000 0000
0600 0000 0000 0000 0700 0000 0000 0000
The magic (MGRF), version, and the visible little-endian counters all look internally plausible — nothing looks visibly torn/truncated in this window. That's consistent with the checksum itself (presumably stored elsewhere in the header, or computed over this region) being updated in a separate write from the page-count/metadata fields it covers — i.e. the header is likely updated via something like "write new counts, then separately (re)compute and write a checksum over the header" rather than as one atomic unit (e.g. write-new-page-then-atomic-rename, or a checksum-then-commit ordering with the counts written last). A kill landing between those two steps leaves a header whose declared checksum doesn't match its own content, with no shadow/backup copy to fall back on — even though a .wal exists for (presumably) data pages.
Impact
This is significantly more severe than #304 (the transient multi-writer page-bounds bug), which is at least recoverable — retrying eventually succeeds and no data is lost. Here, a single ungraceful termination during any write has a ~1-in-4 chance (in our sample; likely varies with exact timing relative to internal write phases) of destroying the entire database permanently, including everything ever successfully committed and checkpointed before that point — not just the in-flight transaction. Given how commonly processes get killed ungracefully in real deployments (container orchestration SIGKILL after a grace period, OOM killer, kill -9 during operator troubleshooting — see project-minigraf/temporal_reasoning#108 and #109 for a real incident this session where we killed live server processes multiple times), this seems likely to bite real users.
Suggested investigation focus
- Confirm whether the header (page 0) is updated via separate writes for its data fields vs. its checksum, and if so, make that update atomic — e.g. compute the checksum first and write the whole header page in one
pwrite, or write a new header to a fresh location and atomically rename/swap a pointer to it. - Investigate why the
.walfile's size never changes across any of these runs (constant 3189 bytes) — confirm whether it's actually participating in crash recovery for the header, or only for data pages (or not being used for recovery at all despite being written to disk). - Consider whether
MiniGrafDb.open()should attempt any repair/replay pass when it detects a header checksum mismatch, rather than failing hard with no path forward — at minimum, exposing a repair/recovery API (even a manual, explicit one) would let callers attempt to salvage a damaged file instead of losing it outright.
Environment
- Reproduced via direct
minigraf.MiniGrafDbPython FFI bindings (minigrafPyPI package, bundled withtemporal_reasoning), not throughtemporal_reasoning'smcp_server.pywrapper — this is a core storage-engine issue, independent of the MCP layer. - Page size 4096 bytes (confirmed via file-size arithmetic in all cases).
- Reproducible at very small scale (tens of KB) — not dependent on large files or long-running processes.
Activity
ci: run the nightly SIGKILL fuzz suite on all three platforms
fix: make on-disk header and WAL creation crash-safe under SIGKILL (#308