Repository navigation
test: fault-injection matrix through save/WAL/recovery; never retry a failed checkpoint (#390) - #515
Merged
Merged
Conversation
…try a failed checkpoint (#390) A test-only, thread-local fault plan (src/storage/fault.rs) is asked before every write and sync of FileBackend, the WAL (header, entries, torn-tail cut, removal) and sync_parent_dir. It fails the k-th write or sync with EIO, a short or torn write, a full disk that keeps failing later writes, a failed fsync, or a failed fsync that also loses the writes since the last good one (FileBackend keeps pre-images while a plan is armed; the WAL is cut back to its last synced length). src/fault_matrix.rs runs each fault at every call index of a transact, a WAL-creating transact, an automatic checkpoint, checkpoint(), the close-time checkpoint and rebuild_indexes(), then kills the handle and reopens: the facts are exactly the handle's, plus the failed transaction whole or not at all; verify is clean; a commit after recovery survives another kill. Two more tests fill the disk halfway through a checkpoint, and retry a checkpoint whose meta fsync failed, crashing at each write. Decision (written down in ERROR_REFERENCE and Minigraf::checkpoint): a checkpoint that failed after it may have written is never retried on the same handle. PersistentFactStorage sets `failed` before its first write and clears it on success; while set, save and rebuild_indexes return the new STG-046, writes are refused before they take a tx_count, and the close-time save is skipped. Without this, the retry test reopens to STG-029: the retried checkpoint overwrote pages the on-disk meta of the "failed" one referenced. A rebuild refused with STG-041 writes nothing and does not set it. The #513 test hook (tear_next_append) is replaced by the fault plan. Refs #390 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h
…n open file (#390) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h
adityamukho
force-pushed
the
test/390-fault-matrix
branch
from
October 10, 2026 09:55
180b850 to
d701821
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #390 (durability label: stays open until released). Builds on #514 (merged).
What
Fault plan (
src/storage/fault.rs, test builds only). A thread-local plan is consulted before every write and sync ofFileBackend, the WAL (header, entries, torn-tail cut, removal) andsync_parent_dir. It fails the k-th write or sync with:FileBackendkeeps pre-images while armed; the WAL is cut back to its last synced length)Matrix (
src/fault_matrix.rs). Each fault runs at every call index of: transact, a WAL-creating transact, auto-checkpoint,checkpoint(), the close-time checkpoint, andrebuild_indexes(). The database is committed over two checkpoints (multi-level trees, a free list, multi-valued/retracted/long values) with entries pending in the WAL. After each fault:STG-046WAL-007/STG-046verifyis cleanPlus: the disk fills halfway through a checkpoint; and a retried checkpoint after a failed meta fsync, crashing at each of its writes. Runs in about 17 s (debug).
Decision: a failed checkpoint is never retried on the same handle
PersistentFactStoragesetsfailedbefore a commit's first write and clears it on success. While it is set:save/rebuild_indexesreturn the newSTG-046transact/commitare refused before they take atx_countReopen recovers everything: it is all in the WAL. A rebuild refused with
STG-041writes nothing and keeps the handle usable. Documented inERROR_REFERENCE.md(STG-046), theMinigraf::checkpointrustdoc and CHANGELOG.Why it matters: with the refusal disabled,
no_commit_after_a_failed_meta_fsyncreopens toSTG-029. The meta page of the "failed" checkpoint reached the file, and the retry overwrote pages it references.Mutation checks
faults_during_transact*fail (a commit after recovery is lost).Docs: CHANGELOG, ERROR_REFERENCE, TEST_COVERAGE, CLAUDE.md module list, test counts (1,406). The docs-site note is in minigraf-docs#4.
🤖 Generated with Claude Code
https://claude.ai/code/session_012wXxZguv7PgKQjMMH4yJ5h