Repository navigation
fix(wal): replay WAL entries of more than 1,000,000 facts (#686) - #698
Merged
Merged
Conversation
Replay took an entry of more than 1,000,000 facts as damage, and the writer's tail cut at open removed it and every later entry, so those committed transactions were lost after a crash or close without a checkpoint. A fact count is now damage only when the bytes after it cannot hold that many facts (each has at least its 4-byte length), checked before the count is converted to usize. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
Take the entry's offset from valid_end rather than stream_position, and fall back to no bound when the file length is unavailable (a corrupt count still ends at EOF), so the coverage gate's branch floor holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
The byte bound now stops the truncated-entry tests before the fact loop; cover the loop's two truncation exits directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
# Conflicts: # CHANGELOG.md
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
For #686 (data-integrity; the v2.x backport is a separate PR, and the issue stays open until a published release has the fix).
WAL replay treated an entry of more than 1,000,000 facts as damage and stopped there. The writer never had that limit.
scan_entriesalso drives the writer's tail cut at open (#513), so the next open truncated the WAL at that entry and lost it and every later entry. On v3 no transaction writes such an entry since #550, but a WAL left by v2.x can hold one.src/wal.rs: the count cap is gone. A fact count is now damage only when the bytes left in the file cannot hold that many facts (every fact has at least its 4-byte length prefix). The check runs on the rawu64before theusizeconversion, so on 32-bit an impossible count ends replay where it used to fail withWAL-005(Codex review). A corrupt count that passes the bound still ends at EOF, a bad fact or the CRC, as before. Memory use stays bounded by the file's size.MAX_FACTS_PER_ENTRYis removed.MAX_TX_CHUNK_FACTS(1,000,000) keeps thetransaction_chunk_factsclamp as a memory bound; its doc comment and thelarge_tx.rsassertion no longer tie it to the WAL.Tests
test_wal_entry_of_more_than_a_million_facts_replays: an entry of 1,000,001 facts plus a later one replay; reopening the writer cuts off nothing, and an append lands after them. Before the fix, reopening cut the 41 MB WAL back to its 32-byte header.test_wal_entry_with_more_facts_than_bytes_left_ends_replay(replaces the 1,000,001-count test): a count ofleft/4 + 1, and one ofu64::MAX, end replay at that entry, and reopening the writer cuts the WAL off there.cargo clippy --all-features -- -D warningsis clean.Docs
CHANGELOG (Fixed),
docs/TEST_COVERAGE.md, CLAUDE.md test count. The docs site needs no change.🤖 Generated with Claude Code
https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW