Repository navigation
feat(storage): v8 hardening, measurements and docs (PR 5 of 5) - #452
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
…full scan selective_fact_fetch swallowed lookup errors and fell back to a full scan, which could hide a damaged index page behind another index. It now returns the error. Corruption tests damage an index leaf, an internal node, a DICT leaf, a value page and a free-list page: every read or checkpoint that reaches the page fails with STG-029, and the file still opens at its last generation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
144k facts runs in every test run; 1.15M is ignored and runs in the nightly smoke workflow. Both measure 142 bytes per fact, history included, against the spec's limit of 200. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
Committed reads seed their dictionary memo with the entity UUID and attribute name the caller already passed, and each committed reader keeps a bounded cache of ident and transaction-timestamp lookups shared across reads; neither changes after commit. Concurrent point queries at 100k facts are 19-31 % faster than after PR 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions ca
31DA
nnot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 5 of 5 of the v8 storage format: hardening, measurements and docs (spec §10–§12; plan section "PR 5"). The file format does not change. With this PR, every delivery PR in the spec is done.
Changes
DatalogExecutor::selective_fact_fetchturned an indexed-lookup error into a fallback full scan (.ok()?), which could hide a damaged AEVT or EAVT page behind the other index. It now returnsResult<Option<_>>and propagates. A derived-facts read in the executor uses?instead ofunwrap_or_default.checkpoint/after_1_fact/1mand a newcheckpoint/after_100k_facts/{10k,100k}.Measurements
checkpoint/after_1_factcheckpoint/after_100k_factsconcurrent_btree_scan), against PR 4: −31 / −25 / −19 % at 100k with 2/4/8 threads, and −10 to −12 % at 10k. At 100k with 2 threads this is back to PR 2's level (25.5 µs). Single-threaded point lookups are within noise (+2–3 %).Tests
persistent_facts.rs): damage an EAVT leaf, an EAVT internal node, a DICT leaf, a value page, an AEVT leaf and a free-list page in turn.index_corruption_test.rs): queries throughMinigrafon a file with damaged leaves fail with STG-029.size_per_fact_in_433_shape(144k facts, every run) andsize_per_fact_in_433_shape_1m(ignored, nightly).cargo clippy --all-features -- -D warningsis clean.Docs
Architecture.mdgets a full "File format v8" section (pushed to the wiki repo separately).Refs #374 #434 #388 #433
🤖 Generated with Claude Code
https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn