Repository navigation
Add monthly full dumps of the search index to B2 - #537
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
daoudclarke
force-pushed
the
claude/epic-johnson-ppm1oz
branch
from
October 31, 2026 09:01
6722bc6 to
a49d0ff
Compare
The index is the only copy of seed crawls, curation and Staan copies, so it cannot be rebuilt from the crawl batches. backup_index streams the whole file, zstd-compressed and SSE-C encrypted, to a bucket of its own, reading each 4 MB chunk under a shared lock on its byte range so no page is caught half-written. A manifest with the size and SHA-256 is written once the upload finishes. restore_index downloads in ranges, checks the result against the manifest and only then renames it into place. A Dokku cron entry runs the backup monthly; docs/index-backup.md covers setup and restore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jZWrRVSzD1uuafxpb8fW2
…mory From review of the backup PR: - botocore retries a request but not an error reading its body, so a dropped connection killed the whole restore. Each range is now retried with backoff, and a short body counts as a failure. - A failed restore removes its .partial file instead of leaving up to 420 GB on the live volume and blocking the next restore; a killed one's is written over. - Chunks are 64 pages, not 1024, so writers wait on a shared lock for one quick pread. The cron runs at best-effort priority 7 rather than idle, which could leave a lock held while search I/O kept the disk busy. - s3transfer buffered ten 64 MB parts of the non-seekable stream; now five. - Docs: retention by a lifecycle rule on the prefix, since each backup is a new file; cancelling unfinished uploads; checking SSE-C against real B2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jZWrRVSzD1uuafxpb8fW2
Brings the backup module and both management commands to full coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4oCL1aRJtAnHhPyPct
The release check counts any pyproject.toml or uv.lock change as a crawler change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4oCL1aRJtAnHhPyPct
The index is going to be released under an open licence from the same bucket, and /page already serves every document, so encryption protects nothing and losing the key would lose every copy. "Dump" fits an object that is both the backup and the public release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ofi2yAG1Bpw5cakYG
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ofi2yAG1Bpw5cakYG
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ofi2yAG1Bpw5cakYG
daoudclarke
force-pushed
the
claude/epic-johnson-ppm1oz
branch
from
October 31, 2026 11:20
a49d0ff to
98c4374
Compare
daoudclarke
pushed a commit
that referenced
this pull request
Oct 31, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ofi2yAG1Bpw5cakYG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
manage.py dump_index(mwmbl/indexer/dump.py):TinyIndex.page), so no page in the dump is caught half-written. A half-written page would never decode again.<object>.json) with the size and SHA-256 only after the upload finishes, so an object without a manifest is an upload that didn't finish.MWMBL_DUMP_*environment variables when it runs, so the app still starts without them. It uses its own bucket and key, separate from the crawl bucket.manage.py restore_index <key> <output>:app.jsoncron: runs the dump at 03:17 UTC on the 1st of each month, at low I/O priority.docs/index-dumps.md: setup and restore steps.motoadded as a dev dependency for the tests.Why
A dump does two jobs. It is the index's backup: the index is the only copy of seed crawls, curation and Staan copies, so rebuilding it from the crawl batches would recover only part of it. It is also the public release of the index under an open licence. That is why it is called a dump rather than a backup 10000 , and why it isn't encrypted:
/page/{n}already serves the same data one page at a time. Hourly deltas of changed pages, tracked in Redis, would follow in a later PR.How it was verified
make checkpasses.test/test_index_dump.pypasses (13 tests).test/test_index_dump.py, against moto's S3:Before merging
docs/index-dumps.md: create the public bucket and a key without delete rights, then set the environment variables./app/storagemount andDJANGO_SETTINGS_MODULE.Generated by Claude Code