Repository navigation
Add an admin page showing the status of index backups - #538
Closed
daoudclarke wants to merge 5 commits into
Closed
daoudclarke wants to merge 5 commits into
daoudclarke wants to merge 5 commits into
Conversation
The index is the only copy of seed crawls, curation and Staan copies, so it cannot be rebuilt from the crawl batches. backup_index streams the whole file, zstd-compressed and SSE-C encrypted, to a bucket of its own, reading each 4 MB chunk under a shared lock on its byte range so no page is caught half-written. A manifest with the size and SHA-256 is written once the upload finishes. restore_index downloads in ranges, checks the result against the manifest and only then renames it into place. A Dokku cron entry runs the backup monthly; docs/index-backup.md covers setup and restore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jZWrRVSzD1uuafxpb8fW2
Each backup_index run now records itself as an IndexBackup row: RUNNING with its progress updated every gigabyte read, then SUCCEEDED with the SHA-256 or FAILED with the error. /admin/index-backups/ reads only those rows, so it needs no B2 call. It flags no success in 35 days as overdue, and a running backup with no progress for an hour as stalled - which is also how a run the container killed before it could record a failure shows up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jZWrRVSzD1uuafxpb8fW2
…mory From review of the backup PR: - botocore retries a request but not an error reading its body, so a dropped connection killed the whole restore. Each range is now retried with backoff, and a short body counts as a failure. - A failed restore removes its .partial file instead of leaving up to 420 GB on the live volume and blocking the next restore; a killed one's is written over. - Chunks are 64 pages, not 1024, so writers wait on a shared lock for one quick pread. The cron runs at best-effort priority 7 rather than idle, which could leave a lock held while search I/O kept the disk busy. - s3transfer buffered ten 64 MB parts of the non-seekable stream; now five. - Docs: retention by a lifecycle rule on the prefix, since each backup is a new file; cancelling unfinished uploads; checking SSE-C against real B2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jZWrRVSzD1uuafxpb8fW2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jZWrRVSzD1uuafxpb8fW2
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
Author
|
Superceded by #571 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #537. Merge that one first, then retarget this PR to
main.What changed
IndexBackupmodel (migration0050).backup_indexwrites one row per run:RUNNING, and the run updatesbytes_readevery GiB it reads.SUCCEEDED, with the SHA-256. On an exception it becomesFAILED, with the error, and the exception is re-raised./admin/index-backups/, linked from the admin home page's Status table. It shows:RUNNINGrow with no progress for an hour is shown as stalled. That is also how a run shows up when the container was killed before it could record a failure.Why
A monthly cron job that fails quietly looks the same as one that works. This page shows when the last success was and what's running right now.
How it was verified
make checkpasses.test/test_admin_index_backups.py: staff-only access; overdue with no backups and with only an old success; a recent success; a running row versus a stalled one.test/test_index_backup.py: a run is recorded as succeeded with the manifest's SHA-256; a failed run records its error; progress is reported once per interval, not once per chunk.Generated by Claude Code