8000 feat(db): safe database upgrades with named steps, a lock, a backup first and lens migrations by adeelahmad · Pull Request #113 · adeelahmad/lense · GitHub
[go: up one dir, main page]
Skip to content

feat(db): safe database upgrades with named steps, a lock, a backup first and lens migrations - #113

Merged
adeelahmad merged 7 commits into
mainfrom
claude/db-migrations-y5img5
Oct 4, 2026
Merged

feat(db): safe database upgrades with named steps, a lock, a backup first and lens migrations#113
adeelahmad merged 7 commits into
mainfrom
claude/db-migrations-y5img5

Conversation

@adeelahmad
@adeelahmad adeelahmad commented Oct 4, 2026 •
edited
Loading
Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: data upgrades were a list in store.migrate(), tracked by one counter (seq:migrations). Every process that opened the database (API, workers, lens watch) ran pending steps at the same moment, nothing was backed up first, nothing recorded what ran or why it failed, and two branches each adding a step could leave one skipped on a database.

After: upgrades are named steps in app/domain/migrations.py. On start, one process takes a lease lock (migration_lock:run, renewed by a heartbeat, lapses after 5 minutes if the holder dies) and the others wait. If the database holds data (any recording or account), the holder backs it up into <data_dir>/backups/before-upgrade-<time> first (server: SurrealDB's /export, gzipped; embedded: a copy of the database folder taken with it closed; newest three kept). A failed backup stops the upgrade with a clear message unless LENS_UPGRADE_BACKUP=off. Then the steps run in order, each run or failure kept in migration:⟨name⟩.

New commands: lens migrations lists done, pending, failed and unknown (run by a newer Lens) steps without upgrading, so it works while an upgrade is failing; --run runs them. lens backup takes a backup on demand.

Existing installs: the two steps the old counter tracked (access-levels, collection-homes) are adopted from seq:migrations, so nothing runs again and no backup is taken on this update.

How: store.connect(cfg, upgrade=True) calls migrations.run(). New steps are appended to steps() and to SHIPPED in the test, which fails on any rename, reorder or removal. test_every_step_is_safe_to_repeat snapshots every table after one run and checks a second run changes nothing. Backup tests restore the backup (into a fresh server database, or by opening the copied folder) and check it holds the pre-upgrade data. The convention is in docs/database.md (Upgrades, Backups) and CONTRIBUTING.md.

Motivation and Context

Adeel asked how migrations work as we develop and to get them sorted. Users have real data on installs that update by running install.sh again.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Tests: full backend suite in memory (549 passed) and against a SurrealDB 3.2.4 server with surrealkv (549 passed); the one failure in both is the known LibreOffice conversion test. No API changes, so openapi.json is unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CMvFKqsrnJrxxvME87LRk7

@claude
feat(db): named database upgrades with a lock, a run log and lens mig…
a8db7ee 
…rations

Data upgrades were tracked by a position counter, ran in every starting process
at once, and left no record. Steps now have names (adopting the two the counter
tracked), one process runs them under a lease lock while the others wait, each
run or failure is kept in migration:<name>, and lens migrations lists them
without running anything.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMvFKqsrnJrxxvME87LRk7
@adeelahmad adeelahmad self-assigned this Oct 4, 2026
@claude
feat(db): back the database up before upgrading it, and lens backup
41c682a 
A database with data is exported (server) or copied with it closed (embedded)
into <data_dir>/backups before pending upgrade steps run; the newest three are
kept. A failed backup stops the upgrade unless LENS_UPGRADE_BACKUP=off.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMvFKqsrnJrxxvME87LRk7
@adeelahmad adeelahmad changed the title feat(db): named database upgrades with a lock, a run log and lens migrations feat(db): safe database upgrades with named steps, a lock, a backup first and lens migrations Oct 4, 2026
@claude
style(tests): sort imports in the Fedora test (pre-commit is red on m…
7495d39 
…ain)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMvFKqsrnJrxxvME87LRk7
@adeelahmad Claude
adeelahmad commented Oct 4, 2026 •
edited
Loading
Copy link
Copy Markdown
Owner Author

pre-commit failed on ruff (import sort, then ruff format) in tests/api/test_fedora.py and tests/fake_fedora.py, which this PR doesn't touch; main has the same unformatted code. I ported the fixes (sorted imports, then the formatter's output) into this PR so the check can go green; they're no-ops once main has them.

Generated by Claude Code

@adeelahmad Claude
adeelahmad commented Oct 4, 2026
Copy link
Copy Markdown
Owner Author

Next.js failed in its Tests step: 5 Jest tests hit the 5 s timeout (e.g. library-collections-ui.test.tsx). This PR changes no frontend code, and the same step fails on main's latest CI run (run 37179956663) on the same self-hosted runner, so it isn't this PR's. No fix exists yet to port. I'll re-run the failed job once the rest of this run finishes on the backlogged runner.

Generated by Claude Code

claude added 2 commits October 4, 2026 07:22
@claude
style(tests): format the Fedora tests (pre-commit is red on main)
75e8e79 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMvFKqsrnJrxxvME87LRk7
8000
@claude
ci: remove a leftover SurrealDB container before and after the server…
2d0b005 
… tests

The self-hosted runner keeps containers between jobs, so docker run --name
surreal failed on main as well.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMvFKqsrnJrxxvME87LRk7
@adeelahmad Claude
adeelahmad commented Oct 4, 2026
Copy link
Copy Markdown
Owner Author

The FastAPI (SurrealDB server) job failed at "Start SurrealDB" with Conflict. The container name "/surreal" is already in use. The self-hosted runner keeps containers between jobs, and main's last CI run fails the same way. I ported a fix into this PR: ci.yml now removes any leftover surreal container before starting one, and removes it again when the job ends.

Generated by Claude Code

@adeelahmad Claude
adeelahmad commented Oct 4, 2026 •
edited
Loading
Copy link
Copy Markdown
Owner Author

The 4 server-job failures (Chromium setsid, calendar redirect to localhost) are in code this PR doesn't touch.

Update: #112 fixed both on main. I merged main into this PR, which also takes #112's version of the SurrealDB container steps in ci.yml in place of mine.

Generated by Claude Code

@adeelahmad
adeelahmad merged commit 55d1d78 into main Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@adeelahmad adeelahmad

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@adeelahmad @claude
0