adeelahmad · Oct 4, 2026 · Oct 4, 2026 · Oct 4, 2026 · Oct 4, 2026 · Oct 4, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,10 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
 
 ## Unreleased
 
+- **Safer upgrades of existing installs.** Data upgrades are now named steps that run once per database, under a lock
+  so only one process runs them while the others wait, with a record of when each ran and why one failed.
+  `lens migrations` lists them. A database with data is backed up into `<data_dir>/backups` before it is upgraded
+  (the newest three are kept), and `lens backup` takes one on demand. See docs/database.md, Upgrades.
 - **Routine choices on your own Mac with Laya.** `decisions.engine: laya` takes routine choices with a Laya typed
   decision model on MLX (English, multilingual or typed-decisions), fetched by Lens itself, free and offline. Lens in
   Docker on a Mac asks `lens decide-server` on the Mac. Elsewhere it says Laya isn't available and the LLM decides.

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -16,6 +16,8 @@ Thanks for helping. The full guide, with the backend and frontend conventions, i
 - Backend: `make test-backend` and `make lint-backend` (ruff, mypy). Frontend: `make test-frontend` and
   `make lint-frontend`.
 - Add tests with the change, and update `docs/` when behaviour or configuration changes.
+- Changing the shape of stored data? Existing installs have real data: add an upgrade step
+  (docs/database.md, Upgrades) rather than expecting a fresh database.
 - One concern per pull request. Keep generated files (`openapi.json`, the frontend client) in the same PR as the API
   change that produced them.
 

diff --git a/docs/database.md b/docs/database.md
@@ -33,7 +33,8 @@ engine and with SurrealDB 2.3 and 3.2 servers.
 ## Schema
 
 The schema is defined idempotently in `app/domain/store.py` (`SCHEMA`) and applied on every start
-(`DEFINE ... IF NOT EXISTS`), so there are no migration files. Tables are `SCHEMALESS` with indexes on the fields
+(`DEFINE ... IF NOT EXISTS`), so new tables, fields and indexes need no migration (see [Upgrades](#upgrades) for the
+rest). Tables are `SCHEMALESS` with indexes on the fields
 that are queried. Graph edges are real relations: `mentions` (segment → entity) and `same_as` (speaker → speaker).
 
 ```sql
@@ -48,8 +49,35 @@ cosine; servers only) defined when the first vector is stored, since its `DIMENS
 which model and dimension that is. A server searches it with `embedding <|k,ef|> $vec AND …filters`, which 3.2.4
 applies inside the index scan (a filtered search still returns k rows).
 
-When a change needs data rewritten (not just a new index), add an idempotent step to `store.connect()` guarded by a
-version stored in the `seq` table, so it runs once per database.
+## Upgrades
+
+Installs keep their data across updates (running `install.sh` again, a new image, `git pull`). New tables, fields and
+indexes arrive through `SCHEMA` on the next start. Anything else, such as rewriting stored values, moving data between
+tables, or changing or removing an index that already exists, is a named step in `app/domain/migrations.py`:
+
+* Every process that opens the database (the API, workers, `lens watch`, `lens` commands) runs the steps the database
+  hasn't had yet when it starts, in order. One process takes a lock (`migration_lock:run`) and runs them; the others
+  wait, so no code reads data in a shape it doesn't expect. A holder that dies loses the lock after five minutes.
+* Before a database that holds data (any recording or account) runs pending steps, the process holding the lock backs
+  it up into `<data_dir>/backups/before-upgrade-<time>` (see [Backups](#backups)); the newest three are kept. If the
+  backup fails, for example because the disk is full, the upgrade doesn't start and Lens says why; set
+  `LENS_UPGRADE_BACKUP=off` to upgrade without one.
+* Each step run is recorded in `migration:⟨name⟩` with when and how long it took. A step that fails records its error
+  and stops the start; once the cause is fixed, the next start runs it again.
+* `lens migrations` lists every step (done, pending, failed, or unknown when a newer Lens ran it) without running
+  anything, so it works while an upgrade is failing; `lens migrations --run` runs the pending ones.
+
+To add a step:
+
+1. Write a function `fn(db)` next to the code it serves. It must be safe to run twice, since a start that stops half
+   way runs it again; prefer `UPDATE ... WHERE <old shape>` over rewriting every row.
+2. Append `("short-name", module.fn)` to `steps()` in `migrations.py`, and the name to `SHIPPED` in
+   `tests/domain/test_migrations.py`. Never rename, reorder or remove a shipped step: databases remember steps by
+   name, so a renamed one runs again and a removed one never runs.
+3. Keep reads tolerant of the old shape until the step has run (it runs before requests are served, but a worker on
+   another machine may still be on the old version).
+4. Test it on data in the old shape, against a SurrealDB server too (`LENS_TEST_SURREAL_URL`);
+   `test_every_step_is_safe_to_repeat` checks that a second run changes nothing.
 
 ## Concurrency
 
@@ -83,6 +111,14 @@ single statements and for `run()` transactions (which roll back as a whole, so r
 
 ## Backups
 
-With a server: `surreal export --conn http://host:8000 --user root --pass … --ns archive --db main backup.surql`, and
-`surreal import` to restore. Also back up `data_dir` (reports, frames, caches, and `secret.key` unless
-`ARCHIVE_SECRET_KEY` is set; without that key, stored credentials can't be decrypted).
+`lens backup` writes one into `<data_dir>/backups` (Lens also takes one before upgrading the database):
+
+* With a server, `manual-<time>.surql.gz`: SurrealDB's export of the database, gzipped. Restore it into an empty
+  database with `gunzip -k` and `surreal import --conn http://host:8000 --user root --pass … --ns archive --db main
+  <file>.surql`. Taking one by hand: `surreal export --conn http://host:8000 --user root --pass … --ns archive --db main
+  backup.surql`.
+* Embedded, `manual-<time>.surrealkv`: a copy of the database folder, taken with it closed. Restore it by stopping Lens
+  and putting the copy in place of `<data_dir>/surrealdb`.
+
+Also back up `data_dir` (reports, frames, caches, and `secret.key` unless `ARCHIVE_SECRET_KEY` is set; without that
+key, stored credentials can't be decrypted).
diff --git a/docs/get-started.md b/docs/get-started.md
@@ -105,6 +105,8 @@ lens search "capsid" --ns podcasts
 lens search "money worries" --mode semantic   # by meaning (with an embedding model; see configuration.md)
 lens embed                                 # index what isn't yet searchable by meaning
 lens reindex                               # after changing search.stemming
+lens migrations                            # database upgrades: run, pending or failed
+lens backup                                # back the database up into <data_dir>/backups
 ```
 
 Docker on macOS can't use the Apple GPU. Keep the database and API in Docker and run a native worker against the same

diff --git a/fastapi_backend/app/cli.py b/fastapi_backend/app/cli.py
@@ -96,6 +96,9 @@ def _main_base(argv=None):
     p.add_argument("--host")
     p.add_argument("--port", type=int)
     sub.add_parser("status")
+    p = sub.add_parser("migrations", help="list the database upgrades: run, pending or failed (docs/database.md)")
+    p.add_argument("--run", action="store_true", help="run the pending ones now (Lens also runs them when it starts)")
+    sub.add_parser("backup", help="back the database up into <data_dir>/backups (docs/database.md, Backups)")
     sub.add_parser("reindex", help="rebuild the search index (after changing search.tokenizer)")
     p = sub.add_parser("encrypt", help="encrypt the files Lens keeps, and keep encrypting new ones (docs/encryption.md)")
     p.add_argument("--off", action="store_true", help="decrypt them again and stop encrypting new ones")
@@ -122,7 +125,7 @@ def _main_base(argv=None):
             os.environ["ARCHIVE_CONFIG"] = a.config
         uvicorn.run("app.main:app", host=a.host or cfg["server"]["host"], port=a.port or cfg["server"]["port"], proxy_headers=True)
         return
-    conn = store.connect(cfg)
+    conn = store.connect(cfg, upgrade=a.cmd != "migrations")  # so a failed upgrade can still be looked at
     try:
         if a.cmd in ("scan", "transcribe", "diarize", "analyze", "embed", "summarize", "run", "report"):
             which = ["scan", "transcribe", "diarize", "analyze", "embed", "summarize", "report"] if a.cmd == "run" else [a.cmd]
@@ -184,6 +187,25 @@ def _main_base(argv=None):
                 print(f"  {n:<16} {st:<12} {c:>5}  {ms / 3.6e6:6.1f} h")
             for r in conn.rows("SELECT title, error FROM recording WHERE error != NONE LIMIT 10"):
                 print(f"  error: {r['title']}: {r['error']}")
+        elif a.cmd == "migrations":
+            from .domain import migrations
+
+            if a.run:
+                ran = migrations.run(conn)
+                print(f"ran {len(ran)} upgrade(s): {', '.join(ran)}" if ran else "nothing to run")
+            for m in migrations.status(conn):
+                when = {"done": m["done_at"], "failed": m["failed_at"]}.get(m["state"]) or ""
+                took = f" ({m['took_ms']} ms)" if m["state"] == "done" and m.get("took_ms") else ""
+                print(f"  {m['state']:<8} {m['name']:<28} {when}{took}")
+                if m["state"] == "failed":
+                    print(f"           {m['error']}")
+                elif m["state"] == "unknown":
+                    print("           run by a newer Lens than this one")
+        elif a.cmd == "backup":
+            from .domain import migrations
+
+            dst = migrations.backup(conn)
+            print(f"backed up to {dst}" if dst else "nothing to back up: the database is in memory")
         elif a.cmd == "reindex":
             store.reindex(conn, cfg)
             print("search index rebuilt")