8000 feat(backups): restore, download, any schedule and copies to an rclone connection from the web app by adeelahmad · Pull Request #263 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 37 additions & 12 deletions docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,15 +131,40 @@ single statements and for `run()` transactions (which roll back as a whole, so r

## Backups

`lens backup` writes one into `<data_dir>/backups`, as does Back up now in the web app ([Updates and backups](updates.md));
Lens also takes one before upgrading the database or installing a release:

* With a server, `manual-<time>.surql.gz`: SurrealDB's export of the database, gzipped. Restore it into an empty
database with `gunzip -k` and `surreal import --conn http://host:8000 --user root --pass … --ns archive --db main
<file>.surql`. Taking one by hand: `surreal export --conn http://host:8000 --user root --pass … --ns archive --db main
backup.surql`.
* Embedded, `manual-<time>.surrealkv`: a copy of the database folder, taken with it closed. Restore it by stopping Lens
and putting the copy in place of `<data_dir>/surrealdb`.

Also back up `data_dir` (reports, frames, caches, and `secret.key` unless `ARCHIVE_SECRET_KEY` is set; without that
key, stored credentials can't be decrypted).
**Settings › Backups** in the web app takes a backup now, lists them, downloads one, restores one and takes them on a
schedule (the API: `/api/v1/admin/backups`); `lens backup` takes one from the command line, and an upgrade takes one
first ([Updates and backups](updates.md)). Each is one entry in `<data_dir>/backups`, named
`<kind>-<time>`:

| Kind | Taken | Kept |
| --- | --- | --- |
| `manual` | from Settings › Backups, `POST /api/v1/admin/backups` or `lens backup` | until you delete it |
| `scheduled` | on `backups.schedule` (any cron expression in `backups.timezone`, typed or picked in one click; none by default) | the newest `backups.keep` (7), here and on the connection |
| `before-upgrade` | before the database is upgraded, and before a release is installed | the newest three taken before database upgrades |
| `before-restore` | what the database held just before a restore replaced it | the newest three |

* With a server, `<kind>-<time>.surql.gz`: SurrealDB's export of the database, gzipped.
* Embedded, `<kind>-<time>.surrealkv`: a copy of the database folder, taken with it closed (downloaded as a
`.tar.gz`).

**Restoring** replaces the whole database with the backup's. What it held is backed up first (`before-restore`) and
put back if the restore fails; a backup from an older Lens is brought up to date (schema and upgrade steps) afterwards.
People signed in since the backup was taken sign in again. Only an admin signed in with a passkey can download or
restore a backup from Settings (a download there is a link that works for two minutes); an admin API token can take,
list and delete them. `backups.restore(db, cfg, name)` in `app/domain/backups.py` is the one restore everything uses.

**A copy somewhere else.** Choose a storage connection under *Also copy each backup to* (S3 or S3-compatible,
Google Drive, Dropbox, OneDrive, SFTP, SMB, WebDAV or a folder; added under Sources, through rclone), and every new
backup, by hand or scheduled, is copied there under `backups.folder` as `<name>.lensbk`, with the scheduled ones
kept to `backups.keep` there as well. A copy is encrypted (AES-256-GCM) with a key derived from `secret.key` (or
`ARCHIVE_SECRET_KEY`), not with a key kept in the database, so it can be opened after the database is lost. Keep that
key away from the connection. *Copy there* sends one again, and *Bring back* fetches a copy that only the connection
has, so you can then restore or download it. If copying fails, the backup here is still kept, and Settings shows why.

By hand, with a server: `gunzip -k` the file, then `surreal import --conn http://host:8000 --user root --pass … --ns
archive --db main <file>.surql` into an empty database. Embedded: stop Lens and put the copy in place of
`<data_dir>/surrealdb`.

Backups hold the database only. Also back up `data_dir` (uploads, reports, frames, caches), and keep `secret.key`
(unless `ARCHIVE_SECRET_KEY` is set) apart from the backups: without it, stored credentials and server-held file keys
can't be decrypted, and with it a backup opens them ([encryption](encryption.md)).
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
activity,
admin,
auth,
backups,
batches,
budgets,
chats,
Expand Down Expand Up @@ -72,6 +73,7 @@
for module in (
users,
admin,
backups, # before updates: its /admin/backups/remote would otherwise be read as a backup's name
updates,
activity,
budgets,
Expand Down
166 changes: 166 additions & 0 deletions fastapi_backend/app/api/v1/routes/backups.py
10000
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
"""Restoring a backup, downloading one by link, and what the schedule and a restore are doing (app/domain/backups.py,
docs/database.md#backups; admins only). Listing, taking and deleting backups are in routes/updates.py.

Restoring (which replaces the database) and download links (a backup holds everything in it) need an admin signed in
with a passkey. A link works for LINK_SECONDS, so the browser saves the file itself however big it is; an embedded
database's folder comes as a .tar.gz.
"""

from __future__ import annotations

import tarfile
from collections.abc import Iterator

from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import FileResponse, StreamingResponse

from app.api.deps import AdminReader, AdminWriter, Cfg, Db, domain_errors
from app.core.security import sign_path, verify_path
from app.domain import auth, backups
from app.schemas.backups import BackupLink, CopyWork, RestoreStart
from app.schemas.common import Ok
from app.schemas.updates import Backup, RemoteBackups

router = APIRouter(prefix="/admin/backups", tags=["updates"])
LINK_SECONDS = 120


@router.get("/remote")
def remote_backups(user: AdminReader, db: Db, cfg: Cfg) -> RemoteBackups:
"""The copies on backups.connection, how copying the last backup there went, and a copy running now."""
where, st = backups.remote(cfg), backups.remote_state(db)
out = RemoteBackups(
connection=where and where["connection"],
folder=where and where["folder"],
last_at=st.get("at"),
last_name=st.get("name"),
last_error=st.get("error"),
copying=CopyWork(**backups.copying),
)
if where:
try:
out.backups = [Backup(**b) for b in backups.remote_listing(db, cfg)]
except (RuntimeError, ValueError, OSError) as e:
out.error = str(e)[:300] or type(e).__name__
return out


def _copy(db, cfg, what, name):
if not backups.remote(cfg):
raise HTTPException(400, "choose a storage connection for backups first")
try:
backups.start_copy(db, cfg, what, name)
except backups.Busy as e:
raise HTTPException(409, str(e)) from None


@router.post("/{name}/send", status_code=202)
def send_backup(name: str, user: AdminWriter, db: Db, cfg: Cfg) -> Ok:
"""Copy this backup to backups.connection now, in the background."""
with domain_errors():
backups.path(db, name)
_copy(db, cfg, "send", name)
auth.audit(db, user.as_audit(), "backup.send", None, {"name": name})
return Ok()


@router.post("/{name}/fetch", status_code=202)
def fetch_backup(name: str, user: AdminWriter, db: Db, cfg: Cfg) -> Ok:
"""Bring the copy of this backup back from backups.connection, in the background, to restore or download it."""
if not backups.NAME.match(name):
raise HTTPException(400, "no such backup")
_copy(db, cfg, "fetch", name)
auth.audit(db, user.as_audit(), "backup.fetch", None, {"name": name})
return Ok()


@router.delete("/{name}/remote")
def delete_remote_backup(name: str, user: AdminWriter, db: Db, cfg: Cfg) -> Ok:
"""Delete the copy of this backup on backups.connection."""
try:
backups.remote_delete(db, cfg, name)
except (ValueError, RuntimeError) as e:
raise HTTPException(400, str(e)) from None
auth.audit(db, user.as_audit(), "backup.delete_remote", None, {"name": name})
return Ok()


def _signed_in(user) -> None:
if user.via != "access":
raise HTTPException(403, "sign in with a passkey for this; API tokens and apps can't")


@router.post("/{name}/restore", status_code=202)
def restore_backup(name: str, body: RestoreStart, user: AdminWriter, request: Request, db: Db) -> Ok:
"""Replace the whole database with this backup, in the background (GET /admin/backups follows it). What it held
is backed up first (before-restore), and put back if the restore fails. Sign-ins made since the backup end."""
_signed_in(user)
if body.confirm != name:
raise HTTPException(400, "confirm with the backup's name")
with domain_errors():
backups.usable(db, name)
if backups.in_progress(db):
raise HTTPException(409, "a backup is being written; wait for it to finish")
cfg = request.app.state.archive.current()
who = user.as_audit()
auth.audit(db, who, "backup.restore", None, {"name": name})
try:
# the audit log was replaced too: the restore is recorded again in the restored one
backups.start_restore(
db, cfg, name, lambda out: auth.audit(db, who, "backup.restored", None, {"name": name, "before": out and out["name"]})
)
except backups.Busy as e:
raise HTTPException(409, str(e)) from None
return Ok()


@router.post("/{name}/link")
def backup_link(name: str, user: AdminReader, db: Db) -> BackupLink:
"""A link that downloads the backup for the next couple of minutes, for the browser to open."""
_signed_in(user)
with domain_errors():
backups.path(db, name)
auth.audit(db, user.as_audit(), "backup.download", None, {"name": name})
return BackupLink(url=sign_path(f"/api/v1/admin/backups/{name}/file", ttl=LINK_SECONDS), expires_in=LINK_SECONDS)


def _tar(folder) -> Iterator[bytes]:
"""A folder as a .tar.gz, written as it is read."""

class Sink:
def __init__(self):
self.parts: list[bytes] = []

def write(self, b):
self.parts.append(bytes(b))
return len(b)

def take(self):
out, self.parts = b"".join(self.parts), []
return out

sink = Sink()
with tarfile.open(fileobj=sink, mode="w|gz") as tar: # type: ignore[call-overload]
for p in sorted(folder.rglob("*")):
tar.add(p, arcname=f"{folder.name}/{p.relative_to(folder)}", recursive=False)
if chunk := sink.take():
yield chunk
yield sink.take()


@router.get(
"/{name}/file",
response_class=StreamingResponse,
responses={200: {"content": {"application/gzip": {}}, "description": "the backup (an embedded one as a .tar.gz)"}},
)
def backup_file(name: str, db: Db, exp: str | None = None, sig: str | None = None):
"""The backup itself, through a link from POST /admin/backups/{name}/link."""
if not verify_path(f"/api/v1/admin/backups/{name}/file", exp, sig):
raise HTTPException(403, "this download link has expired; download the backup again from Settings")
with domain_errors():
p = backups.path(db, name)
if p.is_dir():
return StreamingResponse(
_tar(p), media_type="application/gzip", headers={"Content-Disposition": f'attachment; filename="{name}.tar.gz"'}
)
return FileResponse(p, media_type="application/gzip", filename=name)
35 changes: 30 additions & 5 deletions fastapi_backend/app/api/v1/routes/updates.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,14 @@

from __future__ import annotations

import contextlib

from fastapi import APIRouter, HTTPException
from fastapi.responses import FileResponse

from app.api.deps import AdminReader, AdminWriter, Cfg, Db, domain_errors
from app.domain import auth, backups, updates
from app.schemas.backups import BackupSchedule, RestoreWork
from app.schemas.common import Ok
from app.schemas.updates import Backup, Backups, UpdateStatus, UpgradeRequest

Expand Down Expand Up @@ -44,18 +47,40 @@ def start_upgrade(user: AdminWriter, db: Db, cfg: Cfg, body: UpgradeRequest | No


@router.get("/admin/backups")
def list_backups(user: AdminReader, db: Db) -> Backups:
"""The database backups in <data_dir>/backups, newest first."""
return Backups(folder=str(backups.folder(db)), backups=[Backup(**b) for b in backups.listing(db)])
def list_backups(user: AdminReader, db: Db, cfg: Cfg) -> Backups:
"""The database backups in <data_dir>/backups, newest first; those being written; the restore running here (or
the last one); and the backup schedule."""
b, st = cfg["backups"], backups.schedule_state(db)
scheme = db.url.split(":", 1)[0]
return Backups(
folder=str(backups.folder(db)),
backups=[Backup(**x) for x in backups.listing(db)],
database="memory" if scheme in ("mem", "memory") else "embedded" if db.embedded else "server",
in_progress=backups.in_progress(db),
restore=RestoreWork(**backups.work),
schedule=BackupSchedule(
schedule=b.get("schedule"),
timezone=b.get("timezone") or "UTC",
keep=b.get("keep") or 7,
next_at=st.get("at") if st.get("plan") == backups.plan(cfg) else backups.next_at(cfg),
last_at=st.get("last_at"),
last_name=st.get("last_name"),
last_error=st.get("last_error"),
),
)


@router.post("/admin/backups", status_code=201)
def take_backup(user: AdminWriter, db: Db) -> Backup:
"""Back the database up now (it can take a while on a big archive)."""
def take_backup(user: AdminWriter, db: Db, cfg: Cfg) -> Backup:
"""Back the database up now (it can take a while on a big archive). With backups.connection set, it is then
copied there in the background (GET /admin/backups/remote follows it)."""
got = backups.take(db)
if got is None:
raise HTTPException(400, "an in-memory database can't be backed up")
auth.audit(db, user.as_audit(), "backup.create", None, {"name": got["name"]})
if backups.remote(cfg):
with contextlib.suppress(backups.Busy): # another copy is running: this one waits for the next backup
backups.start_copy(db, cfg, "send", got["name"])
return Backup(**got)


Expand Down
Loading
Loading
0