8000 feat(setup): first-run setup wizard answered by .env, and startup on a fresh database by adeelahmad · Pull Request #19 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **First-run setup wizard.** A fresh install now walks its first admin through setup in the web app: after the
admin account (still with the one-time setup code, so a stranger can't claim a public server) come the first
namespace, the model provider (with a connection test) and storage (the upload limit, and a folder to watch).
Every step and the whole wizard can be skipped; installs that already had accounts never see it
(docs/configuration.md#first-run-setup).
- Or answer it in `.env`: `LENS_ADMIN_EMAIL` / `LENS_ADMIN_PASSWORD` create the first admin at startup,
`LENS_NAMESPACE` the first namespace, `LENS_LLM_BASE_URL` / `LENS_LLM_MODEL` / `LENS_LLM_API_KEY` set the
model provider, and `LENS_SETUP_WIZARD=off` turns the wizard off. Environment values win over the app, and
the wizard and Settings show them locked; a key from the environment is never shown.
- **Fix: a fresh database with no namespaces in archive.yaml no longer stops the API from starting** with
"table 'seq' does not exist" (SurrealDB 3). The counters table is defined with the rest of the schema now, so
nothing reads it before it exists.

- **Fix: Chat answers no longer break off with "The answer stopped before it finished".** With some model servers an
answer ended mid-stream with nothing saved, so the question sat unanswered in the conversation. Now each one ends
in an answer, or an error that's shown and saved with the conversation.
Expand Down
11 changes: 11 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,17 @@ x-backend: &backend
MAIL_USERNAME: ${MAIL_USERNAME:-}
MAIL_PASSWORD: ${MAIL_PASSWORD:-}
MAIL_FROM: ${MAIL_FROM:-}
# first run (docs/configuration.md#first-run-setup): unset, the web app's setup wizard asks
LENS_SETUP_CODE: ${LENS_SETUP_CODE:-}
LENS_SETUP_WIZARD: ${LENS_SETUP_WIZARD:-}
LENS_ADMIN_EMAIL: ${LENS_ADMIN_EMAIL:-}
LENS_ADMIN_PASSWORD: ${LENS_ADMIN_PASSWORD:-}
LENS_ADMIN_NAME: ${LENS_ADMIN_NAME:-}
LENS_NAMESPACE: ${LENS_NAMESPACE:-}
LENS_LLM_BASE_URL: ${LENS_LLM_BASE_URL:-}
LENS_LLM_MODEL: ${LENS_LLM_MODEL:-}
LENS_LLM_API_KEY: ${LENS_LLM_API_KEY:-}
LENS_LLM_VISION_MODEL: ${LENS_LLM_VISION_MODEL:-}
volumes:
- archive-data:/data
- ${AUDIO_DIR:-./local-audio}:/audio:ro
Expand Down
5 changes: 3 additions & 2 deletions docs/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
## People and roles

There is no public sign-up. The first admin is created with a one-time **setup code** that the API prints in its log
on first start (or `lens users add you@example.com --admin`). Admins then add people and give them roles per
on first start (or `lens users add you@example.com --admin`, or `LENS_ADMIN_EMAIL` and `LENS_ADMIN_PASSWORD` in
`.env`). On a fresh install the web app then opens a short setup wizard ([Configuration](configuration.md#first-run-setup)). Admins then add people and give them roles per
namespace: viewer, editor or owner.

Passwords are hashed with scrypt and need at least 10 characters. Failed sign-ins are throttled per email and address
Expand Down Expand Up @@ -31,7 +32,7 @@ Sign out ──► POST /api/v1/auth/logout (ends the session on the API too)

| Endpoint | |
|---|---|
| `GET /api/v1/auth/status` | `{setup_required}`: the sign-in page shows the setup form when true |
| `GET /api/v1/auth/status` | `{setup_required, wizard_pending}`: the sign-in page shows the setup form while the first admin is missing; admins are taken to the setup wizard while it is pending |
| `POST /api/v1/auth/setup` | first admin, with the setup code |
| `POST /api/v1/auth/login` · `/refresh` · `/logout` | token pairs |
| `GET /api/v1/auth/me` | the account, roles by namespace, and how the caller authenticated |
Expand Down
23 changes: 23 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,34 @@ with AES-GCM and are write-only: the API reports whether one is set, never its v
| `ARCHIVE_ALLOWED_HOSTS` | | break-glass override of allowed Host headers if a bad setting locks everyone out |
| `RUN_BACKGROUND` | follows `workers.inline` | run job workers and folder watching inside the API process |
| `LENS_SETUP_CODE` | random | fix the first-run setup code (automation) |
| `LENS_ADMIN_EMAIL` / `LENS_ADMIN_PASSWORD` / `LENS_ADMIN_NAME` | | create the first admin at startup, with no setup code ([First-run setup](#first-run-setup)) |
| `LENS_NAMESPACE` | | create the first namespace at startup, while there is none |
| `LENS_LLM_BASE_URL` / `LENS_LLM_MODEL` / `LENS_LLM_API_KEY` / `LENS_LLM_VISION_MODEL` | | the model provider; wins over Settings, which show these locked |
| `LENS_SETUP_WIZARD` | | `off`: never show the setup wizard |
| `FRONTEND_URL` | `http://localhost:3000` | links in emails |
| `CORS_ORIGINS` | `["http://localhost:3000"]` | origins allowed to call the API from a browser |
| `OPENAPI_URL` | `/openapi.json` | `""` disables `/docs` and the schema |
| `MAIL_SERVER`, `MAIL_PORT`, `MAIL_USERNAME`, `MAIL_PASSWORD`, `MAIL_FROM`, `MAIL_STARTTLS`, `MAIL_SSL_TLS`, `USE_CREDENTIALS`, `VALIDATE_CERTS` | | SMTP for password reset and for telling owners about requests for access; without `MAIL_SERVER` the links are logged |

## First-run setup

A fresh install (no accounts when the API first starts) walks its first admin through setup in the web app:

1. **Admin account**, with the one-time setup code from the log (`make setup-code`), so a stranger who finds a new
public server can't claim it. Skipped when `LENS_ADMIN_EMAIL` and `LENS_ADMIN_PASSWORD` create the admin at
startup; then you sign in with those.
2. **Namespace**: the first one, with its knowledge graph shared or isolated. Already done when archive.yaml or
`LENS_NAMESPACE` names namespaces, or an install script made one.
3. **Model provider**: an OpenAI-compatible server (OpenAI, Ollama, llama.cpp, LM Studio, vLLM), its model and key,
with a test.
4. **Storage**: where data lives (shown; set in archive.yaml and `SURREAL_URL`), the largest upload, and optionally a
folder inside `sources.local_roots` to watch.

Every step can be skipped, and the whole wizard too; all of it stays in Settings. Values from the environment win and
show locked. Installs that already had accounts never see the wizard. The API side is `GET /api/v1/setup`,
`POST /api/v1/setup/namespace`, `PUT /api/v1/setup/llm`, `PUT /api/v1/setup/storage` and `POST /api/v1/setup/finish`
(admins); `GET /api/v1/auth/status` says whether it is still pending (`wizard_pending`).

## Frontend environment

| Variable | |
Expand Down
4 changes: 3 additions & 1 deletion docs/get-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ On first start the API log prints a setup code:
No accounts yet. Create the first admin in the web app with setup code: …
```

Open the web app, choose **Set up**, and create the admin account with it.
Open the web app, choose **Set up**, and create the admin account with it. A short wizard then asks for the first
namespace, the model provider and storage; skip any of it and change it later in Settings
([Configuration](configuration.md#first-run-setup) lists the `.env` values that answer it instead).

Put audio under `./local-audio/podcasts` and `./local-audio/interviews` (or set `AUDIO_DIR`), or import transcripts
from the web app. Namespaces and folders are configured in `fastapi_backend/docker/archive.yaml`.
Expand Down
10 changes: 10 additions & 0 deletions fastapi_backend/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,15 @@ USE_CREDENTIALS=False
FRONTEND_URL=http://localhost:3000
CORS_ORIGINS=["http://localhost:3000"]

# --- First run (docs/configuration.md#first-run-setup). Without these, the web app's setup wizard asks. ---
# LENS_SETUP_CODE=choose-one # the first admin's one-time setup code (default: random, in the log)
# LENS_ADMIN_EMAIL=you@example.com # create the first admin at startup (with LENS_ADMIN_PASSWORD)
# LENS_ADMIN_PASSWORD=
# LENS_NAMESPACE=archive # create the first namespace at startup
# LENS_LLM_BASE_URL=http://localhost:11434/v1 # the model provider; these win over Settings and show locked
# LENS_LLM_MODEL=llama3.1
# LENS_LLM_API_KEY=
# LENS_SETUP_WIZARD=off # never show the wizard

# Uncomment to disable /docs and /openapi.json
# OPENAPI_URL=""
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
requests,
search,
searches,
setup,
sources,
speakers,
templates,
Expand All @@ -40,6 +41,7 @@
for module in (
users,
admin,
setup,
namespaces,
hierarchy,
recordings,
Expand Down
9 changes: 7 additions & 2 deletions fastapi_backend/app/api/v1/routes/auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from app.config import settings
from app.core.security import create_access_token
from app.domain import auth, store
from app.domain import setup as first_run
from app.email import send_reset_password_email
from app.schemas.auth import (
ApiToken,
Expand Down Expand Up @@ -43,8 +44,12 @@ def _pair(db: Db, cfg: Cfg, request: Request, user: dict) -> TokenPair:

@router.get("/status")
def status(request: Request, db: Db) -> AuthStatus:
"""Whether the archive still needs its first admin (the sign-in page shows the setup form instead)."""
return AuthStatus(setup_required=request.app.state.archive.setup_code is not None and auth.account_count(db) == 0)
"""Whether the archive still needs its first admin (the sign-in page shows the setup form instead), and whether
the first-run wizard is still to be finished (the web app takes admins there)."""
return AuthStatus(
setup_required=request.app.state.archive.setup_code is not None and auth.account_count(db) == 0,
wizard_pending=first_run.pending(db),
)


@router.post("/setup")
Expand Down
74 changes: 74 additions & 0 deletions fastapi_backend/app/api/v1/routes/setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
"""The first-run setup wizard (admins): the first namespace, the model provider and storage, then finishing it.

The first admin is created before this, with the setup code (POST /auth/setup) or from the environment. Fields the
environment sets are left out of every save and reported as locked (domain/setup.py).
"""

from __future__ import annotations

from fastapi import APIRouter, HTTPException, Request

from app.api.deps import AdminReader, AdminWriter, Db
from app.domain import auth, setup
from app.schemas.setup import SetupFinish, SetupLlm, SetupNamespace, SetupSaved, SetupStorage, SetupView

router = APIRouter(prefix="/setup", tags=["setu 10BC8 p"])


@router.get("")
def get_setup(user: AdminReader, request: Request, db: Db) -> SetupView:
"""What is set already, and which fields .env or archive.yaml locks."""
return SetupView.model_validate(setup.view(db, request.app.state.archive.base))


@router.post("/namespace")
def save_namespace(body: SetupNamespace, user: AdminWriter, request: Request, db: Db) -> SetupSaved:
"""Create the first namespace (or set the graph mode of one that exists)."""
if setup.view(db, request.app.state.archive.base)["namespace"]["locked"]:
raise HTTPException(409, "the namespaces are set in archive.yaml or LENS_NAMESPACE")
try:
setup.save_namespace(db, body.name, body.graph)
except ValueError as e:
raise HTTPException(400, str(e)) from None
auth.audit(db, user.as_audit(), "namespace.create", body.name.strip().lower(), ["setup"])
request.app.state.graph_cache.clear()
return SetupSaved(saved=["namespace"])


@router.put("/llm")
def save_llm(body: SetupLlm, user: AdminWriter, request: Request, db: Db) -> SetupSaved:
"""The model provider: an OpenAI-compatible server's address, the model, and its key (empty: unchanged)."""
values: dict = {"base_url": (body.base_url or "").strip() or None, "model": (body.model or "").strip() or None}
if body.api_key:
values["api_key"] = body.api_key
try:
saved = setup.save_llm(db, request.app.state.archive.base, values, user.email)
except ValueError as e:
raise HTTPException(400, str(e)) from None
if saved:
auth.audit(db, user.as_audit(), "settings.save", "llm", saved)
return SetupSaved(saved=saved)


@router.put("/storage")
def save_storage(body: SetupStorage, user: AdminWriter, request: Request, db: Db) -> SetupSaved:
"""The upload limit, and optionally a folder on this machine (inside sources.local_roots) to watch."""
try:
wid = setup.save_storage(db, request.app.state.archive.base, body.max_upload_mb, body.folder, body.namespace, user.email)
except KeyError:
raise HTTPException(400, "no such namespace") from None
except ValueError as e:
raise HTTPException(400, str(e)) from None
if body.max_upload_mb is not None:
auth.audit(db, user.as_audit(), "settings.save", "uploads", ["max_mb"])
if wid is not None:
auth.audit(db, user.as_audit(), "watch.create", f"watch_path:{wid}")
return SetupSaved(saved=[k for k in ("max_upload_mb", "folder") if getattr(body, k) is not None], watch=wid)


@router.post("/finish")
def finish(body: SetupFinish, user: AdminWriter, db: Db) -> SetupSaved:
"""Finish (or skip) the wizard; the web app stops showing it. Everything stays changeable in Settings."""
setup.finish(db, user.email, body.skipped)
auth.audit(db, user.as_audit(), "setup.finish", None, ["skipped"] if body.skipped else None)
return SetupSaved()
5 changes: 4 additions & 1 deletion fastapi_backend/app/core/runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from typing import Any

from app.config import settings as env
from app.domain import auth, jobs, settings, sources, store, templates
from app.domain import auth, jobs, settings, setup, sources, store, templates

log = logging.getLogger("lens")

Expand All @@ -32,6 +32,9 @@ def prepare(self) -> None:
templates.seed(self.db)
# Pay for the embedded engine's full-text repair at startup rather than in someone's first search.
self.db.ready_fulltext()
if auth.account_count(self.db) == 0:
setup.mark_fresh(self.db) # a fresh install: the web app walks the first admin through setup
setup.apply_env(self.db)
if auth.account_count(self.db) == 0:
self.setup_code = os.environ.get("LENS_SETUP_CODE") or secrets.token_urlsafe(9)
log.warning("No accounts yet. Create the first admin in the web app with setup code: %s", self.setup_code)
Expand Down
36 changes: 31 additions & 5 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,32 @@
("video", "face_engine"): {"opencv", "insightface", "none"},
("video", "object_engine"): {"yolox", "ultralytics", "off"},
}
ENV_OVERRIDES = {("server", "allowed_hosts"): "ARCHIVE_ALLOWED_HOSTS"}
# Settings the environment (.env) sets, which win over archive.yaml and the app and show as locked there: the
# break-glass allowed hosts, and the model provider so an install can be configured without the setup wizard.
ENV_OVERRIDES = {
("server", "allowed_hosts"): "ARCHIVE_ALLOWED_HOSTS",
("llm", "base_url"): "LENS_LLM_BASE_URL",
("llm", "model"): "LENS_LLM_MODEL",
("llm", "api_key"): "LENS_LLM_API_KEY",
("llm", "vision_model"): "LENS_LLM_VISION_MODEL",
}


def env_value(section, key):
"""The environment's value for a setting in ENV_OVERRIDES, parsed; None when it isn't set."""
raw = os.environ.get(ENV_OVERRIDES.get((section, key)) or "", "").strip()
if not raw:
return None
if isinstance(store.DEFAULTS.get(section, {}).get(key), list):
return [x.strip() for x in raw.split(",") if x.strip()]
return raw


def locked(section):
"""The keys of a section the environment sets."""
return [k for (s, k) in ENV_OVERRIDES if s == section and env_value(s, k) is not None]


# The types uploads.extensions may name: what the folder scans import, a few more that ffmpeg reads, and documents and
# images.
UPLOAD_TYPES = frozenset([*store.MEDIA_EXT, ".aif", ".aiff", ".wma", ".mpg", ".mpeg", ".3gp", *store.DOCUMENT_EXT, *store.IMAGE_EXT])
Expand Down Expand Up @@ -143,9 +168,10 @@ def effective(db, base, reveal=True):
cfg[sec][k] = unseal(base, sealed, f"setting:{sec}.{k}")
except Exception: # noqa: BLE001 - key changed since it was saved
cfg[sec][k] = None
for (sec, key), env in ENV_OVERRIDES.items():
if os.environ.get(env):
cfg[sec][key] = [x.strip() for x in os.environ[env].split(",") if x.strip()]
for sec, key in ENV_OVERRIDES:
v = env_value(sec, key)
if v is not None:
cfg[sec][key] = {"secret": True, "set": True} if not reveal and key in SECRETS.get(sec, ()) else v
return cfg


Expand Down Expand Up @@ -178,7 +204,7 @@ def view(db, base):
"values": vals,
"updated_at": (meta.get(sec) or {}).get("updated_at"),
"updated_by": (meta.get(sec) or {}).get("updated_by"),
"locked": [k for (s, k), env in ENV_OVERRIDES.items() if s == sec and os.environ.get(env)],
"locked": locked(sec),
}
out["bootstrap"] = {
"database": db.url,
Expand Down
Loading
Loading
0