adeelahmad · Oct 2, 2026 · Oct 2, 2026 · Oct 2, 2026 · Oct 2, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,15 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
 
 ## Unreleased
 
+- **Opt-in telemetry.** Lens can send OpenTelemetry traces and metrics about its own work to a collector you choose:
+  API requests by route, jobs and each step, routines, workflow runs, and model calls with their tokens and an
+  estimated cost (from per-model prices you set). It is off by default and never on unless you turn it on, in
+  Settings → Telemetry, the setup wizard's new last step, or `LENS_TELEMETRY=on` with `LENS_TELEMETRY_ENDPOINT` in
+  .env (`LENS_TELEMETRY=off` keeps it off). It goes only to that endpoint, over OTLP/HTTP; there is no built-in
+  destination. Spans and metrics carry ids, step types, model names, counts and timings, never transcript, prompt or
+  answer text, file names, titles, people, paths or addresses. Settings shows whether it is on and how the last
+  exports went, and can send a test span; the API and each worker follow a change without a restart
+  (docs/telemetry.md).
 - **Security: frontend dependencies patched.** The web app's lockfile now pulls fixed versions of form-data, ws,
   brace-expansion, minimatch, picomatch, glob, js-yaml, flatted, browserslist, Babel and the other packages GitHub
   flagged, each kept inside the major version its parent asks for (`overrides` in `nextjs-frontend/pnpm-workspace.yaml`).

diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml
@@ -38,6 +38,10 @@ x-backend: &backend
     LENS_LLM_MODEL: ${LENS_LLM_MODEL:-}
     LENS_LLM_API_KEY: ${LENS_LLM_API_KEY:-}
     LENS_LLM_VISION_MODEL: ${LENS_LLM_VISION_MODEL:-}
+    # opt-in telemetry (docs/telemetry.md): unset, it stays off unless turned on in Settings
+    LENS_TELEMETRY: ${LENS_TELEMETRY:-}
+    LENS_TELEMETRY_ENDPOINT: ${LENS_TELEMETRY_ENDPOINT:-}
+    LENS_TELEMETRY_HEADERS: ${LENS_TELEMETRY_HEADERS:-}
   volumes:
     - archive-data:/data
     - ${AUDIO_DIR:-./local-audio}:/audio:ro

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -31,6 +31,7 @@ with AES-GCM and are write-only: the API reports whether one is set, never its v
 | `LENS_NAMESPACE` | | create the first namespace at startup, while there is none |
 | `LENS_LLM_BASE_URL` / `LENS_LLM_MODEL` / `LENS_LLM_API_KEY` / `LENS_LLM_VISION_MODEL` | | the model provider; wins over Settings, which show these locked |
 | `LENS_SETUP_WIZARD` | | `off`: never show the setup wizard |
+| `LENS_TELEMETRY` / `LENS_TELEMETRY_ENDPOINT` / `LENS_TELEMETRY_HEADERS` | off | opt-in OpenTelemetry traces and metrics, sent only to this OTLP/HTTP endpoint ([Telemetry](telemetry.md)); `LENS_TELEMETRY=off` keeps it off whatever Settings say |
 | `FRONTEND_URL` | `http://localhost:3000` | the web app's address as people use it: links in emails, and where apps send people to sign in ([OAuth](authentication.md#oauth)) |
 | `CORS_ORIGINS` | `["http://localhost:3000"]` | origins allowed to call the API from a browser |
 | `OPENAPI_URL` | `/openapi.json` | `""` disables `/docs` and the schema |

diff --git a/docs/telemetry.md b/docs/telemetry.md
@@ -0,0 +1,113 @@
+# Telemetry
+
+Lens can send traces and metrics about its own work to an [OpenTelemetry](https://opentelemetry.io/) collector, to see
+how it performs: which API requests are slow, how long each job step takes, how often jobs fail, and how many tokens
+model calls use and what they cost.
+
+**It is off unless you turn it on.** Nothing is collected while it is off, and Lens has no built-in destination: it
+never sends telemetry to the Lens project or anyone else. Turned on, it sends only to the endpoint you set, typically
+a collector on the same machine or network, which can pass it on to Jaeger, Grafana Tempo and Prometheus, SigNoz,
+Honeycomb, or whatever you use.
+
+## Turning it on
+
+Any of these:
+
+- **Settings → Telemetry**: turn on **Send telemetry** and set the **OTLP endpoint**. **Send a test span** checks the
+  address and headers first, even while it is off. The section shows whether it is on, where it sends, and how the
+  last exports went.
+- **The setup wizard** asks on a fresh install (its last step); it is off unless you choose on.
+- **.env**, for the API and every worker:
+
+  ```sh
+  LENS_TELEMETRY=on                                 # off keeps it off, whatever Settings say
+  LENS_TELEMETRY_ENDPOINT=http://localhost:4318     # the collector's OTLP/HTTP address
+  LENS_TELEMETRY_HEADERS=Authorization=Bearer%20xyz # optional: key=value pairs, comma-separated, URL-encoded
+  ```
+
+  What .env sets wins over Settings, which show those fields locked.
+
+Each process (the API, each `lens worker`, `lens watch`) follows the settings on its own and picks up a change within a
+few seconds, with no restart. The endpoint is the collector's base address: traces go to `/v1/traces` and metrics to
+`/v1/metrics` under it, over OTLP/HTTP with protobuf.
+
+In Docker, `localhost` is the container itself: point the endpoint at a collector on the `lens` network (for example
+`http://otel-collector:4318`, below) or at `http://host.docker.internal:4318` for one on the host.
+
+## What is sent
+
+| Signal | What | Attributes |
+| --- | --- | --- |
+| Trace | each API request, named by its route template (`GET /api/v1/recordings/{rid}`) | method, route, status code |
+| Trace | each job, with a span per step (`step transcribe`) | job, recording and pipeline ids; step type and outcome |
+| Trace | routines and their actions, workflow runs | routine, workflow and recording ids; action type; trigger |
+| Trace | each call to the model server (`chat llama3.1`) | model, server host, temperature, tokens in and out, estimated cost |
+| Metric | `http.server.request.duration` | method, route, status code |
+| Metric | `lens.job.step.duration`, `lens.jobs` | step type, outcome |
+| Metric | `lens.routine.runs`, `lens.workflow.runs` | outcome |
+| Metric | `gen_ai.client.operation.duration`, `gen_ai.client.token.usage` | model, token type |
+| Metric | `lens.llm.cost` (USD) | model |
+
+Every span and metric carries `service.name` (Settings → Telemetry, default `lens`), `service.version`, a random
+`service.instance.id` per process, and `lens.process.role` (`api`, `worker` or `watcher`). Model calls follow the
+OpenTelemetry [GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
+
+**Never sent:** transcript text, prompts or answers, file names, titles, namespace or speaker names, people, email
+addresses, client IP addresses, request paths or query strings (only route templates), host names, or error messages
+(only the error's type, since a message can quote content).
+
+### Tokens and cost
+
+Token counts come from the model server's reply (`usage`). Most servers report them; a streamed chat answer has them
+only when the server sends usage in its last chunk.
+
+Cost is an estimate from prices you set: **Model prices** in Settings → Telemetry, one line per model with its input
+and output price in dollars per million tokens (`gpt-4o-mini 0.15 0.60`). The prices in Settings → AI assistant count
+for the configured model too. A model without a price gets no cost, so local models cost nothing.
+
+## Other settings
+
+| Setting | Default | |
+| --- | --- | --- |
+| Traces / Metrics | on / on | send either or both |
+| Share of traces kept | `1` | `0.1` keeps one trace in ten (head sampling) |
+| Send metrics every | `60` seconds | |
+| Service name | `lens` | how this server shows in your tracing tool |
+
+In archive.yaml they live under `telemetry:` (`enabled`, `endpoint`, `traces`, `metrics`, `sample_ratio`,
+`export_seconds`, `service_name`, `prices`).
+
+## A collector next to Lens
+
+A minimal [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) that takes Lens's traces and metrics and
+prints them (swap the `debug` exporter for your backend's):
+
+```yaml
+# otel-collector.yaml
+receivers:
+  otlp:
+    protocols:
+      http:
+        endpoint: 0.0.0.0:4318
+exporters:
+  debug:
+    verbosity: basic
+service:
+  pipelines:
+    traces: { receivers: [otlp], exporters: [debug] }
+    metrics: { receivers: [otlp], exporters: [debug] }
+```
+
+Beside `docker-compose.prod.yml`, in a `docker-compose.override.yml` or with `-f`:
+
+```yaml
+services:
+  otel-collector:
+    image: otel/opentelemetry-collector:latest
+    command: ["--config=/etc/otel-collector.yaml"]
+    volumes:
+      - ./otel-collector.yaml:/etc/otel-collector.yaml:ro
+    networks: [lens]
+```
+
+Then set the endpoint to `http://otel-collector:4318` and turn telemetry on.
diff --git a/fastapi_backend/.env.example b/fastapi_backend/.env.example
@@ -52,5 +52,10 @@ CORS_ORIGINS=["http://localhost:3000"]
 # LENS_LLM_API_KEY=
 # LENS_SETUP_WIZARD=off                 # never show the wizard
 
+# --- Telemetry (docs/telemetry.md): off unless turned on here or in Settings, and sent only to this endpoint ---
+# LENS_TELEMETRY=on                     # on, or off to keep it off whatever Settings say
+# LENS_TELEMETRY_ENDPOINT=http://localhost:4318   # your OpenTelemetry collector's OTLP/HTTP address
+# LENS_TELEMETRY_HEADERS=               # optional: key=value pairs, comma-separated, URL-encoded
+
 # Uncomment to disable /docs and /openapi.json
 # OPENAPI_URL=""
diff --git a/fastapi_backend/app/api/v1/routes/admin.py b/fastapi_backend/app/api/v1/routes/admin.py
@@ -12,8 +12,8 @@
 
 from app.api.deps import AdminReader, AdminWriter, Cfg, Db, domain_errors
 from app.core.middleware import host_name
-from app.domain import auth, jobs, llm, settings, sources, store
-from app.schemas.admin import AuditEntry, Health, LlmTestResult, Started
+from app.domain import auth, jobs, llm, settings, sources, store, telemetry
+from app.schemas.admin import AuditEntry, Health, LlmTestResult, Started, TelemetryStatus, TelemetryTestResult
 from app.schemas.auth import AccountToken
 from app.schemas.common import Ok
 
@@ -54,6 +54,19 @@ def test_llm(user: AdminWriter, cfg: Cfg) -> LlmTestResult:
     return LlmTestResult(ok=True, reply=reply.strip()[:40], ms=int((time.time() - t0) * 1000), model=cfg["llm"]["model"])
 
 
+@router.get("/settings/telemetry/status")
+def telemetry_status(user: AdminReader, cfg: Cfg) -> TelemetryStatus:
+    """Whether telemetry is on, where it goes, and how the API process's last exports went. Off by default."""
+    return TelemetryStatus.model_validate(telemetry.status(cfg))
+
+
+@router.post("/settings/telemetry/test")
+def test_telemetry(user: AdminWriter, cfg: Cfg) -> TelemetryTestResult:
+    """Send one test span to the saved endpoint now (on or off), to check the address and headers."""
+    ok, error, ms = telemetry.test_export(cfg)
+    return TelemetryTestResult(ok=ok, error=error, ms=ms)
+
+
 @router.get("/audit")
 def list_audit(user: AdminReader, db: Db, limit: int = Query(200, ge=1, le=1000)) -> list[AuditEntry]:
     """Who changed what, newest first."""

diff --git a/fastapi_backend/app/api/v1/routes/setup.py b/fastapi_backend/app/api/v1/routes/setup.py
@@ -1,4 +1,4 @@
-"""The first-run setup wizard (admins): the first namespace, the model provider and storage, then finishing it.
+"""The first-run setup wizard (admins): the first namespace, the model provider, storage and telemetry, then finishing it.
 
 The first admin is created before this, with the setup code (POST /auth/setup) or from the environment. Fields the
 environment sets are left out of every save and reported as locked (domain/setup.py).
@@ -10,7 +10,7 @@
 
 from app.api.deps import AdminReader, AdminWriter, Db
 from app.domain import auth, setup
-from app.schemas.setup import SetupFinish, SetupLlm, SetupNamespace, SetupSaved, SetupStorage, SetupView
+from app.schemas.setup import SetupFinish, SetupLlm, SetupNamespace, SetupSaved, SetupStorage, SetupTelemetry, SetupView
 
 router = APIRouter(prefix="/setup", tags=["setup"])
 
@@ -66,6 +66,18 @@ def save_storage(body: SetupStorage, user: AdminWriter, request: Request, db: Db
     return SetupSaved(saved=[k for k in ("max_upload_mb", "folder") if getattr(body, k) is not None], watch=wid)
 
 
+@router.put("/telemetry")
+def save_telemetry(body: SetupTelemetry, user: AdminWriter, request: Request, db: Db) -> SetupSaved:
+    """Opt in to telemetry: traces and metrics sent only to this endpoint (an OTLP collector). Off unless chosen."""
+    try:
+        saved = setup.save_telemetry(db, request.app.state.archive.base, body.enabled, body.endpoint, user.email)
+    except ValueError as e:
+        raise HTTPException(400, str(e)) from None
+    if saved:
+        auth.audit(db, user.as_audit(), "settings.save", "telemetry", saved)
+    return SetupSaved(saved=saved)
+
+
 @router.post("/finish")
 def finish(body: SetupFinish, user: AdminWriter, db: Db) -> SetupSaved:
     """Finish (or skip) the wizard; the web app stops s
10BC8
howing it. Everything stays changeable in Settings."""

diff --git a/fastapi_backend/app/cli.py b/fastapi_backend/app/cli.py
@@ -188,7 +188,7 @@ def platform_main(argv, config):
     import threading
     import time
 
-    from .domain import auth, jobs, notify, routines, settings, sources
+    from .domain import auth, jobs, notify, routines, settings, sources, telemetry
 
     ap = argparse.ArgumentParser(prog="lens")
     sub = ap.add_subparsers(dest="cmd", required=True)
@@ -250,6 +250,7 @@ def password():
                     auth.update_account(db, acct["id"], password=password())
                     print(f"password changed for {a.email}")
         elif a.cmd == "worker":
+            telemetry.set_role("worker")
             wk = jobs.Worker(db, C, a.name, a.steps.split(",") if a.steps else None, log=print)
             if a.once:
                 print(f"ran {wk.drain()} job(s)")
@@ -263,9 +264,11 @@ def password():
                 except KeyboardInterrupt:
                     stop.set()
         elif a.once:
+            telemetry.set_role("watcher")
             print(f"scanned {sources.poll_due(db, C(), print)} folder(s)")
             print(f"ran {routines.run_due(db, C(), print)} routine(s)")
         else:
+            telemetry.set_role("watcher")
             print("watching storage sources and running routines; Ctrl-C to stop")
             _stop_on_term()
             try:
@@ -278,6 +281,7 @@ def password():
     except (ValueError, KeyError) as e:
         raise SystemExit(str(e)) from None
     finally:
+        telemetry.shutdown()  # sends what is buffered
         db.close()
 
 

diff --git a/fastapi_backend/app/domain/analyze.py b/fastapi_backend/app/domain/analyze.py
@@ -9,7 +9,7 @@
 import urllib.request
 from collections import Counter, defaultdict
 
-from . import store
+from . import store, telemetry
 
 STOP = set(
     """a about above after again against all almost also am an and any are aren as at be because been before being
@@ -628,8 +628,8 @@ def _llm(cfg, user, system=SUMMARY_SYSTEM):
     req = urllib.request.Request(
         l["base_url"].rstrip("/") + "/chat/completions", data=json.dumps(body).encode(), headers=headers, method="POST"
     )
-    with urllib.request.urlopen(req, timeout=l.get("timeout", 300)) as r:
-        text = json.loads(r.read().decode())["choices"][0]["message"]["content"]
+    with telemetry.model_call(cfg, body) as call, urllib.request.urlopen(req, timeout=l.get("timeout", 300)) as r:
+        text = call.reply(json.loads(r.read().decode()))["choices"][0]["message"]["content"]
     text = re.sub(r"^```(?:json)?|```$", "", text.strip()).strip()
     obj = json.loads(text[text.find("{") : text.rfind("}") + 1])
     missing = [k for k in SUMMARY_NEEDS if k not in obj]

diff --git a/fastapi_backend/app/domain/jobs.py b/fastapi_backend/app/domain/jobs.py
@@ -13,7 +13,7 @@
 import threading
 import time
 
-from . import analyze, ingest, pipelines, render, speakers as spk, store
+from . import analyze, ingest, pipelines, render, speakers as spk, store, telemetry
 
 R = store.R
 PIPELINE = ["transcribe", "diarize", "shots", "ocr", "faces", "objects", "describe", "analyze", "summarize", "report"]
@@ -435,6 +435,7 @@ def started(k):
         return time.time()
 
     def finished(k, outcome, note, t0, outputs=None):
+        telemetry.record("lens.job.step.duration", time.time() - t0, {"lens.step": _spec(steps[k])["type"], "lens.step.outcome": outcome})
         runs[k] = store.clean(
             {
                 **runs[k],
@@ -512,15 +513,17 @@ def beat():
                 i, t0 = i + 1, None
                 continue
             before = _saved(db, rid)
-            try:
-                STEPS[step](db, cfg_fn(), rid, say, spec)
-            except Skip as e:
-                say(f"{step} skipped: {e}")
-                finished(i, "skipped", str(e), t0, _outputs(before, _saved(db, rid)))
-            else:
-                note = said[0]
-                say(f"{spec.get('name') or step} done in {time.time() - t0:.1f}s")
-                finished(i, "done", note, t0, _outputs(before, _saved(db, rid)))
+            with telemetry.span(f"step {step}", {"lens.step": step, "lens.step.index": i, "lens.job.id": str(jid)}) as sp:
+                try:
+                    STEPS[step](db, cfg_fn(), rid, say, spec)
+                except Skip as e:
+                    say(f"{step} skipped: {e}")
+                    finished(i, "skipped", str(e), t0, _outputs(before, _saved(db, rid)))
+                else:
+                    note = said[0]
+                    say(f"{spec.get('name') or step} done in {time.time() - t0:.1f}s")
+                    finished(i, "done", note, t0, _outputs(before, _saved(db, rid)))
+                sp.set_attribute("lens.step.outcome", runs[i]["outcome"])
             _timed(db, rid, spec, runs[i]["seconds"], runs[i]["outcome"] == "skipped")
             i, t0 = i + 1, None
     except (Exception, SystemExit) as e:  # noqa: BLE001 - recorded on the job; SystemExit too (a missing engine says so)
@@ -738,7 +741,17 @@ def run_once(self):
         if not job:
             return False
         self.register(job["id"])
-        run_job(self.db, self.cfg_fn, job, self.name, self.can, self.log)
+        pipe = job.get("pipeline") if isinstance(job.get("pipeline"), dict) else {}
+        attrs = {
+            "lens.job.id": str(job["id"]),
+            "lens.recording.id": str(job["recording"]),
+            "lens.pipeline.id": str(pipe["id"]) if pipe.get("id") is not None else ("standard" if pipe else None),
+            "lens.pipeline.version": pipe.get("version"),
+        }
+        with telemetry.span("job", attrs) as sp:
+            outcome = run_job(self.db, self.cfg_fn, job, self.name, self.can, self.log)
+            sp.set_attribute("lens.job.outcome", outcome)
+        telemetry.record("lens.jobs", 1, {"lens.job.outcome": outcome})
         self.register()
         return True