adeelahmad · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026
diff --git a/README.md b/README.md
@@ -202,8 +202,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
   schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity
   ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage
   groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today;
-  AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching
-  once the mesh's install line lands), and Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
+  AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, with
+  machines that join over the mesh to transcribe and tell voices apart, not yet tried on a real cloud), and Linux and
+  model hosts are planned. ([Auto scaling](docs/scaling.md);
   [checks](docs/requirements.md#scale-out))
 - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a
   notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine,
@@ -268,7 +269,7 @@ These are open pull requests, not yet part of Lens:
   ([#406](https://github.com/adeelahmad/lense/pull/406)).
 - **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
 - **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates
-  for ([#591](https://github.com/adeelahmad/lense/pull/591)).
+  for ([#595](https://github.com/adeelahmad/lense/pull/595)).
 
 ## Planned
 

diff --git a/changelog.d/mesh-node.md b/changelog.d/mesh-node.md
@@ -1,5 +1,4 @@
 - **Joining a node** (in progress). Another machine joins with the install line and `LENS_JOIN` and `LENS_JOIN_TOKEN`:
   it asks nothing, publishes no port, and runs Nebula and `lens node`, which joins once and then follows the primary's
-  config, renews its certificate and reports over the mesh. Adding a node token now returns that install line. Worker
-  nodes running jobs and the Settings pages follow in their own pull requests
-  ([On a node](docs/mesh.md#on-a-node)).
+  config, renews its certificate and reports over the mesh. Adding a node token now returns that install line. The
+  Settings pages follow in their own pull request ([On a node](docs/mesh.md#on-a-node)).
diff --git a/changelog.d/mesh-workers.md b/changelog.d/mesh-workers.md
@@ -0,0 +1,7 @@
+- **Worker nodes run jobs** (in progress). On a worker node, `lens node` also runs `lens worker`, against this
+  install's database through the mesh, signed in as a user of that one database made for the node (removed with it;
+  `SURREAL_AUTH=database`). Its jobs fetch their files from the primary over the mesh, plain, never a vault's or
+  another namespace's, and it never gets the archive's secret, so it runs `transcribe` and `diarize` with the models
+  on that machine, skipping any other step `LENS_WORKER_STEPS` names. A worker that serves some namespaces takes only
+  their jobs, and an auto scaling group on machines elsewhere counts only the jobs a worker node takes, so it never
+  launches one for others ([Worker nodes](docs/mesh.md#worker-nodes)).
diff --git a/docker-compose.node.yml b/docker-compose.node.yml
@@ -1,6 +1,7 @@
 # A machine on another Lens install's mesh (docs/mesh.md, Nodes): what install.sh starts with LENS_JOIN and
 # LENS_JOIN_TOKEN. No database, no web app, nothing published: `mesh` runs Nebula, and `node` (`lens node`) joins once,
-# then follows the primary's config over the mesh and reports every minute.
+# then follows the primary's config over the mesh and reports every minute; on a worker node it runs `lens worker` too,
+# against the primary's database over the mesh.
 #   LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN=lensjoin1.… docker compose -p lens-node -f docker-compose.node.yml up -d --build
 name: lens-node
 
@@ -36,9 +37,8 @@ services:
     depends_on: [mesh]
     security_opt: ["no-new-privileges:true"]
     cap_drop: [ALL]
-    read_only: true
-    tmpfs: [/tmp]
-    mem_limit: 512m
+    # not read-only and no memory cap: on a worker node it also runs `lens worker`, which fetches its speech model
+    # into the image's folders and needs the machine's memory, as the primary's worker does
     healthcheck:
       disable: true  # the image's check is the API's, which a node doesn't run
     environment:

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -22,6 +22,7 @@ with AES-GCM and are write-only: the API reports whether one is set, never its v
 | `PASSWORD_RESET_EXPIRE_MINUTES` | `60` | |
 | `SURREAL_URL` | embedded | `ws://host:8000` for a server; see [Database](database.md) |
 | `SURREAL_USER` / `SURREAL_PASS` / `SURREAL_NS` / `SURREAL_DB` / `SURREAL_POOL_SIZE` | `root` / `root` / `archive` / `main` / `8` | |
+| `SURREAL_AUTH` | root | `database`: `SURREAL_USER` is a user of that one database, not the server's root (a [worker node](mesh.md#worker-nodes) signs in so) |
 | `ARCHIVE_SECRET_KEY` | generated in `data_dir/secret.key` | encrypts stored credentials; keep it stable and backed up |
 | `ARCHIVE_CONFIG` | `archive.yaml` | processing configuration file |
 | `ARCHIVE_ALLOWED_HOSTS` | | break-glass override of allowed Host headers if a bad setting locks everyone out |

diff --git a/docs/mesh.md b/docs/mesh.md
@@ -3,8 +3,8 @@
 Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the
 token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings, the API
 under `/api/v1/mesh`, Nebula running on the primary in the `mesh` container, relaying the database and the API, and
-`lens node` with the install line that joins another machine. Coming in their own pull requests: worker nodes running
-jobs, Settings › Network and Nodes in the web app, and running code on compute nodes.
+`lens node` with the install line that joins another machine, and worker nodes running jobs. Coming in their own pull
+requests: Settings › Network and Nodes in the web app, and running code on compute nodes.
 
 Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code
 on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a
@@ -21,7 +21,7 @@ install.
 | Role | What it does | What it reaches on the primary |
 | --- | --- | --- |
 | **primary** | This install: the lighthouse other machines find each other through, the relay for machines that can't reach each other directly, and the certificate authority | |
-| **worker** | Runs jobs (transcription, OCR, the model steps it can) against this install's database, as `lens worker` does on this machine | the database port (8000) and the API (8080) |
+| **worker** | Runs jobs against this install's database, as `lens worker` does on this machine: transcribing and telling voices apart with the models on that machine ([Worker nodes](#worker-nodes)) | the database port (8000) and the API (8080) |
 | **compute** | Runs code in sandboxes for the assistant. It gets nothing of Lens's: no database, no keys | the API (8080) |
 
 A worker node needs this install's database to be a SurrealDB server, which Docker installs have (`SURREAL_URL`);
@@ -91,6 +91,31 @@ environment only, never in `.env`, and isn't needed again: running the line agai
 same node. A node that's removed keeps asking, so an undo in Settings › Nodes brings it back; `docker compose -f
 docker-compose.node.yml exec node lens node status` says what it is on the mesh, without its credentials.
 
+## Worker nodes
+
+On a worker node, `lens node` also runs `lens worker` once the node is on the mesh, and keeps it running: it's started
+again when it stops, stopped while the node is removed, and started with the new password after an undo. Its errors
+and its last lines are in the node's report on the Nodes page.
+
+- **The database through the mesh.** It reaches this install's SurrealDB at the primary's mesh address (port 8000,
+  open to worker nodes only), signed in as a user of that one database made for the node (`lens_node_<id>`, its
+  owner): never the server's root, and nothing on the server beyond Lens's database. The node asks for it
+  (`GET /api/v1/mesh/node/worker`, with its node token) and hands it to the worker's environment only; it isn't kept
+  on the node's disk. Removing the node removes the user, and an undo makes it again with a new password.
+  `SURREAL_AUTH=database` is what makes `lens worker` sign in so ([Configuration](configuration.md)).
+- **Files from here.** A job's file is fetched from the primary over the mesh (`GET /api/v1/mesh/node/media/<id>`),
+  plain, and kept on the node in `data_dir/cache/mesh` until it's unused for 6 hours; it's fetched again only when it
+  changed. A vault's files never leave this machine, nor those of a namespace the node doesn't serve, whose jobs it
+  doesn't take either.
+- **No secrets of Lens's.** It never gets the archive's secret (`ARCHIVE_SECRET_KEY`): namespace keys and settings'
+  secrets (a cloud speech or chat model's key) stay sealed. So by default it runs the steps that need neither and
+  write only to the database: `transcribe` and `diarize`, with the models on that machine (fetched there on first
+  use). `LENS_WORKER_STEPS` on the install line picks among those; any other step it names is skipped (the node's
+  report says so), since its jobs would fail there, and stays with this machine's worker. An auto scaling group on
+  machines elsewhere counts only the jobs these steps run, so it never launches a machine for work it wouldn't take.
+- **Its name** is the node's, or the auto scaling unit's (`LENS_SCALING_UNIT`), and it runs only the steps it's
+  given, never routines or the other services this machine's worker runs.
+
 ## On this machine
 
 Lens's server can't create network devices, and nothing in Lens but Nebula needs to, so on the primary Nebula runs in
@@ -169,22 +194,29 @@ removes it (undo with `POST /nodes/{id}/restore` while its certificate is good).
 - Only a machine with a token an admin made joins; the token is spent on use, expires, and never crosses the wire.
 - A machine's certificate names only its own address, role and namespaces; it can't widen them by renewing.
 - Nebula's firewall, rendered by Lens, opens only what each role needs. Lens itself never trusts the mesh's address:
-  a worker node signs in to the database with its credential, a node's calls carry its node token, and the primary's
-  calls to a node carry its agent key.
+  a worker node signs in to the database as its own user of Lens's database, a node's calls carry its node token, and
+  the primary's calls to a node carry its agent key.
+- A worker node never gets the archive's secret, a vault's files or another namespace's: it reads the files of the
+  namespaces it serves, plain, over the mesh.
 - Removing a node blocks every certificate it was given, on every machine, and its credentials stop at once.
 - The CA's key and the tokens' secrets are sealed in the database like any setting's secret (`ARCHIVE_SECRET_KEY` or
   `secret.key`); node tokens are kept only as hashes.
 - Nebula's download is pinned to one version with its checksum kept in Lens, unlike programs fetched from a "latest"
   release.
 
-What it doesn't do: a worker node holds the database credential, so it is a trusted peer as a worker on this machine
-is; a compute node is not, and gets no database. The lighthouse must be reachable on UDP `mesh.port` from the nodes
+What it doesn't do: a worker node owns Lens's database (its rows, which aren't encrypted by Lens) and keeps plain
+copies of the files it worked on for a few hours, so it is a trusted peer as a worker on this machine is; a compute
+node is not, and gets no database. The lighthouse must be reachable on UDP `mesh.port` from the nodes
 (a Raspberry Pi behind a carrier's NAT can be a node, not the primary a node reaches directly, unless `mesh.hosts`
 names an address that forwards to it). Nebula's own isolation is upstream's.
 
 ## Refine later
 
 - The upgrade agent on nodes: today a node is updated by running its install line again.
+- Worker nodes running steps that write files (pages, frames, renditions), once those go through the file store, and
+  steps that need a provider's key, through calls to the primary rather than the key.
+- A job a worker node can't do (a cloud speech provider set since it started) handed back to this machine.
+- An auto scaling unit draining its worker on a spot instance's two-minute notice.
 
 - A second transport for networks that block UDP.
 - Rotating the CA.

diff --git a/docs/requirements.md b/docs/requirements.md
@@ -248,6 +248,7 @@ request, to another check that proves the same thing; loosening a requirement or
 | Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` |
 | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` |
 | DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` |
+| A group of machines elsewhere counts only the jobs a worker node takes, and never launches one for others | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_group_elsewhere_counts_only_the_jobs_a_node_takes` |
 | Proxmox VE: clones of a VM template, started and booted through the guest agent, found by tag, stopped and deleted; a clone whose agent never answers says so | [test_scaling_proxmox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_proxmox.py) `test_proxmox_clones_boots_through_the_guest_agent_and_deletes`, `test_a_clone_whose_agent_never_answers_says_so`, `test_a_clone_still_booting_is_pending` |
 | AWS EC2 units launch as spot or on-demand instances, hardened (IMDSv2, encrypted disk, terminate on shutdown), with only the mesh's one-time join line in their user-data | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_a_spot_unit_launches_hardened_and_priced_in_its_zone`, `test_the_driver_is_offered_and_its_settings_are_checked` |
 | A spot unit falls back to on demand when AWS has none; without a price or the mesh, nothing launches | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to`, `test_a_unit_without_a_price_or_the_mesh_launches_nothing` |

diff --git a/docs/scaling.md b/docs/scaling.md
@@ -10,8 +10,8 @@ Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the A
 Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that
 runs more `lens worker` processes where Lens runs. In progress: **AWS EC2** (spot or on demand), **DigitalOcean**
 and **Hetzner Cloud** providers are built (credentials checked, prices, launching, watching and deleting machines),
-and launch once the mesh's install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is
-**Proxmox VE**, clones of a VM template on your own server. Planned: providers for any Linux host over SSH; model hosts
+and their machines join over the [mesh](mesh.md) as worker nodes that transcribe and tell voices apart; so is
+**Proxmox VE**, clones of a VM template on your own server. They haven't been tried on a real cloud machine yet. Planned: providers for any Linux host over SSH; model hosts
 (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making providers and groups in Settings.
 
 Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
@@ -67,8 +67,11 @@ it runs.
 
 A cloud provider's unit is a new machine. Its first boot runs a script (user-data) that sets `LENS_SCALING_UNIT` (its
 name) and `LENS_WORKER_STEPS` (the template's steps), then the mesh's install line with a join token made for that
-unit, usable once within an hour. The machine joins Lens over the [mesh](mesh.md) and starts `lens worker`; the
-database credential travels inside the sealed join answer, so the user-data holds no long-lived secret. Each machine
+unit, usable once within an hour. The machine joins Lens over the [mesh](mesh.md) as a worker node and starts
+`lens worker`; it asks for its own database user over the mesh once it has joined, so the user-data holds no
+long-lived secret. A worker node runs only `transcribe` and `diarize` (it never gets the archive's key:
+[Worker nodes](mesh.md#worker-nodes)), so a group on these providers counts only those steps' jobs in its metrics
+(`unit_steps`), and one whose template names none of them waits and launches nothing. Each machine
 carries Lens's tag (`scaling.tag`: a DigitalOcean tag, a Hetzner label `lens=<tag>`), so Lens's machines can be told
 apart. Letting a unit go deletes its machine; a machine switched off at the provider counts as gone and is deleted.
 
@@ -80,7 +83,7 @@ deleted. Letting a unit go stops the clone and deletes it with its disks. The AP
 start, stop and delete VMs and use their guest agent (the `PVEVMAdmin` role on the template and the clones' pool is
 enough).
 
-Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches
+Until the mesh is on (Settings › Network), a group on these providers waits (`waiting_for` says so) and launches
 nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing.
 
 ## AWS EC2