adeelahmad · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026
diff --git a/README.md b/README.md
@@ -184,8 +184,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
   schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity
   ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage
   groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today;
-  DigitalOcean and Hetzner Cloud providers are in progress (built, launching once the mesh's install line lands), and
-  AWS, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
+  DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching once the mesh's install
+  line lands), and AWS, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
   [checks](docs/requirements.md#scale-out))
 - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a
   notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine,
@@ -262,7 +262,7 @@ Next, and not in a pull request yet; each links to where it is tracked:
   topics, recordings and the other things that have pages ([notes](docs/notes.md)).
 - **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
   ([local models](docs/local-models.md)).
-- **Auto scaling:** providers for AWS, Proxmox and Linux hosts over SSH, joining over the Nebula mesh, model hosts
+- **Auto scaling:** providers for AWS and Linux hosts over SSH, joining over the Nebula mesh, model hosts
   (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings
   ([auto scaling](docs/scaling.md)).
 - **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the

diff --git a/docs/requirements.md b/docs/requirements.md
@@ -224,6 +224,7 @@ 
8000
request, to another check that proves the same thing; loosening a requirement or
 | Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` |
 | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` |
 | DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` |
+| Proxmox VE: clones of a VM template, started and booted through the guest agent, found by tag, stopped and deleted; a clone whose agent never answers says so | [test_scaling_proxmox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_proxmox.py) `test_proxmox_clones_boots_through_the_guest_agent_and_deletes`, `test_a_clone_whose_agent_never_answers_says_so`, `test_a_clone_still_booting_is_pending` |
 | Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` |
 
 ## Keep it up to date and backed up

diff --git a/docs/scaling.md b/docs/scaling.md
@@ -10,8 +10,8 @@ Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the A
 Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that
 runs more `lens worker` processes where Lens runs. In progress: **DigitalOcean** and **Hetzner Cloud** providers are
 built (credentials checked, list prices, launching, watching and deleting machines), and launch once the mesh's
-install line, which another machine joins Lens with, lands ([the mesh](mesh.md)). Planned: providers for AWS, Proxmox
-and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making
+install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is **Proxmox VE**, clones of a
+VM template on your own server. Planned: providers for AWS and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making
 providers and groups in Settings.
 
 Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
@@ -56,6 +56,7 @@ carries none of them.
 | --- | --- | --- | --- |
 | `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) |
 | `digitalocean` | droplets in `region`, with `token` (sealed), optional `ssh_keys` and `vpc_uuid` | DigitalOcean's hourly list price | `size` (`s-1vcpu-2gb`), `image` (`ubuntu-24-04-x64`), `steps` |
+| `proxmox` (Proxmox VE) | clones of a VM template on `node`, at `url` with an API token (`token_id`, `token_secret` sealed); `verify_tls` (on) and `ca_cert` for a private CA | nothing, unless you give it a price | `vm_template` (its id), `full_clone` (off: linked), `storage`, `pool`, `steps` |
 | `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` |
 
 A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where
@@ -70,6 +71,14 @@ database credential travels inside the sealed join answer, so the user-data hold
 carries Lens's tag (`scaling.tag`: a DigitalOcean tag, a Hetzner label `lens=<tag>`), so Lens's machines can be told
 apart. Letting a unit go deletes its machine; a machine switched off at the provider counts as gone and is deleted.
 
+A Proxmox unit is a clone of `vm_template`, tagged with Lens's tag and started. Its template needs the QEMU guest
+agent (`qemu-guest-agent` installed, and the VM's Guest Agent option on): once the agent answers, Lens runs the same
+first-boot script in the clone through it, then tags it `lens-booted`, so nothing is put on the server's storage. The
+script stays in Lens's memory until then; a clone whose boot was cut short, by Lens restarting, never joins and is
+deleted. Letting a unit go stops the clone and deletes it with its disks. The API token needs to clone, configure,
+start, stop and delete VMs and use their guest agent (the `PVEVMAdmin` role on the template and the clones' pool is
+enough).
+
 Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches
 nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing.
 
@@ -161,7 +170,7 @@ Turning scaling on (`change_settings scaling`) always asks.
 
 ## Refine later
 
-Providers for Proxmox, SSH hosts and model hosts; the AWS provider (in its own pull request); revoking a unit's mesh
+Providers for SSH hosts and model hosts; Proxmox LXC containers; the AWS provider (in its own pull request); revoking a unit's mesh
 certificate when it goes; reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro
 rate; spot and preemptible instances; warm pools; predictive scaling; health checks beyond
 joining; prices fetched from providers' pricing APIs.
diff --git a/fastapi_backend/app/domain/scaling.py b/fastapi_backend/app/domain/scaling.py
@@ -44,7 +44,7 @@
 import time
 
 from . import activity, budgets, jobs, schedule, settings, store
-from . import scaling_clouds  # noqa: F401 - registers the cloud drivers
+from . import scaling_clouds, scaling_proxmox  # noqa: F401 - registers their drivers
 from . import scaling_drivers as drivers
 from .store import R
 

diff --git a/fastapi_backend/app/domain/scaling_proxmox.py b/fastapi_backend/app/domain/scaling_proxmox.py
@@ -0,0 +1,207 @@
+"""Proxmox VE as an auto scaling provider (scaling_drivers.py, docs/scal
8000
ing.md): each unit is a clone of a VM template
+on your own Proxmox server, which joins Lens over the mesh and runs `lens worker`; letting the unit go stops and
+deletes the clone. It costs nothing unless you give the provider a price.
+
+The template needs the QEMU guest agent (`qemu-guest-agent`, with the VM's Guest Agent option on): once the clone is
+up, Lens runs the first-boot script (scaling_drivers.user_data) in it through the agent, so nothing has to be put on
+the Proxmox server's storage. Lens calls the Proxmox API with an API token (`user@realm!name` and its secret), over
+httpx, so a provider of this kind installs nothing.
+"""
+
+from __future__ import annotations
+
+import ssl
+import threading
+import time
+from urllib.parse import quote
+
+import httpx
+
+from . import jobs
+from .scaling_drivers import Driver, DriverError, _field, register, user_data
+
+TIMEOUT = 30.0
+CLONE_SECONDS = 600  # a full clone of a big disk takes a while; a linked clone, seconds
+AGENT_SECONDS = 600  # how long the clone has to boot far enough for its guest agent to answer
+BOOTED = "lens-booted"  # the tag a clone gets once its first-boot script ran
+
+
+class Proxmox(Driver):
+    id = "proxmox"
+    label = "Proxmox VE"
+    about = "Clones of a VM template on your Proxmox server, which join Lens over the mesh and run Lens workers."
+    priced = False
+    needs_join = True
+    fields = {
+        "url": _field("Address", required=True, help="the Proxmox API, like https://pve.home:8006"),
+        "node": _field("Node", required=True, help="the Proxmox node clones run on, like pve"),
+        "token_id": _field("API token id", required=True, help="user@realm!name, like lens@pve!scaling"),
+        "token_secret": _field("API token secret", required=True, secret=True),
+        "verify_tls": _field("Check the certificate", "boolean", default=True, help="off only for a test server"),
+        "ca_cert": _field("CA certificate", help="the PEM of the CA that signed the server's certificate, when it isn't public"),
+    }
+    template = {
+        "vm_template": _field("VM template", "integer", required=True, help="the id of the template to clone"),
+        "full_clone": _field("Full clone", "boolean", default=False, help="a full copy of the disk; off: a linked clone"),
+        "storage": _field("Storage", help="where a full clone's disk goes; none: the template's"),
+        "pool": _field("Pool", help="the resource pool clones join"),
+        "steps": _field(
+            "Steps",
+            "list",
+            help="the steps these workers run (transcribe, analyze, embed...); none: every step",
+            options=sorted(jobs.STEPS),
+        ),
+    }
+    transport: httpx.BaseTransport | None = None  # tests hand in an httpx.MockTransport
+    background = True  # tests run the boot inline
+    pause = staticmethod(time.sleep)

+    def __init__(self):
+        self.booting: dict[str, str | None] = {}  # clone name -> None while it boots, or why its boot failed
+        self.lock = threading.Lock()
+
+    # ---------- the API ----------
+    def _verify(self, p):
+        s = p["settings"]
+        if not s.get("verify_tls", True):
+            return False
+        if s.get("ca_cert"):
+            try:
+                return ssl.create_default_context(cadata=s["ca_cert"])
+            except ssl.SSLError as e:
+                raise DriverError(f"the CA certificate can't be read: {e.reason}") from None
+        return True
+
+    def _client(self, p):
+        s = p["settings"]
+        return httpx.Client(
+            base_url=s["url"].rstrip("/") + "/api2/json",
+            headers={"Authorization": f"PVEAPIToken={s['token_id']}={p['secrets'].get('token_secret') or ''}"},
+            timeout=TIMEOUT,
+            verify=self._verify(p) if self.transport is None else True,
+            transport=self.transport,
+        )
+
+    def call(self, p, method, path, gone_ok=False, **kw):
+        """The answer's `data`; None when `gone_ok` and the VM isn't there. DriverError otherwise."""
+        try:
+            with self._client(p) as c:
+                r = c.request(method, path, **kw)
+        except httpx.HTTPError as e:
+            raise DriverError(f"couldn't reach Proxmox: {type(e).__name__}") from None
+        said = (r.reason_phrase or "").strip()
+        if gone_ok and (r.status_code == 404 or (r.status_code == 500 and "does not exist" in (said + r.text))):
+            return None
+        if r.status_code >= 400:
+            raise DriverError(f"Proxmox said {r.status_code}" + (f": {said}" if said else ""))
+        return (r.json() if r.content else {}).get("data")
+
+    def _vm(self, node):
+        where, _, vmid = (node.get("external_id") or "").partition("/")
+        return where, vmid
+
+    def _wait(self, p, where, upid, seconds):
+        """Until the task ends; DriverError if it failed or took too long."""
+        end = time.monotonic() + seconds
+        while True:
+            st = self.call(p, "GET", f"/nodes/{where}/tasks/{quote(upid, safe='')}/status") or {}
+            if st.get("status") == "stopped":
+                if st.get("exitstatus") != "OK":
+                    raise DriverError(f"Proxmox's task failed: {st.get('exitstatus')}")
+                return
+            if time.monotonic() > end:
+                raise DriverError(f"Proxmox's task didn't finish within {seconds} seconds")
+            self.pause(2)
+
+    # ---------- the driver ----------
+    def check(self, p):
+        version = (self.call(p, "GET", "/version") or {}).get("version")
+        nodes = {n.get("node") for n in self.call(p, "GET", "/nodes") or []}
+        if p["settings"]["node"] not in nodes:
+            raise DriverError(f"Proxmox has no node {p['settings']['node']}; it has {', '.join(sorted(nodes)) or 'none'}")
+        return {"version": version}
+
+    def launch(self, p, template, name, boot):
+        where = p["settings"]["node"]
+        vmid = int(self.call(p, "GET", "/cluster/nextid"))
+        body = {"newid": vmid, "name": name, "full": 1 if template.get("full_clone") else 0}
+        if template.get("full_clone") and template.get("storage"):
+            body["storage"] = template["storage"]
+        if template.get("pool"):
+            body["pool"] = template["pool"]
+        upid = self.call(p, "POST", f"/nodes/{where}/qemu/{template['vm_template']}/clone", data=body)
+        with self.lock:
+            self.booting[name] = None
+        script = user_data(boot, template)
+        if self.background:
+            threading.Thread(target=self._boot, args=(p, where, vmid, upid, name, boot["tag"], script), daemon=True).start()
+        else:
+            self._boot(p, where, vmid, upid, name, boot["tag"], script)
+        return {"id": f"{where}/{vmid}"}
+
+    def _boot(self, p, where, vmid, upid, name, tag, script):
+        """Wait for the clone, tag it, start it, and once its guest agent answers, run the first-boot script in it. The
+        script lives only here, never on the Proxmox server: a clone whose boot was cut short (Lens restarted) never
+        joins, and the scaler deletes it."""
+        vm = f"/nodes/{where}/qemu/{vmid}"
+        try:
+            if upid:
+                self._wait(p, where, upid, CLONE_SECONDS)
+            self.call(p, "PUT", f"{vm}/config", data={"tags": tag})
+            self._wait(p, where, self.call(p, "POST", f"{vm}/status/start"), 120)
+            end = time.monotonic() + AGENT_SECONDS
+            while True:
+                try:
+                    self.call(p, "POST", f"{vm}/agent/ping")
+                    break
+                except DriverError:
+                    if time.monotonic() > end:
+                        raise DriverError(f"the clone's guest agent didn't answer within {AGENT_SECONDS} seconds") from None
+                    self.pause(5)
+            self.call(p, "POST", f"{vm}/agent/exec", data={"command": ["sh", "-c", script]})
+            self.call(p, "PUT", f"{vm}/config", data={"tags": f"{tag};{BOOTED}"})
+            with self.lock:
+                self.booting.pop(name, None)
+        except Exception as e:  # noqa: BLE001 - shown on the unit when the scaler next looks
+            with self.lock:
+                self.booting[name] = str(e)[:300] if isinstance(e, DriverError) else f"{type(e).__name__}: {e}"[:300]
+
+    def describe(self, p, node):
+        with self.lock:
+            boot = self.booting.get(node.get("name"), ...)
+        if boot is None:
+            return {"state": "pending"}  # still cloning or booting here
+        if boot is not ...:
+            raise DriverError(boot)  # the boot failed: shown on the unit, which fails when it doesn't join
+        where, vmid = self._vm(node)
+        st = self.call(p, "GET", f"/nodes/{where}/qemu/{vmid}/status/current", gone_ok=True)
+        if st is None or st.get("status") != "running":
+            return {"state": "gone"}  # stopped, or a clone whose boot this process never finished
+        return {"state": "running"}
+
+    def terminate(self, p, node):
+        with self.lock:
+            self.booting.pop(node.get("name"), None)
+        where, vmid = self._vm(node)
+        if not vmid:
+            return
+        vm = f"/nodes/{where}/qemu/{vmid}"
+        st = self.call(p, "GET", f"{vm}/status/current", gone_ok=True)
+        if st is None:
+            return
+        if st.get("status") == "running":
+            self._wait(p, where, self.call(p, "POST", f"{vm}/status/stop"), 120)
+        upid = self.call(p, "DELETE", vm, gone_ok=True, params={"purge": 1, "destroy-unreferenced-disks": 1})
+        if upid:
+            self._wait(p, where, upid, 300)
+
+    def listing(self, p, tag):
+        vms = self.call(p, "GET", "/cluster/resources", params={"type": "vm"}) or []
+        return [
+            {"id": f"{v.get('node')}/{v.get('vmid')}", "name": v.get("name")}
+            for v in vms
+            if tag in (v.get("tags") or "").split(";") and not v.get("template")
+        ]
+
+
+register(Proxmox())