adeelahmad · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026
diff --git a/README.md b/README.md
@@ -186,8 +186,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
   schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity
   ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage
   groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today;
-  DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching once the mesh's install
-  line lands), and AWS, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
+  AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching
+  once the mesh's install line lands), and Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
   [checks](docs/requirements.md#scale-out))
 - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a
   notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine,
@@ -264,7 +264,7 @@ Next, and not in a pull request yet; each links to where it is tracked:
   topics, recordings and the other things that have pages ([notes](docs/notes.md)).
 - **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
   ([local models](docs/local-models.md)).
-- **Auto scaling:** providers for AWS and Linux hosts over SSH, joining over the Nebula mesh, model hosts
+- **Auto scaling:** providers for Linux hosts over SSH, joining over the Nebula mesh, model hosts
   (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings
   ([auto scaling](docs/scaling.md)).
 - **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the

diff --git a/docs/requirements.md b/docs/requirements.md
@@ -228,6 +228,9 @@ request, to another check that proves the same thing; loosening a requirement or
 | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` |
 | DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` |
 | Proxmox VE: clones of a VM template, started and booted through the guest agent, found by tag, stopped and deleted; a clone whose agent never answers says so | [test_scaling_proxmox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_proxmox.py) `test_proxmox_clones_boots_through_the_guest_agent_and_deletes`, `test_a_clone_whose_agent_never_answers_says_so`, `test_a_clone_still_booting_is_pending` |
+| AWS EC2 units launch as spot or on-demand instances, hardened (IMDSv2, encrypted disk, terminate on shutdown), with only the mesh's one-time join line in their user-data | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_a_spot_unit_launches_hardened_and_priced_in_its_zone`, `test_the_driver_is_offered_and_its_settings_are_checked` |
+| A spot unit falls back to on demand when AWS has none; without a price or the mesh, nothing launches | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to`, `test_a_unit_without_a_price_or_the_mesh_launches_nothing` |
+| When AWS takes a spot instance back, the scaler says why and launches another | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_describe_says_when_aws_took_a_spot_instance_back`, `test_the_scaler_replaces_a_spot_unit_aws_took_back`, `test_terminate_check_and_listing` |
 | Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` |
 
 ## Keep it up to date and backed up

diff --git a/docs/scaling.md b/docs/scaling.md
@@ -8,11 +8,11 @@ launched. Everything it does is in the [activity ledger](activity.md), and its c
 
 Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the API, MCP and assistant tools,
 Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that
-runs more `lens worker` processes where Lens runs. In progress: **DigitalOcean** and **Hetzner Cloud** providers are
-built (credentials checked, list prices, launching, watching and deleting machines), and launch once the mesh's
-install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is **Proxmox VE**, clones of a
-VM template on your own server. Planned: providers for AWS and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making
-providers and groups in Settings.
+runs more `lens worker` processes where Lens runs. In progress: **AWS EC2** (spot or on demand), **DigitalOcean**
+and **Hetzner Cloud** providers are built (credentials checked, prices, launching, watching and deleting machines),
+and launch once the mesh's install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is
+**Proxmox VE**, clones of a VM template on your own server. Planned: providers for any Linux host over SSH; model hosts
+(RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making providers and groups in Settings.
 
 Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
 signed in to the web app. Agents can propose and manage groups, but never approve one.
@@ -55,6 +55,7 @@ carries none of them.
 | Driver | Units | Costs | Template |
 | --- | --- | --- | --- |
 | `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) |
+| `aws` (AWS EC2) | EC2 instances that join over the mesh and run `lens worker` | the spot or on-demand price, an estimate | `instance_type`, `market` (`spot` or `on-demand`), `max_price`, `fallback`, `image`, `disk_gb`, `steps` |
 | `digitalocean` | droplets in `region`, with `token` (sealed), optional `ssh_keys` and `vpc_uuid` | DigitalOcean's hourly list price | `size` (`s-1vcpu-2gb`), `image` (`ubuntu-24-04-x64`), `steps` |
 | `proxmox` (Proxmox VE) | clones of a VM template on `node`, at `url` with an API token (`token_id`, `token_secret` sealed); `verify_tls` (on) and `ca_cert` for a private CA | nothing, unless you give it a price | `vm_template` (its id), `full_clone` (off: linked), `storage`, `pool`, `steps` |
 | `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` |
@@ -82,6 +83,66 @@ enough).
 Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches
 nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing.
 
+## AWS EC2
+
+Lens grows onto AWS when work waits: from a Raspberry Pi at home to as many EC2 instances as the group's max and
+budget allow, and back to none. Each unit is an instance that joins Lens over the [mesh](mesh.md) and runs its worker.
+
+```http
+POST /api/v1/scaling/providers
+{"name": "AWS", "driver": "aws", "settings": {"region": "eu-west-1"},
+ "secrets": {"access_key_id": "AKIA...", "secret_access_key": "..."}}
+```
+
+| Provider setting | Default | |
+| --- | --- | --- |
+| `region` | `us-east-1` | the region units launch in |
+| `access_key_id`, `secret_access_key` | | an IAM user's keys, sealed |
+| `subnet_id` | the default VPC's | where units launch |
+| `security_group_ids` | the VPC's default group | units only reach out (to the mesh's lighthouse), so they need no inbound rule |
+| `key_name` | none | an EC2 key pair, to sign in to units yourself |
+
+| Unit (template) | Default | |
+| --- | --- | --- |
+| `instance_type` | `t4g.medium` | any EC2 type; Arm (Graviton) and x86 both work |
+| `market` | `spot` | `spot`: AWS's spare capacity, often far cheaper, which AWS may take back with two minutes' notice; `on-demand` |
+| `max_price` | the on-demand price | the most a spot unit may cost per hour |
+| `fallback` | `true` | when AWS has no spot capacity of that type, or none under the max price, launch on demand instead |
+| `image` | the latest Ubuntu 24.04 | an AMI; by default Canonical's for the type's architecture, found in the region |
+| `disk_gb` | `30` | the unit's disk (gp3, encrypted) |
+| `steps` | every step | the steps its worker runs |
+
+**Spot instances.** When AWS takes a spot instance back, the scaler sees it gone, records why ("AWS took the spot
+instance back") and launches another, on demand if spot has run out and `fallback` allows. The job its worker was
+running goes back in the queue once its heartbeat stops, so nothing is lost.
+
+**What it costs.** Each unit's price is an estimate: the spot price in the zone it landed in (from
+`DescribeSpotPriceHistory`), or the on-demand list price (from the AWS Price List API). A price you give the provider
+wins over both. Without any price Lens launches nothing, because a unit with no price would never count against the
+budget. AWS bills separately; the ledger's figures are estimates.
+
+**Hardened by default.** Every instance requires IMDSv2, has an encrypted disk that goes with it, terminates itself
+when it shuts down (a stopped one is terminated too, so its disk doesn't keep costing), and is tagged `lens-unit` with
+its name and `lens` with Lens's tag. Its user-data is the one every cloud unit boots with ([machines
+elsewhere](#machines-elsewhere)).
+
+The keys need these permissions (an IAM policy for a user only Lens uses):
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {"Effect": "Allow", "Resource": "*", "Action": [
+      "ec2:DescribeAvailabilityZones", "ec2:DescribeInstanceTypes", "ec2:DescribeImages",
+      "ec2:DescribeSpotPriceHistory", "ec2:DescribeInstances", "ec2:RunInstances",
+      "ec2:CreateTags", "ec2:TerminateInstances", "pricing:GetProducts"]}
+  ]
+}
+```
+
+`pricing:GetProducts` is only needed for on-demand prices you don't give yourself. Lens talks to AWS through botocore
+(Apache-2.0), fetched the first time an AWS provider is saved.
+
 ## Groups
 
 ```http
@@ -170,7 +231,10 @@ Turning scaling on (`change_settings scaling`) always asks.
 
 ## Refine later
 
-Providers for SSH hosts and model hosts; Proxmox LXC containers; the AWS provider (in its own pull request); revoking a unit's mesh
-certificate when it goes; reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro
-rate; spot and preemptible instances; warm pools; predictive scaling; health checks beyond
+Providers for SSH hosts and model hosts; Proxmox LXC containers; revoking a unit's mesh certificate when it goes;
+reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro rate; preemptible and spot
+instances at clouds other than AWS; draining an AWS spot unit on its two-minute notice (the node agent reading the
+instance's metadata) and handing its job back at once; following the spot price as it changes; AWS sign-in with an
+instance role or IAM Identity Center instead of keys; mixed instance types per group; warm pools; predictive scaling;
+health checks beyond
 joining; prices fetched from providers' pricing APIs.
diff --git a/fastapi_backend/app/domain/scaling.py b/fastapi_backend/app/domain/scaling.py
@@ -1188,7 +1188,7 @@ def _reconcile(self, cfg, g, drv, p, n, now, did):
             db.q("UPDATE $r SET error = $e", r=R("scale_node", n["id"]), e=f"{type(e).__name__}: {e}"[:300])
             return True
         if d.get("state") == "gone":
-            why = "stopped at the provider" if state != "launching" else "never came up at the provider"
+            why = d.get("why") or ("stopped at the provider" if state != "launching" else "never came up at the provider")
             _terminate(db, cfg, drv, p, {**n, "external_id": None}, why, state="failed" if state == "launching" else "terminated", now=now)
             did["terminated"] += 1
             return False