8000 feat(scaling): AWS EC2 units, as spot instances or on demand by adeelahmad · Pull Request #559 · adeelahmad/lense · GitHub
[go: up one dir, main page]
Skip to content

feat(scaling): AWS EC2 units, as spot instances or on demand - #559

Merged
adeelahmad merged 7 commits into
mainfrom
claude/project-thread-dtagi8
Oct 18, 2026
Merged

feat(scaling): AWS EC2 units, as spot instances or on demand#559
adeelahmad merged 7 commits into
mainfrom
claude/project-thread-dtagi8

Conversation

@adeelahmad
@adeelahmad adeelahmad commented Oct 18, 2026 •
edited
Loading
Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Builds on #556 (the auto scaling core, merged).

Description

Before: auto scaling could only run more workers on the machine Lens runs on.

After: a scaling group can use an AWS EC2 provider. Lens launches EC2 instances that join it over the Nebula mesh and run lens worker, as spot instances (the default, falling back to on demand when AWS has no spot capacity or none under the max price) or on demand. When AWS takes a spot instance back, the scaler records "AWS took the spot instance back" and launches another; the job goes back in the queue through the existing heartbeat reaper.

How: app/domain/scaling_aws.py implements the core's Driver interface on botocore (Apache-2.0), the new aws extra fetched when the first AWS provider is saved. Prices are labelled estimates: the spot price in the zone the instance landed in, or the on-demand price from the AWS Price List API; a price set on the provider wins, and with no price nothing launches (an unpriced unit would never count against the budget). Instances require IMDSv2, encrypt their disk, terminate on shutdown (a stopped one is terminated too), are tagged lens-unit, and default to the newest Canonical Ubuntu 24.04 AMI for the type's architecture. User-data is only the mesh's one-time join line. The core's reconcile step now keeps the reason a driver gives when a unit is gone.

Units launch only once the mesh can join machines (the Nebula thread's install line). Until then a launch fails before any AWS call, saying to turn the mesh on.

Motivation and Context

Adeel asked for Lens to start on a laptop or a Raspberry Pi and expand onto AWS and other clouds automatically when needed, including spot instances. The Auto scaling thread owns the core and the other providers; this PR is the AWS and spot part, as agreed between the two threads.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (ruff, ruff format, mypy clean).
  • I added or updated tests: tests/domain/test_scaling_aws.py, 7 tests on botocore's Stubber, so every call is checked against EC2's own API model, including one through the scaler.
  • My change requires documentation updates, and I have updated docs/ accordingly (docs/scaling.md "AWS EC2", with the IAM policy).
  • CHANGELOG: n/a, the title says it.

Done questions (AGENTS.md)

  • Goal: AWS and spot are built; units join once the mesh install line lands. Merges on green.
  • Agents: providers and groups are made through /api/v1/scaling, MCP and the assistant's scaling tools (the core's).
  • Zero setup: nothing new for installs; botocore is fetched only when an AWS provider is saved.
  • Fewest clicks: one provider (region and two keys) and a group with defaults (t4g.medium, spot, fallback, Ubuntu AMI).
  • Configurable: every field has a default and is set through the API, MCP or assistant; the web form for providers and groups is the Auto scaling thread's planned Settings work.
  • Light: no new default dependency; nothing runs until scaling is on and a group is approved.
  • Private and secure: keys sealed by the core; IMDSv2, encrypted disks, no long-lived secret in user-data; threat model P-SCALING notes the one-time join token in user-data.
  • Cost: spot by default, on-demand fallback, every unit priced (estimate) into the ledger and budget; no price means no launch.
  • User decides: a person approves each group's terms (the core); scaling is off by default; no existing default changes.
  • Upgrades: n/a, no stored-data shape changes beyond the core's tables.
  • Honest: README and docs say AWS units launch once the mesh joins machines; prices labelled estimates.
  • Tests, docs and generated files updated (uv lock for the extra; no API shape change, so no OpenAPI change).
  • Open standards; botocore is Apache-2.0, so Lens stays MIT.
  • Refine later in docs/scaling.md: drain on the two-minute spot notice, following the spot price, instance roles and Identity Center, mixed instance types, spot at other clouds.
  • Reuse: the core's driver interface, sealing, ledger and budgets; botocore rather than hand-written AWS signing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SETHpH6NUGKT1Bwd6xyPCA

@adeelahmad adeelahmad self-assigned this Oct 18, 2026
Base automatically changed from claude/project-thread-azm2mb to main October 18, 2026 06:33
@claude
feat(scaling): AWS EC2 units, as spot instances or on demand
928e0bc 
Lens can grow from one machine onto AWS when work waits: a scaling group on an AWS
provider launches EC2 instances that join over the mesh and run its worker. Spot
units fall back to on demand when AWS has no spot capacity; when AWS takes one back
the scaler says why and launches another. Prices are estimates (zone spot price or
the Price List API); with no price nothing launches. Instances require IMDSv2,
encrypt their disk and terminate on shutdown. botocore is the new `aws` extra,
fetched when the first AWS provider is saved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SETHpH6NUGKT1Bwd6xyPCA
@adeelahmad
adeelahmad force-pushed the claude/project-thread-dtagi8 branch from 2d618ff to 928e0bc Compare October 18, 2026 06:34
@adeelahmad
adeelahmad marked this pull request as ready for review October 18, 2026 06:34
@adeelahmad adeelahmad mentioned this pull request Oct 18, 2026
Merged
19 of 23 tasks
@adeelahmad Claude
adeelahmad commented Oct 18, 2026
Copy link
Copy Markdown
Owner Author

Clicks and keystrokes per feature failed on 928e0bc: the computer survey's first-question journey timed out (locator.waitFor: Timeout 60000ms exceeded) waiting for the first answer. The phone survey in the same job passed that journey, and every other journey passed on both.

I don't think this is this PR's failure. The diff only adds the AWS driver (scaling_aws.py, which imports no SDK until a provider is saved), one line in the scaler's reconcile step, and docs. Nothing in it touches chat or the setup wizard. With this PR merged into today's main, I ran ci/clicks/run.sh --only install,first-admin,setup-wizard,first-question locally three times, and first-question passed each time (0 clicks, 43 keys, as in the baseline). PRs #569, #553 and #565 on main after #556 passed this job too.

No fix exists for it yet. I'm re-running the failed job once.

Generated by Claude Code

claude added 5 commits October 18, 2026 07:45
@claude
Merge remote-tracking branch 'origin/main' into claude/project-thread…
173a311 
…-dtagi8
@claude
Merge remote-tracking branch 'origin/main' into claude/project-thread…
c4acd90 
…-dtagi8
@claude
docs(readme): list answering questions about Lens itself as built (po…
4460b97 
…rt of #578)

The requirements check fails on main since #400 was released; the same
change as #578, a no-op once that merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SETHpH6NUGKT1Bwd6xyPCA
@claude
Merge remote-tracking branch 'origin/main' into claude/project-thread…
7fb3819 
…-dtagi8

# Conflicts:
#	README.md
@claude
Merge remote-tracking branch 'origin/main' into claude/project-thread…
254c491 
…-dtagi8

# Conflicts:
#	README.md
#	docs/requirements.md
#	docs/scaling.md
@adeelahmad
adeelahmad merged commit b9ad68d into main Oct 18, 2026
20 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-dtagi8 branch October 18, 2026 12:05
@adeelahmad adeelahmad mentioned this pull request Oct 18, 2026
Merged
21 of 23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@adeelahmad adeelahmad

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@adeelahmad @claude
0