8000 feat(scaling): Lambda GPU instances as a model host, over HTTPS Lens signs by adeelahmad · Pull Request #675 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
< 8000 react-partial partial-name="global-nav-bar" data-ssr="true" data-attempted-ssr="true" data-react-profiling="false" >

feat(scaling): Lambda GPU instances as a model host, over HTTPS Lens signs - #675

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-azm2mb
Oct 19, 2026
Merged

adeelahmad merged 1 commit into
mainfrom
claude/project-thread-azm2mb

Conversation

@adeelahmad
Copy link
Copy Markdown
Owner

Description

Before: Lambda could only be a worker-node provider. A model group could scale on RunPod, Hugging Face, SageMaker or Bedrock, not on Lambda GPU instances. Two providers on the same cloud account also flagged each other's units as strays.

After: a lambda-models provider launches Lambda GPU instances that run vLLM and serve the group's model. Each unit serves over HTTPS with a certificate Lens signs from a per-group CA. Lens pins that CA (new llm.tls_ca setting) while the unit serves, so calls to a unit's public address are encrypted and no other server can stand in for it. The firewall opens only port 8000, and the group's vLLM key still guards every call. Units count as known when any provider has them, so two providers on one account no longer flag each other's units.

How: scaling.unit_tls creates the group's EC P-256 CA once, keeps its key sealed on scale_group.unit_ca, and signs a leaf certificate per unit with SKI and AKI. LambdaModels (a Lambda subclass) makes a <tag>-models firewall ruleset per region once. Its user-data writes the cert and key and starts vLLM with --ssl-certfile/--ssl-keyfile, with tensor parallel size taken from the instance type. describe checks /health over HTTPS against the CA before the unit serves. Scaler._serve saves and restores llm.tls_ca with the provider, and _others spreads only across units with the same CA. In llm.py, _tls builds a pinned SSL context for chat, listing and llama.cpp calls, and shared() skips providers with a pinned CA.

Tests use the Lambda Cloud OpenAPI spec (refreshed with firewall-rulesets) and a real local TLS server. They cover the firewall rule, one key per group, the CA key staying sealed, a real HTTPS chat against the issued certificate, refusal with no CA or another group's CA, bad tls_ca rejected, no strays across two providers, and restore on scale-in.

Motivation and Context

Adeel's auto-scaling ask named Lambda among the model providers. This was the last named provider not yet built.

Types of changes

  • Bug fix (non-breaking change which fixes an issue): strays across two providers on one account
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (ruff, eslint, prettier).
  • I added or updated tests (test_a_lambda_model_group_serves_https_with_a_certificate_lens_signed).
  • Documentation updated (docs/scaling.md, docs/configuration.md, docs/requirements.md, README).
  • Changelog fragment in changelog.d/scaling-lambda-models.md.

Done questions (AGENTS.md)

  • Goal: merges on green CI.
  • Agents: providers and groups use the existing scaling API and assistant tools. llm.tls_ca is an ordinary setting.
  • Zero setup: off by default. It needs a Lambda API key, like the Lambda node provider.
  • Fewest clicks: same provider and group forms as other drivers.
  • Configurable: llm.tls_ca defaults to none, and the scaler keeps it.
  • Light: no new dependency; cryptography was already requi 8000 red.
  • Private and secure: traffic to units is TLS with a pinned CA, the CA key is sealed, and a unit holds only its own leaf key. Threat model P-SCALING (md and yaml) updated with what voids it.
  • Cost: units launch only within the group's budget and approval rules, as before.
  • User decides: a person's provider pick still wins; defaults are unchanged.
  • Upgrades: new optional unit_ca on scale_group and an optional setting; no migration needed.
  • Honest: docs list it as built, with what's left under refine later.
  • Tested and documented.
  • Open: X.509 and TLS; no new dependency.
  • Reported in the thread.
  • Reuse: the existing Lambda driver, sealing, _serve and spread.

Generated by Claude Code

…signs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ALjNbLobvPe7VwvDL8j7V3
@adeelahmad
adeelahmad merged commit a2c7a90 into main Oct 19, 2026
12 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-azm2mb branch October 19, 2026 05:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0