Repository navigation
feat(scaling): Lambda GPU instances as a model host, over HTTPS Lens signs - #675
Merged
Merged
Conversation
…signs Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ALjNbLobvPe7VwvDL8j7V3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Before: Lambda could only be a worker-node provider. A model group could scale on RunPod, Hugging Face, SageMaker or Bedrock, not on Lambda GPU instances. Two providers on the same cloud account also flagged each other's units as strays.
After: a
lambda-modelsprovider launches Lambda GPU instances that run vLLM and serve the group's model. Each unit serves over HTTPS with a certificate Lens signs from a per-group CA. Lens pins that CA (newllm.tls_casetting) while the unit serves, so calls to a unit's public address are encrypted and no other server can stand in for it. The firewall opens only port 8000, and the group's vLLM key still guards every call. Units count as known when any provider has them, so two providers on one account no longer flag each other's units.How:
scaling.unit_tlscreates the group's EC P-256 CA once, keeps its key sealed onscale_group.unit_ca, and signs a leaf certificate per unit with SKI and AKI.LambdaModels(aLambdasubclass) makes a<tag>-modelsfirewall ruleset per region once. Its user-data writes the cert and key and starts vLLM with--ssl-certfile/--ssl-keyfile, with tensor parallel size taken from the instance type.describechecks/healthover HTTPS against the CA before the unit serves.Scaler._servesaves and restoresllm.tls_cawith the provider, and_othersspreads only across units with the same CA. Inllm.py,_tlsbuilds a pinned SSL context for chat, listing and llama.cpp calls, andshared()skips providers with a pinned CA.Tests use the Lambda Cloud OpenAPI spec (refreshed with firewall-rulesets) and a real local TLS server. They cover the firewall rule, one key per group, the CA key staying sealed, a real HTTPS chat against the issued certificate, refusal with no CA or another group's CA, bad
tls_carejected, no strays across two providers, and restore on scale-in.Motivation and Context
Adeel's auto-scaling ask named Lambda among the model providers. This was the last named provider not yet built.
Types of changes
!in the title)Checklist:
test_a_lambda_model_group_serves_https_with_a_certificate_lens_signed).docs/scaling.md,docs/configuration.md,docs/requirements.md, README).changelog.d/scaling-lambda-models.md.Done questions (AGENTS.md)
llm.tls_cais an ordinary setting.llm.tls_cadefaults to none, and the scaler keeps it.cryptographywas already requi 8000 red.unit_caonscale_groupand an optional setting; no migration needed._serveand spread.Generated by Claude Code