8000 feat(scaling): Lambda GPU instances as a model host, over HTTPS Lens signs by adeelahmad · Pull Request #675 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
runs more workers on this machine today; AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud, Lambda GPU
instances, Proxmox VE and Linux hosts over SSH providers are in progress (built, with machines that join over the mesh to transcribe and tell
voices apart, not yet tried on a real cloud), so are RunPod GPUs, Hugging Face Inference Endpoints and Amazon
SageMaker endpoints and Amazon Bedrock provisioned throughput that serve Lens's model calls while they run, in turn
SageMaker endpoints, Lambda GPU instances and Amazon Bedrock provisioned throughput that serve Lens's model calls while they run, in turn
across a group's units;
Bedrock's models can be the LLM provider too, and more model hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out))
- **Run on more machines.** Add a Mac with a GPU, a box for the assistant's code or a server with more disk, and
Expand Down Expand Up @@ -345,7 +345,6 @@ Next, and not in a pull request yet; each links to where it is tracked:
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
- **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
([local models](docs/local-models.md)).
- **Auto scaling:** Lambda instances serving a model ([auto scaling](docs/scaling.md)).
- **Speech providers:** their summaries, chapters and entities as optional imports
([speech providers](docs/speech-providers.md)).
- **Storage:** uploads and assets' media in the file store, and a storage key to download
Expand Down
4 changes: 4 additions & 0 deletions changelog.d/scaling-lambda-models.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **Lambda as a model host.** A `lambda-models` provider runs a model with vLLM on Lambda GPU instances, which serve
Lens's model calls while they run. Each instance serves HTTPS with a certificate signed by its group's own
certificate authority, which Lens makes and keeps sealed, and Lens trusts only that authority for it (the new
`llm.tls_ca` setting); a firewall ruleset Lens makes once per region opens the port ([Auto scaling](docs/scaling.md)).
5 changes: 5 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,11 @@ of their own then: they never borrow the AWS key. The keys need `bedrock:InvokeM
and others serving the same model with the same key. It is used only while `llm.base_url` is that `base_url`, and
auto scaling keeps it while a group's units serve ([Auto scaling](scaling.md)).

`llm.tls_ca` is the one certificate authority (PEM) the provider's certificate must be signed by, whatever name the
certificate carries; the usual public authorities are then not trusted for it. Auto scaling sets it while a Lambda
model unit serves (its group's own authority, [Auto scaling](scaling.md)); search by meaning and the voice don't borrow
such a provider.

## Chat models

People can choose the model a conversation uses, and ask a question again with another (Retry with another model).
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Amazon Bedrock provisioned throughput: made with no commitment, named and tagged with the unit, running once in service, failed ones gone with Bedrock's reason, found by name across pages and deleted when it goes; nothing launches without the admin's price; while it serves, Lens's model calls go to it signed; every request and answer held to AWS's model of Bedrock's API and its name patterns | [test_scaling_bedrock.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_bedrock.py) `test_a_unit_is_a_provisioned_throughput_held_until_it_goes`, `test_one_that_fails_says_bedrocks_reason_and_none_launch_without_a_price`, `test_a_long_name_keeps_its_number_and_bedrocks_pattern`, `test_a_bedrock_group_serves_lenss_model_calls_until_it_scales_in` |
| A running model unit is Lens's LLM provider until none runs, then the provider from before comes back; a provider a person picks meanwhile stays | [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_a_running_model_unit_serves_lenss_model_calls_until_none_runs`, `test_a_provider_a_person_picks_while_a_unit_serves_stays` |
| Model calls go to a group's running units in turn (one key per group, sealed on it); a unit that can't be reached hands its call on, and a spread left from another provider sends nothing elsewhere | [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_model_calls_go_to_a_groups_running_units_in_turn`, `test_llm_spread_is_checked` |
| Lambda as a model host: GPU instances run vLLM over HTTPS with a certificate the group's own CA signed (its key sealed on the group), behind a firewall ruleset made once per region and the group's key; Lens's calls trust only that CA, and fail with none or another group's, tried against a real HTTPS server | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_lambda_model_group_serves_https_with_a_certificate_lens_signed`, `test_lens_reads_only_fields_the_clouds_answer_with` |
| Provider fakes take only the requests the real API would, answer only what it could, and Lens reads only fields the provider documents; AWS runs from launch to scale-in on moto | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_lens_reads_only_fields_the_clouds_answer_with`; [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_lens_reads_only_fields_runpod_answers_with`; [test_scaling_aws_moto.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws_moto.py) `test_a_spot_instance_launches_tagged_hardened_and_is_found_by_its_tag`, `test_an_instance_someone_stopped_is_terminated_so_its_disk_stops_costing`, `test_the_scaler_runs_a_group_on_moto_from_launch_to_scale_in` |
| Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` |

Expand Down
24 changes: 19 additions & 5 deletions docs/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ checked, prices, launching, watching and deleting machines), and their machines
worker nodes that transcribe and tell voices apart; so are **Proxmox VE**, clones of a VM template on your own server,
and **Linux hosts over SSH**, workers started on machines you have. They haven't been tried on a real cloud machine
yet. Also in progress: **RunPod** GPUs, **Hugging Face** Inference Endpoints and ** F438 Amazon SageMaker** real-time
endpoints, which serve a model with vLLM, and **Amazon Bedrock** provisioned throughput; each takes Lens's model calls
while one runs. Planned: Lambda instances serving a model.
endpoints and **Lambda** GPU instances, which serve a model with vLLM, and **Amazon Bedrock** provisioned throughput;
each takes Lens's model calls while one runs.

Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
signed in to the web app. Agents can propose and manage groups, but never approve one.
Expand Down Expand Up @@ -85,6 +85,7 @@ carries none of them.
| `ssh` (Linux hosts over SSH) | `lens worker` started on `hosts` (`user@host` or `user@host:port`), signed in with `private_key` (sealed; `passphrase` if it has one) | nothing, unless you give it a price | `per_host` (1): workers on each host; `steps` |
| `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` |
| `lambda` (Lambda, GPU instances) | instances in `region` (`us-east-1`), with `token` (a Lambda Cloud API key, sealed) and `ssh_key` (the name of a key in your Lambda account; none: its first) | Lambda's `price_cents_per_hour` for the type | `instance_type` (`gpu_1x_a10`), `steps` |
| `lambda-models` (Lambda, a model host) | GPU instances serving `model` with vLLM over HTTPS, in `region` with `token` (sealed) and `ssh_key` as `lambda`; `hf_token` (sealed) for gated models | Lambda's `price_cents_per_hour` for the type | `model`, `instance_type` (`gpu_1x_a10`), `max_model_len`, `image` (vLLM's), `args`, `use_as_provider` (on) |

A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where
it runs.
Expand All @@ -100,7 +101,9 @@ go itself, since it may not have started them: a person signed in to the web app
stays a few seconds) or `POST /api/v1/scaling/providers/{id}/strays/let-go` with the stray's `id`, which terminates it
at the provider. An agent asks for that (`let_go_scaling_stray` in the assistant), and the ask waits for the person's
yes on the Approvals page; an API key or an app gets 403. A unit Lens let go within the last hour isn't counted, since a provider may
still show it while it goes. When the provider can't be asked, `list_error` says why.
still show it while it goes. A unit of any of Lens's providers counts, so two providers on one account (Lambda's
workers and its model hosts) don't take each other's units for strays. When the provider can't be asked, `list_error`
says why.

### Machines elsewhere

Expand Down Expand Up @@ -163,6 +166,15 @@ Face's address, and vLLM asks for the unit's own key, which goes to the endpoint
Face says the endpoint runs. One that fails, is paused or scaled to zero at Hugging Face counts as gone and is
deleted, with Hugging Face's reason (a gated model it may not download, say) in the unit's `why`.

A Lambda model unit is a GPU instance whose first boot runs vLLM's image with Docker (which Lambda's image comes with),
serving the template's `model` over HTTPS on port 8000, split across the instance's GPUs (`gpu_8x_...` runs eight
tensor-parallel shards). Lambda gives an instance no name a public certificate authority would vouch for, so each
group has a certificate authority of its own: Lens makes it once, keeps its key sealed on the group, and signs a
certificate for each unit, which goes to the instance in its first-boot script with the unit's key. While the unit
serves, `llm.tls_ca` holds the group's certificate, and Lens's calls trust only a certificate it signed. A firewall
ruleset Lens makes once per region (`<tag>-models`) opens port 8000 on the instances launched with it; vLLM asks every
call for the group's key. The unit is running once vLLM answers its health check over HTTPS.

A SageMaker unit is a real-time endpoint in your AWS account serving the template's `model` with AWS's vLLM container
for SageMaker on one `instance_type` instance. It is three things at SageMaker, a model, an endpoint configuration and
the endpoint, all named after the unit and tagged `lens-unit` with its name and `lens` with Lens's tag; terminating a
Expand Down Expand Up @@ -216,7 +228,9 @@ answers for Bedrock, holding each request and answer to AWS's model of Bedrock's
model's patterns; Lens's Converse calls are held to it the same way.
Proxmox's fake is held the same way to Proxmox's own schema of its API, converted from its API viewer (forms
read by the types the schema gives). SSH hosts have no API document: each test host is a real SSH server (Paramiko's, a
test-only dependency) that checks the client's key and runs the driver's commands in a real shell.
test-only dependency) that checks the client's key and runs the driver's commands in a real shell. A Lambda model
unit's certificate is tried against a real HTTPS server on the test machine: Lens's calls go through with the group's
certificate authority and fail with no `llm.tls_ca` or another group's.

While a model group's unit runs, it is Lens's LLM provider (the template's `use_as_provider`, on by default): the
scaler points Settings › LLM provider at the unit's address, model and key, keeping the provider that was there
Expand Down Expand Up @@ -382,7 +396,7 @@ which only that yes in the web app does.
## Refine later

Serving only some kinds of call (chat, decisions); more model
hosts (Lambda instances serving a model); Bedrock's price per unit-hour from AWS and Bedrock API keys as the
hosts; Bedrock's price per unit-hour from AWS and Bedrock API keys as the
provider's key; Hugging Face's scale to zero and private endpoints; RunPod network volumes, so a model is downloaded once; Proxmox LXC containers; a live euro rate;
preemptible and spot instances at clouds other than AWS and RunPod; draining an AWS spot unit on its two-minute notice
(the node agent reading the instance's metadata) and handing its job back at once; following the spot price as it
Expand Down
26 changes: 22 additions & 4 deletions fastapi_backend/app/domain/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import json
import os
import re
import ssl
import time
import urllib.error
import urllib.request
Expand Down Expand Up @@ -77,7 +78,7 @@ def _post(cfg, payload, whole=False):
for n, at in enumerate(tries, 1):
req = chat_request(at, payload, whole)
try:
r = urllib.request.urlopen(req, timeout=l.get("timeout") or 300)
r = urllib.request.urlopen(req, timeout=l.get("timeout") or 300, **_tls(l))
if "vnd.amazon.eventstream" in (r.headers.get("Content-Type") or ""):
return PayloadParts(r)
return r
Expand All @@ -97,6 +98,23 @@ def _post(cfg, payload, whole=False):
raise LLMError(f"can't reach the LLM server at {at['llm']['base_url']}: {e}") from None


_CONTEXTS = {} # llm.tls_ca -> its SSL context


def _tls(l):
"""urlopen's TLS for the provider: with `llm.tls_ca` (an auto scaling unit's group CA, scaling.unit_tls), only a
certificate that CA signed, whatever name it carries; else Python's defaults."""
ca = l.get("tls_ca")
if not ca:
return {}
if ca not in _CONTEXTS:
ctx = ssl.create_default_context(cadata=ca)
ctx.check_hostname = False # the CA is the group's own and signs only its units, which have no name to check
_CONTEXTS.clear()
_CONTEXTS[ca] = ctx
return {"context": _CONTEXTS[ca]}


_TURN = itertools.count() # whose turn it is among the addresses of llm.spread


Expand Down Expand Up @@ -145,7 +163,7 @@ def shared(cfg):
they have no server of their own; (None, None) for SageMaker or Bedrock, which Lens calls for chat only and whose
AWS key is only ever used to sign."""
l = cfg.get("llm") or {}
if aws(cfg):
if aws(cfg) or l.get("tls_ca"): # an auto scaling unit with its own CA serves chat only
return None, None
return l.get("base_url"), _key(l)

Expand Down Expand Up @@ -317,7 +335,7 @@ def _listing(cfg, ttl=60):
if hit and time.monotonic() - hit[0] < ttl:
return hit[1]
try:
with urllib.request.urlopen(urllib.request.Request(url + "/models", headers=_headers(cfg)), timeout=10) as r:
with urllib.request.urlopen(urllib.request.Request(url + "/models", headers=_headers(cfg)), timeout=10, **_tls(l)) as r:
listed = {str(m["id"]): m for m in json.load(r).get("data") or [] if isinstance(m, dict) and m.get("id")}
except (urllib.error.URLError, TimeoutError, OSError, ValueError, AttributeError) as e:
raise LLMError(f"can't list the models at {l['base_url']}: {e}") from None
Expand Down Expand Up @@ -354,7 +372,7 @@ def _llamacpp_window(cfg):
give (only the context the model was trained for)."""
root = re.sub(r"/v1/?$", "", cfg["llm"]["base_url"].rstrip("/"))
try:
with urllib.request.urlopen(urllib.request.Request(root + "/props", headers=_headers(cfg)), timeout=10) as r:
with urllib.request.urlopen(urllib.request.Request(root + "/props", headers=_headers(cfg)), timeout=10, **_tls(cfg["llm"])) as r:
j = json.load(r)
except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
raise LLMError(f"can't ask llama.cpp's server for its context: {e}") from None
Expand Down
Loading
Loading
0