8000 feat(scaling): spread model calls across a group's running units by adeelahmad · Pull Request #673 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
runs more workers on this machine today; AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud, Lambda GPU
instances, Proxmox VE and Linux hosts over SSH providers are in progress (built, with machines that join over the mesh to transcribe and tell
voices apart, not yet tried on a real cloud), so are RunPod GPUs, Hugging Face Inference Endpoints and Amazon
SageMaker endpoints and Amazon Bedrock provisioned throughput that serve Lens's model calls while they run;
SageMaker endpoints and Amazon Bedrock provisioned throughput that serve Lens's model calls while they run, in turn
across a group's units;
Bedrock's models can be the LLM provider too, and more model hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out))
- **Run on more machines.** Add a Mac with a GPU, a box for the assistant's code or a server with more disk, and
Lens joins it over a private network of its own, a Nebula mesh it issues the certificates for: one install line
Expand Down Expand Up @@ -344,7 +345,7 @@ Next, and not in a pull request yet; each links to where it is tracked:
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
- **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
([local models](docs/local-models.md)).
- **Auto scaling:** spreading model calls across GPUs, and Lambda instances serving a model ([auto scaling](docs/scaling.md)).
- **Auto scaling:** Lambda instances serving a model ([auto scaling](docs/scaling.md)).
- **Speech providers:** their summaries, chapters and entities as optional imports
([speech providers](docs/speech-providers.md)).
- **Storage:** uploads and assets' media in the file store, and a storage key to download
Expand Down
4 changes: 4 additions & 0 deletions changelog.d/scaling-spread.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **Model calls spread across a group's units.** While a model group serves Lens's model calls, every running unit of
it takes them in turn, not only the one Settings › LLM provider shows, so scaling a group out adds capacity. A
group's units now share one key, sealed on the group, and the scaler keeps the new `llm.spread` setting at their
addresses; a unit that can't be reached hands its call to the next ([Auto scaling](docs/scaling.md)).
4 changes: 4 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,10 @@ of their own then: they never borrow the AWS key. The keys need `bedrock:InvokeM
(InvokeEndpointWithResponseStream), as the container writes them. Keys allowed only `bedrock:InvokeModel` or `sagemaker:InvokeEndpoint` still work: their answers come whole. Picking a provider changes nothing until you save; from then on, what Lens asks its model
(summaries, chat, descriptions, with the text they need) goes to that provider.

`llm.spread` sends those calls to more addresses in turn: `{"base_url": ..., "urls": [...]}`, the provider's address
and others serving the same model with the same key. It is used only while `llm.base_url` is that `base_url`, and
auto scaling keeps it while a group's units serve ([Auto scaling](scaling.md)).

## Chat models

People can choose the model a conversation uses, and ask a question again with another (Retry with another model).
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Amazon Bedrock as the LLM provider: chat, tool calls and their results, images, JSON answers and summaries go to Bedrock's Converse API signed with the AWS key, Bedrock's answers and refusals read as an OpenAI-compatible server's (a context too long, tools a model doesn't take), chat answers stream as they are written (ConverseStream; whole when the key may only invoke), and embeddings and speech never borrow the key; each request and answer is held to botocore's model of the API | [test_llm_bedrock.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_llm_bedrock.py) `test_a_chat_is_a_converse_call_and_reads_as_an_openai_reply`, `test_tool_calls_and_their_results_go_as_converse_turns`, `test_a_json_answer_asks_for_the_schema_in_the_system_prompt`, `test_a_summary_goes_to_bedrock_too`, `test_a_streamed_answer_is_converse_stream_as_it_is_written`, `test_a_model_error_mid_stream_says_so`, `test_keys_that_may_only_invoke_get_the_answer_whole`, `test_bedrocks_refusals_read_as_lenss`, `test_bedrock_lends_its_key_to_nothing_and_lists_the_model_it_was_given` |
| Amazon Bedrock provisioned throughput: made with no commitment, named and tagged with the unit, running once in service, failed ones gone with Bedrock's reason, found by name across pages and deleted when it goes; nothing launches without the admin's price; while it serves, Lens's model calls go to it signed; every request and answer held to AWS's model of Bedrock's API and its name patterns | [test_scaling_bedrock.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_bedrock.py) `test_a_unit_is_a_provisioned_throughput_held_until_it_goes`, `test_one_that_fails_says_bedrocks_reason_and_none_launch_without_a_price`, `test_a_long_name_keeps_its_number_and_bedrocks_pattern`, `test_a_bedrock_group_serves_lenss_model_calls_until_it_scales_in` |
| A running model unit is Lens's LLM provider until none runs, then the provider from before comes back; a provider a person picks meanwhile stays | [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_a_running_model_unit_serves_lenss_model_calls_until_none_runs`, `test_a_provider_a_person_picks_while_a_unit_serves_stays` |
| Model calls go to a group's running units in turn (one key per group, sealed on it); a unit that can't be reached hands its call on, and a spread left from another provider sends nothing elsewhere | [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_model_calls_go_to_a_groups_running_units_in_turn`, `test_llm_spread_is_checked` |
| Provider fakes take only the requests the real API would, answer only what it could, and Lens reads only fields the provider documents; AWS runs from launch to scale-in on moto | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_lens_reads_only_fields_the_clouds_answer_with`; [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_lens_reads_only_fields_runpod_answers_with`; [test_scaling_aws_moto.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws_moto.py) `test_a_spot_instance_launches_tagged_hardened_and_is_found_by_its_tag`, `test_an_instance_someone_stopped_is_terminated_so_its_disk_stops_costing`, `test_the_scaler_runs_a_group_on_moto_from_launch_to_scale_in` |
| Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` |

Expand Down
12 changes: 10 additions & 2 deletions docs/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,15 @@ While a model group's unit runs, it is Lens's LLM provider (the template's `use_
scaler points Settings › LLM provider at the unit's address, model and key, keeping the provider that was there
sealed, and puts it back once no unit of such a group runs. When that unit goes and another runs, the other takes
over. A provider a person picks meanwhile stays: Lens doesn't point at a unit again until none has run. Each switch
is in the activity ledger (`scaling.serve`, `scaling.serve.end`). One unit serves at a time; the others stand by.
is in the activity ledger (`scaling.serve`, `scaling.serve.end`).

Model calls go to every running unit of the group in turn, not only the one Settings shows: a group's units serve
with one key (made once, sealed on the group), and the scaler keeps `llm.spread` at the shown unit's address and the
others' (`{"base_url": ..., "urls": [...]}`), so a group scaled out to three units takes three times the calls. A
unit that can't be reached hands its call to the next. A spread is used only while Settings still shows the address
it was made for, so a provider a person picks never sends calls to a unit. Units launched before groups had one key
(Lens before this) keep their own and only serve as the shown unit until they are replaced. Bedrock's units are
models at one address rather than addresses, so a Bedrock group serves from one unit at a time.

## AWS EC2

Expand Down Expand Up @@ -373,7 +381,7 @@ which only that yes in the web app does.

## Refine later

Spreading model calls across a group's units, and serving only some kinds of call (chat, decisions); more model
Serving only some kinds of call (chat, decisions); more model
hosts (Lambda instances serving a model); Bedrock's price per unit-hour from AWS and Bedrock API keys as the
provider's key; Hugging Face's scale to zero and private endpoints; RunPod network volumes, so a model is downloaded once; Proxmox LXC containers; a live euro rate;
preemptible and spot instances at clouds other than AWS and RunPod; draining an AWS spot unit on its two-minute notice
Expand Down
58 changes: 39 additions & 19 deletions fastapi_backend/app/domain/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from __future__ import annotations

import http.client
import itertools
import json
import os
import re
Expand Down Expand Up @@ -72,25 +73,44 @@ def _post(cfg, payload, whole=False):
raise LLMError("no language model is configured (Settings → LLM provider)")
if region := bedrock.region(l.get("base_url")):
return _bedrock(cfg, payload, region)
req = chat_request(cfg, payload, whole)
try:
r = urllib.request.urlopen(req, timeout=l.get("timeout") or 300)
if "vnd.amazon.eventstream" in (r.headers.get("Content-Type") or ""):
return PayloadParts(r)
return r
except urllib.error.HTTPError as e:
said = e.read().decode("utf-8", "replace")
if e.code == 403 and req.full_url.endswith("/invocations-response-stream"):
return _post(cfg, payload, whole=True) # keys that may only InvokeEndpoint, as before answers streamed
if e.code in (400, 413) and FULL_SAYS.search(said):
m = WINDOW_SAYS.search(said)
window = _tokens(next(g for g in m.groups() if g)) if m else None
if window: # what the server said, rather than what it lists or nothing
_WINDOWS[(l["base_url"].rstrip("/"), payload.get("model") or l["model"])] = (time.monotonic(), window)
raise ContextFull(f"{e.code} from the LLM server: {said[:300]}", window) from None
raise LLMError(f"{e.code} from the LLM server: {said[:300]}") from None
except (urllib.error.URLError, TimeoutError, OSError) as e:
raise LLMError(f"can't reach the LLM server at {l['ba 8 se_url&# 10BC8 39;]}: {e}") from None
tries = _spread(cfg)[:2] # a unit that can't be reached hands the call to the next
for n, at in enumerate(tries, 1):
req = chat_request(at, payload, whole)
try:
r = urllib.request.urlopen(req, timeout=l.get("timeout") or 300)
if "vnd.amazon.eventstream" in (r.headers.get("Content-Type") or ""):
return PayloadParts(r)
return r
except urllib.error.HTTPError as e:
said = e.read().decode("utf-8", "replace")
if e.code == 403 and req.full_url.endswith("/invocations-response-stream"):
return _post(at, payload, whole=True) # keys that may only InvokeEndpoint, as before answers streamed
if e.code in (400, 413) and FULL_SAYS.search(said):
m = WINDOW_SAYS.search(said)
window = _tokens(next(g for g in m.groups() if g)) if m else None
if window: # what the server said, rather than what it lists or nothing
_WINDOWS[(l["base_url"].rstrip("/"), payload.get("model") or l["model"])] = (time.monotonic(), window)
raise ContextFull(f"{e.code} from the LLM server: {said[:300]}", window) from None
raise LLMError(f"{e.code} from the LLM server: {said[:300]}") from None
except (urllib.error.URLError, TimeoutError, OSError) as e:
if n == len(tries):
raise LLMError(f"can't reach the LLM server at {at['llm']['base_url']}: {e}") from None


_TURN = itertools.count() # whose turn it is among the addresses of llm.spread


def _spread(cfg):
"""Where a chat call may go, in order: the provider alone, or with `llm.spread` (auto scaling's units serving the
provider's model with its key, scaling.py `_serve`) every one of them, starting with the next in turn. A spread
left from another provider is ignored."""
l = cfg["llm"]
s = l.get("spread") or {}
if not s.get("urls") or s.get("base_url") != l.get("base_url"):
return [cfg]
urls = [s["base_url"], *s["urls"]]
i = next(_TURN) % len(urls)
return [cfg if u == l["base_url"] else {**cfg, "llm": {**l, "base_url": u}} for u in urls[i:] + urls[:i]]


_MODELS = {} # base_url -> (when, {name: what the server says of it}): the server's list, kept a minute
Expand Down
52 changes: 49 additions & 3 deletions fastapi_backend/app/domain/scaling.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import re
import threading
import time
from secrets import token_urlsafe

from . import activity, budgets, jobs, schedule, settings, store
from . import scaling_bedrock, scaling_clouds, scaling_models, scaling_proxmox, scaling_sagemaker, scaling_ssh # noqa: F401 - registers their drivers
Expand Down Expand Up @@ -1056,14 +1057,29 @@ def can_join(cfg):
return bool(getattr(mesh, "make_token", None) and getattr(mesh, "join_line", None))


def unit_key(db, cfg, gid):
"""The API key a model group's units serve with (RunPod and Hugging Face give it to vLLM), made once and kept
sealed on the group: one key for all of them, so Lens's model calls can go to any of its units (Scaler._serve)."""
row = db.one("SELECT unit_key FROM $r", r=R("scale_group", gid)) or {}
ctx = f"scale_group:{gid}.unit_key"
if row.get("unit_key"):
return settings.unseal(cfg, row["unit_key"], ctx)
key = token_urlsafe(32)
db.q("UPDATE $r SET unit_key = $k", r=R("scale_group", gid), k=settings.seal(cfg, key, ctx))
return key


def bootstrap(db, cfg, g, name):
"""How a unit joins Lens: a worker started here gets its name and inherits the rest; a machine elsewhere gets the
mesh's install line (`LENS_JOIN=… LENS_JOIN_TOKEN=… sh install.sh`) with a one-time join token good for an hour,
named for the unit, so the provider's user-data carries no long-lived secret: the node asks for its database
user over the mesh once it has joined (docs/mesh.md, Worker nodes). {env, line, token_id, tag} (line and token_id
None without the mesh)."""
boot = {"env": {"LENS_SCALING_UNIT": name}, "line": None, "token_id": None, "tag": conf(cfg)["tag"]}
if g.get("kind") == "model" or not can_join(cfg): # a model server is reached over HTTPS: it joins nothing
if g.get("kind") == "model": # a model server is reached over HTTPS: it joins nothing
boot["key"] = unit_key(db, cfg, g["id"])
return boot
if not can_join(cfg):
return boot
mesh = mesh_module()
make, line = mesh.make_token, mesh.join_line
Expand Down Expand Up @@ -1190,13 +1206,17 @@ def _serve(self, cfg):
self.log("scaling: no model unit runs; the LLM provider is the one from before again")
if serving or row.get("previous"):
db.q("UPSERT $r SET serving = NONE, previous = NONE", r=ROW)
self._spread(cfg, None)
return
if serving.get("base_url") and llm.get("base_url") != serving["base_url"]:
if not serving.get("held"): # a person picked another provider: theirs stays
db.q("UPSERT $r SET serving.held = true, previous = NONE", r=ROW)
self._spread(cfg, None)
return
served = [n for n in units if n["endpoint"]["base_url"] == serving.get("base_url")]
if served: # still served by a unit that runs
self._spread(cfg, self._others(cfg, units, endpoint_of(cfg, served[0])))
return
if serving.get("base_url") in {n["endpoint"]["base_url"] for n in units}:
return # still served by a unit that runs
n = units[0]
e = endpoint_of(cfg, n)
if not serving:
Expand All @@ -1209,6 +1229,32 @@ def _serve(self, cfg):
refs = [_ref("scale_node", n["id"]), _ref("scale_group", n["group"])]
activity.record("run", "scaling.serve", refs, cfg, db, detail={"unit": n["name"], "model": e["model"]})
self.log(f"scaling: model calls go to {n['name']} ({e['model']})")
self._spread(cfg, self._others(cfg, units, e))

@staticmethod
def _others(cfg, units, e):
"""The served unit's endpoint and the addresses of the other running units that serve the same model with the
same key (a group's units share one: unit_key), which take Lens's model calls in turn with it."""
urls = set()
for n in units:
o = endpoint_of(cfg, n)
if (
o
and o["base_url"] != e["base_url"]
and o.get("model") == e.get("model")
and (o.get("api_key") or "") == (e.get("api_key") or "")
):
urls.add(o["base_url"])
return {"base_url": e["base_url"], "urls": sorted(urls)} if urls else None

def _spread(self, cfg, want):
"""Keeps `llm.spread` (llm.py `_spread`) at `want`: the served address and the others its calls go to in turn."""
if (cfg.get("llm") or {}).get("spread") != want:
settings.save(self.db, cfg, "llm", {"spread": want}, "auto scaling")
n = len(want["urls"]) + 1 if want else 1
self.log(
f"scaling: model calls go to {n} unit{'s' if n > 1 else ''} in turn" if want else "scaling: model calls go to one place"
)

def _group(self, cfg, g, now, did):
db = self.db
Expand Down
4 changes: 2 additions & 2 deletions fastapi_backend/app/domain/scaling_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def check(self, p):
return {"pods": len(pods) if isinstance(pods, list) else 0}

def launch(self, p, template, name, boot):
s, key = p["settings"], secrets.token_urlsafe(32)
s, key = p["settings"], boot.get("key") or secrets.token_urlsafe(32)
model = template["model"]
args = ["--model", model, "--served-model-name", model, "--port", str(PORT)]
if (template.get("gpu_count") or 1) > 1:
Expand Down Expand Up @@ -204,7 +204,7 @@ def price(self, p, template):
return float(hw["pricePerHour"]) if hw.get("pricePerHour") is not None else None

def launch(self, p, template, name, boot):
ns, key, model = self.namespace(p), secrets.token_urlsafe(32), template["model"]
ns, key, model = self.namespace(p), boot.get("key") or secrets.token_urlsafe(32), template["model"]
hw = self.hardware(p, ns, template["hardware"])
if len(name) > 32: # endpoint names are at most 32 characters: keep the unit's number on the end
n = name.rsplit("-", 1)[-1]
Expand Down
Loading
Loading
0