8000 feat(scaling): a unit that goes leaves the mesh by adeelahmad · Pull Request #669 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog.d/scaling-unmesh.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **A scaling unit that goes leaves the mesh.** When auto scaling lets a unit go, or one fails or never comes up, its
unspent join token is withdrawn and the machine that joined with it is taken off the mesh, as removing it on the
Nodes page does: its certificates go on the blocklist and its database user is removed
([Machines elsewhere](docs/scaling.md#machines-elsewhere)).
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` |
| Settings › Auto scaling adds a provider from its driver's fields (credentials sent apart, never shown), shows its check in words, removes it with Undo, and makes a group, with a budget when the provider charges | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `adds a provider from its driver's fields, with the secret sent apart`, `shows a failed check in words, after saving and in the list`, `makes a group with a budget on a priced provider`, `asks no budget on a free provider and ticks a list's options`, `removes a provider with Undo instead of a question` |
| Units a provider runs with Lens's tag that Lens has no unit for show on the provider and in the activity ledger once; Lens lets none go itself, a person signed in to the web app does (with Undo), and the assistant only asks | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_units_lens_lost_track_of_show_on_their_provider`, `test_a_person_lets_a_stray_go_and_an_agent_cant`; [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_asks_to_let_a_stray_go_and_the_person_says_yes`; [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows the units a provider runs that Lens lost track of, and why it couldn't ask`, `lets a stray go with Undo instead of a question` |
| A unit that goes (let go, failed, or never came up) leaves the mesh: its unspent join token is withdrawn and the node that joined with it is removed, its worker stopped and its database user gone | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_a_unit_let_go_or_failed_leaves_the_mesh`; [test_mesh_workers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mesh_workers.py) `test_a_scaling_unit_let_go_leaves_the_mesh_with_its_token` |
| This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` |
| DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` |
| Lambda: GPU instances launched with the account's SSH key and a one-use join token in their user-data, tagged with Lens's tag and the unit, priced from Lambda's list, watched (an instance Lambda takes back counts as gone), found by tag across pages and terminated; every request and answer held to Lambda's published API document | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_lambda_group_launches_gpu_instances_that_join_over_the_mesh`, `test_lambda_needs_an_ssh_key_and_pages_through_instances`, `test_lens_reads_only_fields_the_clouds_answer_with` |
Expand Down
7 changes: 4 additions & 3 deletions docs/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,9 @@ A cloud provider's unit is a new machine. Its first boot runs a script (user-dat
name) and `LENS_WORKER_STEPS` (the template's steps), then the mesh's install line with a join token made for that
unit, usable once within an hour. The machine joins Lens over the [mesh](mesh.md) as a worker node and starts
`lens worker`; it asks for its own database user over the mesh once it has joined, so the user-data holds no
long-lived secret. A worker node runs only `transcribe` and `diarize` (it never gets the archive's key:
long-lived secret. When the unit goes (let go, failed, or never came up), it leaves the mesh too: its join token is
withdrawn if unspent, and the node that joined with it is taken off (its certificates on the blocklist, its database
user removed), as removing it on the Nodes page does. A worker node runs only `transcribe` and `diarize` (it never gets the archive's key:
[Worker nodes](mesh.md#worker-nodes)), so a group on these providers counts only those steps' jobs in its metrics
(`unit_steps`), and one whose template names none of them waits and launches nothing. Each machine
carries Lens's tag (`scaling.tag`: a DigitalOcean tag, a Hetzner label `lens=<tag>`, a Lambda tag `lens` with the tag
Expand Down Expand Up @@ -373,8 +375,7 @@ which only that yes in the web app does.
Spreading model calls across a group's units, and serving only some kinds of call (chat, decisions); more model
hosts (Lambda instances serving a model); Bedrock's price per unit-hour from AWS, its streamed answers
(`ConverseStream`) and Bedrock API keys as the provider's key; SageMaker's streamed invocations (`InvokeEndpointWithResponseStream`), so a SageMaker answer
streams as it is written; Hugging Face's scale to zero and private endpoints; RunPod network volumes, so a model is downloaded once; Proxmox LXC containers; revoking a unit's mesh
certificate when it goes; a live euro rate;
streams as it is written; Hugging Face's scale to zero and private endpoints; RunPod network volumes, so a model is downloaded once; Proxmox LXC containers; a live euro rate;
preemptible and spot instances at clouds other than AWS and RunPod; draining an AWS spot unit on its two-minute notice
(the node agent reading the instance's metadata) and handing its job back at once; following the spot price as it
changes; AWS sign-in with an instance role or IAM Identity Center instead of keys; mixed instance types per group;
Expand Down
17 changes: 17 additions & 0 deletions fastapi_backend/app/domain/mesh.py
Original file line number Diff line number Diff line change
Expand Up @@ -959,6 +959,23 @@ def remove(db, nid, by=None):
return node(db, nid)


def let_go_unit(db, name, by="scaling"):
"""An auto scaling unit let go (scaling.py): the join tokens made for it that weren't spent are withdrawn, and the
machine that joined with one is taken off the mesh (its certificates on the blocklist, its database user gone),
so a machine let go can't come back on what it was given. Returns the ids of the nodes taken off."""
tids = db.values("SELECT VALUE record::id(id) FROM mesh_token WHERE name = $n AND created_by = 'scaling'", n=name)
if not tids:
return []
db.q("UPDATE mesh_token SET revoked_at = $t WHERE record::id(id) IN $ids AND revoked_at = NONE", ids=tids, t=store.now())
gone = []
for nid in db.values(
"SELECT VALUE record::id(id) FROM mesh_node WHERE token IN $ids AND revoked_at = NONE AND role != 'primary'", ids=tids
):
remove(db, nid, by)
gone.append(nid)
return gone


def undo_remove(db, nid, cfg):
n = node(db, nid)
if not n["revoked_at"]:
Expand Down
17 changes: 16 additions & 1 deletion fastapi_backend/app/domain/scaling.py
Original file line number Diff line number Diff line change
Expand Up @@ -884,6 +884,7 @@ def _terminate(db, cfg, drv, p, n, why, by=None, state="terminated", now=None):
t=_iso(now),
)
db.q("DELETE $w", w=R("worker", n["name"]))
off = _unmesh(db, n["name"])
activity.record(
"run",
"scaling.terminate" if state == "terminated" else "scaling.fail",
Expand All @@ -892,11 +893,24 @@ def _terminate(db, cfg, drv, p, n, why, by=None, state="terminated", now=None):
db,
actor=by,
ok=state == "terminated",
detail={"why": why, "unit": n.get("name")},
detail=store.clean({"why": why, "unit": n.get("name"), "off_mesh": off or None}),
)
return True


def _unmesh(db, name):
"""A unit gone takes its place on the mesh with it: its join token, if unspent, is withdrawn and the node that
joined with it is taken off (mesh.let_go_unit). Trouble here is logged; the unit is gone either way."""
mesh = mesh_module()
if mesh is None or not hasattr(mesh, "let_go_unit"):
return []
try:
return mesh.let_go_unit(db, name)
except Exception as e: # noqa: BLE001 - logged; the node can still be removed in Settings › Nodes
log.warning("scaling: couldn't take unit %s off the mesh: %s: %s", name, type(e).__name__, e)
return []


def accrue(db, cfg, n, now, final=False):
"""What a unit cost since its last accrual, as a ledger row counted for it, its group and its provider: its price
per hour for the hours since, always an estimate (a list price, not a bill). Every accrue_minutes, and when it
Expand Down Expand Up @@ -1318,6 +1332,7 @@ def _launch(self, cfg, g, drv, p, now, why):
except Exception as e: # noqa: BLE001 - the unit failed; three in a row pause the group
err = f"{type(e).__name__}: {e}"[:300] if not isinstance(e, drivers.DriverError) else str(e)[:300]
db.q("UPDATE $r SET state = 'failed', error = $e, ended_at = $t", r=R("scale_node", nid), e=err, t=_iso(now))
_unmesh(db, name) # its join token goes unspent
fails = (g.get("failures") or 0) + 1
db.q("UPDATE $r SET failures = $f, error = $e", r=R("scale_group", g["id"]), f=fails, e=err)
activity.record("run", "scaling.fail", refs, cfg, db, ok=False, error=type(e).__name__, detail={"why": err, "unit": name})
Expand Down
29 changes: 29 additions & 0 deletions fastapi_backend/tests/api/test_mesh_workers.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,35 @@ def test_a_scaling_unit_names_its_worker_and_its_steps(client, db, cfg, nebula,
assert Proc.started[-1].cmd[1:6] == ["worker", "--name", "hetzner-3", "--steps", "transcribe"]


def test_a_scaling_unit_let_go_leaves_the_mesh_with_its_token(client, db, cfg, nebula, server_db, folders, monkeypatch): # noqa: F811
from app.domain import scaling

h = admin(client, db)
client.post("/api/v1/mesh/tokens", json={"role": "worker"}, headers=h) # the mesh is on
_, joins = mesh.make_token(db, cfg, by="scaling", role="worker", name="lens-w-7", hours=1, uses=1)
unspent, _ = mesh.make_token(db, cfg, by="scaling", role="worker", name="lens-w-8", hours=1, uses=1)
by_hand, _ = mesh.make_token(db, cfg, by="root@x.io", role="worker", name="lens-w-9", hours=1, uses=1)
monkeypatch.setenv("LENS_SCALING_UNIT", "lens-w-7")
a = agent(client, folders, joins, name="lens-w-7")
a.tick()
(folders[1] / "host.pub").write_text(mesh.keygen(cfg)[1])
running(folders[1])
a.tick()
nid = a.node["node"]["id"]
assert f"lens_node_{nid}" in users(db)

assert scaling._unmesh(db, "lens-w-7") == [nid]
assert mesh.node(db, nid)["revoked_at"] and f"lens_node_{nid}" not in users(db)
due(a)
a.tick()
assert a.removed and Proc.started[-1].stopped # off the mesh: its worker stops

assert scaling._unmesh(db, "lens-w-8") == [] # it never joined: its token can't be used now
revoked = {t["id"]: bool(t.get("revoked_at")) for t in db.rows("SELECT record::id(id) AS id, revoked_at FROM mesh_token")}
assert revoked[unspent["id"]] is True and revoked[by_hand["id"]] is False # a token a person made isn't Lens's to withdraw
assert scaling._unmesh(db, "lens-w-9") == [] and scaling._unmesh(db, "nobody") == []


def test_a_node_runs_only_the_steps_a_node_can(client, db, cfg, nebula, server_db, folders, monkeypatch): # noqa: F811
monkeypatch.setenv("LENS_WORKER_STEPS", "summarize,diarize")
a = on_mesh(client, cfg, admin(client, db), folders)
Expand Down
25 changes: 25 additions & 0 deletions fastapi_backend/tests/api/test_scaling.py
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,31 @@ def test_protected_and_terminated_units(client, db, cfg, h, key, fake):
assert fake.machines == {} and scaling.nodes(db, gid, alive=True) == []


def test_a_unit_let_go_or_failed_leaves_the_mesh(client, db, cfg, h, fake, monkeypatch):
class Mesh: # the mesh's side of it (mesh.let_go_unit) is in test_mesh_workers.py
off = []

def let_go_unit(self, db, name, by="scaling"):
self.off.append(name)
return [len(self.off)]

mesh = Mesh()
monkeypatch.setattr(scaling, "mesh_module", lambda: mesh)
cfg = on(db, cfg)
pid = provider(client, h)["id"]
gid = group(client, h, pid, min=1, max=1, cooldown_seconds=0, warmup_seconds=0).json()["id"]
tick(db, cfg)
(n,) = scaling.nodes(db, gid, alive=True)
client.post(f"/api/v1/scaling/units/{n['id']}/terminate", headers=h, json={})
assert mesh.off == [n["name"]]
(gone,) = [a for a in activity.history(db, f"scale_node:{n['id']}", 20) if a["action"] == "scaling.terminate"]
assert gone["detail"]["off_mesh"] == [1]
fake.fail = "out of capacity" # a launch that fails leaves no join token behind either
tick(db, cfg, NOW + dt.timedelta(minutes=1))
(failed,) = [x for x in scaling.nodes(db, gid) if x["state"] == "failed"]
assert mesh.off == [n["name"], failed["name"]]


def test_the_settings_section(client, db, cfg, h):
assert settings.effective(db, cfg)["scaling"]["enabled"] is False
for bad in ({"tick_seconds": 1}, {"join_minutes": 500}, {"tag": "Not A Tag"}):
Expand Down
Loading
Loading
0