8000 feat(scaling): add providers and make groups in Settings by adeelahmad · Pull Request #621 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
8000
Diff view
17 changes: 8 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,13 +238,12 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[the meter](docs/budgets.md#the-meter-and-the-costs-page); [checks](docs/requirements.md#see-what-it-costs))
- **Scale out.** Scaling groups add workers when work waits and let them go when it doesn't, by target, step and
schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity
ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage
groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today;
AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud, Proxmox VE and Linux hosts over SSH providers are in
progress (built, with machines that join over the mesh to transcribe and tell voices apart, not yet tried on a real
cloud), so is RunPod for GPUs that serve Lens's model calls while they run, and more model hosts are planned.
([Auto scaling](docs/scaling.md);
[checks](docs/requirements.md#scale-out))
ledger. You add providers and make groups in Settings › Auto scaling, and a group launches nothing until you approve
its terms there; agents propose and manage groups over the API, MCP and the assistant, and never approve one. It
runs more workers on this machine today; AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud, Proxmox VE and
Linux hosts over SSH providers are in progress (built, with machines that join over the mesh to transcribe and tell
voices apart, not yet tried on a real cloud), so is RunPod for GPUs that serve Lens's model calls while they run,
and more model hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out))
- **Run on more machines.** Add a Mac with a GPU, a box for the assistant's code or a server with more disk, and
Lens joins it over a private network of its own, a Nebula mesh it issues the certificates for: one install line
with a one-time token, and the machine needs nothing else. Worker nodes run jobs as their own user of the
Expand Down Expand Up @@ -318,8 +317,8 @@ Next, and not in a pull request yet; each links to where it is tracked:
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
- **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
([local models](docs/local-models.md)).
- **Auto scaling:** spreading model calls across GPUs, more model hosts (Lambda, Hugging Face, SageMaker, Bedrock),
and making providers and groups in Settings ([auto scaling](docs/scaling.md)).
- **Auto scaling:** spreading model calls across GPUs, and more model hosts (Lambda, Hugging Face, SageMaker,
Bedrock) ([auto scaling](docs/scaling.md)).
- **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the
activity ledger ([speech providers](docs/speech-providers.md)).
- **Storage:** uploads and assets' media in the file store, and a storage key to download
Expand Down
6 changes: 6 additions & 0 deletions changelog.d/scaling-forms.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
- **Add providers and make scaling groups in Settings.** Settings › Auto scaling lists the providers with their last
check in words, Check again, and Remove with Undo. Add a provider picks the kind, then fills a form drawn from the
driver's fields (defaults filled, credentials as password fields, Enter saves) and shows the check it runs. New group
on a provider asks for a name, the minimum and maximum, a monthly budget when the provider charges, and what each
unit is; a group made there is approved as it's made. Rules stay with the assistant, MCP and the API for now
([In Settings](docs/scaling.md#in-settings)).
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,7 @@ request, to another check that proves the same thing; loosening a requirement or
| The scaler launches units, sees them join, drains busy workers before letting them go, and fails and pauses on units that don't come up | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_the_scaler_launches_joins_drains_and_terminates`, `test_a_unit_that_never_joins_fails_and_failures_pause_the_group`, `test_protected_and_terminated_units` |
| Each unit's cost goes into the activity ledger as an estimate, and a group stops at its budget's cap | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_the_scaler_launches_joins_drains_and_terminates`, `test_at_its_budgets_cap_a_group_stops` |
| Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` |
| Settings › Auto scaling adds a provider from its driver's fields (credentials sent apart, never shown), shows its check in words, removes it with Undo, and makes a group, with a budget when the provider charges | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `adds a provider from its driver's fields, with the secret sent apart`, `shows a failed check in words, after saving and in the list`, `makes a group with a budget on a priced provider`, `asks no budget on a free provider and ticks a list's options`, `removes a provider with Undo instead of a question` |
| This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` |
| DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` |
| A group of machines elsewhere counts only the jobs a worker node takes, and never launches one for others | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_group_elsewhere_counts_only_the_jobs_a_node_takes` |
Expand Down
36 changes: 27 additions & 9 deletions docs/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ launched. Everything it does is in the [activity ledger](activity.md), and its c
[budget](budgets.md).

Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the API, MCP and assistant tools,
Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that
runs more `lens worker` processes where Lens runs. In progress: **AWS EC2** (spot or on demand), **DigitalOcean**
and **Hetzner Cloud** providers are built (credentials checked, prices, launching, watching and deleting machines),
and their machines join over the [mesh](mesh.md) as worker nodes that transcribe and tell voices apart; so are
**Proxmox VE**, clones of a VM template on your own server, and **Linux hosts over SSH**, workers started on machines
you have. They haven't been tried on a real cloud machine yet. Also in progress: **RunPod**, GPUs that serve a model
with vLLM, which Lens's model calls go to while one runs. Planned: more model hosts (Lambda, Hugging Face, SageMaker,
Bedrock); and making providers and groups in Settings.
Settings › Auto scaling (the switch, [adding providers and making groups](#in-settings), and each group's terms to
approve or pause), and **This machine**, a provider that runs more `lens worker` processes where Lens runs. In
progress: **AWS EC2** (spot or on demand), **DigitalOcean** and **Hetzner Cloud** providers are built (credentials
checked, prices, launching, watching and deleting machines), and their machines join over the [mesh](mesh.md) as
worker nodes that transcribe and tell voices apart; so are **Proxmox VE**, clones of a VM template on your own server,
and **Linux hosts over SSH**, workers started on machines you have. They haven't been tried on a real cloud machine
yet. Also in progress: **RunPod**, GPUs that serve a model with vLLM, which Lens's model calls go to while one runs.
Planned: more model hosts (Lambda, Hugging Face, SageMaker, Bedrock).

Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
signed in to the web app. Agents can propose and manage groups, but never approve one.
Expand All @@ -40,6 +40,23 @@ One process runs the scaler at a time: the API and every `lens worker` (except t
`--no-schedule`, as scaled workers are) take turns holding its lease. While scaling is off and nothing is left to let
go, the scaler doesn't even take the lease.

## In Settings

**Settings › Auto scaling** lists the providers, each with its kind, which credentials are set and whether its last
check passed (and why not, in words when it failed), with **Check again** and **Remove**. Remove takes it out of the
list at once with **Undo** on the notice; it's deleted a few seconds later, or as you leave the page, since its sealed
credentials can't be put back once it's gone. A provider a group uses can't be removed until the group goes.

**Add a provider** asks for the kind (each with what it does), then a form drawn from the driver's `fields`: defaults
filled in, the name first and focused, credentials as password fields (sent as `secrets`, never shown again), lists
one per line, on and off settings as checkboxes, and Enter saves. A driver that needs packages says which, and their
licence, before it fetches them on the first save. The check runs as it's saved; when it fails, the dialog says why.

**New group** on a provider asks for a name (`<provider> workers`), the minimum (0) and maximum (1), a monthly budget
in USD only when the provider charges money (then it's required), and the driver's `template` with its defaults. A
group made here is approved as it's made. Its rules are set by the assistant, MCP or the API for now; without rules it
keeps its minimum running.

## Providers

```http
Expand Down Expand Up @@ -271,4 +288,5 @@ preemptible and spot instances at clouds other than AWS and RunPod; draining an
(the node agent reading the instance's metadata) and handing its job back at once; following the spot price as it
changes; AWS sign-in with an instance role or IAM Identity Center instead of keys; mixed instance types per group;
warm pools; predictive scaling; health checks beyond
joining; prices fetched from providers' pricing APIs.
joining; prices fetched from providers' pricing APIs. In Settings: rules, changing a provider or a group, removing a
group, and a provider's price and note.
1 change: 1 addition & 0 deletions nextjs-frontend/__tests__/settings-kokoro.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ const view = (voice: Record<string, unknown>) =>
untrusted_asks: true,
injection_check: "patterns",
disabled_tools: [],
answer_comments: true,
max_steps: 6,
max_tool_calls: 50,
browser_steps: 25,
Expand Down
Loading
Loading
0