8000 feat(scaling): spread model calls across a group's running units by adeelahmad · Pull Request #673 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content

feat(scaling): spread model calls across a group's running units - #673

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-azm2mb
Oct 19, 2026
Merged

adeelahmad merged 1 commit into
mainfrom
claude/project-thread-azm2mb

Conversation

@adeelahmad
Copy link
Copy Markdown
Owner

Description

Before: a model group (RunPod, Hugging Face, SageMaker) served Lens's model calls from one unit only. A group scaled out to three units paid for three GPUs and used one; the others stood by.

After: every running unit of the group takes model calls in turn. A group's units share one API key, made once and sealed on the group, and the scaler keeps a new llm.spread setting ({"base_url", "urls"}) at the shown unit's address and the others'. A unit that can't be reached hands its call to the next. A spread is used only while llm.base_url is still the address it was made for, so a provider a person picks never sends calls to a unit.

How: scaling.unit_key makes or opens the group's key, and bootstrap hands it to RunPod and Hugging Face launches as boot["key"]. Scaler._serve computes the other running units with the same model and key (_others) and saves llm.spread only when it changes. llm._post walks _spread(cfg), starting with the next address in turn and trying at most one more when an address can't be reached. Units launched before this keep their own keys and serve only as the shown unit until replaced. Bedrock's units are models at one address, so a Bedrock group still serves from one unit.

Motivation and Context

Adeel asked for auto scaling of model providers. Scaling out a model group should add capacity. This was the first Refine later item in docs/scaling.md.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (ruff, eslint, prettier).
  • I added or updated tests (test_model_calls_go_to_a_groups_running_units_in_turn, test_llm_spread_is_checked).
  • Documentation updated (docs/scaling.md, docs/configuration.md, docs/requirements.md, README).
  • Changelog fragment in changelog.d/scaling-spread.md.

Done questions (AGENTS.md)

  • Goal: merges on green CI.
  • Agents: llm.spread shows and is settable in the settings API like any setting; the scaler keeps it.
  • Zero setup: nothing to set up.
  • Fewest clicks: no click.
  • Configurable: llm.spread defaults to none and is an ordinary checked setting.
  • Light: one comparison per tick, and a save only when units change.
  • Private and secure: the key stays sealed (on the group and on each unit). Threat model P-SCALING (md and yaml) says the RunPod and Hugging Face key is now per group. Units are all in one provider account, so no one new can read it. A spread from another provider is ignored.
  • Cost: it uses the GPUs already paid for. Calls stay in the ledger as before.
  • User decides: a person's provider pick still wins. Defaults are unchanged.
  • Upgrades: a new optional setting (default none) and an optional unit_key on scale_group. Old units keep their keys.
  • Honest: docs say which units spread and which don't (Bedrock, units from before).
  • Tested and documented.
  • Open: n/a, no new dependency.
  • Reported in the thread.
  • Reuse: the existing settings store, sealing and _serve.

Generated by Claude Code

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ALjNbLobvPe7VwvDL8j7V3
@adeelahmad
adeelahmad merged commit f85b5b7 into main Oct 19, 2026
12 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-azm2mb branch October 19, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0