Repository navigation
Tasks: bind multiple Models and let the agent discover them (/v1/models) #577
Description
Problem
Right now a task effectively gets one model. TaskSpec has no model field at all, so the model is whatever the runner is told in spec.config (e.g. model for the Claude Code runner) plus a key in spec.env. The proposals we have for fixing that (#348's spec.modelRef, and the Model changes in #516 and #475) all keep the same shape: one task, one Model, one provider + model + key.
I think that is too restrictive. Agents regularly want more than one model in a single task: a cheap model for subagents or summarization and a stronger one for planning, a fallback when a provider is rate limited, or just letting the harness pick. Most harnesses already know how to do this if they can ask "which models do I have?", which is what GET /v1/models is in the OpenAI and Anthropic APIs.
Proposal
Two parts, and the second is more open than the first.
1. A task binds a list of models, not one.
kind: Task
spec:
models:
- name: claude-model
- name: default-modelEach entry references a Model in the task's atespace. If we add a singular modelRef first, moving to a list later is an API break, whereas a list with one entry covers the single-model case today. So I'd rather start with the list.
2. The agent can discover what it was given.
At the very least, the bound models should be readable inside the sandbox, next to the existing metadata endpoints:
GET $AX_METADATA_URL/metadata/v1alpha1/ax/models
returning name, provider and model ID for each bound Model (no credentials).
What I actually want is an OpenAI-compatible /v1/models that the agent can point its SDK at, so existing harnesses work without knowing anything about AX. That raises the question of what serves it and whether it also proxies the model calls. Options I can see:
- Metadata only: the runner serves the list, the agent calls providers directly, and credentials come from egress injection (Credentials via egress injection: keep real keys out of tasks (rotation + multi-tenant) #475).
- The runner serves
/v1/modelsin the provider's format on loopback, still only a list. - A model gateway serves
/v1/modelsand proxies the inference calls, which also gives us one place for budgets and usage (Task budgets: add spec.budget (active deadline, token ceiling) and populate the unused status.usage #471).
Option 1 is small and doesn't close off 2 or 3, so it seems like the right first step.
A second axis: Model itself holds multiple models
Another way to get at this is to make Model one provider with many models, instead of one provider + one model:
kind: Model
metadata:
name: anthropic
spec:
provider: anthropic
secretKey: { name: anthropic-api-secret, key: ANTHROPIC_API_KEY }
models:
- name: claude-opus-5
parameters: { maxTokens: 16000 }
- name: claude-haiku-5I like this because it matches what a provider's /v1/models already returns: one endpoint and one credential, many models. It also lines up with egress injection, since it is one credential per host, which is the limit #475 calls out. And nothing consumes Model today, so changing spec.model into a list is cheap now and gets harder once tasks bind it.
The costs are that parameters has to move to each entry, and the kind is really a provider at that point, so the name Model gets misleading (see #356).
I don't think this replaces the list on the task. A task that wants Anthropic and Gemini still binds two of these, unless each entry carries its own provider and secret, and then Model is a catalog and we lose the one-credential-per-host benefit. So I see the two as complementary: Model holds many models per provider, and the task binds one or more Models.
Open questions
- Should a task list models explicitly, or bind by label selector so it sees "everything in the atespace I'm allowed to use"?
- Does the list need a role or alias per entry (
default,fast), or is that the runner's business inspec.config? - Task specs are immutable and gateway rules are copied at create time. Is the model list also fixed at create, or does a later
Modelchange show up in a running task? - How does this line up with Credentials via egress injection: keep real keys out of tasks (rotation + multi-tenant) #475? With several models, each one needs its own egress rule and placeholder key, and two models on the same host with different keys hit the "one credential per host" limit called out there. A provider-shaped
Modelwould mostly avoid that. - If
Modelholds many models per provider, should it be renamed?
Activity