Repository navigation
Description
Problem
AX has no way to bound how long a Task runs or how many tokens it spends. The README names this risk ("can burn money in a loop if nobody is watching"), and docs/roadmap.md lists "token/timeout budgets" under Stabilize Core Specs. Today:
TaskSpec(pkg/apis/v1alpha1/ax.proto) has no deadline or budget field. A Task staysRunninguntil someone runsax suspendorax delete.TaskStatus.usage(UsageStats) andTaskStatus.pending_approvalare declared, but nothing ininternal/orrunner/ever sets them.ax get taskalways shows them empty, which suggests features that don't exist. (pending_approvallooks like a leftover from the earlier harness architecture, see Verify Tool Policy Approvals Configuration #111.)internal/workspace/planner.gocallsmodel.Client.Generate, and the response carriesUsage(internal/model/client.go:141), but the value is discarded. AX doesn't count its own platform model spend either.UsageStatsusesint32. A long-running agent can pass 2^31 prompt tokens.
Why this matters
- Every unattended Task is unbounded. The only stop today is a human or an external watcher. That works against AX's goal of running very large numbers of tasks, where nobody watches each one.
- Worker capacity. A Task stuck in a loop holds its worker indefinitely. On a pool with fewer workers than Tasks, new Tasks fail with
ResourceExhausted(same symptom as Proposal: suspend a Task automatically when its command exits (spec.onCompletion) #420, but for Tasks that never exit rather than Tasks that already have). - Cost. Agents call metered model APIs. Without a ceiling, a stuck tool loop is open-ended spend.
- It's already on the roadmap, and the status fields already exist in the API. This fills them in rather than adding a new concept.
Example workload: an unattended agent that places outbound calls and holds long conversations against a metered LLM API (for example, candidate screening). A call that never ends or a retry loop is direct cost, and the operator needs AX to enforce a hard limit.
Proposal
Phased; each phase is useful on its own.
1. Active deadline
spec:
budget:
activeDeadline: 30m # counts Running time only; Suspended time is excludedWhen the deadline is exceeded, the runner stops the command (SIGTERM, then the grace period from #446). The Task gets condition Ready=False, reason=DeadlineExceeded and then follows spec.onCompletion from #420, or moves to Failed if #420 hasn't landed.
2. Usage reporting
The harness owns model calls; AX doesn't proxy them. So:
- The metadata server gets
POST /metadata/v1alpha1/ax/usage, which accepts token deltas from the harness. - The planner's own
Generateusage is added to the same totals. - The control plane copies the totals into
status.usagewhen it reads task status, next to the exit status proposed in Proposal: suspend a Task automatically when its command exits (spec.onCompletion) #420 (/metadata/v1alpha1/ax/status). UsageStatsfields change toint64while the API is still v1alpha1.
3. Token ceiling
spec.budget.maxTokens. When reported usage crosses it, the Task is stopped the same way as phase 1, with reason=TokenBudgetExceeded.
This is cooperative enforcement, since it relies on the harness reporting usage. Strict enforcement would require routing model egress through an AX-controlled gateway, which overlaps with #449 and is probably a separate decision.
4. pending_approval
Until an approval policy is designed, either remove this field or mark it reserved in v1alpha1, so the API doesn't advertise a feature that isn't there.
Open questions
- Where the deadline clock lives. docs/runner.md says suspend sends SIGTERM, but suspend freezes the sandbox without any signal #451 shows the command restarts after resume, so a runner-side timer would reset every time. Accumulated running time probably needs to live in the Task record (updated on suspend/resume), with the runner told the remaining budget when it starts.
- Enforcement point. Since ac23328 there is no background controller. Options are runner-side enforcement only, or reusing the retry loop proposed in Retry and waiting policy for Substrate errors #468 as a sweep.
- Timing relative to Revisit the API server and Task lifecycle #433. If the lifecycle rework will change how status is stored and reported, phase 1 could land first and phases 2–3 after.
- Naming.
budget.activeDeadlinevs. a Kubernetes-styleactiveDeadlineSeconds.
Related
#420 (completion policy and exit reporting), #346 (status after command exit), #446 (stop grace period), #433 (Task lifecycle revisit), #468 (background loop), #449 (egress), #111 (earlier approval-policy work)
I'm happy to implement phases 1 and 2 once external contributions open (#448). In the meantime I can share a fork branch with the change and tests if that helps the design discussion.
Activity
anudeep11k commentedon Oct 14, 2026
I implemented phase 1 on my fork so there's something concrete to review: https://github.com/anudeep11k/ax/tree/feat/task-active-deadline
It answers two of the open questions above differently from the proposal text:
status.activeDurationwhen a task leavesRunning, and tracks the current run instatus.runningSince. Suspended time doesn't count, and the client can't set either field onCreateTask.FailedwithReady=False, reasonDeadlineExceeded, and the workspace is kept.ResumeTaskreturnsFailedPreconditiononce the deadline is used up. The enforcer's suspend runs on a context that server shutdown can't cancel, to avoid losing the snapshot as inax suspendof a large workspace can lose the workspace while every command reports success #453.Enforcement is a sweep (
ax-server --budget-check-interval, default 10s). I noticed #468 proposes a single background loop that owns retry pacing and locking. If you'd rather this sweep run inside that loop instead of being separate, I can rework it once #465 lands.Tests cover accounting across suspend/resume, the deadline boundary, retrying a failed suspend, and refusing resume.
go test -race ./...passes. I'll open a PR when external contributions open (#448), or sooner if you'd like one.Full diff against main: main...anudeep11k:ax:feat/task-active-deadline
mayawang commentedon Oct 14, 2026
Thanks @anudeep11k, this is a really useful write-up, especially the point that status.usage and pending_approval are declared but never set. Budgets are on the roadmap we're finalizing this week, and this issue will be an input to that design. We'll follow up here once the shape is settled.
anudeep11k commentedon Oct 14, 2026
Thanks @mayawang, glad it's useful. Some context on where this came from: I build a voice agent product that phones people, holds a screening conversation, and writes a report. In it, a single call once ran 40 minutes against a 10-minute target, and I had to enforce limits in my own code to stop it happening again. That's why the missing budget controls in AX stood out when I read through the code: anyone running unattended, metered agents on AX would need to rebuild the same guard outside the platform.
I'll keep the phase 1 branch rebased on main in the meantime, and I'm happy to adapt it to whatever shape you settle on (including folding the sweep into the #468 loop) or to take on the usage reporting piece once the design is set. When you're ready for it, I'd be glad to open the PR myself if you're able to add me as a collaborator, or to hand the branch to whoever picks this up, whichever is easier for you.
MushuEE commentedon Oct 14, 2026
Budgets can be wide and complex:
Even within those, rate limits and token budgets are very hard to manage, because output token usage is generally unknown and unbounded. A call's cost is only known after it finishes, and thinking tokens make it worse. So a token ceiling can only be a soft limit ("stop after the call that crosses it"), unless the call is metered somewhere the harness can't bypass, such as the egress gateway (#449). Per-tenant is really a grouping of the other budgets, so it probably belongs above a single Task.
On timing enforcement, @anudeep11k's server-side clock works at smaller scale. Three gaps to consider in the design:
Traffic can wake a task without AX knowing. Substrate's router resumes a suspended actor when a request arrives for it, and AX isn't told. The sweep only checks tasks AX has recorded as Running, so these runs aren't counted, and a task that already hit its deadline can run again with no limit.
A failed suspend is reported as success. The reconciler logs a SuspendActor error and still records the task as Suspended (internal/controller/reconciler.go).
Stopping tasks one at a time doesn't scale. If many tasks expire together (e.g. a burst of tasks created with the same deadline), the delay grows linearly: 10k tasks at a few seconds per suspend takes hours.
anudeep11k commentedon Oct 14, 2026
Thanks @MushuEE, these are all real gaps. Taking them in order:
Tokens / per-tenant: agreed. A token ceiling can only be soft (stop after the call that crosses it) unless metering happens at the egress gateway (#449), and per-tenant belongs above a single Task as a grouping of the other budgets. Time is the one budget AX can enforce exactly today, which is why I started there.
1. Router wakes. AX can detect these even if it can't see them happen: each sweep, call
ListActorsonce per atespace and compare actual state with what AX recorded. An actor that's running while AX has it as Suspended or Failed starts the clock at the time it's observed (undercounting by at most one sweep interval), and if its deadline is already used, it's suspended again. That bounds the overrun but doesn't prevent it. Preventing it needs Substrate either to let the router refuse to resume a flagged actor, or to emit actor lifecycle events. Is either of those on Substrate's roadmap? If not, I can open an issue there.2. Failed suspend. That's #441. The enforcer already treats a suspend error as "retry next sweep" and leaves the task Running (there's a test for it), so once #464 propagates the error, this path becomes correct for budgets too.
3. Scale. Two changes: (a) keep a Redis sorted set of Running budgeted tasks scored by project 8D9F ed expiry time (
runningSince + remaining), so a sweep reads only tasks that are due, not every task; (b) suspend due tasks with bounded concurrency instead of one at a time. Total time is still bound by Substrate's snapshot throughput, but it no longer grows with the number of tasks sequentially.Gaps 1 and 3 don't depend on how tokens or per-tenant budgets end up, so I'd like to own those: the expiry index, parallel suspends and wake detection, built against whatever time-budget shape you settle on. If that works for you, feel free to assign this issue to me so it's clear who's on it. I'll hold off on pushing changes until the design is settled.
rakyll commentedon Oct 16, 2026
I merged #485 that removes usage stats. We should come back to this topic later once Task spec is more mature. There may be one more Task kind that we want to introduce and we need to align the bugdet spec with it first.
anudeep11k commentedon Oct 16, 2026
Thanks @rakyll, makes sense to align budgets with the new Task kind first. Glad the usage / pending_approval cleanup helped. I've rebased the branch on main after #485 so it's ready when you revisit, and I'm happy to help with the budget spec once the new Task kind takes shape.
If the new Task kind gets its own design issue, I'd be glad to follow along there.