8000 Task budgets: add spec.budget (active deadline, token ceiling) and populate the unused status.usage · Issue #471 · google/ax · GitHub
[go: up one dir, main page]

Skip to content

Task budgets: add spec.budget (active deadline, token ceiling) and populate the unused status.usage #471

Open

Description

@anudeep11k

Problem

AX has no way to bound how long a Task runs or how many tokens it spends. The README names this risk ("can burn money in a loop if nobody is watching"), and docs/roadmap.md lists "token/timeout budgets" under Stabilize Core Specs. Today:

  • TaskSpec (pkg/apis/v1alpha1/ax.proto) has no deadline or budget field. A Task stays Running until someone runs ax suspend or ax delete.
  • TaskStatus.usage (UsageStats) and TaskStatus.pending_approval are declared, but nothing in internal/ or runner/ ever sets them. ax get task always shows them empty, which suggests features that don't exist. (pending_approval looks like a leftover from the earlier harness architecture, see Verify Tool Policy Approvals Configuration #111.)
  • internal/workspace/planner.go calls model.Client.Generate, and the response carries Usage (internal/model/client.go:141), but the value is discarded. AX doesn't count its own platform model spend either.
  • UsageStats uses int32. A long-running agent can pass 2^31 prompt tokens.

Why this matters

  • Every unattended Task is unbounded. The only stop today is a human or an external watcher. That works against AX's goal of running very large numbers of tasks, where nobody watches each one.
  • Worker capacity. A Task stuck in a loop holds its worker indefinitely. On a pool with fewer workers than Tasks, new Tasks fail with ResourceExhausted (same symptom as Proposal: suspend a Task automatically when its command exits (spec.onCompletion) #420, but for Tasks that never exit rather than Tasks that already have).
  • Cost. Agents call metered model APIs. Without a ceiling, a stuck tool loop is open-ended spend.
  • It's already on the roadmap, and the status fields already exist in the API. This fills them in rather than adding a new concept.

Example workload: an unattended agent that places outbound calls and holds long conversations against a metered LLM API (for example, candidate screening). A call that never ends or a retry loop is direct cost, and the operator needs AX to enforce a hard limit.

Proposal

Phased; each phase is useful on its own.

1. Active deadline

spec:
  budget:
    activeDeadline: 30m   # counts Running time only; Suspended time is excluded

When the deadline is exceeded, the runner stops the command (SIGTERM, then the grace period from #446). The Task gets condition Ready=False, reason=DeadlineExceeded and then follows spec.onCompletion from #420, or moves to Failed if #420 hasn't landed.

2. Usage reporting

The harness owns model calls; AX doesn't proxy them. So:

  • The metadata server gets POST /metadata/v1alpha1/ax/usage, which accepts token deltas from the harness.
  • The planner's own Generate usage is added to the same totals.
  • The control plane copies the totals into status.usage when it reads task status, next to the exit status proposed in Proposal: suspend a Task automatically when its command exits (spec.onCompletion) #420 (/metadata/v1alpha1/ax/status).
  • UsageStats fields change to int64 while the API is still v1alpha1.

3. Token ceiling

spec.budget.maxTokens. When reported usage crosses it, the Task is stopped the same way as phase 1, with reason=TokenBudgetExceeded.

This is cooperative enforcement, since it relies on the harness reporting usage. Strict enforcement would require routing model egress through an AX-controlled gateway, which overlaps with #449 and is probably a separate decision.

4. pending_approval

Until an approval policy is designed, either remove this field or mark it reserved in v1alpha1, so the API doesn't advertise a feature that isn't there.

Open questions

Related

#420 (completion policy and exit reporting), #346 (status after command exit), #446 (stop grace period), #433 (Task lifecycle revisit), #468 (background loop), #449 (egress), #111 (earlier approval-policy work)

I'm happy to implement phases 1 and 2 once external contributions open (#448). In the meantime I can share a fork branch with the change and tests if that helps the design discussion.

Activity

  1. anudeep11k commented on Oct 14, 2026

    @anudeep11k
    Author

    I implemented phase 1 on my fork so there's something concrete to review: https://github.com/anudeep11k/ax/tree/feat/task-active-deadline

    It answers two of the open questions above differently from the proposal text:

    Enforcement is a sweep (ax-server --budget-check-interval, default 10s). I noticed #468 proposes a single background loop that owns retry pacing and locking. If you'd rather this sweep run inside that loop instead of being separate, I can rework it once #465 lands.

    Tests cover accounting across suspend/resume, the deadline boundary, retrying a failed suspend, and refusing resume. go test -race ./... passes. I'll open a PR when external contributions open (#448), or sooner if you'd like one.

    Full diff against main: main...anudeep11k:ax:feat/task-active-deadline

  2. mayawang commented on Oct 14, 2026

    @mayawang
    Collaborator

    Thanks @anudeep11k, this is a really useful write-up, especially the point that status.usage and pending_approval are declared but never set. Budgets are on the roadmap we're finalizing this week, and this issue will be an input to that design. We'll follow up here once the shape is settled.

  3. anudeep11k commented on Oct 14, 2026

    @anudeep11k
    Author

    Thanks @mayawang, glad it's useful. Some context on where this came from: I build a voice agent product that phones people, holds a screening conversation, and writes a report. In it, a single call once ran 40 minutes against a 10-minute target, and I had to enforce limits in my own code to stop it happening again. That's why the missing budget controls in AX stood out when I read through the code: anyone running unattended, metered agents on AX would need to rebuild the same guard outside the platform.

    I'll keep the phase 1 branch rebased on main in the meantime, and I'm happy to adapt it to whatever shape you settle on (including folding the sweep into the #468 loop) or to take on the usage reporting piece once the design is set. When you're ready for it, I'd be glad to open the PR myself if you're able to add me as a collaborator, or to hand the branch to whoever picks this up, whichever is easier for you.

  4. MushuEE commented on Oct 14, 2026

    @MushuEE
    Collaborator

    Budgets can be wide and complex:

    • Tokens
    • CPU
    • Latency
    • Per-tenant

    Even within those, rate limits and token budgets are very hard to manage, because output token usage is generally unknown and unbounded. A call's cost is only known after it finishes, and thinking tokens make it worse. So a token ceiling can only be a soft limit ("stop after the call that crosses it"), unless the call is metered somewhere the harness can't bypass, such as the egress gateway (#449). Per-tenant is really a grouping of the other budgets, so it probably belongs above a single Task.

    On timing enforcement, @anudeep11k's server-side clock works at smaller scale. Three gaps to consider in the design:

    Traffic can wake a task without AX knowing. Substrate's router resumes a suspended actor when a request arrives for it, and AX isn't told. The sweep only checks tasks AX has recorded as Running, so these runs aren't counted, and a task that already hit its deadline can run again with no limit.

    A failed suspend is reported as success. The reconciler logs a SuspendActor error and still records the task as Suspended (internal/controller/reconciler.go).

    Stopping tasks one at a time doesn't scale. If many tasks expire together (e.g. a burst of tasks created with the same deadline), the delay grows linearly: 10k tasks at a few seconds per suspend takes hours.

  5. anudeep11k commented on Oct 14, 2026

    @anudeep11k
    Author

    Thanks @MushuEE, these are all real gaps. Taking them in order:

    Tokens / per-tenant: agreed. A token ceiling can only be soft (stop after the call that crosses it) unless metering happens at the egress gateway (#449), and per-tenant belongs above a single Task as a grouping of the other budgets. Time is the one budget AX can enforce exactly today, which is why I started there.

    1. Router wakes. AX can detect these even if it can't see them happen: each sweep, call ListActors once per atespace and compare actual state with what AX recorded. An actor that's running while AX has it as Suspended or Failed starts the clock at the time it's observed (undercounting by at most one sweep interval), and if its deadline is already used, it's suspended again. That bounds the overrun but doesn't prevent it. Preventing it needs Substrate either to let the router refuse to resume a flagged actor, or to emit actor lifecycle events. Is either of those on Substrate's roadmap? If not, I can open an issue there.

    2. Failed suspend. That's #441. The enforcer already treats a suspend error as "retry next sweep" and leaves the task Running (there's a test for it), so once #464 propagates the error, this path becomes correct for budgets too.

    3. Scale. Two changes: (a) keep a Redis sorted set of Running budgeted tasks scored by project 8D9F ed expiry time (runningSince + remaining), so a sweep reads only tasks that are due, not every task; (b) suspend due tasks with bounded concurrency instead of one at a time. Total time is still bound by Substrate's snapshot throughput, but it no longer grows with the number of tasks sequentially.

    Gaps 1 and 3 don't depend on how tokens or per-tenant budgets end up, so I'd like to own those: the expiry index, parallel suspends and wake detection, built against whatever time-budget shape you settle on. If that works for you, feel free to assign this issue to me so it's clear who's on it. I'll hold off on pushing changes until the design is settled.

  6. added
    devxDeveloper experience: onboarding, defaults, and gaps found building on AX
    on Oct 15, 2026
  7. rakyll commented on Oct 16, 2026

    @rakyll
    Member

    I merged #485 that removes usage stats. We should come back to this topic later once Task spec is more mature. There may be one more Task kind that we want to introduce and we need to align the bugdet spec with it first.

  8. anudeep11k commented on Oct 16, 2026

    @anudeep11k
    Author

    Thanks @rakyll, makes sense to align budgets with the new Task kind first. Glad the usage / pending_approval cleanup helped. I've rebased the branch on main after #485 so it's ready when you revisit, and I'm happy to help with the budget spec once the new Task kind takes shape.
    If the new Task kind gets its own design issue, I'd be glad to follow along there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    configcontrollerdevxDeveloper experience: onboarding, defaults, and gaps found building on AXdocumentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0