8000 Retry and waiting policy for Substrate errors · Issue #468 · google/ax · GitHub
[go: up one dir, main page]

Skip to content

Retry and waiting policy for Substrate errors #468

Open

Description

@mayawang

Several open PRs add handling for Substrate errors at different layers: #457 (adapt to Substrate v0.4, retries in the Substrate client), #463 (transient Substrate unavailability) and #464 (propagate SuspendActor errors). This issue sets one policy so they fit together.

Updated 14 Oct: #465 (wait for worker capacity) and #367 are closed. AX won't run its own queue; Substrate will own queuing (agent-substrate/substrate#2059) and AX will use it.

Policy

Why this split

  • Where a retry lives depends on whether the call could have taken effect. Errors that are safe to repeat and clear in seconds (Aborted, Unavailable) are retried quickly in the client, where the caller never sees them. Anything that can take minutes (ResourceExhausted) or may already have taken effect (DeadlineExceeded) moves to the task level, where the state is stored and visible.
  • A task hit by a Substrate outage shows Pending with a reason, so ax get and ax watch say why it isn't running, and re-running the resume picks it up.
  • No queue in AX. Waiting for capacity belongs in Substrate, the only place that knows when a worker frees up, and anything AX built here would not be forward compatible with it.
  • No template fallback while the Substrate API is alpha. A silent fallback runs the task with settings it didn't ask for (for example, it would drop feat(controller): apply Task resource limits to the ActorTemplate #405's resource limits) and replaces the real error with a misleading one (Custom ActorTemplate creation errors are swallowed; Task fails with misleading 'actor template not found' (e.g. tag-referenced image) #366).

Alternatives considered

Per PR

Merge order: #457, then #464, then #463, then #405 (Task resource limits, rebased onto #457).

Out of scope for now: queuing for capacity (Substrate will own it), and automatic retry of Pending tasks.

If you see a problem with this, comment here by end of Thursday. Otherwise this is the plan.

Activity

  1. mayawang commented on Oct 14, 2026

    @mayawang
    CollaboratorAuthor

    Added the reasoning and alternatives considered.

  2. mayawang commented on Oct 14, 2026

    @mayawang
    CollaboratorAuthor

    Updated after #465 and #367 were closed: AX won't run its own queue, Substrate will own queuing (agent-substrate/substrate#2059). The rest of the policy is unchanged; #463 no longer depends on a loop.

  3. added a commit that references this issue on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0