Repository navigation
Description
Several open PRs add handling for Substrate errors at different layers: #457 (adapt to Substrate v0.4, retries in the Substrate client), #463 (transient Substrate unavailability) and #464 (propagate SuspendActor errors). This issue sets one policy so they fit together.
Updated 14 Oct: #465 (wait for worker capacity) and #367 are closed. AX won't run its own queue; Substrate will own queuing (agent-substrate/substrate#2059) and AX will use it.
Policy
- Substrate client (
internal/substrate): retry only errors where the call is known not to have taken effect, for a few seconds with jittered backoff. That isAborted(busy actor) andUnavailable(Substrate not reachable). Use the single retry helper from fix(substrate): adapt to Substrate v0.4 API and harden actor RPCs #457. DeadlineExceededis not retried in the client, because the call may have succeeded. It is treated likeUnavailableat the task level below, and the next attempt goes through the idempotent ensure calls.- Task level: when client retries run out on
UnavailableorDeadlineExceeded, the task staysPendingwithReady=False, reason=SubstrateUnavailable, and the error is returned so the caller can retry. It is not markedFailedfor these. ResourceExhausted(no free worker) is returned to the caller. AX does not queue or retry it (Task is marked Failed permanently when Substrate returns ResourceExhausted (no free workers) instead of staying Pending and retrying #367, working as intended).- ActorTemplate creation errors (
InvalidArgument,FailedPrecondition) are not retried and do not fall back to another template. The task fails with Substrate's reason, so problems like Custom ActorTemplate creation errors are swallowed; Task fails with misleading 'actor template not found' (e.g. tag-referenced image) #366 (misleading "actor template not found") don't come back. - A failed suspend is returned to the caller, and the stored task keeps its previous state (Propagate SuspendActor error up from substrate #464).
Why this split
- Where a retry lives depends on whether the call could have taken effect. Errors that are safe to repeat and clear in seconds (
Aborted,Unavailable) are retried quickly in the client, where the caller never sees them. Anything that can take minutes (ResourceExhausted) or may already have taken effect (DeadlineExceeded) moves to the task level, where the state is stored and visible. - A task hit by a Substrate outage shows
Pendingwith a reason, soax getandax watchsay why it isn't running, and re-running the resume picks it up. - No queue in AX. Waiting for capacity belongs in Substrate, the only place that knows when a worker frees up, and anything AX built here would not be forward compatible with it.
- No template fallback while the Substrate API is alpha. A silent fallback runs the task with settings it didn't ask for (for example, it would drop feat(controller): apply Task resource limits to the ActorTemplate #405's resource limits) and replaces the real error with a misleading one (Custom ActorTemplate creation errors are swallowed; Task fails with misleading 'actor template not found' (e.g. tag-referenced image) #366).
Alternatives considered
- Retry everything in the Substrate client. Simplest, but a capacity wait can take minutes, which blocks the caller past the 30s resource-lock lease, and repeating a call that may have succeeded is unsafe.
- A background retry loop in AX (Keep tasks pending until Substrate worker capacity becomes available #465). Clean, but it is a queue, and Substrate will own queuing. Closed in favour of waiting for that.
- Let each PR handle its own errors. That's what happened, and it produced the stacked retries above and different states for the same failure.
- Chosen: Substrate queues resumes, and AX uses it when it lands (Resume order under worker-pool saturation: old requests starve, and cancelled resumes wedge workers agent-substrate/substrate#2059). Until then,
ResourceExhaustedgoes back to the caller.
Per PR
- fix(substrate): adapt to Substrate v0.4 API and harden actor RPCs #457: drop the
InvalidArgumentfallback to the base template, and retryResumeActoronAbortedonly. - Keep tasks pending until Substrate worker capacity becomes available #465: closed, see the note at the top.
- controller,substrate: handle transient Substrate unavailability without permanent Task failure #463: rebase after fix(substrate): adapt to Substrate v0.4 API and harden actor RPCs #457. Add
Unavailableto fix(substrate): adapt to Substrate v0.4 API and harden actor RPCs #457's retry helper, keep the taskPendingwithSubstrateUnavailableand return the error, and keep re-apply of aPendingtask limited to an unchanged spec. - Propagate SuspendActor error up from substrate #464: as is, plus a test or a note for a failed suspend on the create path.
Merge order: #457, then #464, then #463, then #405 (Task resource limits, rebased onto #457).
Out of scope for now: queuing for capacity (Substrate will own it), and automatic retry of Pending tasks.
If you see a problem with this, comment here by end of Thursday. Otherwise this is the plan.
Activity
mayawang commentedon Oct 14, 2026
Added the reasoning and alternatives considered.
mayawang commentedon Oct 14, 2026
Updated after #465 and #367 were closed: AX won't run its own queue, Substrate will own queuing (agent-substrate/substrate#2059). The rest of the policy is unchanged; #463 no longer depends on a loop.
lifecycle: remove MaxQueueWaitDuration and Transition stub
store, lifecycle: trim seam to minimal sync-path runner contract
lifecycle: remove MaxQueueWaitDuration and Transition stub
store, lifecycle: trim seam to minimal sync-path runner contract
store: add TaskRow, TaskApplier, and StoredStatus runner seam (#591)
Runner proto lifecycle (#633)