Repository navigation
E2E flaky: ax resume fails with "no worker has room for the actor" right after setup #568
Description
The Kind E2E suite failed once on #566 and passed on re-run:
https://github.com/google/ax/actions/runs/42960996660/job/130327233529
Every task test failed at its first ax resume with ResourceExhausted: no worker has room for the actor.
Cause: the tests start before Substrate's workers are schedulable. A worker pod is Ready as soon as it registers, but the scheduler skips it until atelet has downloaded the gVisor runtimes and the worker has reported them (agent-substrate/substrate#3276). hack/setup-e2e-kind.sh only waits for the Deployment rollout.
Fix (test-only): wait for schedulable workers, or retry resume on ResourceExhausted as Substrate's own e2e does. Also add atelet and worker logs to the failure dump; without them this run cannot show why the workers were slow.
Follow-up: ax-server returns this error as Internal and marks the task Failed. It could be treated as transient, like the errors in #463.
Activity
mayawang commentedon Nov 11, 2026
+1 on the follow-up. Per #468, a capacity error should leave the task Pending rather than Failed, but AX shouldn't retry or queue it itself, Substrate will queue (agent-substrate/substrate#2059)
[-]E2E flake: `ax resume` fails with "no worker has room for the actor" right after setup[/-][+]E2E flaky: `ax resume` fails with "no worker has room for the actor" right after setup[/+]