You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Quick start never creates or mentions a WorkerPool, so the first Task fails with ResourceExhausted #437
AX runs each Task as a Substrate actor, and Substrate needs a WorkerPool with a free worker in the Task's atespace. Nothing in the quick start creates or mentions one:
make deploy does not create one.
The README and docs/ do not mention one.
Substrate's hack/install-ate.sh --deploy-ate-system does not create one either; only its demos do.
So on a fresh install, the first Task fails:
Phase: Failed
Ready False ActorResumeFailed resuming actor default/validate2: rpc error: code = ResourceExhausted desc = no free workers available
Because of #367, the Task then stays Failed even after a pool is added.
Reproduced on
AX v0.3.1 (e70162a) with Agent Substrate v0.2.0, on kind (Docker Desktop, arm64), 2026-09-28.
Earlier, AX v0.3.0 on GKE Standard 1.36.3-gke.1767000.
main (ac23328) still has no WorkerPool manifest under deploy/ or examples/, and the README and docs/ do not mention one.
Workaround
With this pool in the Task's atespace, a new Task ran and ax ssh worked. It is adapted from Substrate's demos/counter/counter.yaml.tmpl and applied with ko apply from a Substrate checkout:
apiVersion: ate.dev/v1alpha1kind: WorkerPoolmetadata:
name: ax-poolnamespace: default # the Task's atespacespec:
replicas: 2workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisortemplate:
nodeSelector:
ate.dev/substrate-version: "v0.2.0"# must match the label the install puts on the nodesresources:
limits: { cpu: "1", memory: 1Gi }requests: { cpu: 250m, memory: 1Gi }
Suggestion
Add a minimal WorkerPool step between "Deploy the control plane" and "Run your first task", or have make deploy create a small default pool.
Mention the ate.dev/substrate-version node label. A pool whose nodeSelector does not match it never schedules its workers, and the Task fails with the same ResourceExhausted.
Fix ready on my fork: docs/quick-start-workerpool-and-examples (diff vs upstream) — PR creation is currently rejected by the repo's collaborators_only policy (as tracked in #358), so I'm leaving the branch here and will open the PR as soon as that allows.
What it does for this issue:
Adds a "Create a WorkerPool" step to the README quick start, between deploying the control plane and running the first task, with the failure mode spelled out: the ResourceExhausted surfaces at ax resume, not ax apply — the task stays Suspended until then.
Adds examples/workerpool.yaml, adapted from Substrate's counter demo, including the ko apply-from-a-Substrate-checkout flow for the ko:// worker image and ate.dev/substrate-version nodeSelector guidance (checked via kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version; source installs label nodes with a git-describe string like v0.2.0-10-g1d7ca8ce, so the example uses a placeholder rather than a hardcoded tag).
One correction to the workaround in the issue body: the pool does not need to live in the task's atespace. Worker selection is cluster-wide — matched by sandbox class and label selectors, with no namespace dimension (see Constraints in cmd/ateapi/internal/scheduling/scheduling.go and the "pool selection is cluster-wide" note in Substrate's multi-template demo). Any namespace works; the namespace only decides where the worker pods are managed.
Friendly nudge on this one. The branch docs/quick-start-workerpool-and-examples (linked above, single commit) still applies: it test-merges cleanly against current main (8888435 — the only main-side change to these files since the branch was posted is the logo swap in #497), and its premises still hold on today's code: CreateTask still stores new tasks as Suspended, and multi-workspace bindings are still valid.
Following the pattern from #487 (branch on a fork, maintainer opens the PR from it with the contributor's commits): would a maintainer be able to open the PR from cloudonly/ax:docs/quick-start-workerpool-and-examples? The commit carries the WorkerPool quick-start step for this issue, the resume-flow fix for #426, and the multi-workspace example for #425. Happy to split it into separate branches if you'd rather have one PR per issue.
Thanks @cloudonly for your patience on this, and @eren23 for verifying it on kind. One change before I open the PR: #496 is moving AX to one workspace per task, so the multi-workspace example won't be valid after it lands. Could you drop that example, or switch it to a single workspace? I'll open the PR from your branch right after.
Done — examples/multi-workspace.yaml now binds a single git-backed Workspace (171dbdb..2b04656 on the branch above), and it still test-merges cleanly against current main (7b3347f).
I kept the filename so the docs/manifests.md link (#425) keeps resolving. One note: the paragraph around that link still describes multi-entry binding ("first entry is the working directory", unique paths), so it will need a pass once #496 lands — happy to adjust that too, or rename the file and update the link, whichever you prefer.
Thanks @cloudonly, opened as #505 from your branch. Yes please on the docs/manifests.md paragraph, now that #496 has landed. Heads-up that we're planning to slim the README down to a low-bar path (local mode in #504, and a single make deploy once images are published), so some of the quick-start steps may move to docs/ later. Your fixes make today's quick start correct, which is still worth landing.
Activity
cloudonly commentedon Sep 28, 2026
Fix ready on my fork:
docs/quick-start-workerpool-and-examples(diff vs upstream) — PR creation is currently rejected by the repo'scollaborators_onlypolicy (as tracked in #358), so I'm leaving the branch here and will open the PR as soon as that allows.What it does for this issue:
ResourceExhaustedsurfaces atax resume, notax apply— the task staysSuspendeduntil then.examples/workerpool.yaml, adapted from Substrate's counter demo, including theko apply-from-a-Substrate-checkout flow for theko://worker image andate.dev/substrate-versionnodeSelector guidance (checked viakubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version; source installs label nodes with a git-describe string likev0.2.0-10-g1d7ca8ce, so the example uses a placeholder rather than a hardcoded tag).One correction to the workaround in the issue body: the pool does not need to live in the task's atespace. Worker selection is cluster-wide — matched by sandbox class and label selectors, with no namespace dimension (see
Constraintsincmd/ateapi/internal/scheduling/scheduling.goand the "pool selection is cluster-wide" note in Substrate's multi-template demo). Any namespace works; the namespace only decides where the worker pods are managed.cloudonly commentedon Oct 19, 2026
Friendly nudge on this one. The branch docs/quick-start-workerpool-and-examples (linked above, single commit) still applies: it test-merges cleanly against current
main(8888435— the only main-side change to these files since the branch was posted is the logo swap in #497), and its premises still hold on today's code:CreateTaskstill stores new tasks asSuspended, and multi-workspace bindings are still valid.Following the pattern from #487 (branch on a fork, maintainer opens the PR from it with the contributor's commits): would a maintainer be able to open the PR from
cloudonly/ax:docs/quick-start-workerpool-and-examples? The commit carries the WorkerPool quick-start step for this issue, the resume-flow fix for #426, and the multi-workspace example for #425. Happy to split it into separate branches if you'd rather have one PR per issue.mayawang commentedon Oct 20, 2026
Thanks @cloudonly for your patience on this, and @eren23 for verifying it on kind. One change before I open the PR: #496 is moving AX to one workspace per task, so the multi-workspace example won't be valid after it lands. Could you drop that example, or switch it to a single workspace? I'll open the PR from your branch right after.
cloudonly commentedon Oct 21, 2026
Done —
examples/multi-workspace.yamlnow binds a single git-backed Workspace (171dbdb..2b04656on the branch above), and it still test-merges cleanly against currentmain(7b3347f).I kept the filename so the
docs/manifests.mdlink (#425) keeps resolving. One note: the paragraph around that link still describes multi-entry binding ("first entry is the working directory", unique paths), so it will need a pass once #496 lands — happy to adjust that too, or rename the file and update the link, whichever you prefer.mayawang commentedon Oct 21, 2026
Thanks @cloudonly, opened as #505 from your branch. Yes please on the docs/manifests.md paragraph, now that #496 has landed. Heads-up that we're planning to slim the README down to a low-bar path (local mode in #504, and a single make deploy once images are published), so some of the quick-start steps may move to docs/ later. Your fixes make today's quick start correct, which is still worth landing.
mayawang commentedon Nov 4, 2026
Fixed by #505: the quick start now has a WorkerPool step and examples/workerpool.yaml. Thanks @cloudonly!