8000 Quick start never creates or mentions a WorkerPool, so the first Task fails with ResourceExhausted · Issue #437 · google/ax · GitHub
[go: up one dir, main page]

Skip to content
  • Quick start never creates or mentions a WorkerPool, so the first Task fails with ResourceExhausted #437

    Closed

    Description

    @ilaigold

    Summary

    AX runs each Task as a Substrate actor, and Substrate needs a WorkerPool with a free worker in the Task's atespace. Nothing in the quick start creates or mentions one:

    • make deploy does not create one.
    • The README and docs/ do not mention one.
    • Substrate's hack/install-ate.sh --deploy-ate-system does not create one either; only its demos do.

    So on a fresh install, the first Task fails:

    Phase:  Failed
    Ready   False   ActorResumeFailed   resuming actor default/validate2: rpc error: code = ResourceExhausted desc = no free workers available
    

    Because of #367, the Task then stays Failed even after a pool is added.

    Reproduced on

    • AX v0.3.1 (e70162a) with Agent Substrate v0.2.0, on kind (Docker Desktop, arm64), 2026-09-28.
    • Earlier, AX v0.3.0 on GKE Standard 1.36.3-gke.1767000.

    main (ac23328) still has no WorkerPool manifest under deploy/ or examples/, and the README and docs/ do not mention one.

    Workaround

    With this pool in the Task's atespace, a new Task ran and ax ssh worked. It is adapted from Substrate's demos/counter/counter.yaml.tmpl and applied with ko apply from a Substrate checkout:

    apiVersion: ate.dev/v1alpha1
    kind: WorkerPool
    metadata:
      name: ax-pool
      namespace: default   # the Task's atespace
    spec:
      replicas: 2
      workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor
      template:
        nodeSelector:
          ate.dev/substrate-version: "v0.2.0"   # must match the label the install puts on the nodes
        resources:
          limits: { cpu: "1", memory: 1Gi }
          requests: { cpu: 250m, memory: 1Gi }

    Suggestion

    • Add a minimal WorkerPool step between "Deploy the control plane" and "Run your first task", or have make deploy create a small default pool.
    • Mention the ate.dev/substrate-version node label. A pool whose nodeSelector does not match it never schedules its workers, and the Task fails with the same ResourceExhausted.

    Activity

    1. cloudonly commented on Sep 28, 2026

      @cloudonly
      Contributor

      Fix ready on my fork: docs/quick-start-workerpool-and-examples (diff vs upstream) — PR creation is currently rejected by the repo's collaborators_only policy (as tracked in #358), so I'm leaving the branch here and will open the PR as soon as that allows.

      What it does for this issue:

      • Adds a "Create a WorkerPool" step to the README quick start, between deploying the control plane and running the first task, with the failure mode spelled out: the ResourceExhausted surfaces at ax resume, not ax apply — the task stays Suspended until then.
      • Adds examples/workerpool.yaml, adapted from Substrate's counter demo, including the ko apply-from-a-Substrate-checkout flow for the ko:// worker image and ate.dev/substrate-version nodeSelector guidance (checked via kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version; source installs label nodes with a git-describe string like v0.2.0-10-g1d7ca8ce, so the example uses a placeholder rather than a hardcoded tag).

      One correction to the workaround in the issue body: the pool does not need to live in the task's atespace. Worker selection is cluster-wide — matched by sandbox class and label selectors, with no namespace dimension (see Constraints in cmd/ateapi/internal/scheduling/scheduling.go and the "pool selection is cluster-wide" note in Substrate's multi-template demo). Any namespace works; the namespace only decides where the worker pods are managed.

    2. cloudonly commented on Oct 19, 2026

      @cloudonly
      Contributor

      Friendly nudge on this one. The branch docs/quick-start-workerpool-and-examples (linked above, single commit) still applies: it test-merges cleanly against current main (8888435 — the only main-side change to these files since the branch was posted is the logo swap in #497), and its premises still hold on today's code: CreateTask still stores new tasks as Suspended, and multi-workspace bindings are still valid.

      Following the pattern from #487 (branch on a fork, maintainer opens the PR from it with the contributor's commits): would a maintainer be able to open the PR from cloudonly/ax:docs/quick-start-workerpool-and-examples? The commit carries the WorkerPool quick-start step for this issue, the resume-flow fix for #426, and the multi-workspace example for #425. Happy to split it into separate branches if you'd rather have one PR per issue.

    3. mayawang commented on Oct 20, 2026

      @mayawang
      Collaborator

      Thanks @cloudonly for your patience on this, and @eren23 for verifying it on kind. One change before I open the PR: #496 is moving AX to one workspace per task, so the multi-workspace example won't be valid after it lands. Could you drop that example, or switch it to a single workspace? I'll open the PR from your branch right after.

    4. cloudonly commented on Oct 21, 2026

      @cloudonly
      Contributor

      Done — examples/multi-workspace.yaml now binds a single git-backed Workspace (171dbdb..2b04656 on the branch above), and it still test-merges cleanly against current main (7b3347f).

      I kept the filename so the docs/manifests.md link (#425) keeps resolving. One note: the paragraph around that link still describes multi-entry binding ("first entry is the working directory", unique paths), so it will need a pass once #496 lands — happy to adjust that too, or rename the file and update the link, whichever you prefer.

    5. mayawang commented on Oct 21, 2026

      @mayawang
      Collaborator

      Thanks @cloudonly, opened as #505 from your branch. Yes please on the docs/manifests.md paragraph, now that #496 has landed. Heads-up that we're planning to slim the README down to a low-bar path (local mode in #504, and a single make deploy once images are published), so some of the quick-start steps may move to docs/ later. Your fixes make today's quick start correct, which is still worth landing.

    6. mayawang commented on Nov 4, 2026

      @mayawang
      Collaborator

      Fixed by #505: the quick start now has a WorkerPool step and examples/workerpool.yaml. Thanks @cloudonly!

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      No labels
      No labels

      Type

      No type

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions

        0