Repository navigation
Custom ActorTemplate creation errors are swallowed; Task fails with misleading 'actor template not found' (e.g. tag-referenced image) #366
Description
Summary
When the custom ActorTemplate for a Task cannot be created on Substrate, the controller logs a WARN and falls back to the configured default template (--template, default default-template in atespace ax-system). In a stock deployment that template does not exist, so the Task fails with ActorCreationFailed: actor template not found, which points at the wrong problem. The real cause is only visible in the controller log.
The most common trigger is a tag-referenced image: Substrate rejects it with actor_template.containers[0].image: Invalid value: "...": must be pinned by digest (changing the image invalidates snapshots). Nothing in the AX docs or CLI mentions the digest requirement.
AX d8ed0fe, Agent Substrate on a single-node kind cluster.
Reproduction
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
name: tagged-image
atespace: default
spec:
image: "localhost:5001/my-agent:v1" # tag, not digest
command: ["python3", "/app/agent.py"]ax describe task tagged-image:
Conditions:
TYPE STATUS REASON MESSAGE
Ready False ActorCreationFailed creating actor default/tagged-image: rpc error: code = FailedPrecondition desc = actor template not found
Controller log at the same time:
level=WARN msg="could not create custom ActorTemplate, falling back to default template" error="creating actor template default/tagged-image-tmpl-...: rpc error: code = InvalidArgument desc = actor_template.containers[0].image: Invalid value: \"localhost:5001/my-agent:v1\": must be pinned by digest (changing the image invalidates snapshots)"
Switching the image to localhost:5001/my-agent@sha256:... makes the Task run.
Expected
The Task should surface the template creation error itself, e.g. Ready=False, reason=TemplateCreationFailed, message=<Substrate error>, and the reconcile should stop there. Falling back to a template that carries a different image (or does not exist) is never what the user asked for.
Suggested fix
In internal/controller/reconciler.go, the branch after EnsureActorTemplateWithImage currently only warns and continues. Treat the error as terminal for that reconcile: set the condition, set phase: Failed, return. Optionally validate the image reference in ax-server (reject tag-only references, or resolve them to a digest) and mention the digest requirement in docs/manifests.md under Task.
Activity
Update contribution guidelines and explain the upcoming refactor (goo…
ericcurtin commentedon Sep 27, 2026
Fix ready: template creation errors now fail the task (Ready=False, reason TemplateCreationFailed) instead of a silent fallback. Also notes the digest requirement in docs/manifests.md.
Branch: https://github.com/ericcurtin/ax/tree/fix/terminal-template-creation-failure
go test ./...passes, includes a new regression test.PR creation looks restricted to collaborators here, so I can't open one directly. Happy to if given access, or feel free to cherry-pick the branch.
Unrelated, but if useful for your workflow: https://github.com/llmmanorg/llmman
fix(controller): fail task when ActorTemplate creation fails
fix(controller): fail task when ActorTemplate creation fails
fix(controller): fail task when ActorTemplate creation fails
fix(controller): fail task when ActorTemplate creation fails
fix(controller): fail task when ActorTemplate creation fails
fix(controller): fail task when ActorTemplate creation fails (google#472