You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Task reports phase=Running and Ready=True indefinitely after its command has exited #346
A Task reports phase: Running with Ready: True and the message "Task is running and its workspace is ready" indefinitely after its command has exited. This holds for a clean non-zero exit, for a process that died on an unhandled exception, and for a task whose Substrate actor has moved to ACTOR_STATE_CRASHED.
TaskStatus carries no exit code and no condition describing the command, so there is no field that distinguishes "working" from "exited an hour ago".
AX v0.3.0 (d8ed0fe), Agent Substrate on a 2-node kind cluster.
Conditions:
TYPE STATUS REASON MESSAGE
GatewayReady True PoliciesApplied Network policies active
WorkspaceReady True SetupComplete Workspace setup completed at 10.244.1.65
Ready True TaskRunning Task is running and its workspace is ready
status.phase is Running. The exit code 42 appears nowhere.
Two further cases
A command that died on an exception. A python process exited with a traceback; six minutes later ax get task still reported Running/Ready=True. ps aux inside the sandbox showed only PID 1 /usr/local/bin/ax-task-runner and a defunct [git] — no application process at all.
An actor Substrate already knows has crashed. For another task, ax ssh refused with:
actor default/killtest unavailable: AssignWorker prerequisite not met for Actor:
default/killtest (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED or ACTOR_STATE_PAUSED)
At the same moment ax get task killtest reported phase: Running, Ready: True. So Substrate had the correct state and the Task resource did not reflect it. That last case looks like the most clearly actionable one — the information already exists a layer down.
Why this is being reported rather than worked around
docs/concepts.md is explicit that AX deliberately does not model the agent's shape, and I think that boundary is right — we are not asking for AX to supervise the agent. The gap is narrower: the conditions table documents Ready as "the task is running and WorkspaceReady is True", and ax watch/ax describe are presented as the way to follow a task, but neither can answer whether the command is still alive. A long-running task that dies quietly is indistinguishable from one doing slow work, for as long as you care to wait.
Some options, in what I would guess is increasing order of cost:
surface the actor state when it is ACTOR_STATE_CRASHED, since Substrate already reports it
a terminal phase (Succeeded/Failed) plus the exit code in TaskStatus when the command exits
a CommandRunning condition, leaving Ready meaning what it means now
Even the first would be enough to stop a dead task reading as a healthy one.
docs/runner.md on v0.3.0 already says the controller never starts spec.command as PID 1. It always launches /usr/local/bin/ax-task-runner, the command arrives only through AX_TASK_YAML, and "the control plane does not currently read the command's exit status back from the container." Stay-up-after-exit is required so /readyz and ax ssh keep working after the child dies.
Your ACTOR_STATE_CRASHED vs phase: Running case is the extra one: Substrate already has a terminal actor state, and the Task resource still prints Ready. I read the runner page at d8ed0fe; I did not run ax.
Fix ready: implemented on my fork (branch fix/346-task-phase-after-command-exit, regression test included, make test green) and ready to PR — but this repo's collaborators_only PR creation policy rejects PRs from non-collaborators (see #358), so I cannot open it. If PR creation opens for external contributors, I will open the PR immediately. Happy to adjust scope per maintainer feedback.
Activity
Skip white space only text part. (google#346)
foma-agent commentedon Sep 21, 2026
docs/runner.mdon v0.3.0 already says the controller never startsspec.commandas PID 1. It always launches/usr/local/bin/ax-task-runner, the command arrives only throughAX_TASK_YAML, and "the control plane does not currently read the command's exit status back from the container." Stay-up-after-exit is required so/readyzandax sshkeep working after the child dies.Your
ACTOR_STATE_CRASHEDvsphase: Runningcase is the extra one: Substrate already has a terminal actor state, and the Task resource still prints Ready. I read the runner page atd8ed0fe; I did not run ax.arbazkhan971 commentedon Sep 22, 2026
Fix ready: implemented on my fork (branch
fix/346-task-phase-after-command-exit, regression test included,make testgreen) and ready to PR — but this repo'scollaborators_onlyPR creation policy rejects PRs from non-collaborators (see #358), so I cannot open it. If PR creation opens for external contributors, I will open the PR immediately. Happy to adjust scope per maintainer feedback.api,cli: report task residency, actor UID, and outcome
api,cli: report task residency, actor UID, and outcome