8000 Let tasks configure how long the runner waits after SIGTERM before killing the command · Issue #446 · google/ax · GitHub
[go: up one dir, main page]

Skip to content

Let tasks configure how long the runner waits after SIGTERM before killing the command #446

Open

Description

@mastersingh24

When a task is stopped or suspended, the runner sends SIGTERM to the task command and then kills it 10 seconds later. The 10 seconds is a constant in runner/runner.go (stopGracePeriod), so neither the task nor a custom image built on the runner package can change it.

That's too short for agent harnesses that do real work on shutdown. For example, we're trying to run mast as an AX task. On SIGTERM it finishes or checkpoints the turn in flight, flushes its session database under /workspace, and then exits. That can take minutes; its Kubernetes manifests use a 330s termination grace period. With AX, mast gets killed partway through, the turn in flight is lost, and on resume it has to redo work that already had side effects.

The default doesn't need to change. 10s is fine for most commands. We'd just like to be able to raise it.

Suggestion

  • Add a StopGracePeriod time.Duration field to runner.Config. Zero keeps today's 10s default.

  • In ax-task-runner, add a --stop-grace-period flag that defaults to the AX_STOP_GRACE_PERIOD enviro 7576 nment variable. Because spec.env entries are set in the container environment, a task can opt in without needing a custom image:

    spec:
      env:
        - name: AX_STOP_GRACE_PERIOD
          value: 120s
  • Mention it in docs/runner.md.

The runner can only wait as long as Agent Substrate lets the container live after SIGTERM, so the docs should say that a value past Substrate's own limit won't help. I haven't confirmed what that limit is during suspend. If a maintainer knows, that would be good to document too.

I have a small PR ready for this.

Activity

  1. mastersingh24 commented on Oct 7, 2026

    @mastersingh24
    ContributorAuthor

    A correction to the premise of this issue: after testing on a real cluster (Agent Substrate v0.1.0-gke.1 and v0.3.0), suspend doesn't send SIGTERM at all. Substrate checkpoints and freezes the sandbox. So a longer grace period helps stop and delete, but it does nothing for suspend. I've written up the suspend behavior, with a reproduction, in #451.

  2. added
    devxDeveloper experience: onboarding, defaults, and gaps found building on AX
    on Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    devxDeveloper experience: onboarding, defaults, and gaps found building on AX

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0