8000 boot-agents: codex resume-on-relaunch, with a verified fallback by simon-inkie · Pull Request #18 · simon-inkie/inkie-deck · GitHub
[go: up one dir, main page]

Skip to content

boot-agents: codex resume-on-relaunch, with a verified fallback - #18

Merged
simon-inkie merged 2 commits into
mainfrom
chore/port-boot-agents-codex-resume
Sep 4, 2026
Merged

simon-inkie merged 2 commits into
mainfrom
chore/port-boot-agents-codex-resume

Conversation

@simon-inkie
Copy link
Copy Markdown
Owner

Summary

Ports the internal fleet's largest fix of the three landed this week: Codex-runtime agents now get resume-on-relaunch (a relaunched agent whose session crashed or needs restarting resumes cleanly rather than losing state), plus auto-dismissal of Codex's own "update available" dialog on launch so a fresh boot doesn't get stuck behind a dialog nobody's there to click through.

Four new functions plus a dedicated codex branch in launch_agent. The core safety property: a session file is only ever quarantined (renamed aside) on a positive observation of a crash AND a successful fallback send, never on ambiguous evidence — an empty process read, mid-poll or single-check, is always classified inconclusive, never assumed to be a crash. A single transient hiccup can't quarantine a healthy session.

Verification note

Every ported invariant was mutation-checked: broke each one in turn on a scratch copy (dropping the crash+send AND-gate, misclassifying an empty read as crash, weakening the healthy-streak threshold, dropping launch flags from the fallback, headline-only dialog matching, not resetting the streak) and confirmed the suite catches each one specifically, not just generally. I independently re-read the crash-classification function myself and confirmed the AND-gate and dual-path empty-read handling directly in the code, not just from the report.

Found and NOT silently fixed, flagging separately: porting this surfaced a genuine latent bug in the internal source file itself (not this repo) — cmd is declared without an initialiser and read back in the inconclusive-path message; under set -u, if the poll's max-checks is ever configured to 0, that's an unbound-variable crash. This OSS port initialises it defensively. Flagging back to the internal side separately, out of scope for this PR.

Test plan

  • node scripts/leak-gate.mjs, licence-gate.mjs both clean
  • boot-agents.test.sh: new suite, 31/31 passing, registered in shell-suites.test.ts
  • Mutation-tested against 6 distinct invariant breaks, each caught by a specific case
  • Unaffected suites still green: pane-idle (30), inbox-push (80), inbox (69), activity-sweep (84), stuck-pane-sweep (82)

Ported from the internal fleet deployment this package was extracted from.

A codex-runtime agent had no resume path at all. claude agents resume with
`claude --resume <name>`, but a codex agent launched a bare `codex`, so every
relaunch started genuinely fresh: a crash, a manual restart, or an incidental
relaunch as a side effect of another agent's boot silently discarded the whole
conversation. It bit live, where an unrelated boot run reset an agent mid-task
and read as a wedged pane rather than as a lost session.

WHAT LANDS

codex_latest_session_for_workspace scans the Codex sessions tree
mtime-descending, reads only the first line of each rollout file (a
session_meta record), and applies three filters that are all load-bearing: an
interactive originator, a user-owned thread, and an exact workspace cwd matched
through its closing quote so a short agent name cannot match a longer one that
starts with it. The id comes from the parsed payload rather than the filename,
and is cross-checked against the filename's own uuid so a format change cannot
quietly start yielding wrong ids. CODEX_SESSIONS_DIR is the test seam.

codex_verify_resume_or_fallback then confirms the launch actually worked. A
selected id is not a promise that `codex resume` accepts it: a corrupt session
file, a version incompatibility, or a session locked elsewhere drops the pane
to a bare shell, which is worse than never resuming at all, since the old
behaviour at least always produced a live agent. A pane that fails to come up
as codex/node gets a fresh codex re-sent into it carrying the same name and
flags as the primary send, and the offending session file is renamed, never
deleted, so the next boot stops re-selecting it.

The same verification window dismisses Codex's own update-available dialog.
Whenever a newer release is published, every launch shows a blocking dialog
waiting on a keystroke a headless boot never sends, and because Codex is itself
a node process the pane still reads as a live `node` while it sits frozen
behind it. The dismissal selects "skip until next version" rather than the
default-highlighted "update now", which would run an unattended global npm
install in the middle of a boot. It is deliberately not gated on a selected
session id: the dialog belongs to codex startup in general, so a first boot
meets it too.

CLASSIFYING THE OUTCOME IS THE HARD PART

Most of the care here is in telling three situations apart, because getting
them wrong is expensive in both directions.

  - A dismissed dialog only proves the dialog is gone. It sits in front of
    session loading, so a single check straight after it can read a live node
    a moment before the load fails. That path therefore polls, bounded, and
    requires three CONSECUTIVE healthy reads before trusting the pane, because
    a crash in progress can still show one transient good read.
  - A crash is a NON-EMPTY read that is not codex/node. An EMPTY read is not a
    crash, on either path: it means the pane could not be observed at all, it
    resets the healthy streak, and it classifies as inconclusive.
  - Inconclusive never returns success. It logs wording distinct from a
    confirmed crash and takes the same safe recovery, because declaring an
    uncertain success is the one forbidden outcome.

The quarantine gate follows from that. Attempting a fallback send is always
worth it. But the rename requires BOTH a positively observed crash AND a
successful send. Send success alone answers only "did tmux accept keys
somewhere", and target existence only answers "is there somewhere to send";
neither converts an unobserved pane into a session fault. Quarantining a
healthy session silently drops the next boot onto an older conversation, which
is the exact continuity loss this mechanism exists to prevent. On an
inconclusive outcome the target's existence is checked directly with `tmux
list-panes`, a different question from the process-name read, and a
confirmed-gone window is escalated with no send attempted and no file touched.
Recreating a lost window is deliberately out of scope: creating windows belongs
to launch_agent, which already knows the name, directory and session.

ADAPTATIONS FOR THIS REPOSITORY

Paths are AGENTS_HOME-relative rather than assuming where agent homes live. The
fallback re-send carries this package's own launch prefix and the per-agent
model flag, which is what the internal version's environment prefix amounts to
here: the point of that argument is that a crashed process takes its launch
context down with it, so a bare re-send boots an agent that does not know what
it is or which model it should be running.

TESTS

New packages/base/test/boot-agents.test.sh, 31 assertions, registered in
shell-suites.test.ts so it actually runs under vitest rather than sitting where
it can never go red. No live tmux and no live codex: boot-agents.sh is sourced
behind a BOOT_AGENTS_LIB_ONLY guard that stops it before any live tmux call,
session fixtures live in a scratch directory, and the pane check and tmux are
stubbed with controllable exit codes so the send-success and target-existence
gates are exercisable.

Mutation-checked rather than assumed. Each of these goes red on exactly the
cases named, and nothing else:

  drop the outcome == crash half of the quarantine gate -> D11b, T13
  classify a no-dialog empty read as a crash            -> T13
  lower the consecutive-healthy threshold to one        -> D10, D12
  drop the launch flags from the fallback re-send       -> 7 cases
  match the dialog on its headline alone                -> D2
  stop resetting the streak on an empty read            -> D12

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@simon-inkie
Copy link
Copy Markdown
Owner Author

Model-Diverse Peer Review: PR #18 (inkie-deck, commit 5fbed1a)

Verdict: FULL APPROVAL


1. Audit of Codex Resume & Session Quarantine Architecture (packages/base/bin/boot-agents.sh)

  • Candidate Discovery & Robust Filtering (codex_latest_session_for_workspace):

    • Scans $CODEX_SESSIONS_DIR (~/.codex/sessions) ordered by mtime descending, reading only line 1 (session_meta).
    • Strict filtering criteria prevent cross-talk and subagent confusion:
      1. originator == "codex-tui" excludes non-interactive headless executions.
      2. thread_source == "user" isolates primary interactive conversations from subagent threads.
      3. cwd == "$AGENTS_HOME/<agent>" matches through the trailing quote, guaranteeing prefix isolation (e.g. wren cannot accidentally claim wren-scratch sessions).
      4. Cross-checks parsed payload id against the filename's UUID suffix; skips malformed entries safely.
    • Returns empty gracefully on missing directories or malformed files, allowing first-time boots to start fresh.
  • Non-Interactive Update Dialog Dismissal:

    • Detects the blocking "update available" modal in pane_text and automatically sends keystroke 3 ("Skip until next version") without Enter.
    • Solves the silent freeze where headless boots previously stalled indefinitely awaiting user input while still presenting a live node process.
  • Multi-Poll Health Verification & Quarantine Boundaries:

    • codex_verify_resume_or_fallback polls the pane command repeatedly over a structured verification window.
    • Distinguishes between healthy startup, transient states, and confirmed crashes.
    • Quarantine Discipline: Quarantines (.corrupt-<timestamp>) strictly upon confirmed crash (pane dropped to bare bash/shell).
    • Preserves session files uncorrupted if the failure is unobservable or inconclusive (e.g. missing target pane or empty capture read).

2. Test Verification

  • Executed packages/base/test/boot-agents.test.sh: exactly 31 passed / 31 tests.
  • Comprehensive coverage across all discovery filters (T1 to T9), healthy resumes (T10), crash fallbacks (T11), update dialog dismissals (D1 to D8), and crash/quarantine classifications (D9 to D13).

codex_pane_has_update_dialog required both of its anchors somewhere in the
last 20 lines of the pane. A 20-line window is not a small place: a resumed
Codex session's most recent exchange can quote or discuss this very dialog a
few lines above a clean, ready composer, and both the quote and the composer
then sit inside the same window. That pane read as "dialog pending" when
nothing was pending.

The consequence is worse than a missed dialog. The caller responds to a true
return by sending the bare keystroke "3", Codex's "skip until next version",
with no Enter. Against a live dialog the digit selects and confirms. Against
an idle pane there is no dialog to swallow it, so it lands as a stray
character typed into the composer text box. It is never submitted, so nothing
errors and nothing is logged: it just sits there as staged text, and staged
text is exactly what the live-push composer-clean gate refuses to type over.
One false positive on one boot therefore takes that agent off the delivery
path indefinitely, and the only evidence is a single "3" nobody typed.

The footer anchor now has to be the last non-blank line of the tail window
rather than merely present in it. A live dialog's footer IS the bottom of the
pane, because Codex draws the dialog where the composer would be; a reference
to the dialog in transcript text always has something under it, if only the
composer prompt itself. Position alone rejects the whole class. The headline
stays a loose match inside the window, since it sits several lines above the
footer in a real render and is not the anchor doing the work. Containment on
that line rather than equality with it, so indentation or a future boxed
render cannot turn this into a false negative that leaves an agent frozen
behind a real dialog.

New negative case D3b covers the detector, and D6b covers what actually
matters: no keystroke is sent for that pane. Both fail against the previous
implementation, D6b with "send-keys -t <target> 3" in the log. The existing
live-dialog positive case is unchanged and still passes.

Found by independent review.
@simon-inkie
Copy link
Copy Markdown
Owner Author

Model-Diverse Re-Review Confirmation: PR #18 (inkie-deck, commit 73b03db)

Verdict: FULL APPROVAL


Audit of Bottom-Anchoring Fix (packages/base/bin/boot-agents.sh)

  • Last Non-Blank Line Property Verified:

    • codex_pane_has_update_dialog now extracts the last non-blank line of the 20-line window (grep -v '^[[:space:]]*$' | tail -n 1) and verifies containment of Press enter to continue.
    • The headline Update available! remains a loose match in the 20-line window, but requiring the footer on the last non-blank line ensures quotes discussed earlier in chat scrollback (with an idle composer prompt beneath) return false.
    • Suffix containment accommodates indentation variants while safely preventing false positives.
  • Test Suite Verification:

    • New negative tests D3b (detector returns false when update dialog is quoted above an idle composer) and D6b (confirms no stray 3 keystroke is typed into a ready composer) pass cleanly.
    • boot-agents.test.sh: 33 passed / 33 tests.

@simon-inkie
simon-inkie merged commit 303a456 into main Sep 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0