Repository navigation
The live mission run: make e2e-mission, first runs measured, GOAL MET (M9.13) - #115
Open
adeelahmad wants to merge 13 commits into
Open
adeelahmad wants to merge 13 commits into
adeelahmad wants to merge 13 commits into
Conversation
scripts/smoke/e2e-mission.sh seeds the to-do fixture (scripts/smoke/fixtures/todo-mission: an add() that drops the task, one passing seed test, the pinned single-file journey runner) as `init`, then `aloop init`, and records the seed sha. Before that it refuses: no --yes, an unknown arg or host, a missing CLI, an E2E_DIR that is not empty, inside a git repo or under a CLAUDE.md, and codex/opencode live (dry run only). The environment is scrubbed by prefix for the seed and every session (AGENTIC_*, ALADDIN_*, ALOOP_*, GATE_*, SKIP_HOOKS, TASK_ID, ATTEMPT, AGENT_ROLE, *_SPRINT_LOCK_TTL, the parent session's ids, GIT_DIR). Zero-token self-checks run before anything is spent: the runner twice gives the same see:false, pytest (when installed), no __pycache__ under journeys/, tick --peek SET_MISSION (exit 3), exactly 2 commits, the runner tracked and docs/agents/ ignored, the seed left clean. --dry-run (make e2e-mission YES=1 DRY=1) writes plan.json through scripts/smoke/mission_plan.py: the first prompt and its sha256, the seed sha, argv for the first, reply, surface and cron sessions built from cron.host_cmd plus the measured flags (a new --session-id each, never --resume), the caps, the allowlist (mission approve, baseline confirm), the stop reasons and a failing crontab shim first on the sessions' PATH. It never runs a host. The live path runs the zero-token preflight (no enabled aloop plugin, the network, pytest), then stops with "driver not built" until scripts/smoke/mission_driver.py exists. scripts/smoke/mission_summary.py is run-summary's mission mode (--sessions DIR --kind mission; plugin/ is untouched here): each session's last result line summed, a session without one flagged cost_complete false, gate lines parsed as JSON (a stuck release in either spelling is counted and never a pass), the ledger row. docs/evidence/README.md gets a `## Mission runs` table with no rows and a header naming both run scripts; evidence.test.py checks `| mission N |` rows, rejects a placeholder row, and counts mission rows in the public-figure check. Tests (zero tokens): scripts/test/mission-e2e-live.test.py, 9 checks: the fixture through the real mission CLI gives 1 journey and 1 gap at baseline, then GOAL MET with 1 earned after fix.diff in one sprint; the guards; the dry run and its plan; an env spy on every git the seed runs; the preflight; the summary. evidence.test.py 6, e2e-live-hosts 14, bash-portable 2, mission-e2e 8, make test-hosts, pycheck, shellcheck and check-tracking green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
scripts/smoke/mission_driver.py drives the live run from plan.json. The first session gets the typed answers. After that every step is a new session (never --resume), each in its own process group and holding the mission-loop lock, with ALOOP_MISSION_LOOP set to the driver pid, the environment scrubbed by prefix and the crontab shim first on PATH. The relay types only approve and confirm lines. Each line is rebuilt from disk (the `mission check` sha256 for the mission, the baseline digest for the baseline) and must match the peek's subject. It must also have been shown in the previous session's final result text; otherwise the run gets one surface session and then stops. Before anything is typed the pre-grant checks run: the typed answers and set_by; one journey; the seeded runner unchanged; measured twice, agreeing and failing. After a reply, "hook did not capture" and "captured, not granted" are reported as separate stops. Any other human action, ASK_CONSENT included, stops the run. A WAIT sleeps once per key, at most 15 minutes, and never past the wall-time cap. A role or supervisor tick runs a cron session only when due (record=False, at the peek's now). The driver never writes a mission.tick or a ledger row. Caps: sessions, a per-session timeout (rc 124/137/142, the group killed, no retry), total wall time, no progress, and an outer USD cap on max(host-reported, tick-priced) with --max-budget-usd = min(10, remaining). After goal.met, a RESEARCH, DEFINE_JOURNEYS or BASELINE tick kills the session. 22 post-run checks, each a function, all printed, exit 1 on any failure and no traceback. They cover: mission.json and the grants cite the relayed lines; every human record is by scripted-human (e2e); RED, GREEN and final pass from JSON gate lines (a stuck release is never a pass); goal.met and evaluate check; pytest on main; no suppressed tests; every commit after the seed traces to the mission scripted-human (e2e) approved; crontab unchanged; the cost within the cap. The driver writes driver.json. e2e-mission.sh runs the driver and then mission_summary.py (summary.md gains the stop, the path and the check list). The poll interval moves to E2E_POLL_S. scripts/test/mission-e2e-live.test.py (17 checks) adds a stub host (scripts/test/fixtures/mission_host.py). It runs the real plugin CLI through first/reply/reply/cron to GOAL MET with 22/22 checks, and covers an idle host, forged and wrong lines, no hook and no grant, the sessions, USD, timeout and no-progress caps, gray ethics, tampering, two journeys, a changed runner, stuck gate releases and WAIT. A mutation pass (shown always true, kill the leader only, host cost only, stuck counted as a pass, no pre-grant, a reply without the lead) fails the test each time. Open: the live run (make e2e-mission YES=1) and its ledger row; M9.51 (mission.set carries `by`) and M9.52 (fold the summary into run-summary) added to the ROADMAP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
Environment. The session and seed scrub now also drops the gate contract (REPO_DIR, STORY_DIR, SPRINT_DIR, BASE_REF, SCOPE_GLOBS, ... plus every name in _gatelib's TASK_ENV_KEYS, read from the file), BASH_ENV, GIT_AUTHOR_*/GIT_COMMITTER_*/GIT_CONFIG*, PYTEST_*, and the host's model, effort, thinking and compaction settings. plan.json and summary.json name the CLAUDE*/ANTHROPIC* variables a session still inherits (auth, proxy), never a value. A cap that is nan or inf is refused. Grants. The pre-grant checks compare mission.json whole against the typed answers (the default veto window, no extra key, provenance). They refuse a benchmark, a rival, a looser tolerance, a runner command other than `python3 <runner>`, and any file beside the runner. Each relay records the subject and sha256 it checked. Post-run, exactly one human row must cite the line and cover that content, and the content on disk must still match. New checks: ledger integrity (ledger_problems, B.current_intact, `mission check`) and commit authorship. A restore or retire is outside the allowlist. Every mission.set must be of the mission.json checked. Gates. A pass is the host's own run (selfcheck false), exit 0, no `released` field, and not a budget release (a task.*_hard_limit event or a budget-exceeded marker for that task attempt). The driver and the summary share this rule. The summary's grants are the ledger rows that verify, and it lists the ledger's problems. Runs. The seeded runner drops its own dir from sys.path before it imports anything. The suppressed-tests check scans every .py file pytest would read and the pytest configs, in the repo and above it. pytest runs with -rA, and the check wants a PASSED line for every test. The repo's own code (runner, pytest) runs as a session would: shim first on PATH, its own process group. The outcome is read before that code runs. A SIGTERM, SIGINT or SIGHUP, or a failure of the driver itself, kills the session's group and still writes its record and driver.json. e2e-mission.sh passes its signals on to the driver. Tests. mission-e2e-live.test.py: 35 checks, offline, zero tokens. New blocks cover each finding: every pre-grant check alone, every post-run check broken alone, gate passes, WAIT, the grant, total-time and cut caps, host failures, refusals and preflight, children and SIGKILL, the loop lock, a planted goal.met test, signals, a driver failure, and nan caps. Each fix was mutation-checked: 69 mutations, each reverting one fix, and every one fails the suite. ROADMAP: M9.60 (_gatelib logs released: budget) and M9.61 (post-run code on a throwaway clone) added as gaps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
…9.13) F14, the post-run code. contained() now has its own test: code that exits and leaves a child behind, code that times out (124), and a child that ignores SIGTERM all leave nothing alive in the group (t_contained). t_plant also runs the happy cycle with a planted test that, while the post-run pytest runs it, cites the relayed confirm line again from another session. driver.json's grants still name their reply sessions. mission_summary.py still reads the ledger after that code has run, so M9.61 stays open. Finding 5, suppressed tests. The scan reads every file except .git and the sessions' worktrees, not only what pytest collects, so a hook in build/, dist/ or a virtualenv is found. It counts any pytest_* name, _pytest, pluggy, hookimpl/hookwrapper, sys.modules, an aliased or from-imported skip, pytest.exit and SkipTest. A conftest.py may hold only a docstring, os/sys/pathlib imports and a sys.path insert of a path, checked on its AST. Anything else is reported. So are a module that shadows the product (tests/todo.py, lib/todo/), a module no scan can read (.pyc outside __pycache__, .so, .pyd) and pytest 9's pytest.toml. The checks' runs of the repo's code set PYTHONPYCACHEPREFIX to an empty dir, so they never read a planted __pycache__ entry (other code under the source's own mtime and size). t_pytest_wiring drives check_pytest through a fake pytest. It checks -rA, no cache, rootdir and confcutdir, and that a skipped, xfailed, xpassed or deselected count, a missing PASSED line or exit 1 fails. The stand-in pytest prints PASSED lines only with -rA, as pytest does. Flaky tests. The stub writes its child's pid to a temp file and renames it, and starts the child before the prompt hooks. read_pid() waits for a pid, not just a file. In t_driver_bug the stub starts its child 1s late, and the injected bug strikes only once the child exists. A test's cleanup never kills its own process group, so a broken driver that runs a session in the test's group fails a check instead of killing the suite. ROADMAP: M9.62 added. The gates also read a planted bytecode cache. Tests: mission-e2e-live.test.py has 38 checks (was 35) and passes with the stand-in and with a real pytest (9.1.1), and in 4 copies run at once. evidence, e2e-live-hosts, mission-e2e, make pycheck, make test-hosts and check-tracking pass. 17 mutations each revert one fix or take back one race fix; each one fails the suite (run 3 at a time, with no other block failing). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
…(a name in a regex comment is no read) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
…rney (M9.68) The first live mission run stopped safely at the baseline's pre-grant check (11/24, $0.33). What it found: - A reply session knows no name: the supervisor wrote --by from the account's context. Every reply prompt now names who types the line, and the stub host picks a name of its own when none is given. - The pre-grant checks judged the first journey file, a stray draft, instead of the baseline's own journey. - The rival constraint now says the baseline lists none. - journeys new wrote three broken drafts the researcher could neither fix, delete nor retire, so its stop gate was stuck-released. It now refuses a draft that journeys check would block and writes nothing. ROADMAP: this branch's M9.60-64 become M9.63-67 (#105 uses M9.56-62); new M9.68 (done) and M9.69 (a grant's by is bound to its typed line). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
…second On a fresh clone a captured line from the second of the previous approval reads as older than it (stale), so a fast CI runner got the ask again. later(p) makes the order explicit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
# Conflicts: # scripts/test/mission-tick.test.py
…tree; file M9.70 Run 2 reached GOAL MET (22/24, 4 sessions, 1 sprint, $1.80 by tick's estimate). Its two misses: - The seed-test check compared bytes, though RED appends its tests to that file. It now passes when every statement of the seed's file is kept in order and only imports, defs and classes were added. - GREEN's stop gate was stuck-released: the sprint's gate matrix runs the journey runner, which worktree-hygiene sparse-checks out of the task worktree (docs/agents/). Filed as M9.70. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
# Conflicts: # CHANGELOG.md
…t once, ids M9.80-87 An adversarial review of the live-run fixes confirmed 9 findings: - seed_test_kept accepted an added def that rebinds the seed's test name (replacing it), an import or class shadowing TodoList, and code run at import through a decorator, default, annotation, base or class body. Added statements must now rebind none of the seed's names and run no code at import (pytest's own decorators with plain arguments). - No test pinned check_product's wiring: it now has one on the stub run's repo (an append passes, a redefinition fails). - journeys new returned on a runner problem before the other problems and the "nothing written" footer; it now names every problem at once. Its usage offers --from as the one form. - ROADMAP ids M9.63-70 clash with #105 and #111: now M9.80-87. - Docs: 39 checks, the M9.87 block count, run 1's three stop causes, and a test that pins the rival constraint's new wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Before: M9.13's offline half (#91) proved that
mission tickcan drive a mission to a met goal at zero tokens, but nothing could run one real cycle on a host and measure it.make e2e-mission, which the ROADMAP names, did not exist.After:
make e2e-mission YES=1runs one live mission cycle on the claude host, andmake e2e-mission YES=1 DRY=1plans one without spending anything. The script seeds a to-do repo whereadd()drops the task and the journey runner is pinned. It then plays the human in three ways only:Every human record names
scripted-human (e2e), so a run proves the relay and the loop, not a human's review. The run stops atgoal.metand runs 24 post-run checks. Its evidence row goes in a new## Mission runstable, which now holds the first two runs.How:
Fixture (
scripts/smoke/fixtures/todo-mission/): the buggy app, one passing seed test, and the runnerdocs/agents/mission/journeys/runners/todo_journey.pywith steps open, add, see. It is committed beforealoop-initignoresdocs/agents/.fix.diffis used only by the offline test.e2e-mission.shandmission_plan.py:--yes, the host, the CLI, and anE2E_DIRthat is non-empty, inside a git repo or under aCLAUDE.md. Codex and OpenCode are dry-run only.AGENTIC_*,ALADDIN_*,ALOOP_*,GATE_*, the gate contract,GIT_*and the parent session's variables.ALOOP_MISSION_LOOPis then set as a recorded deviation.plan.json.mission_driver.py:/aloop:mission, the line, and the name for the records; there is no--resume.cron.host_cmdwith--setting-sources project --strict-mcp-config, runs in its own process group with a timeout, and has a failingcrontabshim first on PATH.goal.metis killed.mission.tickor a ledger row.Post-run checks (24): a few examples follow; the full list is
CHECKSinmission_driver.py.conftest.pywith code, configaddopts, shadowed modules).crontab -lis unchanged.Evidence:
mission_summary.py --sessions DIR --kind missionwritessummary.json,summary.mdand the row.evidence.test.pychecks| mission N |rows.ROADMAP: the M9.13 row says the harness is done, records the two live runs, and leaves a 24/24 run open. New gap rows, each with its reason:
mission.sethas noby.run-summary.byis not bound to the human who typed its line.(The door rule on every host and role (M9.31) #105 and The mission loop hands out a goal's commands only when the gate takes them (M9.53) #111 use ids up to M9.70, so these take M9.80 to M9.87.)
The live runs' fixes. Run 1 stopped safely at the baseline's pre-grant check; run 2 reached GOAL MET. What they found, each fixed with a check that fails without it:
mission journeys newwrote broken drafts that the researcher could neither fix, delete nor retire, so its stop gate was stuck-released. It now refuses a draft thatjourneys checkwould block, names every problem at once, and writes nothing (M9.85). This is the one change underplugin/, with a line in the researcher's prompt.--by. Every reply prompt now says who types the line, and the stub host picks a name of its own when none is given.Review before opening. Three adversarial reviewers (acting as the human, gate invariants, cost and test validity) raised 25 findings. Skeptics confirmed 24, and all 24 are fixed. A recheck then reverted each fix and confirmed its covering check goes red (69 single-fix reverts, plus 17 more for the recheck's follow-ups). That recheck also found a race in two of the new tests under load, which is fixed.
Evidence
scripts/test/mission-e2e-live.test.py: 39 checks, zero tokens, with a stubclaudethat captures each prompt through the real UserPromptSubmit hook and drives the real mission CLI. It passes with the stand-in pytest and with real pytest 9.1.1, and 4 of 4 with four copies running at once.evidence.test.py6,e2e-live-hosts.test.py14 andmission-e2e.test.py8 all pass. So domake test-hosts,make pycheck shellcheck json version-check eval-validate, andcheck-tracking.docs/evidence/README.md, ## Mission runs):scripted-human (e2e); the reply sessions do fire UserPromptSubmit. Its misses: the seed-test check (fixed here) and one GREEN stuck release (M9.87, the next PR).make test-hostspasses on the merged tree, and so domission-e2e-live.test.py(39 checks) andmission-journeys.test.py(11).check_product's wiring,journeys newnames every problem at once, the ids moved to M9.80-87, and the doc numbers are corrected.🤖 Generated with Claude Code
https://claude.ai/code/session_018RxETAY6S1gcmdctBK8a8Z