8000 The live mission run: make e2e-mission, first runs measured, GOAL MET (M9.13) by adeelahmad · Pull Request #115 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
- The mission layer's six fixed roles (ROADMAP §4.12, M9.9): `market-researcher`, `product-tester`, `state-assessor`, `evaluator`, `mission-planner` and `ethics-reviewer`, each a persona with a four-case eval suite, a model route (the four that judge on the planning route, the two that measure on the worker route), hook matchers on Claude Code, Codex, OpenCode and dsh, and Codex/OpenCode agents and dsh presets from `aloop install` (which also renders the mission skill for OpenCode as `aloop-mission` once it ships). `plugin/mission/roles.json` names the paths each role writes. `gate-mission-scope` (PreToolUse) is the role path guard: a mission role never writes source, mission.json or another role's dir, never commits and never runs a human's approval; a builder never writes `docs/agents/mission/` or runs a writing `mission` command; the supervisor never hand-writes a role's artifact. No one hand-writes `docs/agents/mission/ledger.jsonl` or the captured prompts a grant reads (`<run data>/transcripts/`), runs a hook's `transcripts prompt|record|…`, or starts a host session (`claude -p`, `codex exec`, `opencode run`, `dsh <profile>`) from a tool call (in every aloop project since M0.9; its prompt would count as the human's words, M9.17), and no dispatched agent runs `mission baseline restore`, `events authority` or an `events emit` of a human event. `gate-mission-artifact` records what exists when a role starts and, at its stop, requires an artifact in its own paths that passes the core contract and its area's check, refuses a deleted artifact, a change to another role's or a ledger that no longer verifies, and records `artifact.created|updated` as provenance. A mission write never arms the sprint lock. The scope gate reads a Bash command as the shell would (`_shellread.py`: quotes, comments, line continuations, heredocs, wrappers such as `timeout` and `env`, `bash -c`, `eval`, a script run by an interpreter, variables and `$(mktemp)`). Each area's check runs once per artifact with its id or subject, an ethics hard line is the reviewer's finding, not a defect, and a derived goal whose only blocks are its pending ethics review waits on the Ethics Reviewer, not on the planner. Test: `mission-roles.test.py` (33 checks, with each role run through the real journeys, baseline, assess, evaluate and ethics, goal, research and judge areas), `mission-scope-bash.test.py` (718 commands), `opencode-plugin.test.mjs`.
- Its gates after a second adversarial review (each finding now a test) and M9.42: a human's confirm, restore or veto-window apply during a role's run, and what its command changed (journeys frozen, a version superseded), is no role's write and never blocks it, while a status no record explains still does. An overlapping run excuses only what it answers for as the file is now, and a run counts as open while it is active (its last tool call). The ledger is append-only during a run, with or without the run log. No role rewrites the human's or intake's intent, and a `source: mission` intent is the Mission Planner's. A research report belongs to the role its `role` field names, so the Product Tester files its own (`research new --role product-tester`); a judge run log and a usage report are the writing role's work, checked by their area; a draft in a role's own dir may be removed. The telemetry check's instrumentation problems and ledger problems that predate the run are notes, not the role's block; a malformed intent field and a timed-out check are ordinary blocks the stuck release can end; an OpenCode child continued for a new task starts a new run. Test: `mission-roles.test.py` (47 checks), `opencode-plugin.test.mjs`.
- Its gates after a third review (each finding now a test with its allow-control): a human event, or intake's own `intent.received`/`intent.normalized`, of the run that names an intent makes its change, a new intent or a deleted one theirs, while one no event names still blocks; the supervisor's restore of the ledger (HEAD's committed copy, or its rows before the first bad line) is no rewrite; a run counts as active by its last tool call in the transcript store, not by a file a role can touch; a judge run log or usage report counts only with its command's event of the run (`judge.run`, `telemetry.report`); a research snapshot belongs to the roles whose reports cite it; deleting an artifact another run's pass accepted during this run blocks; a run that wrote nothing on a ledger broken before it started gets a note. The Product Tester's eval suite names `baseline new --supersedes`, and every persona and eval verb is checked against `mission <area> --help`. Test: `mission-roles.test.py` (57 checks).
- The live mission run's harness (ROADMAP §4.12, M9.13): `make e2e-mission YES=1` seeds a to-do repo (an `add()` that drops the task, a pinned journey runner) and runs one mission cycle on the claude host, where the script types only the first message and the grant lines. `scripts/smoke/mission_driver.py` is the loop: each reply is a new session whose prompt is `/aloop:mission` and the line; a line is typed only when it is the one the tick's reason names and the disk recomputes, passes the pre-grant checks (the mission as typed; one journey on the seeded runner, measured twice, failing at baseline) and was shown to the human; only mission approve and baseline confirm, never consent or `--force`. Caps on sessions, grants, a line relayed once, the outer USD cap (the larger of the host's cost and tick's estimate), timeouts (the session's process group is killed), no progress and a WAIT; after goal.met a session that raises the bar is killed. 24 post-run checks, every one printed; the repo's own code (the runner, pytest) runs as a session would, in its own process group, after the outcome and the grants are read, with no bytecode cache read from the repo, and the suppressed-tests check flags any pytest hook or plugin, a `conftest.py` with code, a swapped or shadowed product module and a module no scan can read. `make e2e-mission YES=1 DRY=1` plans it for free. Tests: `mission-e2e-live.test.py` (39 checks, zero tokens, on a stub host that captures its prompt through the plugin's hooks), `evidence.test.py`.
- `mission journeys new` writes only a complete journey (ROADMAP M9.85): a skeleton or a draft that `journeys check` would block is refused with every problem, and nothing is written or recorded. Reason: M9.13's first live run, where three broken drafts were written that the researcher could neither fix, delete nor retire, so its stop gate was stuck-released. The same run fixed the harness: a reply's prompt says who types the line (the supervisor had written the account's name), the pre-grant checks judge the baseline's own journey, and the rival constraint says the baseline lists none. The second live run reached GOAL MET (22/24 checks, $1.80, 15m32s); the seed-test check now reads the file's parse tree, since RED appends its tests to that file.
- The mission loop end to end, offline (ROADMAP §4.12, M9.13): `mission-e2e.test.py` drives `mission tick` from a new mission through research, a journey, a confirmed baseline, a gap list, a goal, its intents and one Aladdin sprint per intent to GOAL MET and the next raise of the bar, with zero tokens and each step done through the CLI the way its role or the human does it. A fresh clone hands out the same action. Ethics reviews and hard lines are now ledger rows (M9.24), so a clone no longer asks for every review again or forgets a hard line; the ledger row wins over its run-log copy, and a broken ledger makes every ethics verdict `broken`. Tests: `mission-e2e.test.py`, `mission-ethics.test.py`.
- A mission baseline binds the mission's direction (ROADMAP §4.12, M9.25): `direction_sha256` covers the mission id, statement, mode and constraints, so raising a budget cap or changing the cadence (and approving it) keeps the confirmed baseline, while a new direction still asks for a new version. Reason: M9.8's budget test, a raised cap asked for a new baseline though "best" had not changed. Test: `mission-baseline.test.py`.
- Mission approvals are the human's own words (ROADMAP §4.12, M9.17): `mission approve`, `baseline confirm|restore`, `journeys approve`, `mission ethics consent` and `mission telemetry consent` record only when a prompt the `UserPromptSubmit` hook captured has a line such as `approve msn_… 3f2a9c1b` (the verb and the first 8 hex of the content's sha256); until then they ask (exit 3) with that line. `--by` alone, a negated line, another code, a line older than a later decision about the subject (for a baseline, any version's confirmation) and a hand-appended ledger row without such a line grant nothing. Reason: an approval was only as strong as `--by`, so a role could approve its own mission change. Tests: `mission-core.test.py`, `mission-baseline.test.py`.
Expand Down
8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ pycheck: ## Byte-compile the Python hook scripts
@python3 -m py_compile plugin/bin/budget plugin/bin/stats plugin/bin/aloop-init plugin/bin/gate-commit-author plugin/bin/_paths.py \
plugin/bin/_usage.py plugin/bin/_prices.py plugin/bin/_events.py plugin/bin/events plugin/bin/intent plugin/bin/lessons plugin/bin/ids plugin/bin/targets plugin/bin/impact plugin/bin/timebox plugin/bin/sweep plugin/bin/capabilities plugin/bin/models plugin/bin/_routes.py plugin/bin/host plugin/bin/_host.py plugin/bin/config plugin/bin/_config.py plugin/bin/layout plugin/bin/_lsp.py scripts/test/fixtures/fake-lsp.py plugin/bin/_timebox.py plugin/bin/host-adapter plugin/bin/install \
plugin/bin/_approval.py plugin/bin/mission plugin/bin/_mission.py plugin/bin/_mission_roles.py plugin/bin/_shellread.py plugin/bin/gate-mission-scope plugin/bin/gate-mission-artifact $$(ls plugin/mission/*.py 2>/dev/null) \
&& rm -rf plugin/bin/__pycache__ plugin/mission/__pycache__ && echo "python ok"
$$(ls scripts/smoke/*.py 2>/dev/null) \
&& rm -rf plugin/bin/__pycache__ plugin/mission/__pycache__ scripts/smoke/__pycache__ && echo "python ok"

.PHONY: test-hosts
test-hosts: ## Budget/worktree regression + Codex hooks emulation + OpenCode and dsh plugins end-to-end (no network, no tokens)
Expand Down Expand Up @@ -149,6 +150,11 @@ e2e-live: ## Live end-to-end run: one-line intent -> planned, tested, merged cod
@test -n "$(YES)" || { echo "spends tokens: make e2e-live YES=1"; exit 2; }
scripts/smoke/e2e-live.sh --yes $(if $(INTENT),--intent $(INTENT)) $(if $(HOST),--host $(HOST)) $(if $(MODEL),--model $(MODEL))

.PHONY: e2e-mission
e2e-mission: ## Live mission cycle on a seeded to-do repo: the script types only the first message and the grant lines (YES=1; spends dollars; DRY=1 plans it for free)
@test -n "$(YES)" || { echo "spends tokens: make e2e-mission YES=1"; exit 2; }
scripts/smoke/e2e-mission.sh --yes $(if $(DRY),--dry-run) $(if $(HOST),--host $(HOST)) $(if $(MODEL),--model $(MODEL))

.PHONY: smoke-live
smoke-live: ## Live Claude Code session: hooks fire, supervisor write blocked, transcripts + ledger (YES=1; spends ~$0.005)
@test -n "$(YES)" || { echo "spends tokens: make smoke-live YES=1"; exit 2; }
Expand Down
1 change: 1 addition & 0 deletions docs/OVERVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ The model can ignore a skill; it cannot ignore an exit code.
- The door rule (ROADMAP §4.12, M9.30): every agent gets the core principles at session start, and a
PreToolUse gate stops a one-way door (a push to the default branch, a release, a deploy, a secret, …)
until the human records an authorization; two-way doors pass without a question.
- The mission run, `make e2e-mission YES=1 [DRY=1]`: one live mission cycle on a seeded to-do repo, where the script types only the first message and the grant lines it checked on disk; the harness is tested at zero tokens in CI. Measured: mission 2 reached GOAL MET in 4 sessions and one sprint for $1.80 (22/24 checks; the open miss is M9.87), see `docs/evidence/README.md` (ROADMAP M9.13).
- One measured real run (*TheNoteBook*, a Go backend): ~17K LOC, 322 tests, 74 commits in 6
sprints for ~1.55M output tokens, 76% of it on Sonnet. See the
[README](../README.md#evidence).
Expand Down
Loading
Loading
0