8000 feat(e2e-live): the north-star run on Codex and OpenCode (--host); run-summary records the host by adeelahmad · Pull Request #38 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
every figure to the ledger; `scripts/test/evidence.test.py` fails on an unsourced figure.
- `run-summary` counts a `<plugin>/schemas/…` reference as schema use, not a plugin-source read, also
when the path is bound to a shell variable first (run 19: 5 → 0).
- `e2e-live.sh --host codex|opencode` (`make e2e-live YES=1 HOST=…`): the north-star run on the other
hosts: `aloop install <host>` into the seeded repo, then `codex exec --json --full-auto` or
`opencode run --format json` with the host's model routes, the same intent, approval and checks;
`--dry-run` prints the command. `run-summary --host` records the host and names it in the reproduce
command; a transcript with no `result` line records the cost as unmeasured (—), never $0.00.
Test: `scripts/test/e2e-live-hosts.test.py`.
- Natural-language auto-trigger (ROADMAP §4.11): `bin/intent-trigger` on UserPromptSubmit adds context
to run a plain build/fix request as `/aladdin "<intent>"` when it lands in an aloop project with no
sprint running. Quiet for questions, slash commands, short replies, a live sprint lock and non-aloop
Expand Down
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ test-hosts: ## Budget/worktree regression + Codex hooks emulation + OpenCode plu
python3 scripts/test/fail-closed.test.py
python3 scripts/test/intent-trigger.test.py
python3 scripts/test/evidence.test.py
python3 scripts/test/e2e-live-hosts.test.py
@if command -v node >/dev/null; then node scripts/test/opencode-plugin.test.mjs; else echo "node not found — skipped OpenCode test (CI runs it)"; fi

.PHONY: conformance
Expand All @@ -133,7 +134,7 @@ conformance: ## ROADMAP §6 conformance fixtures on today's gates (needs cargo;
.PHONY: e2e-live
e2e-live: ## Live end-to-end run: one-line intent -> planned, tested, merged code (YES=1; spends dollars)
@test -n "$(YES)" || { echo "spends tokens: make e2e-live YES=1"; exit 2; }
scripts/smoke/e2e-live.sh --yes $(if $(INTENT),--intent $(INTENT))
scripts/smoke/e2e-live.sh --yes $(if $(INTENT),--intent $(INTENT)) $(if $(HOST),--host $(HOST))

.PHONY: smoke-live
smoke-live: ## Live Claude Code session: hooks fire, supervisor write blocked, transcripts + ledger (YES=1; spends ~$0.005)
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ aladdin "add CSV export to the report" # records the intent, tells you how
## Demo

A one-line intent, start to merged code, in a real Claude Code session with the approval
given up front — reproducible with `make e2e-live YES=1` (spends about a dollar):
given up front — reproducible with `make e2e-live YES=1` (spends about a dollar; `HOST=codex` or
`HOST=opencode` runs the same loop on those hosts):

| Intent | Result | Time | Cost |
|---|---|-- 10BC8 -|---|
Expand Down Expand Up @@ -194,7 +195,7 @@ structural, not luck:
These mechanisms are codebase-agnostic — greenfield is simply what's measured here.

Live runs: `make smoke-live YES=1` (hook wiring, ~$0.004) and `make e2e-live YES=1` (the whole
loop, ~$1; each run writes `summary.json` and a row for [docs/evidence/](docs/evidence/README.md)). Offline: `make ci` — Rust fmt/clippy/tests, shellcheck, 43 Python/Node suites,
loop, ~$1; each run writes `summary.json` and a row for [docs/evidence/](docs/evidence/README.md)). Offline: `make ci` — Rust fmt/clippy/tests, shellcheck, 44 Python/Node suites,
the gate conformance fixtures (`make conformance`), eval-suite validation. Every ROADMAP
requirement is tracked with its test in [docs/TRACKING.md](docs/TRACKING.md).

Expand Down
2 changes: 1 addition & 1 deletion docs/OVERVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ The model can ignore a skill; it cannot ignore an exit code.
--strict`, eval-suite and version-lockstep checks,
the tracking check and `host check`, plus the newer suites (targets render, host swap,
metrics, config, the aladdin CLI, the site).
- The north-star run, `make e2e-live YES=1 [INTENT=wordfreq]`: a one-line intent becomes
- The north-star run, `make e2e-live YES=1 [INTENT=wordfreq] [HOST=codex|opencode]`: a one-line intent becomes
planned, test-first, gate-verified, merged code in a real Claude Code session with the
approval given up front. Two intents pass every check (semver parser: 11/11, ~$0.65–1.15,
11–19 min; word frequencies with two dependent stories: 13/13, $0.79–1.56, 13–45 min; the last three runs $0.95–1.03, 1–6 supervisor wakes, 0–5 plugin-source reads,
Expand Down
2 changes: 1 addition & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ original plan.
| **M5 — Narrow context** | v0.16 | Session 6: semantic execution map, shadow workspace, `target` / `context` CLI (§4.6) | done: targets, shadow workspace, `target`/`context`, LSP seam, GREEN scope from targets, and tasks.md/validate.md/plan.md rendered from the semantic plan (`targets render`; tests: `semantic-targets`, `lsp-adapter`, `targets-scope`, `plan-render`) |
| **M6 — Audit** | v0.17 | Sessions 7–8: event stream, rationale, authority, lineage query (§4.7); provider seam (§4.8) | done: `aladdin.event/1` stream, decisions, authority, lineage, `events trace SHA`, commit messages and graph export (§4.7); impact provider seam (§4.8) |
| **M7 — Any host** | v0.18 | Session 9: `HostAdapter`, capability matrix, `.aladdin/config.yaml`, host swap, metrics (§4.9–4.10) | done: `HostAdapter` + `host check` in CI, capability matrix, usage accuracy, `.aladdin/config.yaml` with migration, `host swap`, the full metrics table incl. context-window share (tests: `host-adapter`, `portable-config`, `host-swap`, `metrics`) |
| **M8 — Aladdin 1.0** | v1.0 | Session 10: `/aladdin` + `aladdin` CLI with aliases, site, measured public runs (§4.11). The rename to aloop shipped early in v0.11.0 | in progress: `bin/aladdin` + `/aladdin` with `aloop`/`agentic` aliases (test: `aladdin-cli.test.py`); natural-language auto-trigger on every host (`intent-trigger`, test: `intent-trigger.test.py`); README reordered product-first; site at `docs/index.html` (Pages switch is the owner's). evidence ledger `docs/evidence/README.md` + `run-summary` (test: `evidence.test.py`). Open: measured public runs on Codex and OpenCode (`e2e-live.sh` is Claude-only and this environment has no `codex`/`opencode` CLI: an owner's machine with both installed is needed; reason: a number we cannot measure is not published) |
| **M8 — Aladdin 1.0** | v1.0 | Session 10: `/aladdin` + `aladdin` CLI with aliases, site, measured public runs (§4.11). The rename to aloop shipped early in v0.11.0 | in progress: `bin/aladdin` + `/aladdin` with `aloop`/`agentic` aliases (test: `aladdin-cli.test.py`); natural-language auto-trigger on every host (`intent-trigger`, test: `intent-trigger.test.py`); README reordered product-first; site at `docs/index.html` (Pages switch is the owner's). evidence ledger `docs/evidence/README.md` + `run-summary` (test: `evidence.test.py`). `e2e-live.sh --host codex|opencode` runs the same loop on those hosts (`aloop install <host>` into the run repo, then `codex exec --json` / `opencode run --format json`; `--dry-run` prints the command; test: `e2e-live-hosts.test.py`). Open: the measured public runs themselves (this environment has no `codex`/`opencode` CLI: run `make e2e-live YES=1 HOST=codex` and `HOST=opencode` on a machine with both and add the rows; reason: a number we cannot measure is not published) |

**Order rationale:** identity and conformance fixtures first, so every later change is
measured against today's behaviour; budgets and handoff next, because they shape every run;
Expand Down
1 change: 1 addition & 0 deletions docs/TRACKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,4 @@ Status of every requirement in [ROADMAP.md](ROADMAP.md), in the four states §8
| 4.11 | Site (`docs/index.html`, GitHub Pages from `/docs`) with the lamp motif, RED explainer and audit tree; numbers sourced from §0 | IMPLEMENTED_TESTED | `scripts/test/site.test.py` |
| 4.11 | Natural-language auto-trigger inside hosts (`intent-trigger` on UserPromptSubmit; Claude Code, Codex, OpenCode) | IMPLEMENTED_TESTED | `scripts/test/intent-trigger.test.py` |
| 4.11 | Evidence: every public claim links to a reproducible measured run (`docs/evidence/README.md`, `run-summary`) | IMPLEMENTED_TESTED | `scripts/test/evidence.test.py` |
| 4.11 | The north-star run reproducible on every host (`e2e-live.sh --host codex|opencode`: host install + headless CLI, same checks) | IMPLEMENTED_TESTED | `scripts/test/e2e-live-hosts.test.py` (measured Codex/OpenCode rows still pending: no CLIs here) |
6 changes: 4 additions & 2 deletions docs/evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,17 @@ Every time or cost figure in the [README](../../README.md), on the [site](../ind
[ROADMAP §0](../ROADMAP.md#0-at-a-glance) is a row here. Each row is one live run of the whole
loop (`scripts/smoke/e2e-live.sh`): a one-line intent through intake, standards, planning,
RED/SCAFFOLD/GREEN per task, structural review and the final gate, to merged, tested code in a
real Claude Code session with the approval given up front. The numbers come from the harness
real host session (Claude Code, or Codex/OpenCode with `HOST=`) with the approval given up front. The numbers come from the harness
(`run-summary` reads the session's `result` lines and the gate log), never from the model.
`scripts/test/evidence.test.py` fails if a public figure has no row, or a row has no reproduce
command.

**Reproduce:** install the plugin, then run the command in the last column (spends about a
dollar, 11–22 minutes). The run leaves `summary.json`, `summary.md` and a ready-to-paste row in
its run dir; add the row here. Intents: `semver` (one story) and `wordfreq` (two dependent
stories). What each run found and fixed is in ROADMAP M0.8.
stories). `HOST=codex` / `HOST=opencode` run the loop on those hosts (`aloop install <host>` into the
run repo, then `codex exec --json` / `opencode run --format json`); their CLIs print no cost line, so
the cost cell reads — and the intent cell names the host. What each run found and fixed is in ROADMAP M0.8.

| run | intent | plugin | checks | wall | cost | wakes | plugin reads | gate blocks | reproduce |
|---|---|---|---|---|---|---|---|---|---|
Expand Down
2 changes: 1 addition & 1 deletion plugin/bin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ ctx-symbols) → WARN + grep fallback; never a silent pass.
| `worktree-hygiene` | SubagentStart · exec roles — sparse-checks `docs/agents/` out of the new worktree (workers reach it via the absolute `STORY_DIR` back into the main tree, never their own copy) + bootstraps the main tree's `.gitignore` with `.agentic/`/`.transcripts/` if missing. Runs regardless of whether the harness's built-in isolation or `worktree-create` made the worktree. |
| `worktree-create` | OPTIONAL `git worktree add` helper — wire into **user `settings.json`** as a `WorktreeCreate` hook for a non-git VCS (NOT a valid plugin-manifest event). Built-in `isolation: "worktree"` covers plain git. Also applies the same sparse-checkout + `.gitignore` bootstrap as `worktree-hygiene`. |
| `worktree-remove` | OPTIONAL `git worktree remove` cleanup companion for the above (user `settings.json` `WorktreeRemove`) |
| `run-summary` | `<run-dir> [--intent] [--checks N/M] [--secs] [--plugin]` — a live run's measured record: `summary.json`, `summary.md` and a `docs/evidence/README.md` row from the session's `result` lines and `gates.jsonl` (cost, wakes, plugin reads, blocks); called by `e2e-live.sh` |
| `run-summary` | `<run-dir> [--intent] [--host claude|codex|opencode] [--checks N/M] [--secs] [--plugin]` — a live run's measured record: `summary.json`, `summary.md` and a `docs/evidence/README.md` row from the session's `result` lines and `gates.jsonl` (cost, wakes, plugin reads, blocks); called by `e2e-live.sh` |
| `log-execution` | `wave-K activity SN-NN transition [detail…] [execution.log]` — append one transition line (default ledger: the latest sprint's); `--help` |
| `ensure-tools` | SessionStart — builds md-db + ctx-symbols into `$CLAUDE_PLUGIN_DATA/bin` on first run / version change (background), tells the model where the tools are. `--sync` build now · `--wait` · `--resolve T` real binary path |
| `md-db`, `ctx-symbols` | PATH shims (this `bin/` is on PATH in every session): exec the real binary, building it once if needed. Gates test presence with `ensure-tools --resolve`, never `command -v` |
Expand Down
25 changes: 16 additions & 9 deletions plugin/bin/run-summary
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
#!/usr/bin/env python3
"""run-summary — the measured record of one live run (ROADMAP §4.11: evidence, not claims).

run-summary <run-dir> [--intent NAME] [--checks N/M] [--secs WALL] [--plugin DIR]
run-summary <run-dir> [--intent NAME] [--host claude|codex|opencode] [--checks N/M] [--secs WALL] [--plugin DIR]

Reads <run-dir>/out.jsonl (the supervisor's stream-json transcript) and the run repo's
gate log (<run-dir>/repo/.aladdin/logs/gates.jsonl, or .agentic/), writes
<run-dir>/summary.json and <run-dir>/summary.md, and prints the summary with a ready-to-paste
row for docs/evidence/README.md. Numbers come from the harness, never from the model:
cost_usd / supervisor_turns the LAST `result` line (a resumed session reports again)
cost_usd / supervisor_turns the LAST `result` line (a resumed session reports again);
Codex and OpenCode print none, so the cost is null and the row
says — (unmeasured), never $0.00
wakes how many `result` lines: one per supervisor wake
plugin_reads sub-agent tool calls naming a plugin path, except a
`<plugin>/schemas/…` file (md-db's schema: a legitimate use,
Expand Down Expand Up @@ -40,7 +42,9 @@ def main():
print("run-summary: no out.jsonl in %s" % run, file=sys.stderr); return 64
plugin = os.path.abspath(arg("--plugin") or os.path.join(os.path.dirname(os.path.realpath(__file__)), ".."))
pat = re.compile(re.escape(plugin) + r"/schemas/\S*")
cost = turns = wakes = reads = 0
cost = None
turns = wakes = reads = 0
host = arg("--host") or "claude"
secs = arg("--secs")
for l in open(out):
try:
Expand Down Expand Up @@ -82,21 +86,24 @@ def main():
except (OSError, ValueError):
version = ""
intent = arg("--intent") or ""
s = {"run_dir": run, "intent": intent, "plugin_version": version, "checks": arg("--checks") or "",
"wall": wall(secs or 0), "cost_usd": round(float(cost), 2), "supervisor_turns": turns, "wakes": wakes,
s = {"run_dir": run, "intent": intent, "host": host, "plugin_version": version, "checks": arg("--checks") or "",
"wall": wall(secs or 0), "cost_usd": None if cost is None else round(float(cost), 2),
"supervisor_turns": turns, "wakes": wakes,
"plugin_reads": reads, "gate_verdicts": verdicts, "gate_blocks": blocks, "run_repo_head": head,
"reproduce": "make e2e-live YES=1" + (" INTENT=%s" % intent if intent else ""),
"reproduce": "make e2e-live YES=1" + (" INTENT=%s" % intent if intent else "") + (" HOST=%s" % host if host != "claude" else ""),
"recorded_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}
row = "| %s | %s | v%s | %s | %s | $%.2f | %s | %s | %s | `%s` |" % (
os.path.basename(run), intent or "?", version, s["checks"] or "?", s["wall"], s["cost_usd"], wakes, reads, blocks, s["reproduce"])
cost_cell = "—" if s["cost_usd"] is None else "$%.2f" % s["cost_usd"]
label = (intent or "?") + ("" if host == "claude" else " (%s)" % host)
row = "| %s | %s | v%s | %s | %s | %s | %s | %s | %s | `%s` |" % (
os.path.basename(run), label, version, s["checks"] or "?", s["wall"], cost_cell, wakes, reads, blocks, s["reproduce"])
md = ("# Live run %s\n\n| run | intent | plugin | checks | wall | cost | wakes | plugin reads | gate blocks | reproduce |\n"
"|---|---|---|---|---|---|---|---|---|---|\n%s\n\nrun repo head: `%s` · gate verdicts: %d · supervisor turns: %d\n"
% (os.path.basename(run), row, head, verdicts, turns))
with open(os.path.join(run, "summary.json"), "w") as f:
json.dump(s, f, indent=1)
with open(os.path.join(run, "summary.md"), "w") as f:
f.write(md)
print("cost: $%.2f · supervisor turns: %s · wall: %s" % (s["cost_usd"], turns, s["wall"]))
print("cost: %s · supervisor turns: %s · wall: %s" % (cost_cell, turns, s["wall"]))
print("supervisor wakes: %d · sub-agent reads of the plugin's own source: %d" % (wakes, reads))
if verdicts:
print("gate verdicts: %d (%d blocks)" % (verdicts, blocks))
Expand Down
Loading
Loading
0