8000 fix: live e2e run 10 gaps — report append from a worktree, stale task.env copy, stuck-worker release, test temp-dir leak by adeelahmad · Pull Request #13 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,14 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
overrides it (`breach:` line) and acts per class (SKILL.md "Breach classes").

### Added
- Stuck release: a worker that hits the same gate block 3 times in a row (one gate, task,
attempt and worktree; `AGENTIC_STUCK_BLOCKS`) is released **unverified** instead of
looping until its time box (live run 10 lost 29 minutes to a RED worker that stopped 29
times on one unchanged reason). The release is not a pass: the gate leaves
`.agentic/state/stuck/<task>.<attempt>` with the reason, `budget watch` prints a `STUCK`
line once, and the supervisor re-dispatches with feedback or re-plans. A different reason
or a new attempt resets the count; a self-check never releases. Every gate verdict now
logs its worktree and a checksum of the block reason. 7 conformance checks (§6.9).
- **Portable project config** (ROADMAP §4.9). `.aladdin/config.yaml` (budgets, context,
semantic, models, provenance, memory, git, plus per-role budget profiles) replaces
`docs/agents/defaults.md` as the human's file. `aloop config migrate` writes it from the
Expand Down Expand Up @@ -237,6 +245,28 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
reading it stays allowed. 2 checks failed before.

### Fixed
- `gate-validate-artifact`'s block now carries md-db's diagnostics (the missing section or field
and its hint): a PostToolUse block hands the model only stderr, and the bare "failed the
schema" made the planner guess (live e2e run 11: every planning artifact was written twice,
12 of 25 gate verdicts).
- `gate-dispatch` refuses a task contract with no `STORY_DIR`, or one naming a dir that does
not exist, and tells a whole-wave structural review to use the sprint dir
`docs/agents/sprintN/` (run 11: the reviewer's contract had none, 2 blocks the worker could
not fix). 5 new dispatch checks.
- OpenCode: a stuck release (exit 0 with `STUCK … NOT verified`) is reported to the supervisor
as "released UNVERIFIED, do not merge", never as a silent pass; 2 new checks.
- `make test-hosts` masked a failing OpenCode test as "node not found".
- The host tests (Codex hooks, OpenCode plugin, rename-compat, budget, fail-closed, scope,
dispatch) each leaked a temp dir, and the Codex and OpenCode ones started a real background
cargo build over their fake tools (~500MB per run; 57 leaked dirs filled the disk and killed
e2e run 11 with ENOSPC). They now remove their temp dir on exit and set
`AGENTIC_NO_AUTOINSTALL=1`; `e2e-live` drops its cargo target dirs after the run.
- The comms gate's "no output.md" block now says HOW to append the report: a shell heredoc
from the worktree, because Claude Code's Write/Edit refuse a path outside it and
`docs/agents/` lives only in the main tree (e2e run 10: a RED worker stopped 29 times on
it). The worker personas say the same.
- A worktree `task.env` copy OLDER than the supervisor's contract (a re-dispatch bumped
`ATTEMPT`) is stale and is ignored, not reported as tampering (e2e run 10: 3 blocks).
- `ensure-tools` rebuilds the bundled md-db / ctx-symbols when their sources change, not
only on a version bump; e2e run 9 ran a stale ctx-symbols that still had the duplicate bug.
- `gate-stage2-complete` names the expected `sN-NN-<slug>/` story dir (and a misnamed one it
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ test-hosts: ## Budget/worktree regression + Codex hooks emulation + OpenCode plu
python3 scripts/test/dispatch.test.py
python3 scripts/test/ensure-tools.test.py
python3 scripts/test/fail-closed.test.py
@command -v node >/dev/null && node scripts/test/opencode-plugin.test.mjs || { echo "node not found — skipped OpenCode test (CI runs it)"; }
@if command -v node >/dev/null; then node scripts/test/opencode-plugin.test.mjs; else echo "node not found — skipped OpenCode test (CI runs it)"; fi

.PHONY: conformance
conformance: ## ROADMAP §6 conformance fixtures on today's gates (needs cargo; no network, no tokens)
Expand Down
13 changes: 10 additions & 3 deletions docs/OVERVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,16 @@ The model can ignore a skill; it cannot ignore an exit code.
**Working and tested:**
- The full planning → execution loop on Claude Code; Codex and OpenCode via generated
agents and the same gates.
- CI on every PR: Rust fmt/clippy/tests for both backends, shellcheck on every gate, Codex
hooks emulation + OpenCode end-to-end (18 checks), `claude plugin validate --strict`,
eval-suite and version-lockstep checks.
- CI on every PR: Rust fmt/clippy/tests for both backends, shellcheck on every gate, the
ROADMAP §6 conformance fixtures (real gates on a fixture crate, 54 checks), the host tests
(budget/worktree, Codex hooks emulation, supervisor scope, hook matchers, dispatch,
ensure-tools, fail-closed: 73 checks; OpenCode end-to-end: 20), `claude plugin validate
--strict`, eval-suite and version-lockstep checks.
- The north-star run, `make e2e-live YES=1 [INTENT=wordfreq]`: a one-line intent becomes
planned, test-first, gate-verified, merged code in a real Claude Code session with the
approval given up front. Two intents pass every check (semver parser: 11/11, ~$0.65–1.15,
11–19 min; word frequencies with two dependent stories: 12/12, $0.86–1.56, 13–45 min). Every run so
far has found a gate false positive that unit tests missed; ROADMAP M0.8 lists them.
- One measured real run (*TheNoteBook*, a Go backend): ~17K LOC, 322 tests, 74 commits in 6
sprints for ~1.55M output tokens, 76% of it on Sonnet. See the
[README](../README.md#token-efficiency-a-measured-run).
Expand Down
6 changes: 3 additions & 3 deletions docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ spec; this table is the plan. Versions are targets, not promises.
| Milestone | Target | Contents | Status |
|---|---|---|---|
| **M0 — Hygiene** | v0.10.x | Mark `DESIGN` / `ARCHITECTURE` / `KICKSTART` historical; fix stale counts (8 → 9 roles) and the README CI section; add the `CONTRIBUTING.md` the README badge links to; one live-session smoke test of the hook wiring (open since `DEVLOG` v0.1) | done: live smoke test `make smoke-live YES=1` (6 checks, ~$0.004 on haiku); its first run found the Linux fail-open in `gate-supervisor-scope` (fixed, regression test added) |
| **M0.8 — North-star run** | v0.11.x | `make e2e-live YES=1`: a one-line intent (semver parser) goes through retrospective, intake, standards, planner, RED/SCAFFOLD/GREEN per task, structural review and the final gate to merged, tested code in a real Claude Code session, with the approval given up front. Reason: nothing had ever run the whole loop live; the first run found that no role gate fired on plugin installs | run 1: done end to end, 19 min, $1.15 (sonnet supervisor, haiku workers), 22 tests on main, 9/11 checks; found and fixed the `aloop:<role>` matcher bug, a selfcheck crash and copy-merging. Run 2: **11/11**, 12m46s, $0.92, host gates on every role, worktree branches merged with `--no-ff`; found gates looping on the supervisor's contract slips (fixed). Run 3: 11/11, 12m00s, **$0.79**; found GREEN looping on RED's unformatted tests (fixed: RED/SCAFFOLD run the format check) and selfcheck missing the contract in Claude Code worktrees (fixed: the contract is copied in). Run 4 (all fixes): 11/11, 17m56s, $1.10, 12 gate blocks and no loops (28 → 17 → 12). Cost varies $0.79–1.15 per run; next: cut it (scaffold symbol parsing slips such as `(enum,`, supervisor turns), then a second intent beyond the semver fixture. Run 5: 11/11 but 29 min, $1.83, 44 blocks (the planner made 5 tasks; 27 false isolation blocks after workers `cd`'d to the main tree, fixed; subagents also read the plugin's own source, which costs tokens). Run 6: 11/11, **11m10s, $0.65**, plugin-source reads 30 → 2 after the agent prompts said not to; found a dispatch-before-contract loop (25 blocks, fixed by `gate-dispatch`) and annotated scaffold-symbol lines (fixed). Run 7 (after #9): 11/11, 13m46s, $0.98, 20 blocks, no dispatch loop; found RED blocked for listing an earlier task's passing tests, scaffold-symbols tables with a header row and trait-impl lines, and a guessed memory.md frontmatter (all fixed; `aloop init` now seeds memory.md). The re-plan coverage loss is now caught by `gate-final` (every planned test must be on main). Run 8, **second intent** (`INTENT=wordfreq`, two dependent stories): 12/12, 18m05s, $1.53, 54 blocks; found scope globs expanded against the worktree (29 GREEN blocks) and `mod common;` across test crates reported as a HIGH duplicate (57 review halts), both fixed. Run 9 (wordfreq again): 12/12, 19m42s, $1.56, 29 blocks; the ctx-symbols fix had not reached the run (`ensure-tools` rebuilt only on a version bump, fixed) and the planner named story dirs `S1-01/` (block message now names `s1-01-<slug>/`). Next: rerun wordfreq with the rebuilt tools, then supervisor-written GREEN instructions overreaching the next task |
| **M0.8 — North-star run** | v0.11.x | `make e2e-live YES=1`: a one-line intent (semver parser) goes through retrospective, intake, standards, planner, RED/SCAFFOLD/GREEN per task, structural review and the final gate to merged, tested code in a real Claude Code session, with the approval given up front. Reason: nothing had ever run the whole loop live; the first run found that no role gate fired on plugin installs | run 1: done end to end, 19 min, $1.15 (sonnet supervisor, haiku workers), 22 tests on main, 9/11 checks; found and fixed the `aloop:<role>` matcher bug, a selfcheck crash and copy-merging. Run 2: **11/11**, 12m46s, $0.92, host gates on every role, worktree branches merged with `--no-ff`; found gates looping on the supervisor's contract slips (fixed). Run 3: 11/11, 12m00s, **$0.79**; found GREEN looping on RED's unformatted tests (fixed: RED/SCAFFOLD run the format check) and selfcheck missing the contract in Claude Code worktrees (fixed: the contract is copied in). Run 4 (all fixes): 11/11, 17m56s, $1.10, 12 gate blocks and no loops (28 → 17 → 12). Cost varies $0.79–1.15 per run; next: cut it (scaffold symbol parsing slips such as `(enum,`, supervisor turns), then a second intent beyond the semver fixture. Run 5: 11/11 but 29 min, $1.83, 44 blocks (the planner made 5 tasks; 27 false isolation blocks after workers `cd`'d to the main tree, fixed; subagents also read the plugin's own source, which costs tokens). Run 6: 11/11, **11m10s, $0.65**, plugin-source reads 30 → 2 after the agent prompts said not to; found a dispatch-before-contract loop (25 blocks, fixed by `gate-dispatch`) and annotated scaffold-symbol lines (fixed). Run 7 (after #9): 11/11, 13m46s, $0.98, 20 blocks, no dispatch loop; found RED blocked for listing an earlier task's passing tests, scaffold-symbols tables with a header row and trait-impl lines, and a guessed memory.md frontmatter (all fixed; `aloop init` now seeds memory.md). The re-plan coverage loss is now caught by `gate-final` (every planned test must be on main). Run 8, **second intent** (`INTENT=wordfreq`, two dependent stories): 12/12, 18m05s, $1.53, 54 blocks; found scope globs expanded against the worktree (29 GREEN blocks) and `mod common;` across test crates reported as a HIGH duplicate (57 review halts), both fixed. Run 9 (wordfreq again): 12/12, 19m42s, $1.56, 29 blocks; the ctx-symbols fix had not reached the run (`ensure-tools` rebuilt only on a version bump, fixed) and the planner named story dirs `S1-01/` (block message now names `s1-01-<slug>/`). Run 10 (rebuilt tools): 12/12, $1.20, but 44m48s and 45 blocks: the duplicate false positive is gone (0 HIGH); a RED worker's Write/Edit to the story's output.md was refused by Claude Code's worktree isolation and it stopped 29 times without trying a shell append (fixed: the block and the personas say how), plus 3 stale-`ATTEMPT` tamper blocks (fixed). Fixed after run 10: a worker that repeats an identical block 3 times is released unverified with a `STUCK` marker the supervisor acts on (reason: 29 minutes lost in run 10). Run 11 (all run-10 fixes): 12/12, **12m56s, $0.86, 6 blocks** (from 44m48s, $1.20, 45): no r 8BD0 epeated block, one RED report-append block fixed on the first retry, one scaffold fmt block, two structural-reviewer comms blocks (STORY_DIR unset for a sprint-wide review, then a stale block) and 12 `gate-validate-artifact` write blocks on the planner's artifacts. Fixed after run 11: the artifact-write block carries md-db's diagnostics (the planner guessed twice per artifact: 12 of 25 verdicts), `gate-dispatch` requires an existing `STORY_DIR` and names the sprint dir for a whole-wave structural review (2 blocks), and OpenCode reports a stuck release as unverified. Next: rerun wordfreq to measure, then supervisor-written GREEN instructions overreaching the next task |
| **M0.6 — Gates fail closed** | v0.11.x | A gate that crashes (exit 1, e.g. an unbound variable) is a non-blocking hook error on Claude Code, so the action goes through. Make every gate trap unexpected errors into exit 2 with the reason, and add a conformance case per gate. Reason: the smoke run showed one real instance | done: `_failclosed.sh` on all 10 sub-agent gates + supervisor-scope inside a sprint; `scripts/test/fail-closed.test.py` (5 checks). Left fail-open on purpose: `gate-commit-author` and `gate-validate-artifact` outside a sprint, which see every session on the machine |
| **M0.7 — Supervisor-owned task contract** | v0.11.x | The contract lives in the main tree, `.agentic/state/tasks/<TASK_ID>.env`, and the dispatch prompt's first line names the task; gates and the time box find it from the host's transcript, a worktree `task.env` that disagrees blocks, and sub-agents may not Write/Edit `.agentic/state/` or `task.env`. Settled by a live probe: Claude Code's `isolation: "worktree"` makes `.claude/worktrees/agent-<id>` at dispatch, so a pre-written in-worktree contract was impossible there. Bash writes are guarded too (a command naming the contract with a write verb is blocked; a heuristic, so an obfuscated write still gets through). On hosts with no sub-agent transcript the worktree's `TASK_ID` (or its `.agentic/worktrees/<id>` path) picks the contract | done: `make conformance` (30) + `scripts/test/{budget-worktree,supervisor-scope}.test.py` |
| **M0.7 — Supervisor-owned task contract** | v0.11.x | The contract lives in the main tree, `.agentic/state/tasks/<TASK_ID>.env`, and the dispatch prompt's first line names the task; gates and the time box find it from the host's transcript, a worktree `task.env` that disagrees blocks, and sub-agents may not Write/Edit `.agentic/state/` or `task.env`. Settled by a live probe: Claude Code's `isolation: "worktree"` makes `.claude/worktrees/agent-<id>` at dispatch, so a pre-written in-worktree contract was impossible there. Bash writes are guarded too (a command naming the contract with a write verb is blocked; a heuristic, so an obfuscated write still gets through). On hosts with no sub-agent transcript the worktree's `TASK_ID` (or its `.agentic/worktrees/<id>` path) picks the contract | done: `make conformance` (54 today) + `scripts/test/{budget-worktree,supervisor-scope}.test.py` |
| **M0.5 — Rename to aloop** | v0.11.0 | Plugin, marketplace, skill, CLI (`aloop`, `aloop init`), OpenCode plugin file and docs renamed; `agentic`, `agentic init`, `agentic-init` and `agentic bind` kept as aliases; OpenCode/Codex installs clean up pre-rename files. Data layout and settings keys unchanged (moved to M1). GitHub repo rename is the owner's call | done: `scripts/test/rename-compat.test.py` |
| **M1 — Trustworthy state** | v0.12 | Session 1: conformance fixtures around current behaviour, canonical IDs and the reference remap (§4.5); then move the data layout to the new name (`.agentic/` → `.aloop/`, `AGENTIC_*` → `ALOOP_*`) with a read-old fallback, proven by those fixtures | in progress: §6.2–6.9 fixtures pass on today's gates (`make conformance`, 30 checks incl. task.env tampering and the main-tree contract; found and fixed the whole-directory schema-gate bug); next: canonical IDs and the reference remap (§4.5) |
| **M1 — Trustworthy state** | v0.12 | Session 1: conformance fixtures around current behaviour, canonical IDs and the reference remap (§4.5); then move the data layout to the new name (`.agentic/` → `.aloop/`, `AGENTIC_*` → `ALOOP_*`) with a read-old fallback, proven by those fixtures | in progress: §6.2–6.9 fixtures pass on today's gates (`make conformance`, 54 checks incl. task.env tampering, the main-tree contract and every false positive the live runs found; found and fixed the whole-directory schema-gate bug); next: canonical IDs and the reference remap (§4.5) |
| **M2 — Budgets that plan** | v0.13 | Sessions 2 and 4: budget semantics (§3.1–3.5), sprint envelope and admission control (§4.2), breach classes, failure envelope, grace rule, lesson candidates (§4.3) | not started |
| **M3 — Clean boundaries** | v0.14 | Session 3: canonical handoff, rotation and the resume rule (§4.4) | not started |
| **M4 — Intent in, autonomy optional** | v0.15 | Session 5: Intent Contract, clarification policy, autonomous mode (§4.1) | not started |
Expand Down
3 changes: 2 additions & 1 deletion plugin/agents/green-worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ your init block's `### Budget` line has the exact numbers). Plan for it:
Your brief is the LAST `## … · green-worker · …` block in `$STORY_DIR/init.md`; read the
whole file plus the red-worker and scaffolder blocks in `$STORY_DIR/output.md` (what was
tested, what was stubbed). When done you MUST **append** your report as a new block to
`$STORY_DIR/output.md` — never rewrite an earlier block:
`$STORY_DIR/output.md` — never rewrite an earlier block (append with a shell heredoc, `cat >> "$STORY_DIR/output.md" <<'EOF'`: the Write/Edit
tools refuse a path outside your worktree, and docs/agents/ lives only in the main tree):

## <task> · attempt <N> · green-worker · <ISO8601>
status: ok (or retry|escalate)
Expand Down
3 changes: 2 additions & 1 deletion plugin/agents/red-worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ your init block's `### Budget` line has the exact numbers). Plan for it:
Your brief is the LAST `## … · red-worker · …` block in `$STORY_DIR/init.md`; read the
whole file for prior context (earlier feedback, the chain so far). When done you MUST
**append** your report as a new block to `$STORY_DIR/output.md` — create it with a
`type: output` frontmatter if absent, and NEVER rewrite an earlier block:
`type: output` frontmatter if absent, and NEVER rewrite an earlier block (append with a shell heredoc, `cat >> "$STORY_DIR/output.md" <<'EOF'`: the Write/Edit
tools refuse a path outside your worktree, and docs/agents/ lives only in the main tree):

## <task> · attempt <N> · red-worker · <ISO8601>
status: ok (or retry|escalate)
Expand Down
3 changes: 2 additions & 1 deletion plugin/agents/scaffolder.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ GREEN. Scaffolding is safe to repeat.
Your brief is the LAST `## … · scaffolder · …` block in `$STORY_DIR/init.md`; read the
red-worker block in `$STORY_DIR/output.md` for the symbols the tests reference. When done
you MUST **append** your report as a new block to `$STORY_DIR/output.md` — never rewrite
an earlier block:
an earlier block (append with a shell heredoc, `cat >> "$STORY_DIR/output.md" <<'EOF'`: the Write/Edit
tools refuse a path outside your worktree, and docs/agents/ lives only in the main tree):

## <task> · attempt <N> · scaffolder · <ISO8601>
status: ok
Expand Down
4 changes: 3 additions & 1 deletion plugin/agents/structural-reviewer.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,11 @@ Everything is on PATH — never search for it: `selfcheck`, `md-db`, `ctx-symbol
- Read-only. Writes only output.md findings; fixes nothing itself.

## Comms — the channel (story-bound, append-only)
`$STORY_DIR` is the story's dir, or the sprint dir (`docs/agents/sprintN/`) for a whole-wave review.
Read the story's chain in `$STORY_DIR/output.md` (the red/scaffold/green blocks) plus the
merged code. When done you MUST **append** your findings as a new block to
`$STORY_DIR/output.md` — never rewrite an earlier block:
`$STORY_DIR/output.md` — never rewrite an earlier block (append with a shell heredoc, `cat >> "$STORY_DIR/output.md" <<'EOF'`: the Write/Edit
tools refuse a path outside your worktree, and docs/agents/ lives only in the main tree):

## <task> · attempt <N> · structural-reviewer · <ISO8601>
status: ok (or reject — foundation-poisoning)
Expand Down
Loading
Loading
0