8000 feat: gate-final blocks when a planned test is missing from main by adeelahmad · Pull Request #11 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content

Merged

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
## [Unreleased]

### Added
- **`gate-final` checks every planned test is on main**: each `path::fn` bullet in a story's
`plan.md` must exist at its path. `plan-ready.md` lists only what RED wrote, so a test a
re-plan dropped used to vanish unnoticed (e2e run 6).
- **`aloop init` seeds `docs/agents/memory.md`** as a schema-valid empty table, and
`gate-memory` prints a valid skeleton when it blocks. e2e run 7's archivist guessed
`type: agent-memory` and the supervisor's stop was blocked 15 times.
Expand Down
2 changes: 1 addition & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ spec; this table is the plan. Versions are targets, not promises.
| Milestone | Target | Contents | Status |
|---|---|---|---|
| **M0 — Hygiene** | v0.10.x | Mark `DESIGN` / `ARCHITECTURE` / `KICKSTART` historical; fix stale counts (8 → 9 roles) and the README CI section; add the `CONTRIBUTING.md` the README badge links to; one live-session smoke test of the hook wiring (open since `DEVLOG` v0.1) | done: live smoke test `make smoke-live YES=1` (6 checks, ~$0.004 on haiku); its first run found the Linux fail-open in `gate-supervisor-scope` (fixed, regression test added) |
| **M0.8 — North-star run** | v0.11.x | `make e2e-live YES=1`: a one-line intent (semver parser) goes through retrospective, intake, standards, planner, RED/SCAFFOLD/GREEN per task, structural review and the final gate to merged, tested code in a real Claude Code session, with the approval given up front. Reason: nothing had ever run the whole loop live; the first run found that no role gate fired on plugin installs | run 1: done end to end, 19 min, $1.15 (sonnet supervisor, haiku workers), 22 tests on main, 9/11 checks; found and fixed the `aloop:<role>` matcher bug, a selfcheck crash and copy-merging. Run 2: **11/11**, 12m46s, $0.92, host gates on every role, worktree branches merged with `--no-ff`; found gates looping on the supervisor's contract slips (fixed). Run 3: 11/11, 12m00s, **$0.79**; found GREEN looping on RED's unformatted tests (fixed: RED/SCAFFOLD run the format check) and selfcheck missing the contract in Claude Code worktrees (fixed: the contract is copied in). Run 4 (all fixes): 11/11, 17m56s, $1.10, 12 gate blocks and no loops (28 → 17 → 12). Cost varies $0.79–1.15 per run; next: cut it (scaffold symbol parsing slips such as `(enum,`, supervisor turns), then a second intent beyond the semver fixture. Run 5: 11/11 but 29 min, $1.83, 44 blocks (the planner made 5 tasks; 27 false isolation blocks after workers `cd`'d to the main tree, fixed; subagents also read the plugin's own source, which costs tokens). Run 6: 11/11, **11m10s, $0.65**, plugin-source reads 30 → 2 after the agent prompts said not to; found a dispatch-before-contract loop (25 blocks, fixed by `gate-dispatch`) and annotated scaffold-symbol lines (fixed). Run 7 (after #9): 11/11, 13m46s, $0.98, 20 blocks, no dispatch loop; found RED blocked for listing an earlier task's passing tests, scaffold-symbols tables with a header row and trait-impl lines, and a guessed memory.md frontmatter (all fixed; `aloop init` now seeds memory.md). Next: the gaps run 6 showed (a test-coverage loss when a RED re-plan drops tests; supervisor-written GREEN instructions overreaching the next task), then a second intent |
| **M0.8 — North-star run** | v0.11.x | `make e2e-live YES=1`: a one-line intent (semver parser) goes through retrospective, intake, standards, planner, RED/SCAFFOLD/GREEN per task, structural review and the final gate to merged, tested code in a real Claude Code session, with the approval given up front. Reason: nothing had ever run the whole loop live; the first run found that no role gate fired on plugin installs | run 1: done end to end, 19 min, $1.15 (sonnet supervisor, haiku workers), 22 tests on main, 9/11 checks; found and fixed the `aloop:<role>` matcher bug, a selfcheck crash and copy-merging. Run 2: **11/11**, 12m46s, $0.92, host gates on every role, worktree branches merged with `--no-ff`; found gates looping on the supervisor's contract slips (fixed). Run 3: 11/11, 12m00s, **$0.79**; found GREEN looping on RED's unformatted tests (fixed: RED/SCAFFOLD run the format check) and selfcheck missing the contract in Claude Code worktrees (fixed: the contract is copied in). Run 4 (all fixes): 11/11, 17m56s, $1.10, 12 gate blocks and no loops (28 → 17 → 12). Cost varies $0.79–1.15 per run; next: cut it (scaffold symbol parsing slips such as `(enum,`, supervisor turns), then a second intent beyond the semver fixture. Run 5: 11/11 but 29 min, $1.83, 44 blocks (the planner made 5 tasks; 27 false isolation blocks after workers `cd`'d to the main tree, fixed; subagents also read the plugin's own source, which costs tokens). Run 6: 11/11, **11m10s, $0.65**, plugin-source reads 30 → 2 after the agent prompts said not to; found a dispatch-before-contract loop (25 blocks, fixed by `gate-dispatch`) and annotated scaffold-symbol lines (fixed). Run 7 (after #9): 11/11, 13m46s, $0.98, 20 blocks, no dispatch loop; found RED blocked for listing an earlier task's passing tests, scaffold-symbols tables with a header row and trait-impl lines, and a guessed memory.md frontmatter (all fixed; `aloop init` now seeds memory.md). The re-plan coverage loss is now caught by `gate-final` (every planned test must be on main). Next: supervisor-written GREEN instructions overreaching the next task, then a second intent |
| **M0.6 — Gates fail closed** | v0.11.x | A gate that crashes (exit 1, e.g. an unbound variable) is a non-blocking hook error on Claude Code, so the action goes through. Make every gate trap unexpected errors into exit 2 with the reason, and add a conformance case per gate. Reason: the smoke run showed one real instance | done: `_failclosed.sh` on all 10 sub-agent gates + supervisor-scope inside a sprint; `scripts/test/fail-closed.test.py` (5 checks). Left fail-open on purpose: `gate-commit-author` and `gate-validate-artifact` outside a sprint, which see every session on the machine |
| **M0.7 — Supervisor-owned task contract** | v0.11.x | The contract lives in the main tree, `.agentic/state/tasks/<TASK_ID>.env`, and the dispatch prompt's first line names the task; gates and the time box find it from the host's transcript, a worktree `task.env` that disagrees blocks, and sub-agents may not Write/Edit `.agentic/state/` or `task.env`. Settled by a live probe: Claude Code's `isolation: "worktree"` makes `.claude/worktrees/agent-<id>` at dispatch, so a pre-written in-worktree contract was impossible there. Bash writes are guarded too (a command naming the contract with a write verb is blocked; a heuristic, so an obfuscated write still gets through). On hosts with no sub-agent transcript the worktree's `TASK_ID` (or its `.agentic/worktrees/<id>` path) picks the contract | done: `make conformance` (30) + `scripts/test/{budget-worktree,supervisor-scope}.test.py` |
| **M0.5 — Rename to aloop** | v0.11.0 | Plugin, marketplace, skill, CLI (`aloop`, `aloop init`), OpenCode plugin file and docs renamed; `agentic`, `agentic init`, `agentic-init` and `agentic bind` kept as aliases; OpenCode/Codex installs clean up pre-rename files. Data layout and settings keys unchanged (moved to M1). GitHub repo rename is the owner's call | done: `scripts/test/rename-compat.test.py` |
Expand Down
13 changes: 12 additions & 1 deletion plugin/bin/gate-final
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,21 @@ for d in "$root" ${sprint_dir:+"$sprint_dir"}; do
fi
done

# 2b) Every test a story's plan.md names (`- \`path::fn\``) exists on main. plan-ready.md lists
# only what RED wrote, so a test a re-plan dropped would otherwise vanish unnoticed.
if [ -n "$sprint_dir" ]; then
while IFS= read -r ref; do
path="${ref%%::*}"; fn="${ref##*::}"
grep -qE "fn[[:space:]]+${fn}[[:space:]]*\(" "$root/$path" 2>/dev/null \
|| fail "planned test \`$ref\` is not on main (in its story's plan.md): write it, or have the planner re-plan the story without it"
done < <(find "$sprint_dir" -name plan.md -path '*/s[0-9]*' -exec grep -hoE '^[[:space:]]*- `[^`:]+(::[A-Za-z_][A-Za-z0-9_]*)+`' {} + 2>/dev/null \
| sed -E 's/^[^`]*`//; s/`$//' | sort -u)
fi

# 3) Full gate matrix (from standards.md) green.
run_standards_matrix

note "FINAL gate passed: matrix green, zero suppressions, all plan-ready ticked"
note "FINAL gate passed: matrix green, zero suppressions, all plan-ready ticked, every planned test on main"

# 4) Sprint close — by the harness, only on the real SubagentStop run (never a self-check):
# sprintN/stats.md, docs/agents/NEXT.md (the post-/clear handoff), token totals.
Expand Down
11 changes: 11 additions & 0 deletions scripts/conformance/claude-code.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,17 @@ def tamper(lines):
results.append((9, "a hard-breached worker is released as 'not a pass, do not merge'", rc == 0 and said,
"exit %d %s" % (rc, (lines[0] if lines else "")[:140])))

# FINAL: every test the story's plan.md names must exist on main. A RED re-plan that drops a
# planned test otherwise loses coverage silently (live e2e run 6), since plan-ready.md only
# lists what RED actually wrote.
PLAN = "---\ntype: plan\nstory: S1-01\n---\n# plan\n\n## T1 tests\n\n- `tests/add.rs::add_works`\n — adds.\n"
write(os.path.join(STORY, "plan.md"), PLAN)
fenv = {"STORY_DIR": STORY, "AGENTIC_SELFCHECK": "1"}
case(6, "FINAL passes when every planned test exists on main", "gate-final", MAIN, 0, extra=fenv)
write(os.path.join(STORY, "plan.md"), PLAN + "- `tests/sub.rs::sub_handles_negatives`\n — dropped by a re-plan.\n")
case(6, "FINAL blocks when a planned test is missing from main", "gate-final", MAIN, 2, "sub_handles_negatives", extra=fenv)
os.remove(os.path.join(STORY, "plan.md"))

# §6.2 Planner artifacts are schema-valid (gate-validate-artifact runs md-db on every planner write)
def md_db():
"""The real md-db binary, put where ensure-tools finds it; built from the bundled source if needed."""
Expand Down
Loading
0