8000 Gates refuse a sprint in the run-data dir; record run 24 (haiku-supervisor experiment) by adeelahmad · Pull Request #41 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
- The planner's stop gate (`gate-stage2-complete` on SubagentStop) now logs its verdicts to `gates.jsonl` and the event stream like every other gate, and the stuck release covers the planner: the same block three times in a row releases its stop unverified with a `state/stuck/plan-sprintN.<agent>` marker the supervisor acts on (a re-dispatch is a new agent id and starts over). A direct call (the supervisor's own check, `selfcheck`) is logged as a self-check and never releases. Reason: live e2e run 19's re-planning planner was blocked 9 times on one unchanged reason, none counted. Test: `scripts/test/planner-stuck.test.py`.

### Fixed
- `gate-stage2-complete` and `gate-dispatch` refuse a sprint (or a `STORY_DIR`) inside the run-data dir (`.aladdin/`, `.agentic/`) and name `docs/agents/sprintN/`; with no argument the stage-2 gate looks in `docs/agents/` first. Reason: live run 24 (a haiku-supervisor experiment) planned and built the whole sprint under `.aladdin/sprint1/`, which is git-ignored and never committed, and every gate passed it. Tests: `scripts/test/planner-stuck.test.py`, `scripts/test/dispatch.test.py`.
- Final-gate persona: the suppression check is the gate's (`#[ignore]`, `.skip(`, `xit(`,
`#[cfg(not(test))]`, run by `selfcheck final-gate`), not an open-ended list; an `#[allow(dead_code)]`
in `tests/common/` is not a suppression (live run 22: the persona failed the sprint on it and cost a
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ conformance: ## ROADMAP §6 conformance fixtures on today's gates (needs cargo;
.PHONY: e2e-live
e2e-live: ## Live end-to-end run: one-line intent -> planned, tested, merged code (YES=1; spends dollars)
@test -n "$(YES)" || { echo "spends tokens: make e2e-live YES=1"; exit 2; }
scripts/smoke/e2e-live.sh --yes $(if $(INTENT),--intent $(INTENT)) $(if $(HOST),--host $(HOST))
scripts/smoke/e2e-live.sh --yes $(if $(INTENT),--intent $(INTENT)) $(if $(HOST),--host $(HOST)) $(if $(MODEL),--model $(MODEL))

.PHONY: smoke-live
smoke-live: ## Live Claude Code session: hooks fire, supervisor write blocked, transcripts + ledger (YES=1; spends ~$0.005)
Expand Down
2 changes: 1 addition & 1 deletion docs/ROADMAP.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/TRACKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,4 @@ Status of every requirement in [ROADMAP.md](ROADMAP.md), in the four states §8
| 4.11 | Evidence: every public claim links to a reproducible measured run (`docs/evidence/README.md`, `run-summary`) | IMPLEMENTED_TESTED | `scripts/test/evidence.test.py` |
| 7 | The final-gate persona's suppression check is the gate's (`assert_no_suppression`), never its own list | IMPLEMENTED_TESTED | `scripts/test/suppression-contract.test.py` |
| 4.11 | The north-star run reproducible on every host (`e2e-live.sh --host codex|opencode`: host install + headless CLI, same checks) | IMPLEMENTED_TESTED | `scripts/test/e2e-live-hosts.test.py` (measured Codex/OpenCode rows still pending: no CLIs here) |
| 7 | The sprint and every STORY_DIR live in `docs/agents/sprintN/`, never in the git-ignored run-data dir (`gate-stage2-complete`, `gate-dispatch`) | IMPLEMENTED_TESTED | `scripts/test/planner-stuck.test.py`, `scripts/test/dispatch.test.py` |
3 changes: 3 additions & 0 deletions docs/evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ the cost cell reads — and the intent cell names the host. What each run found
| run 21 | semver | v0.12.0 + #33 (1ad94e2) | 12/12 | 8m17s | $0.66 | 1 | 0 | 3 | `make e2e-live YES=1 INTENT=semver` |
| run 22 | wordfreq | v0.12.0 + #37 (361f8d7) | 13/13 | 16m18s | $1.14 | 2 | 0 | 3 | `make e2e-live YES=1 INTENT=wordfreq` |
| run 23 | wordfreq | v0.12.0 + #33 (7417bb9) | 13/13 | 10m58s | $0.76 | 4 | 1 | 0 | `make e2e-live YES=1 INTENT=wordfreq` |
| run 24 | wordfreq, haiku supervisor | v0.12.0 + #39 (9826566) | 10/13 | 19m09s | $0.66 | 4 | 1 | 6 | `make e2e-live YES=1 INTENT=wordfreq MODEL=haiku` |

Columns: *checks* are the e2e script's own assertions (12 for `semver`, 13 for `wordfreq` since
the commit-trace check); *wakes* are supervisor turns that ended and resumed (each re-reads
Expand All @@ -38,5 +39,7 @@ source; *gate blocks* are `exit 2` verdicts in the run's `gates.jsonl` (host hoo
self-checks). A dash means the run predates that counter.
Run 20 cost $2.79 because the supervisor dispatched planners on opus and workers on sonnet against a
project set to haiku; `gate-dispatch` now refuses that (#33), and run 21 is the same intent on haiku.
Run 24 is an experiment, not a public number: a haiku supervisor cost the same as sonnet in twice
the turns and planned the sprint in the run-data dir (now refused by the gates).
Run 19's reads were first reported as 5: all five were the planner's `md-db describe --schema` on
the plugin's schemas through a shell variable, which the counter now recognises as schema use.
2 changes: 1 addition & 1 deletion plugin/bin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ ctx-symbols) → WARN + grep fallback; never a silent pass.
| gate-structural-integrity | SubagentStop · structural-reviewer | appended output.md block; orphan/parallel/duplicate via ctx-symbols | no/stale output.md block / foundation-poisoning finding |
| gate-final | SubagentStop · final-gate | full matrix; zero suppressions; all plan-ready ticked | matrix red / suppression / unticked |
| gate-standards-cited | SubagentStop · standards | standards.md citations resolve | dangling citation |
| gate-stage2-complete | supervisor self-check + SubagentStop · planner | every story Stage-2 valid; no TBW; sprint self-contained (no later-sprint story IDs, no work deferred to a later sprint); verdicts logged like every gate, and the stuck release covers the planner (3 identical blocks → stop released unverified, marker `state/stuck/plan-sprintN.<agent>`) | incomplete or spilling planning |
| gate-stage2-complete | supervisor self-check + SubagentStop · planner | the sprint is in `docs/agents/sprintN/`, never the run-data dir; every story Stage-2 valid; no TBW; sprint self-contained (no later-sprint story IDs, no work deferred to a later sprint); verdicts logged like every gate, and the stuck release covers the planner (3 identical blocks → stop released unverified, marker `state/stuck/plan-sprintN.<agent>`) | incomplete or spilling planning |
| gate-plan-shape | called by gates | each plan checkbox carries `path::fn` | malformed plan |
| gate-ledger-format | called by gates | execution.log line shape | malformed ledger |

Expand Down
9 changes: 9 additions & 0 deletions plugin/bin/gate-dispatch
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,15 @@ fi
# reviewer's contract had none, since a per-wave review has no single story). A whole-wave review
# uses the sprint dir, whose init.md/output.md carry its dispatch and report blocks.
sd="$(sed -nE 's/^[[:space:]]*(export[[:space:]]+)?STORY_DIR=["'"'"']?([^"'"'"'#]*[^"'"'"'#[:space:]])["'"'"']?[[:space:]]*(#.*)?$/\2/p' "$contract" | tail -1)"
case "/$sd/" in
*/.aladdin/*|*/.agentic/*)
# The comms live with the plan in docs/agents/sprintN/ (live e2e run 24: a haiku supervisor
# put the whole sprint under .aladdin/sprint1/, git-ignored and never committed).
_home="$(printf '%s' "/$sd/" | grep -oE '/\.(aladdin|agentic)/' | head -1 | tr -d /)"
echo "BLOCK[$GATE_NAME]: STORY_DIR=$sd is in the run-data dir $_home/, which is git-ignored and never committed:" >&2
echo " the sprint and its story dirs live in docs/agents/sprintN/ (\`aloop layout show\`); move the sprint there and point STORY_DIR at it." >&2
exit 2 ;;
esac
if [ -z "$sd" ] || [ ! -d "$sd" ]; then
echo "BLOCK[$GATE_NAME]: the task contract $contract must set STORY_DIR to an existing dir${sd:+ (STORY_DIR=$sd does not exist)}:" >&2
echo " the ABSOLUTE path of the story's comms dir, docs/agents/sprintN/sN-NN-<slug>/ (holds init.md + output.md)." >&2
Expand Down
13 changes: 12 additions & 1 deletion plugin/bin/gate-stage2-complete
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

sprint_dir="${1:-${SPRINT_DIR:-}}"
if [ -z "$sprint_dir" ]; then
sprint_dir="$(find . -maxdepth 5 -type d -name 'sprint*' 2>/dev/null | sort | tail -1)"
# The layout's place first (docs/agents/sprintN/); anywhere else is found only to be refused below.
sprint_dir="$(find ./docs/agents -maxdepth 1 -type d -name 'sprint*' 2>/dev/null | sort -V | tail -1)"
[ -n "$sprint_dir" ] || sprint_dir="$(find . -maxdepth 5 -type d -name 'sprint*' 2>/dev/null | sort | tail -1)"
fi
if [ -z "$sprint_dir" ] || [ ! -d "$sprint_dir" ]; then
echo "WARN[$GATE_NAME]: no sprint dir found; stage-2 check skipped (weaker)" >&2; exit 0
Expand All @@ -31,6 +33,15 @@ if [ -z "${TASK_ID:-}" ] && [ -n "$GATE_PAYLOAD" ]; then
fi
fi

# 0) The sprint lives in docs/agents/sprintN/ (bin/_paths.sh), never in the run-data dir, which
# is git-ignored and never committed (live e2e run 24: a haiku supervisor had the whole sprint
# planned and built under .aladdin/sprint1/ and this gate passed it).
case "/$(cd "$sprint_dir" && pwd -P)/" in
*/.aladdin/*|*/.agentic/*)
_home="$(printf '%s' "/$(cd "$sprint_dir" && pwd -P)/" | grep -oE '/\.(aladdin|agentic)/' | head -1 | tr -d /)"
fail "the sprint is in the run-data dir $_home/ ($sprint_dir), which is git-ignored and never committed: plan it in docs/agents/$(basename "$sprint_dir")/ (\`aloop layout show\` prints the layout): mv '$sprint_dir' docs/agents/" ;;
esac

# 1) No TBW placeholders anywhere.
if grep -rl 'TBW' "$sprint_dir" >/dev/null 2>&1; then
fail "TBW remains in $sprint_dir (partial Stage-2 forbidden)"
Expand Down
3 changes: 3 additions & 0 deletions plugin/skills/aloop/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ project that still has `.agentic/` keeps using it until `aloop layout migrate` m
| `docs/agents/NEXT.md` | orchestrator handoff, reloaded after `/clear` | ignored unless tracked |
| `docs/agents/sprintN/` | sprint artifacts + `stats.md` | ignored unless tracked |

The sprint is always `docs/agents/sprintN/`, never under `.aladdin/` (run data, never
committed): `gate-stage2-complete` and `gate-dispatch` refuse a sprint or `STORY_DIR` there.

Commits are authored by the configured human author only; `gate-commit-author` blocks a
commit with another author or (unless allowed) a Claude co-author / attribution trailer.
Build each task's commit message with `events commit-msg <task>` (task title, recorded
Expand Down
16 changes: 16 additions & 0 deletions scripts/test/dispatch.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,22 @@ def run(subagent_type, prompt, agent_type=None, model="haiku"):
assert r.returncode == 2 and "sonnet" in r.stderr, r
ok("the resolved model (pinned id or route, by family) is accepted; a cheaper-than-confirmed pick is denied too")

# The sprint lives in docs/agents/sprintN/, never in the run-data dir (live e2e run 24: a haiku
# supervisor planned and built the whole sprint under .aladdin/sprint1/, which is git-ignored and
# never committed; no gate noticed, and the run's planning checks found no sprint at all).
open(os.path.join(T, "docs", "agents", "defaults.md"), "w").write(
"---\ntype: agentic-defaults\n---\n```conf\nAGENTIC_MODEL_PLANNING=haiku\nAGENTIC_MODEL_WORKER=haiku\n```\n")
for home in (".aladdin", ".agentic"):
hidden = os.path.join(T, home, "sprint1", "s1-01-x")
os.makedirs(hidden, exist_ok=True)
open(contract, "w").write("TASK_ID=S1-01-T1\nSTORY_DIR=%s\n" % hidden)
r = run("aloop:green-worker", "TASK_ID: S1-01-T1\nDo GREEN.")
assert r.returncode == 2 and "docs/agents/sprint" in r.stderr and home in r.stderr, (home, r)
open(contract, "w").write("TASK_ID=S1-01-T1\nSTORY_DIR=%s\n" % story)
r = run("aloop:green-worker", "TASK_ID: S1-01-T1\nDo GREEN.")
assert r.returncode == 0, r
ok("a STORY_DIR inside the run-data dir (.aladdin/, .agentic/) is denied with the docs/agents/sprintN/ path")

r = run("aloop:green-worker", "TASK_ID: S1-09-T9\nDo GREEN.", agent_type="aloop:planner")
assert r.returncode == 0, r
ok("a sub-agent's own Agent calls are not this gate's business")
Expand Down
20 changes: 20 additions & 0 deletions scripts/test/planner-stuck.test.py
6E5A
Original file line number Diff line number Diff line change
Expand Up @@ -92,4 +92,24 @@ def rows():
assert rows()[-1]["exit"] == 0 and rows()[-1]["selfcheck"] is False
ok("a complete plan passes and is logged as a pass")

# The sprint lives in docs/agents/sprintN/, never in the run-data dir (live e2e run 24: a haiku
# supervisor had the whole sprint planned under .aladdin/sprint1/, git-ignored and never
# committed, and this gate passed it). A complete plan there is refused, named or discovered.
H = os.path.join(T, "hidden")
for home in (".aladdin", ".agentic"):
hp = os.path.join(H, home, "sprint1")
for rel, text in (("stories.md", "---\ntype: stories\nsprint: 1\n---\n- S1-01 export\n"),
("s1-01-export/tasks.md", "---\ntype: tasks\nstory: S1-01\n---\n## T1 — a\n"),
("s1-01-export/validate.md", "---\ntype: validate\nstory: S1-01\n---\n## T1\n"),
("s1-01-export/plan.md", "---\ntype: plan\nstory: S1-01\nscope: tests only\n---\n")):
write(os.path.join("hidden", home, "sprint1", rel), text)
if not os.path.isdir(os.path.join(H, ".git")):
subprocess.run(["git", "init", "-q"], cwd=H, check=True)
r = subprocess.run([GATE, hp], cwd=H, env=dict(ENV, AGENTIC_SELFCHECK="1"), capture_output=True, text=True, stdin=subprocess.DEVNULL)
assert r.returncode == 2 and "docs/agents/sprint1" in r.stderr and home in r.stderr, (home, r)
r = subprocess.run([GATE], cwd=H, env=dict(ENV, AGENTIC_SELFCHECK="1"), capture_output=True, text=True, stdin=subprocess.DEVNULL)
assert r.returncode == 2 and "docs/agents/sprint1" in r.stderr, (home, "discovered", r)
subprocess.run(["rm", "-rf", os.path.join(H, home)], check=True)
ok("a sprint planned in the run-data dir (.aladdin/, .agentic/) is refused with the docs/agents/sprintN/ path, named or discovered")

print("\n%d checks passed" % n)
Loading
0