8000 gate-dispatch: the dispatch model is the project's resolved one, not the supervisor's pick (run 20 → run 21: $2.79 → $0.66) by adeelahmad · Pull Request #33 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
## [Unreleased]

### Added
- `gate-dispatch` now refuses a dispatch of any aloop role whose `model` is not the project's confirmed one (`aloop models resolve <role>`, matched by model family so `haiku` and `claude-haiku-5-5` agree), or that passes no model (which would fall back to the agent file's default, opus for the planning roles). Reason: live e2e run 20 dispatched planners on opus and workers on sonnet against a project whose defaults.md said haiku for both, and cost $2.79 instead of $0.99 for the same 12/12 result. Test: `scripts/test/dispatch.test.py`. Live run 21 with the gate: 12/12, 8m17s, $0.66, every dispatch on haiku (ledger rows for runs 20 and 21 in `docs/evidence/README.md`).
- Evidence ledger (ROADMAP §4.11): `docs/evidence/README.md` lists every live run behind a public
number (intent, plugin version, checks, wall, cost, wakes, plugin reads, gate blocks, reproduce
command); `bin/run-summary <run-dir>` writes a run's `summary.json`, `summary.md` and ledger row from
Expand Down
2 changes: 1 addition & 1 deletion docs/ROADMAP.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/TRACKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Status of every requirement in [ROADMAP.md](ROADMAP.md), in the four states §8
| 3.4 | Hard breaches classified with evidence before acting | IMPLEMENTED_TESTED | `scripts/test/budget-classify.test.py` |
| 3.5 | No post-kill narration; harness writes the re-plan block | IMPLEMENTED_TESTED | `scripts/test/budget-hard-stop.test.py` |
| 3.6 | Abstract model routes | IMPLEMENTED_TESTED | `scripts/test/model-routes.test.py` |
| 3.6 | The dispatch model is the project's resolved route, enforced by `gate-dispatch` | IMPLEMENTED_TESTED | `scripts/test/dispatch.test.py` |
| 4.1 | Intent Contract with harness-issued ids | IMPLEMENTED_TESTED | `scripts/test/intent-contract.test.py` |
| 4.1 | Clarification policy (assume / default / ask / spike) | IMPLEMENTED_TESTED | `scripts/test/intent-contract.test.py` |
| 4.1 | Explicit autonomous mode | IMPLEMENTED_TESTED | `scripts/test/intent-contract.test.py` |
Expand Down
4 changes: 4 additions & 0 deletions docs/evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,15 @@ stories). What each run found and fixed is in ROADMAP M0.8.
| run 17 | wordfreq | v0.11.x (master, 2026-10-18) | 13/13 | 12m58s | $0.95 | 2 | 0 | 6 | `make e2e-live YES=1 INTENT=wordfreq` |
| run 18 | wordfreq | v0.12.0 (fd1fd31) | 13/13 | 21m21s | $1.03 | 6 | 2 | 6 | `make e2e-live YES=1 INTENT=wordfreq` |
| run 19 | wordfreq | v0.12.0 + #29 (b93e6fd) | 13/13 | 15m24s | $1.01 | 1 | 0 | 3 | `make e2e-live YES=1 INTENT=wordfreq` |
| run 20 | semver | v0.12.0 + #31 (915810c) | 12/12 | 10m47s | $2.79 | 1 | 2 | 1 | `make e2e-live YES=1 INTENT=semver` |
| run 21 | semver | v0.12.0 + #33 (1ad94e2) | 12/12 | 8m17s | $0.66 | 1 | 0 | 3 | `make e2e-live YES=1 INTENT=semver` |

Columns: *checks* are the e2e script's own assertions (12 for `semver`, 13 for `wordfreq` since
the commit-trace check); *wakes* are supervisor turns that ended and resumed (each re-reads
context; the main cost driver); *plugin reads* are sub-agent tool calls into the plugin's own
source; *gate blocks* are `exit 2` verdicts in the run's `gates.jsonl` (host hooks and
self-checks). A dash means the run predates that counter.
Run 20 cost $2.79 because the supervisor dispatched planners on opus and workers on sonnet against a
project set to haiku; `gate-dispatch` now refuses that (#33), and run 21 is the same intent on haiku.
Run 19's reads were first reported as 5: all five were the planner's `md-db describe --schema` on
the plugin's schemas through a shell variable, which the counter now recognises as schema use.
2 changes: 1 addition & 1 deletion plugin/bin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ctx-symbols) → WARN + grep fallback; never a silent pass.
| script | event · matcher | checks | exit 2 when |
|--------|-----------------|--------|-------------|
| gate-supervisor-scope | PreToolUse · Write\|Edit\|MultiEdit\|Bash | MAIN agent (no `agent_type`) may not write production source mid-sprint, nor hand-author planning artifacts (stories/tasks/validate/plan/intake/standards.md — dispatch the activity), nor `git commit` on main (commits on main arrive through `task-worktree merge`); sub-agents (`agent_type` present) exempt | supervisor writes source / hand-writes a planner-owned artifact / commits on main |
| gate-dispatch | PreToolUse · Agent\|Task | the MAIN agent's dispatch of an execution role opens with `TASK_ID: <id>` and that task's contract `<run-data>/state/tasks/<id>.env` exists (repos with a run-data dir only) | no TASK_ID line / contract not written yet |
| gate-dispatch | PreToolUse · Agent\|Task | the MAIN agent's dispatch of any aloop role passes the project's confirmed model (`aloop models resolve <role>`, matched by family; no model or another model is refused), and a dispatch of an execution role opens with `TASK_ID: <id>` and that task's contract `<run-data>/state/tasks/<id>.env` exists (repos with a run-data dir only) | no TASK_ID line / contract not written yet |
| gate-tooling | SubagentStart · exec roles + supervisor self-check | md-db + ctx-symbols installed (execution may not run grep-degraded); runs `ensure-tools --sync` first | a backend is missing and can't be built |
| gate-validate-artifact | PostToolUse · Write | md-db-validate the written artifact | artifact malformed |
| gate-red-verify | SubagentStop · red-worker | worktree-isolated; appended output.md block (validate_comms); every new test FAILS BY ASSERTION; no regression; diff = tests+shims | no/stale output.md block / shared-tree run / a new test passes / prod code / regression |
Expand Down
33 changes: 29 additions & 4 deletions plugin/bin/gate-dispatch
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,46 @@ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
payload="$(cat 2>/dev/null || true)"

command -v python3 >/dev/null 2>&1 || exit 0
read -r role tid < <(printf '%s' "$payload" | python3 -c '
read -r role tid model < <(printf '%s' "$payload" | python3 -c '
import json, re, sys
p = json.load(sys.stdin)
if p.get("agent_type"):
print("- -"); sys.exit()
print("- - -"); sys.exit()
ti = p.get("tool_input") or {}
role = (ti.get("subagent_type") or "").split(":")[-1]
m = re.match(r"\s*TASK_ID:\s*([A-Za-z0-9._-]+)", ti.get("prompt") or "")
print(role or "-", m.group(1) if m else "-")' 2>/dev/null) || exit 0
model = re.sub(r"\s+", "", str(ti.get("model") or "")) or "-"
print(role or "-", m.group(1) if m else "-", model)' 2>/dev/null) || exit 0

case "$role" in red-worker|scaffolder|green-worker|structural-reviewer|final-gate) ;; *) exit 0 ;; esac
case "$role" in red-worker|scaffolder|green-worker|structural-reviewer|final-gate|planner|intake|standards|archivist) ;; *) exit 0 ;; esac
# shellcheck source=/dev/null
. "$HERE/_paths.sh"
[ -d "$AGENTIC_HOME" ] || exit 0

# The model is the project's confirmed one (docs/agents/defaults.md → `aloop models resolve <role>`,
# ROADMAP §3.6), never the supervisor's pick: live e2e run 20 dispatched planners on opus and
# workers on sonnet with defaults.md saying haiku for both, and cost $2.79 instead of $0.99. A
# pinned id or a route is matched by model family (haiku / sonnet / opus), so `haiku` and
# `claude-haiku-5-5` are the same answer; an absent model would fall back to the agent file's
# default (opus for the planning roles), so it is refused too. An empty resolution (Codex: the
# session's own model) checks nothing.
want="$("$HERE/models" resolve "$role" 2>/dev/null)" || want=""
if [ -n "$want" ]; then
fam() { case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in *haiku*) echo haiku ;; *sonnet*) echo sonnet ;; *opus*) echo opus ;; *) echo "" ;; esac; }
bad=0
if [ "$model" = "-" ]; then bad=1
elif [ "$model" != "$want" ]; then
wf="$(fam "$want")"; mf="$(fam "$model")"
{ [ -z "$wf" ] || [ -z "$mf" ] || [ "$wf" != "$mf" ]; } && bad=1
fi
if [ "$bad" = 1 ]; then
echo "BLOCK[$GATE_NAME]: dispatch $role with model: $want (the project's confirmed model for this role: \`aloop models resolve $role\`, from docs/agents/defaults.md)${model:+; this call passes $( [ "$model" = "-" ] && echo "no model, which would fall back to the agent file's default" || echo "model: $model")}." >&2
echo " The model per role is a project setting (confirmed at init), not a per-dispatch choice. To change it: re-run the init skill or edit .aladdin/config.yaml and \`aloop config sync\`." >&2
exit 2
fi
fi
case "$role" in planner|intake|standards|archivist) exit 0 ;; esac

if [ "$tid" = "-" ]; then
echo "BLOCK[$GATE_NAME]: start the $role dispatch prompt with the line \`TASK_ID: <id>\` — its gates find the task contract through it." >&2
exit 2
Expand Down
6 changes: 4 additions & 2 deletions plugin/skills/aloop/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,10 @@ Roles map to a ROUTE (`strongest` · `strong` · `economical`), and each host re
its own id (`plugin/hosts/routes.json`). The project's confirmed choice lives in
`docs/agents/defaults.md` (`AGENTIC_MODEL_PLANNING`, `AGENTIC_MODEL_WORKER`: a route or a
pinned id). On every dispatch pass `aloop models resolve planning|worker` as the Agent
call's `model`; never hard-code an id. The agent frontmatter carries the Claude Code
resolution of the defaults.
call's `model`; never hard-code an id. `gate-dispatch` refuses a dispatch on any other model,
or with none (live run 20: planners on opus and workers on sonnet against a haiku project cost
$2.79 instead of $0.99). The agent frontmatter carries the Claude Code resolution of the
defaults.

## Project setup comes first (init) — and where everything lives

Expand Down
37 changes: 35 additions & 2 deletions scripts/test/dispatch.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,21 @@ def ok(msg):
print("ok - " + msg)


def run(subagent_type, prompt, agent_type=None):
def run(subagent_type, prompt, agent_type=None, model="haiku"):
p = {"session_id": "s", "cwd": T, "tool_name": "Agent",
"tool_input": {"subagent_type": subagent_type, "prompt": prompt, "isolation": "worktree"}}
if model:
p["tool_input"]["model"] = model
if agent_type:
p["agent_type"] = agent_type
return subprocess.run([GATE], input=json.dumps(p), cwd=T, env=ENV, capture_output=True, text=True)


subprocess.run(["git", "init", "-q"], cwd=T, check=True)
os.makedirs(os.path.join(T, ".aladdin", "state", "tasks"))
os.makedirs(os.path.join(T, "docs", "agents"))
open(os.path.join(T, "docs", "agents", "defaults.md"), "w").write(
"---\ntype: agentic-defaults\n---\n```conf\nAGENTIC_MODEL_PLANNING=haiku\nAGENTIC_MODEL_WORKER=haiku\n```\n")

r = run("aloop:green-worker", "TASK_ID: S1-01-T1\nDo GREEN.")
assert r.returncode == 2 and "S1-01-T1.env" in r.stderr, r
Expand Down Expand Up @@ -75,7 +80,35 @@ def run(subagent_type, prompt, agent_type=None):
for st in ("aloop:planner", "aloop:intake", "general-purpose", "Explore"):
r = run(st, "Plan it.")
assert r.returncode == 0, (st, r)
ok("planning roles and non-aloop agents are not affected")
for st in ("general-purpose", "Explore"):
r = run(st, "Look around.", model="opus")
assert r.returncode == 0, (st, r)
ok("planning roles need no contract; non-aloop agents are not affected at all")

# The dispatch model is the project's confirmed one (ROADMAP §3.6), not the supervisor's pick.
# Live e2e run 20: defaults.md said haiku for every role and `aloop models resolve` printed it,
# yet the supervisor dispatched planners on opus and workers on sonnet: $2.79 instead of $0.99.
for st, m in (("aloop:green-worker", "sonnet"), ("aloop:green-worker", "claude-opus-5-5"), ("aloop:planner", "opus"),
("aloop:intake", "claude-sonnet-5-5"), ("aloop:archivist", "sonnet")):
r = run(st, "TASK_ID: S1-01-T1\nDo it.", model=m)
assert r.returncode == 2 and "model" in r.stderr and "haiku" in r.stderr and "aloop models resolve" in r.stderr, (st, m, r)
r = run("aloop:green-worker", "TASK_ID: S1-01-T1\nDo GREEN.", model=None)
assert r.returncode == 2 and "model: haiku" in r.stderr, r
r = run("aloop:planner", "Plan it.", model=None)
assert r.returncode == 2 and "model: haiku" in r.stderr, r
ok("a dispatch on a model other than the project's resolved one, or with no model, is denied for every aloop role")
for st, m in (("aloop:green-worker", "haiku"), ("aloop:green-worker", "claude-haiku-5-5"), ("aloop:planner", "haiku")):
r = run(st, "TASK_ID: S1-01-T1\nDo it.", model=m)
assert r.returncode == 0, (st, m, r)
open(os.path.join(T, "docs", "agents", "defaults.md"), "w").write(
"---\ntype: agentic-defaults\n---\n```conf\nAGENTIC_MODEL_PLANNING=strongest\nAGENTIC_MODEL_WORKER=strong\n```\n")
r = run("aloop:planner", "Plan it.", model="claude-opus-5-5")
assert r.returncode == 0, r
r = run("aloop:green-worker", "TASK_ID: S1-01-T1\nDo GREEN.", model="sonnet")
assert r.returncode == 0, r
r = run("aloop:green-worker", "TASK_ID: S1-01-T1\nDo GREEN.", model="haiku")
assert r.returncode == 2 and "sonnet" in r.stderr, r
ok("the resolved model (pinned id or route, by family) is accepted; a cheaper-than-confirmed pick is denied too")

r = run("aloop:green-worker", "TASK_ID: S1-09-T9\nDo GREEN.", agent_type="aloop:planner")
assert r.returncode == 0, r
Expand Down
4 changes: 2 additions & 2 deletions scripts/test/layout.test.py
< 4C20 td class="blob-num blob-num-expandable" colspan="2"> Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -113,13 +113,13 @@ def sh_home(t, **env):
assert out.splitlines()[0] == os.path.join(C, ".agentic", "state", "tasks", "T1.env") and out.splitlines()[-1] == "rc=1", out
assert run(C, "python3", "-c", 'import sys; sys.path.insert(0, "%s"); import _paths; print(_paths.contract_path("%s", "T1"))' % (BIN, C)).stdout.strip() == os.path.join(C, ".agentic", "state", "tasks", "T1.env")
payload = json.dumps({"hook_event_name": "PreToolUse", "tool_name": "Agent", "cwd": C,
"tool_input": {"subagent_type": "aloop:green-worker", "prompt": "TASK_ID: T1\ndo it"}})
"tool_input": {"subagent_type": "aloop:green-worker", "prompt": "TASK_ID: T1\ndo it", "model": "sonnet"}})
r = run(C, os.path.join(BIN, "gate-dispatch"), input=payload) if False else subprocess.run(
[os.path.join(BIN, "gate-dispatch")], cwd=C, input=payload, capture_output=True, text=True,
env=dict(os.environ, HOME=os.path.join(C, "home"), AGENTIC_NO_AUTOINSTALL="1"))
assert r.returncode == 0, (r.returncode, r.stderr)
payload2 = json.dumps({"hook_event_name": "PreToolUse", "tool_name": "Agent", "cwd": C,
"tool_input": {"subagent_type": "aloop:green-worker", "prompt": "TASK_ID: T9\ndo it"}})
"tool_input": {"subagent_type": "aloop:green-worker", "prompt": "TASK_ID: T9\ndo it", "model": "sonnet"}})
r = subprocess.run([os.path.join(BIN, "gate-dispatch")], cwd=C, input=payload2, capture_output=True, text=True,
env=dict(os.environ, HOME=os.path.join(C, "home"), AGENTIC_NO_AUTOINSTALL="1"))
assert r.returncode == 2 and ".aladdin/state/tasks/T9.env" in r.stderr, (r.returncode, r.stderr)
Loading
0