8000 Add `aloop budget contract`: the harness writes the task contract by adeelahmad · Pull Request #48 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
## [Unreleased]

### Added
- `aloop budget contract <TASK_ID> --role R --story DIR [--scope G] [--attempt N] [--symbols S]` writes the supervisor's task contract (`<run-data>/state/tasks/<TASK_ID>.env`) with `BASE_REF=HEAD` and the resolved `BUDGET_*` (the story's tasks.md `budget:` line wins), refuses a run-data or missing story dir and an unknown role, and prints one line. SKILL.md tells the supervisor to use it instead of a heredoc. Reason: live run 26's supervisor hand-wrote 9 contracts (about 400 characters each of keys the harness owns), model output re-read on every later turn. Test: `scripts/test/budget-contract.test.py`.
- `gate-dispatch` now refuses a dispatch of any aloop role whose `model` is not the project's confirmed one (`aloop models resolve <role>`, matched by model family so `haiku` and `claude-haiku-5-5` agree), or that passes no model (which would fall back to the agent file's default, opus for the planning roles). Reason: live e2e run 20 dispatched planners on opus and workers on sonnet against a project whose defaults.md said haiku for both, and cost $2.79 instead of $0.99 for the same 12/12 result. Test: `scripts/test/dispatch.test.py`. Live run 21 with the gate: 12/12, 8m17s, $0.66, every dispatch on haiku Run 23 on the two-story `wordfreq` intent: 13/13, 10m58s, $0.76, 0 gate blocks (ledger rows for runs 20–23 in `docs/evidence/README.md`).
- Evidence ledger (ROADMAP §4.11): `docs/evidence/README.md` lists every live run behind a public
number (intent, plugin version, checks, wall, cost, wakes, plugin reads, gate blocks, reproduce
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ test-hosts: ## Budget/worktree regression + Codex hooks emulation + OpenCode plu
python3 scripts/test/budget-hard-stop.test.py
python3 scripts/test/budget-meter.test.py
python3 scripts/test/budget-roles.test.py
python3 scripts/test/budget-contract.test.py
python3 scripts/test/budget-classify.test.py
python3 scripts/test/sprint-admission.test.py
python3 scripts/test/planner-feedback.test.py
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ structural, not luck:
These mechanisms are codebase-agnostic — greenfield is simply what's measured here.

Live runs: `make smoke-live YES=1` (hook wiring, ~$0.004) and `make e2e-live YES=1` (the whole
loop, ~$1; each run writes `summary.json` and a row for [docs/evidence/](docs/evidence/README.md)). Offline: `make ci` — Rust fmt/clippy/tests, shellcheck, 46 Python/Node suites,
loop, ~$1; each run writes `summary.json` and a row for [docs/evidence/](docs/evidence/README.md)). Offline: `make ci` — Rust fmt/clippy/tests, shellcheck, 47 Python/Node suites,
the gate conformance fixtures (`make conformance`), eval-suite validation. Every ROADMAP
requirement is tracked with its test in [docs/TRACKING.md](docs/TRACKING.md).

Expand Down
1 change: 1 addition & 0 deletions docs/TRACKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Status of every requirement in [ROADMAP.md](ROADMAP.md), in the four states §8
| 4.1 | Explicit autonomous mode | IMPLEMENTED_TESTED | `scripts/test/intent-contract.test.py` |
| 4.2 | Sprint token envelope, admission control, rollover | IMPLEMENTED_TESTED | `scripts/test/sprint-admission.test.py` |
| 4.2 | Time budget `auto` mode | IMPLEMENTED_TESTED | `scripts/test/timebox-auto.test.py` |
| 4.2 | Task contract written by the harness (`budget contract`: keys, BASE_REF, resolved budget) | IMPLEMENTED_TESTED | `scripts/test/budget-contract.test.py` |
| 4.2 | Interruptible orchestrator loop (inspect step, token-free wait) | IMPLEMENTED_TESTED | `scripts/test/orchestrator-sweep.test.py` |
| 4.3 | Failure envelope, grace rule, calibration verdict | IMPLEMENTED_TESTED | `scripts/test/planner-feedback.test.py` |
| 4.3 | Lesson candidates promoted only by the recurrence rule | IMPLEMENTED_TESTED | `scripts/test/lesson-candidates.test.py` |
Expand Down
50 changes: 50 additions & 0 deletions plugin/bin/budget
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ Resolution order (first set wins): per-task `budget:` line in tasks.md >

Usage:
budget resolve [--tasks FILE --task Tn] [--worktree WT] the resolved BUDGET_* lines + their source
budget contract TASK_ID --role ROLE --story DIR [--scope GLOBS] [--attempt N] [--symbols S]
before dispatch: write the task contract
(<run-data>/state/tasks/TASK_ID.env) with BASE_REF=HEAD and
the resolved BUDGET_* (DIR's tasks.md `budget:` line for
the id's Tn wins); prints one line
budget hook pre|post|stop hook mode (reads the hook JSON on stdin)
budget watch [--interval S] [--grace S] supervisor: block until a worker must be
killed, one was released STUCK, or none is running;
Expand Down Expand Up @@ -1039,6 +1044,49 @@ def replans(cwd, task):
return 3 if n >= cap else 0


def write_contract(cwd, task_id, opts):
"""`budget contract`: the supervisor's per-task contract, written by the harness that owns
every key in it (live run 26: 9 hand-written heredocs a run, re-read on every later turn)."""
def refuse(msg):
print("budget contract: " + msg, file=sys.stderr)
return 64
if not re.match(r"^S\d+-[A-Za-z0-9._-]+$", task_id or ""):
return refuse("need a TASK_ID like S1-02-T3 (got %r)" % task_id)
role = (opts.get("role") or "").split(":")[-1]
if role not in WORKERS + tuple(ROLE_PROFILES):
return refuse("--role must be one of %s (got %r)" % (", ".join(WORKERS + tuple(ROLE_PROFILES)), role))
attempt = opts.get("attempt") or "1"
if not attempt.isdigit() or int(attempt) < 1:
return refuse("--attempt must be a number >= 1 (got %r)" % attempt)
root = main_tree(cwd)
story = os.path.realpath(os.path.join(cwd, opts.get("story") or ""))
rel = os.path.relpath(story, root)
if not opts.get("story") or rel.split(os.sep)[0] in (_paths.NEW_HOME, _paths.OLD_HOME) or rel.startswith(".."):
return refuse("--story must be the story's sN-NN-<slug>/ dir (or the sprint dir for a whole-sprint "
"role) under docs/agents/sprintN/, never run data (got %r)" % opts.get("story"))
if not os.path.isdir(story):
return refuse("--story %s does not exist" % story)
tn = task_id.rsplit("-", 1)[-1]
tasks = os.path.join(story, "tasks.md")
b, src = resolve(cwd, tasks if os.path.isfile(tasks) else None, tn, None, with_source=True)
lines = [("TASK_ID", task_id), ("ATTEMPT", attempt), ("AGENT_ROLE", role), ("STORY_DIR", story),
("SCOPE_GLOBS", opts.get("scope") or "")]
if opts.get("symbols"):
lines.append(("SCAFFOLD_SYMBOLS", opts["symbols"]))
lines.append(("BASE_REF", sh("git", "rev-parse", "HEAD", cwd=root)))
lines += [("BUDGET_" + k, str(b[k])) for k in ("TOKENS_SOFT", "TOKENS_HARD", "TIME_SOFT_MIN", "TIME_HARD_MIN")]
lines.append(("BUDGET_TOKEN_METER", b["TOKEN_METER"]))
path = os.path.join(root, _paths.home_name(root), "state", "tasks", task_id + ".env")
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path + ".tmp", "w") as f:
f.write("".join('%s="%s"\n' % (k, v) if " " in v else "%s=%s\n" % (k, v) for k, v in lines))
os.replace(path + ".tmp", path)
print("%s: %s attempt %s, budget %d/%d tokens, %d/%d min (from %s)" % (
os.path.relpath(path, root), role, attempt, b["TOKENS_SOFT"], b["TOKENS_HARD"],
b["TIME_SOFT_MIN"], b["TIME_HARD_MIN"], src))
return 0


def main(argv):
cwd = os.getcwd()
if not argv or argv[0] in ("-h", "--help", "help"):
Expand Down Expand Up @@ -1067,6 +1115,8 @@ def main(argv):
print("BUDGET_TOKEN_METER=%s" % b["TOKEN_METER"])
print("# from: %s" % src)
return 0
if cmd == "contract":
return write_contract(cwd, (opts.get("_") or [""])[0], opts)
if cmd == "watch":
return watch(cwd, int(opts.get("interval") or 20), int(opts.get("grace") or 60))
if cmd == "status":
Expand Down
25 changes: 12 additions & 13 deletions plugin/skills/aloop/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -401,8 +401,7 @@ main tree: .aladdin/state/stuck/<task>.<attempt> a gate released a worker (or,
spec. NEVER rewrite an earlier block. The file opens once with `--- type: init ---`.
That list IS the block's whole shape: do not read `pipeline/execution/*/init.template.md`
or `bin/log-execution` to learn it (live runs 13–15 spent 4–6 reads per run on them).
At dispatch you also write the task contract `<run-data>/state/tasks/<TASK_ID>.env` (with
`STORY_DIR` = absolute path to this story dir) and `bin/transcripts stage-in` stages `.transcripts/`.
At dispatch you also write the task contract (`aloop budget contract`, below) and `bin/transcripts stage-in` stages `.transcripts/`.
- `output.md` (story-bound, **append-only**) — the **agent** APPENDS one block per
attempt: `## <task> · attempt N · <role> · <ts>`, `status:`, then `### Summary`, a
`### Result` table (Check/Status/Detail), optional `### Findings`/`### Scaffold`, and
Expand Down Expand Up @@ -488,8 +487,8 @@ Your job during execution is small, and that is on purpose (it keeps your contex

1. **Dispatch** every task that can run now (the wave's RED, or GREEN, …) as
background agents — `run_in_background: true`, `isolation: "worktree"`, `model` from
defaults.md — after writing each `init.md` block + task contract (with
`BUDGET_*`). Log `start` lines. **A wave with ONE dispatchable task runs it in the
defaults.md — after writing each `init.md` block + task contract
(`aloop budget contract`). Log `start` lines. **A wave with ONE dispatchable task runs it in the
foreground** (`run_in_background: false`, no `budget watch`): every wake re-reads your
whole context (live run 15: 16 wakes, 5.5M cached tokens, $1.26 of a $1.42 run; run 14
with foreground dispatch: 2, $0.75), so background dispatch pays only when tasks run in
Expand Down Expand Up @@ -735,15 +734,15 @@ Transcripts are captured by the hooks with no action from you; `reference.md` §
Stale worktrees: `worktree-hygiene` (see `reference.md` § "Worktree hygiene").

## The task contract (<run-data>/state/tasks/<TASK_ID>.env)
Before each dispatch, write the contract in the MAIN tree's run-data dir, i.e.
`.aladdin/state/tasks/<TASK_ID>.env` (`.agentic/state/tasks/...` in a legacy project; `aloop
layout show` prints the dir, and the gates search both), with `TASK_ID`, `ATTEMPT`, `AGENT_ROLE`, `STORY_DIR`
(absolute path to the story dir — where `validate_comms` reads init.md/output.md),
`SCOPE_GLOBS` (GREEN diff-scope), `SCAFFOLD_SYMBOLS` (or the scaffolder writes
`.aladdin/scaffold-symbols` in its worktree), `BASE_REF` (diff base), and the four `BUDGET_*` lines printed by
`budget resolve --tasks <STORY_DIR>/tasks.md --task Tn`. Then make the dispatch prompt's
FIRST line `TASK_ID: <TASK_ID>`. (`gate-dispatch` refuses an execution dispatch without that line or before the contract exists.) Write plain `KEY=value` lines (quotes allowed, lists space- or comma-separated, no `$VAR`
expansion, no commands): the gate library PARSES the contract keys and ignores everything else.
Before each dispatch, write the contract with ONE command (never a heredoc):
`aloop budget contract <TASK_ID> --role <role> --story <STORY_DIR> [--scope "<GREEN globs>"]
[--attempt N] [--symbols "<scaffold symbols>"]`. It writes the MAIN tree's
`<run-data>/state/tasks/<TASK_ID>.env` with `TASK_ID`, `ATTEMPT`, `AGENT_ROLE`, the absolute
`STORY_DIR` (the story dir, or the sprint dir for a whole-sprint role), `SCOPE_GLOBS`,
`SCAFFOLD_SYMBOLS`, `BASE_REF=HEAD` and the resolved `BUDGET_*` (the story's tasks.md
`budget:` line for the id's Tn wins), and prints one line with the budget for your init.md
block. Then make the dispatch prompt's FIRST line `TASK_ID: <TASK_ID>` (`gate-dispatch`
refuses an execution dispatch without that line or before the contract exists).

Why the main tree: on Claude Code `isolation: "worktree"` creates the worktree
(`.claude/worktrees/agent-<id>`) at dispatch, so there is no worktree to write into
Expand Down
2 changes: 1 addition & 1 deletion plugin/skills/aloop/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ All of these are on PATH — call them by bare name.
- `gate-tooling` — execution preflight; BLOCKS the first dispatch unless md-db +
ctx-symbols are installed (auto-installs first; SubagentStart hook + manual check).
- `ensure-tools` — SessionStart installer (`--sync` build now, `--wait`, `--resolve T`).
- `budget` — the time box: `resolve` (BUDGET_* for task.env), `watch` (your sleep),
- `budget` — the time box: `contract` (writes the task contract), `resolve` (BUDGET_* alone), `watch` (your sleep),
`status`, `mark-killed`, `replans` (split-cap check); `hook` is the hook entrypoint.
- `stats show` — token totals (this session, per sprint, whole project); `stats
sprint-close` is run by gate-final, not by you.
Expand Down
97 changes: 97 additions & 0 deletions scripts/test/budget-contract.test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
#!/usr/bin/env python3
"""budget contract: one command writes the supervisor's task contract
(<run-data>/state/tasks/<TASK_ID>.env) with BASE_REF and the resolved BUDGET_* lines.

Reason (live run 26): the supervisor hand-wrote every contract as a heredoc (TASK_ID, ATTEMPT,
AGENT_ROLE, STORY_DIR, SCOPE_GLOBS, BASE_REF and five BUDGET_* lines), 9 times a run, each one
model output re-read on every later turn. The keys and the budget are facts the harness owns.

python3 scripts/test/budget-contract.test.py
"""
import atexit
import json
import os
import shutil
import subprocess
import tempfile

BIN = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..", "plugin", "bin")
T = tempfile.mkdtemp(prefix="aloop-contract-")
atexit.register(shutil.rmtree, T, ignore_errors=True)
ENV = {k: v for k, v in os.environ.items() if not k.startswith(("AGENTIC_", "ALADDIN_")) and k != "SKIP_HOOKS"}
ENV["HOME"] = os.path.join(T, "home")
n = 0


def ok(msg):
global n
n += 1
print("ok - " + msg)


def budget(*args):
return subprocess.run([os.path.join(BIN, "budget"), "contract"] + list(args), cwd=T, env=ENV,
capture_output=True, text=True)


def kv(task):
out = {}
for line in open(os.path.join(T, ".aladdin", "state", "tasks", task + ".env")):
k, _, v = line.rstrip("\n").partition("=")
out[k] = v.strip('"')
return out


subprocess.run(["git", "init", "-q"], cwd=T, check=True)
subprocess.run(["git", "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-q", "--allow-empty", "-m", "init"], cwd=T, check=True)
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=T, capture_output=True, text=True).stdout.strip()
os.makedirs(os.path.join(T, ".aladdin"))
os.makedirs(os.path.join(T, "docs", "agents"))
open(os.path.join(T, "docs", "agents", "defaults.md"), "w").write(
"---\ntype: agentic-defaults\n---\n```conf\nAGENTIC_MODEL_WORKER=haiku\nAGENTIC_TOKENS_HARD=200000\n```\n")
story = os.path.join(T, "docs", "agents", "sprint1", "s1-01-count")
os.makedirs(story)
open(os.path.join(story, "tasks.md"), "w").write(
"---\ntype: tasks\n---\n## T1 — count\nbudget: tokens_hard=90000 time_hard=12\n\n## T2 — top\n")

r = budget("S1-01-T2", "--role", "red-worker", "--story", "docs/agents/sprint1/s1-01-count", "--scope", "tests/**")
assert r.returncode == 0, r
c = kv("S1-01-T2")
assert c["TASK_ID"] == "S1-01-T2" and c["ATTEMPT"] == "1" and c["AGENT_ROLE"] == "red-worker", c
assert c["STORY_DIR"] == os.path.realpath(story) and c["SCOPE_GLOBS"] == "tests/**" and c["BASE_REF"] == head, c
assert c["BUDGET_TOKENS_HARD"] == "200000" and c["BUDGET_TOKEN_METER"] == "fresh", c
assert len(r.stdout.strip().splitlines()) == 1 and "S1-01-T2.env" in r.stdout, r.stdout
ok("writes TASK_ID, ATTEMPT=1, role, absolute STORY_DIR, scope, BASE_REF=HEAD and defaults.md's budget; prints one line")

r = budget("S1-01-T1", "--role", "green-worker", "--story", story, "--attempt", "2", "--symbols", "count_words Counter")
assert r.returncode == 0, r
c = kv("S1-01-T1")
assert c["ATTEMPT"] == "2" and c["SCAFFOLD_SYMBOLS"] == "count_words Counter", c
assert c["BUDGET_TOKENS_HARD"] == "90000" and c["BUDGET_TIME_HARD_MIN"] == "12", c
assert "tasks.md" in r.stdout, r.stdout
ok("the story's tasks.md `budget:` line for the task id's Tn wins; --attempt and --symbols are written")

r = budget("S1-SR-W1", "--role", "structural-reviewer", "--story", os.path.dirname(story))
assert r.returncode == 0 and kv("S1-SR-W1")["STORY_DIR"] == os.path.realpath(os.path.dirname(story)), r
ok("a whole-sprint role is contracted on the sprint dir")

gate = subprocess.run([os.path.join(BIN, "gate-dispatch")], cwd=T, env=ENV, capture_output=True, text=True, input=json.dumps(
{"session_id": "s", "cwd": T, "tool_name": "Agent", "tool_input": {
"subagent_type": "aloop:red-worker", "prompt": "TASK_ID: S1-01-T2\nDo RED.", "isolation": "worktree", "model": "haiku"}}))
assert gate.returncode == 0, gate
ok("gate-dispatch accepts a dispatch on the written contract")

before = sorted(os.listdir(os.path.join(T, ".aladdin", "state", "tasks")))
for args, why in (
(("S1-01-T3", "--role", "red-worker", "--story", "docs/agents/sprint1/nope"), "does not exist"),
(("S1-01-T3", "--role", "red-worker", "--story", ".aladdin"), "docs/agents/sprintN"),
(("S1-01-T3", "--role", "wizard", "--story", story), "role"),
(("S1-01-T3", "--story", story), "role"),
(("../x", "--role", "red-worker", "--story", story), "TASK_ID"),
(("S1-01-T3", "--role", "red-worker", "--story", story, "--attempt", "x"), "attempt")):
r = budget(*args)
assert r.returncode == 64 and why in r.stderr, (args, r)
assert sorted(os.listdir(os.path.join(T, ".aladdin", "state", "tasks"))) == before
ok("a missing or run-data story dir, an unknown role, a bad TASK_ID or attempt is refused (exit 64) and writes nothing")

print("%d checks passed" % n)
Loading
0