8000 dsh cost includes compaction and title calls; macOS CI runs the real dsh test; host tests pass with md-db installed by adeelahmad · Pull Request #118 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# CI — the checks `make ci` + `make validate` run, mirrored on every push/PR.
# Documented in README.md ("Continuous integration"). Four independent jobs:
# Documented in README.md ("Versioning, changelog and CI"). Five independent jobs:
# rust — ctx-symbols + md-db: fmt-check, clippy -D warnings, tests; §6 conformance fixtures
# shellcheck — lints every gate script (errors fail the build)
# plugin — JSON sanity + eval-suite validation + `claude plugin validate --strict`
# hosts — Codex hooks emulation + OpenCode plugin end-to-end + the dsh plugin on the real,
# pinned DeepSeek Harness runtime against a scripted model (no network, no tokens)
# macos — the host tests and the conformance fixtures on macOS (symlinked /var temp dirs,
# BSD userland): the v0.12.0 release run found two bugs Linux CI never saw
# macos — the host tests (the real dsh runtime too) and the conformance fixtures on macOS
# (symlinked /var temp dirs, BSD userland): the v0.12.0 release run found two bugs
# Linux CI never saw
name: CI

on:
Expand Down Expand Up @@ -96,7 +97,13 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: "22"
- name: Host tests (budget, events, Codex hooks, OpenCode plugin)
- name: Install the pinned DeepSeek Harness (dsh) runtime
# dsh is a preview: pinned, and re-verified on each release (ROADMAP §4.9)
run: |
npm install --prefix "$RUNNER_TEMP/dsh" --no-audit --no-fund @deepseek-ai/dsh@0.2.1-rc.2
echo "DSH_BIN=$RUNNER_TEMP/dsh/node_modules/@deepseek-ai/dsh/lib/bin.js" >> "$GITHUB_ENV"
echo "ALOOP_REQUIRE_DSH=1" >> "$GITHUB_ENV"
- name: Host tests (budget, events, Codex hooks, OpenCode and dsh plugins)
run: make test-hosts
env:
BASH32: /bin/bash # stock macOS bash 3.2: bash-portable.test.py runs the gate matrix on it
Expand Down
9 changes: 7 additions & 2 deletions CHANGELOG.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -425,8 +425,8 @@ enforce this. Tag releases `vMAJOR.MINOR.PATCH`.
- **hosts**: byte-compiles the Python hooks, then Codex hooks emulation + the OpenCode
and dsh plugins end-to-end, the dsh one also on the real (pinned) dsh runtime against a
scripted model (no network, no tokens).
- **macos**: the same host tests and the conformance fixtures on macOS, where temp dirs sit
behind a symlink and the userland is BSD.
- **macos**: the same host tests (the real dsh runtime included) and the conformance fixtures
on macOS, where temp dirs sit behind a symlink and the userland is BSD.
- **plugin**: JSON sanity on the manifest + marketplace, eval-suite validation, version
lockstep, then `claude plugin validate ./plugin --strict`.

Expand Down
2 changes: 1 addition & 1 deletion docs/OVERVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ The model can ignore a skill; it cannot ignore an exit code.
ROADMAP §6 conformance suite (real gates on a fixture crate, plus the focused tests that hold
the tool-level items: 19 of the 20 items, 82 checks; §6.15 is measured by live runs), the host tests
(budget/worktree, Codex hooks emulation, supervisor scope, hook matchers, dispatch,
ensure-tools, fail-closed: 85 checks; OpenCode end-to-end: 31; dsh plugin: 46, on the real dsh runtime: 50, dsh install: 10), `claude plugin validate
ensure-tools, fail-closed: 106 checks; OpenCode end-to-end: 40; dsh plugin: 48, on the real dsh runtime: 53, dsh install: 10), `claude plugin validate
--strict`, eval-suite and version-lockstep checks,
the tracking check and `host check`, plus the newer suites (targets render, host swap,
metrics, config, the aladdin CLI, the site).
Expand Down
20 changes: 12 additions & 8 deletions docs/ROADMAP.md

Large diffs are not rendered by default.

8 changes: 5 additions & 3 deletions docs/evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,20 @@ Every time or cost figure in the [README](../../README.md), on the [site](../ind
[ROADMAP §0](../ROADMAP.md#0-at-a-glance) is a row here. Each row is one live run of the whole
loop (`scripts/smoke/e2e-live.sh`): a one-line intent through intake, standards, planning,
RED/SCAFFOLD/GREEN per task, structural review and the final gate, to merged, tested code in a
real host session (Claude Code, or Codex/OpenCode with `HOST=`) with the approval given up front. The numbers come from the harness
real host session (Claude Code, or dsh with `HOST=dsh`; a Codex or OpenCode run measures no cost, see below) with the approval given up front. The numbers come from the harness
(`run-summary` reads the session's `result` lines and the gate log), never from the model.
`scripts/test/evidence.test.py` fails if a public figure has no row, or a row has no reproduce
command.
command or a cost cell other than a measured cost as `run-summary` prints it for the row's host.

**Reproduce:** install the plugin, then run the command in the last column (spends about a
dollar; the latest passing run of each intent took 7–16 minutes). The run leaves `summary.json`,
`summary.md` and a ready-to-paste row in its run dir; add the row here. Intents: `semver` (one
story), `wordfreq` (two dependent stories), `slugify` (Python), `roman` (Go), `camel` (JS),
`extend` and `modify` (brownfield Rust) and `cli` (a Rust binary). `HOST=codex` / `HOST=opencode` run the loop on those hosts (`aloop install <host>` into the
run repo, then `codex exec --json` / `opencode run --format json`); their CLIs print no cost line, so
the cost cell reads — and the intent cell names the host. What each run found and fixed is in ROADMAP M0.8.
the cost cell reads — and the intent cell names the host (an unmeasured cost is no ledger figure, so
`evidence.test.py` refuses such a row: ROADMAP §4.9). On `HOST=dsh`, dsh prints no cost line either, but
`run-summary` prices each call its plugin logged, so a dsh row's cost reads four places (`<$0.0001` below). What each run found and fixed is in ROADMAP M0.8.

| run | intent | plugin | checks | wall | cost | wakes | plugin reads | gate blocks | reproduce |
|---|---|---|---|---|---|---|---|---|---|
Expand Down
17 changes: 10 additions & 7 deletions plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,18 +159,21 @@ dsh --profile aloop "Use the aloop skill. Intent: <one line>"
- **Messages and compaction**: a role is not offered `send_message` (the plugin scopes it out of
the role's tools at its start), so its result reaches the supervisor through its stop gate and
dsh's settlement notice; the supervisor's `send_message` reaches a running role at its next step.
dsh compacts a session in place, and the plugin records `context.compacted` and injects the
handoff again.
- **Cost**: `run-summary --host dsh` prices each logged call from the dated DeepSeek table in
`bin/_prices.py`; a model with no published price leaves the cost unmeasured.
dsh compacts a session in place, and the plugin records `context.compacted` and adds the
handoff to the first request after it (the retry of an overflowed step included).
- **Cost**: `run-summary --host dsh` prices each logged call (dsh's compaction and session-title
calls included) from the dated DeepSeek table in `bin/_prices.py`, to the micro-dollar (the
evidence row to four places, `<$0.0001` below); a model with no published price leaves the cost
unmeasured. A retried attempt and DeepSeek's server-side
Search/Fetch are not in it (ROADMAP §4.9).
- **Unattended runs**: dsh asks before commands unless `DSH_PERMISSION_MODE=danger-full-access`
(no sandbox, no approvals); use that only in a throw-away repo, as `make e2e-live HOST=dsh` does.
The profile disables dsh's session-log upload (`session-log-deepseek`); set
`DSH_TELEMETRY_DISABLED=1` to stop its telemetry too.

Covered in CI by `make test-hosts`: `dsh-plugin.test.mjs` (the plugin against an emulated
dsh context) and `dsh-runtime.test.mjs` (the real dsh runtime, pinned, driven by a scripted
model: no network, no tokens).
Covered in CI by `make test-hosts` on Linux and macOS: `dsh-plugin.test.mjs` (the plugin
against an emulated dsh context) and `dsh-runtime.test.mjs` (the real dsh runtime, pinned,
driven by a scripted model: no network, no tokens).

## Invoking it

Expand Down
2 changes: 1 addition & 1 deletion plugin/bin/README.md
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ ctx-symbols) → WARN + grep fallback; never a silent pass.
| `worktree-hygiene` | SubagentStart · exec roles — sparse-checks `docs/agents/` out of the new worktree (workers reach it via the absolute `STORY_DIR` back into the main tree, never their own copy) + bootstraps the main tree's `.gitignore` with `.agentic/`/`.transcripts/` if missing. Runs regardless of whether the harness's built-in isolation or `worktree-create` made the worktree. |
| `worktree-create` | OPTIONAL `git worktree add` helper — wire into **user `settings.json`** as a `WorktreeCreate` hook for a non-git VCS (NOT a valid plugin-manifest event). Built-in `isolation: "worktree"` covers plain git. Also applies the same sparse-checkout + `.gitignore` bootstrap as `worktree-hygiene`. |
| `worktree-remove` | OPTIONAL `git worktree remove` cleanup companion for the above (user `settings.json` `WorktreeRemove`) |
| `run-summary` | `<run-dir> [--intent] [--host claude|codex|opencode|dsh] [--checks N/M] [--secs] [--plugin]` — a live run's measured record: `summary.json`, `summary.md` and a `docs/evidence/README.md` row from the session's `result` lines and `gates.jsonl` (cost, wakes, plugin reads, blocks); on dsh, which prints no cost, every call its plugin logged is priced from `_prices.py` (`cost_by_model`; an unpriced model leaves the cost unmeasured); called by `e2e-live.sh` |
| `run-summary` | `<run-dir> [--intent] [--host claude|codex|opencode|dsh] [--checks N/M] [--secs] [--plugin]` — a live run's measured record: `summary.json`, `summary.md` and a `docs/evidence/README.md` row from the session's `result` lines and `gates.jsonl` (cost, wakes, plugin reads, blocks); on dsh, which prints no cost, every call its plugin logged is priced from `_prices.py` (`cost_by_model`, the row's cost to four places, `<$0.0001` below; an unpriced model leaves the cost unmeasured); called by `e2e-live.sh` |
| `log-execution` | `wave-K activity SN-NN transition [detail…] [execution.log]` — append one transition line (default ledger: the latest sprint's); `--help` |
| `ensure-tools` | SessionStart — builds md-db + ctx-symbols into `$CLAUDE_PLUGIN_DATA/bin` on first run / version change (background), tells the model where the tools are. `--sync` build now · `--wait` · `--resolve T` real binary path |
| `md-db`, `ctx-symbols` | PATH shims (this `bin/` is on PATH in every session): exec the real binary, building it once if needed. Gates test presence with `ensure-tools --resolve`, never `command -v` |
Expand Down
2 changes: 1 addition & 1 deletion plugin/bin/_gatelib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ validate_comms() {
local block; block="$(awk -v s="$last" 'NR>=s' "$out")"
# Here-strings, never `printf "$block" | grep -q`: grep -q exits at its match, the printf still
# writing the rest dies of SIGPIPE (141), and pipefail read a well-formed block as missing a
# section (conformance §6.6 flaked on "missing '### Next'"; a block past what a pipe holds always did).
# section (conformance §6.6 flaked on "missing '### Next'"; a block past what a pipe holds almost always did).
if [ -n "${AGENT_ROLE:-}" ]; then
grep -q "$AGENT_ROLE" <<<"${block%%$'\n'*}" \
|| fail "the latest output.md block is not from '$AGENT_ROLE' — append YOUR report block for this dispatch (one block per attempt, append-only). How: $how"
Expand Down
7 changes: 4 additions & 3 deletions plugin/bin/_prices.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
"""_prices.py — published per-token prices, for a host whose CLI prints no cost (ROADMAP §4.9).

Claude Code reports its own cost (`total_cost_usd` on its `result` line). DeepSeek Harness (dsh)
reports none, but the aloop dsh plugin logs every call's exact usage per role (_usage rows), so
run-summary prices each call here. A model with no row is UNPRICED: its tokens are reported and
its cost is None, never a guess.
reports none, but the aloop dsh plugin logs each completed call's exact usage per role (_usage
rows; dsh's compaction and session-title calls included, a retried attempt not: ROADMAP §4.9),
so run-summary prices each call here. A model with no row is UNPRICED: its tokens are reported
and its cost is None, never a guess.

DeepSeek, USD per 1M tokens, from https://api-docs.deepseek.com/quick_start/pricing (read 2026-10-20):

Expand Down
17 changes: 13 additions & 4 deletions plugin/bin/_usage.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@
reasoning). One row per increase of the running total.
OpenCode the aloop OpenCode plugin writes Claude-shaped usage records
(.agentic/host/opencode/…), so they read as Claude.
dsh the aloop DeepSeek Harness plugin does the same (.aladdin/host/dsh/…).
dsh the aloop DeepSeek Harness plugin does the same (.aladdin/host/dsh/…); a call
dsh makes outside the conversation (its compaction summary, the session
title) carries `aux`: charged like any call, never the context in use.

Row: {ts, id, model, in, cache_read, cache_write, out} (in = uncached input).
"""
Expand Down Expand Up @@ -64,7 +66,8 @@ def rows(path):
"in": u.get("input_tokens") or 0, "cache_read": u.get("cache_read_input_tokens") or 0,
"cache_write": u.get("cache_creation_input_tokens") or 0, "out": u.get("output_tokens") or 0,
"_unreported": [k for k, f in (("cache_read", "cache_read_input_tokens"),
("cache_write", "cache_creation_input_tokens")) if f not in u]}
("cache_write", "cache_creation_input_tokens")) if f not in u],
"_aux": bool(rec.get("aux"))}
continue
p = rec.get("payload") if isinstance(rec.get("payload"), dict) else None
if not p:
Expand Down Expand Up @@ -100,7 +103,7 @@ def classes(path):
rs = rows(path)
if not rs:
return None
last = rs[-1]
last = _latest(rs)
# every count is read from the host's own log (exact); a class the host never reports is
# named, never estimated or silently zero (ROADMAP §4.9 usage reporting, conformance 12)
return {"in": sum(r["in"] for r in rs), "cache_read": sum(r["cache_read"] for r in rs),
Expand All @@ -117,12 +120,18 @@ def charge(c, meter="fresh"):
return c["context"] + c["out"] if meter == "context" else c["in"] + c["out"]


def _latest(rs):
"""The latest call of the conversation itself: an aux call (dsh's title, a compaction summary) sends
another prompt, so the context in use stays the last real call's (an aux-only log has only those)."""
return next((r for r in reversed(rs) if not r.get("_aux")), rs[-1])


def context_plus_output(path):
"""Budget metric: context in use on the latest call + every output token so far."""
rs = rows(path)
if not rs:
return None
last = rs[-1]
last = _latest(rs)
ctx = last.get("_ctx") or (last["in"] + last["cache_read"] + last["cache_write"])
return ctx + sum(r["out"] for r in rs)

Expand Down
21 changes: 15 additions & 6 deletions plugin/bin/run-summary
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,17 @@ row for docs/evidence/README.md. Numbers come from the harness, never from the m
cost_usd / supervisor_turns the LAST `result` line (a resumed session reports again);
Codex and OpenCode print none, so the cost is null and the row
says — (unmeasured), never $0.00
cost_usd / cost_by_model dsh prints none either, but its plugin logs every call's exact
(--host dsh) usage (<repo>/.aladdin/host/dsh: the supervisor and each role), so
each call is priced from the dated table in _prices.py; a model it
has no price for keeps its tokens with usd null, and then the run's
cost is null too (never a partial sum)
cost_usd / cost_by_model dsh prints none either, but its plugin logs each completed call's
(--host dsh) exact usage (<repo>/.aladdin/host/dsh: the supervisor and each role,
dsh's compaction and session-title calls too), so each call is priced
from the dated table in _prices.py; a model it has no price for keeps
its tokens with usd null, and then the run's cost is null too (never
a partial sum). Kept to the micro-dollar (a short DeepSeek run costs a
fraction of a cent), shown to four places, <$0.0001 below that (read
before rounding), so a measured nonzero cost never reads $0.00 or
$0.0000; scripts/test/evidence.test.py takes exactly these cells (and
Claude's $N.NN). Not in it: a retried attempt and server-side
Search/Fetch (ROADMAP §4.9)
wakes how many `result` lines: one per supervisor wake
plugin_reads sub-agent tool calls naming a plugin path, except a
`<plugin>/schemas/…` file (md-db's schema: a legitimate use,
Expand Down Expand Up @@ -133,7 +139,7 @@ def main():
version = ""
intent = arg("--intent") or ""
s = {"run_dir": run, "intent": intent, "host": host, "plugin_version": version, "checks": arg("--checks") or "",
"wall": wall(secs or 0), "cost_usd": None if cost is None else round(float(cost), 2),
"wall": wall(secs or 0), "cost_usd": None if cost is None else round(float(cost), 6 if host == "dsh" else 2),
"supervisor_turns": turns, "wakes": wakes,
"plugin_reads": reads, "gate_verdicts": verdicts, "gate_blocks": blocks,
"selfcheck_blocks": self_blocks, "run_repo_head": head,
Expand All @@ -142,6 +148,9 @@ def main():
if by is not None:
s.update(cost_by_model=by, prices=_prices.SOURCE)
cost_cell = "—" if s["cost_usd"] is None else "$%.2f" % s["cost_usd"]
if host == "dsh" and s["cost_usd"] is not None: # DeepSeek bills 20-100x below Claude: cents would read a run as free
# judged on the unrounded total: one under half a micro-dollar is stored as 0.0 but was measured
cost_cell = "<$0.0001" if 0 < cost < 0.0001 else "$%.4f" % s["cost_usd"]
label = (intent or "?") + ("" if host == "claude" else " (%s)" % host)
row = "| %s | %s | v%s | %s | %s | %s | %s | %s | %s | `%s` |" % (
os.path.basename(run), label, version, s["checks"] or "?", s["wall"], cost_cell, wakes, reads, blocks, s["reproduce"])
Expand Down
2 changes: 1 addition & 1 deletion plugin/hosts/capabilities.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"rotate_command": "/new"
},
"dsh": {
"usage": "exact; session-title and web-search calls not reported",
"usage": "exact; retried attempts and web-search requests not reported",
"compaction": "advertised",
"live_messaging": "advertised",
"lifecycle": "plugin events + aloop bind",
Expand Down
Loading
Loading
0