Repository navigation
dsh cost includes compaction and title calls; macOS CI runs the real dsh test; host tests pass with md-db installed - #118
Draft
adeelahmad wants to merge 7 commits into
Draft
adeelahmad wants to merge 7 commits into
adeelahmad wants to merge 7 commits into
Conversation
…n the first request after a compaction - The dsh plugin now logs the usage of every call dsh makes outside the agent loop (llm/stream with a `purpose`: the compaction summary, the session title) from the call's own stream, Claude-shaped and marked `aux`. Before, only assistant/message usage was logged, so the ledger, the time box and run-summary --host dsh left those billed calls out and reported the rest as the run's total (runtime: 42 priced of 44 served). _usage keeps an aux call out of the context in use. - run-summary keeps a dsh total to 6 decimals and its row cell to 4 (<$0.0001 below), so a measured $0.0014 run no longer reads cost_usd 0.0 / $0.00. - The SessionStart(compact) context now joins the first request after a compaction (appended in agent/request before dsh builds it), the overflow retry included; an inbox inject reached the model one step late. Tests (each failing before): dsh-runtime (priced calls = served replies, cost_usd > 0 and the sum of cost_by_model, handoff on the retried request), dsh-plugin, budget-meter, e2e-live-hosts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AiLhAWTRtwbgMCDQNJUbea
… principles asserted; suite counts - evidence.test.py required `$N.NN` on every ledger row, so a dsh row as run-summary prints it (`$0.0231`, `<$0.0001`) could not be added without failing CI. The cost cell is now checked against exactly what run-summary prints for a measured cost on the row's host (the HOST= of its reproduce command): `$N.NN` for Claude Code, `$N.NNNN` or `<$0.0001` for dsh. An unmeasured cost is still refused (`—`), and so is a price on a Codex or OpenCode row, which `$N.NN` accepted before. New check: run-summary's own Claude and dsh rows pass, other formats do not. - run-summary chose `<$0.0001` from the rounded total, so a dsh run under half a micro-dollar (cost_usd 0.0) printed `$0.0000`; it reads the unrounded total now. Test: e2e-live-hosts (a $0.00000015 run). - dsh-plugin: the compaction handoff carries the core principles; a role's own request after its compaction gets nothing (the request is now sent); an aborted request and a session log that refuses the append both fall back to inject, once. dsh-runtime: the overflow retry carries the core principles, once. - Suite counts: dsh-plugin 48, dsh-runtime 53 (CHANGELOG, ROADMAP, OVERVIEW); evidence README, ROADMAP and the plugin READMEs name the dsh cost cell; ROADMAP §4.9 gets the open item that a Codex/OpenCode run cannot be a ledger row (no measured cost). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AiLhAWTRtwbgMCDQNJUbea
…ler's tools dir The macos job ran `make test-hosts` with no dsh installed, so dsh-runtime.test.mjs (50 checks) printed "skip" and exited 0 while the README said macOS ran the same host tests. It now installs the pinned dsh exactly as the hosts job does and sets DSH_BIN and ALOOP_REQUIRE_DSH=1. readme.test.py checks that every job running the host tests has that same install step, with the pin the test names, before `make test-hosts` (on the old ci.yml: "CI job 'macos' runs the host tests without installing dsh first"). The conformance harness passed CLAUDE_PLUGIN_DATA and XDG_DATA_HOME through, and ensure-tools looks there before PATH. With a passing ctx-symbols under XDG_DATA_HOME, HEAD scored 72/82. The harness now drops both, for the gates and for the focused suites it runs: 82/82 with the same decoy as XDG_DATA_HOME and as CLAUDE_PLUGIN_DATA. The 6.6 long-report case now has long notes after `### Next`. With only the Next check back in its old `printf | grep -q` form, 6.6 used to pass (81/82; only the 6.3 regressed case caught it) and now fails (80/82). The CHANGELOG, ROADMAP and the 6.3 comment now say what the old gates did. With e2ed5aa's gate hunks reverted, every report longer than a pipe holds was blocked at the role line's `head -1` ("not from '<role>'"), never passed. A regressed RED got through only now and then, on a report short enough to fit in a pipe. Three focused suites (lsp-adapter, plan-render, semantic-targets) still read XDG_DATA_HOME when run directly. This is now a ROADMAP 4.9 open item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AiLhAWTRtwbgMCDQNJUbea
…r counts five jobs
Check 7 only compared the install step's text, so each of these left it
green while the macos dsh runtime test went back to "skip": an `if:` or
`continue-on-error` on both jobs' install steps, `continue-on-error`,
`if:` or `|| true` on the host tests step, a `shell:` or `env:` on the
install step, DSH_BIN cleared or ALOOP_REQUIRE_DSH set to 0 in the
workflow, job or step env or by a later $GITHUB_ENV write, a job-level
`if:` or `continue-on-error`, and the pin moved to rc.20 (substring
match). Now the install step has only name and run, and its run is
exactly the npm install of the pin (a whole token), DSH_BIN to that
install's lib/bin.js and ALOOP_REQUIRE_DSH=1; the tests step is a bare
`run: make test-hosts`; nothing else touches DSH_BIN or sets
ALOOP_REQUIRE_DSH but to 1. Each of 22 ci.yml mutations passed check 7
before (21 of 22) and fails it now; an extra ALOOP_REQUIRE_DSH: "1"
still passes.
New check 8: the ci.yml header said "Four independent jobs" over five
and pointed to a README section ("Continuous integration") that does not
exist. It fails on the old header and passes on the fixed one.
The CHANGELOG headline said `make conformance` no longer depends on the
caller's tools. It only drops the tools dir: an `exit 0` ctx-symbols
first on PATH still gives 74/82 (a current 0.12.0 one 82/82). The
headline and harness comment now say so, and the rest is a ROADMAP 4.9
open item (filtering PATH would also drop cargo, git or python).
The old role-line check blocked almost every report past a pipe's size,
not every one (bash 5.2: 299 of 300 runs at 67 KB, 300 of 300 at
200 KB); CHANGELOG and ROADMAP say "almost every".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AiLhAWTRtwbgMCDQNJUbea
readme check 7 now also refuses a quoted key (YAML reads "if": as if), a defaults: block at workflow or job level, and any env on the host tests step but BASH32 (MAKEFLAGS=-n ran nothing). Check 8 looks for each job's bullet inside the README section the ci.yml header names. Each of these seven edits passed the check before. evidence check 3 now reads a <$0.0001 figure in the README or on the site, so a public dsh figure in that form needs its ledger row too, and a row's HOST= is read as a whole token (HOST=dsh-x is not dsh). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AiLhAWTRtwbgMCDQNJUbea
…db installed Its stage-2 fixture had no Sprint goal, Pre-flight or scope sections, so gate-stage2-complete blocked it wherever md-db resolves (a user's machine with the bundled tools) and passed only on the fallback CI's host jobs run. The fixture now passes planning-artifacts.kdl; the test passes both ways. ROADMAP gains the gap: no CI job runs the host suites with md-db on PATH. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AiLhAWTRtwbgMCDQNJUbea
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AiLhAWTRtwbgMCDQNJUbea
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Before: #104's review left these gaps (4 of its 11 deferred findings), and one more turned up while combining them with master:
cost_usd: 0.0and shown as$0.00, the cell for nothing measured.dsh-runtime.test.mjs: with no dsh installed it printed "skip" and exited 0.make conformancetook a ctx-symbols from the caller'sCLAUDE_PLUGIN_DATA/XDG_DATA_HOME.make test-hostsfailed on any machine with md-db installed.supply-chain.test.py(from GREEN's dependency check reads only the plan text the human's grant pins (M0.9, M10.2) #113) rangate-stage2-completeon planning docs md-db's schema rejects, so it passed only on CI's fallback.After: each is fixed, each with a test that fails on the old code. The other 7 deferred findings (payload robustness and
gate-final) are still in re-review and will come in their own PR.How:
hosts/dsh/aloop.mjs,_usage.py): every call dsh makes outside the agent loop (llm/streamwith apurpose) is logged Claude-shaped and markedaux. An aux call is charged like any call, but it never counts as the context in use.run-summary): for--host dsh, the total keeps 6 decimals and the row cell keeps 4. A cost below that prints<$0.0001, judged on the unrounded total.evidence.test.pynow accepts exactly the cellsrun-summaryprints for a measured cost on the row'sHOST=, read as a whole token.—, or a price on a Codex or OpenCode row.<$0.0001must also be a ledger row.agent/requestbefore dsh builds the request, and only once.ALOOP_REQUIRE_DSH=1, as the hosts job does.readme.test.pycheck 7 fails on the plain edits that skip, forgive or off-pin that step or the host tests:if:,continue-on-error,defaults:,shell:,|| true, quoted keys, an env other thanBASH32, a clearedDSH_BIN, orrc.20forrc.2. Check 8 holds the ci.yml header's job count and its README section.planning-artifacts.kdl. New ROADMAP item: no CI job runs the host suites with md-db on PATH.Review
Two adversarial verifiers per fix, two rounds. Round 1 found real problems, which were fixed with tests. In round 2 both verifiers judged both fixes sound. Their remaining nits are fixed here (553845d). Each new check in that commit fails a mutation that passed before:
HOST=dsh-xandHOST=dsh.xrows;<$0.0001README and site figures.One known misstatement: a9992fa's message says "22 mutations (21 of 22)". The real figures are 21 edits, 20 of which passed before and all 21 fail now. It stays as is to avoid a rewrite.
Evidence
Local on 9ac526e, with
DSH_BINandALOOP_REQUIRE_DSH=1:make test-hostspasses with the bundled md-db and ctx-symbols on PATH, and again with neither. On master 566d806 it fails with md-db present (supply-chain: "md-db rejected …/sprint1"). The bisect points to GREEN's dependency check reads only the plan text the human's grant pins (M0.9, M10.2) #113.make shellcheck pycheck json eval-validate version-check: pass.make conformance: 82/82, in fallback mode as on CI.Suite counts:
dsh-runtime53 (was 50);dsh-plugin48 (was 46);evidence5 (was 4);e2e-live-hosts17;readme8 (was 7);supply-chain8;opencode-plugin40.Not measured: a live DeepSeek run still needs a
DEEPSEEK_API_KEY.Generated by Claude Code