8000 Add Pi as a CLI Base Agent (acting agent and pi Evolver) by veoery · Pull Request #89 · Infini-AI-Lab/hermit · GitHub
[go: up one dir, main page]

Skip to content

Add Pi as a CLI Base Agent (acting agent and pi Evolver) - #89

Merged
haizhongzheng merged 7 commits into
Infini-AI-Lab:mainfrom
veoery:feat/pi
Nov 9, 2026
Merged

haizhongzheng merged 7 commits into
Infini-AI-Lab:mainfrom
veoery:feat/pi

Conversation

@veoery
@veoery veoery commented Nov 8, 2026 •
Copy link
Copy Markdown
Contributor

Change

Adds Pi (pi, v1.5.0) as a third CLI Base Agent beside codex and claude, as one CliAgent subclass. It can act (agent.base: pi) and learn (the new pi Evolver).

  • Layer 1: a <backend>/<model> reference becomes Pi's models.json provider entry (Binding.pi_provider): the key written as $VARIABLE (never the value), the backend's price, and the first dialect the backend serves (chat completions, then Messages, then Responses).
  • Layer 2: Pi(CliAgent) runs pi --mode json in a private home (PI_CODING_AGENT_DIR, HOME), offline.
    • The call's tools are an MCP server with exposure: direct; Pi's default would hide them behind its Code Mode.
    • Resume uses the session file. Usage and the trajectory are read from Pi's events (records.parse_pi, trajectory.pi_steps).
    • Roles, web search and container isolation are refused with Unsupported.
    • Failed sessions: Pi's JSON mode exits 0 however a session ended, so Pi.outcome() reports failed (main's mechanism since Claude Code subscriptions as a shared model backend for acting and learning (#70) #77: exit 0 without a result) for a session that did not settle, aborted, or whose last request failed, was aborted or was cut at its output limit. No change to the shared runner's interface.
  • Layer 3:
    • The acting agent deploys with hosts/pi.py and passes --approve, without which Pi skips the workspace's .pi/ tools and skills.
    • AgentEvolver, the loader and the config accept pi. The new methods/pi/ Evolver copies no login (home: []).
  • hermit deploy --host pi: the adapter's MCP server is now declared exposure: direct. This is a visible change for existing deploy users.
  • Login: a model reference only. Pi's own logins (~/.pi/agent/auth.json) are left for later.
  • Core change: hermit/isolate.py, one word: agent_home.mkdir(parents=True). The task sandbox could not create Pi's two-level user folder (.pi/agent), so every Pi acting call crashed. Under the sandbox checklist this needs core-team agreement (Please confirm here @haizhongzheng )
  • Docs: agent-layers.md, model-backends.md, config-fields.md, task.md, evolver.md, methods/agent.md, custom-evolver.md, compatibility.md, installation.md, CHANGELOG.

After review

Four commits were added on review (details in the review comment):

  • 2361862 merges main (Claude Code subscriptions as a shared model backend for acting and learning (#70) #77). Pi moves onto main's outcome() instead of its own exit_code hook, and takes its home from last_file's directory instead of a home keyword on command(). Pi refuses login and isolation: container.
  • 47fa33c review fixes:
    • a bound model's cost now follows its price (an Anthropic-bound Pi recorded $0);
    • cut, aborted or unsettled sessions are failed;
    • the MCP servers' timeout is 1800 s / 120 s instead of Pi's 60 s;
    • a compaction's request usage is counted;
    • an inherited PI_CODING_AGENT_SESSION_DIR is emptied;
    • RRSI reads a nested Pi log;
    • docs fixes.
  • 68b42b2 tool server fix (also affects Codex and Claude Code): in a task-sandboxed acting run, a deployed workspace's stdio tool server crashed at startup (session.adapter is None in a sandboxed Session), so adapter tools never loaded. It now reads session.manifest.
  • 1f207fd image_input: image_input: true (in agent.pi or evolver.pi) declares a bound model that takes images (Pi's default for an unknown model is text only).

The table below is from before these commits.

Validation

Run on Linux with bubblewrap 0.9.0, Python 3.12 and Pi 1.5.0 (release binary, checked against SHA256SUMS). No real provider was used: a local mock chat-completions server stands in for the model.

Check Result
python -m pytest --integration, pristine origin/main (6b258fb) 3 failed, 940 passed, 57 skipped
python -m pytest --integration, this branch (with 6b258fb merged) 3 failed, 952 passed, 57 skipped
New tests/core/test_pi_agent.py (11 tests, about 5 s) All pass, including 2 integration tests that run the real pi, unsandboxed and in bubblewrap
Codex and Claude unchanged (scratch before/after script, 79 cases) Byte-identical, apart from Pi's additions to CLI_HOMES, STREAMS and the unknown-agent message
hermit train + hermit eval, sandbox enforced, Pi acting and learning (docs' toy task) Test split: untrained 0.0, trained 1.0
  • Failures: the same 3 on both trees, in examples/clbench_database (golden parity), which already fail on main.
  • The +12 passes: the new Pi tests, plus Pi added to test_an_installed_cli_starts_with_only_its_own_paths.
  • Fixtures (tests/fixtures/pi/) are real Pi output; only two paths are changed, as their README says.
  • Not verified: reaching a real provider through the sandbox's allowlist proxy. Pi honoring HTTPS_PROXY is confirmed from its source (an EnvHttpProxyAgent set up in main.ts), not from a run.

Review

  • Tests and documentation cover the affected behavior: see Validation and the docs list.
  • Commands, defaults, and public interfaces are consistent. New: agent.base: pi, evolver.base: pi, a pi: settings section, the provider setting. One visible change: hermit deploy --host pi now writes exposure: direct.
  • Shared logs and examples contain no credentials or private data. The fixtures were checked; models.json holds the key's variable name, never its value.
  • Documentation-only: N/A.
  • Core changes: shared Layer 2 (Pi's outage wording added to UNAVAILABLE), isolate.py (mkdir(parents=True); Pi refuses login) and login.evolver_problems (the same for the pi Evolver).
  • Failures reported accurately: a failed request, an abort, and refused roles, web search or container are covered by tests. A provider outage sets unavailable, so the acting agent retries.
  • Real entry points: make("pi").run, the acting agent's run, hermit train and hermit eval. The stand-ins are only at the edges (the CLI, the model).
  • Test checklist:
    • Tests are in tests/core/; the fixtures are registered in tests/selection.py (CORE) and tests/README.md.
    • The stand-ins are built from recorded Pi 1.5.0 output.
    • The tests use no network, no sign-in, and no installed CLI outside integration tests.
    • test_selection.py's BUILT_IN gains pi.
  • Sandbox checklist, reuse: Pi runs through CliAgent inside the task and evolver sandboxes. All of its variables come from home_env(). Run records:
    rollout-1.json / learner-1.json (excerpt)
  rollout-1: role=rollout enforced=True network=isolated probe=True forwarded=[<model>]
     env: HA_ADAPTER_SANDBOX_ENFORCE, HA_RESULT_FD, HOME, LANG, MOCK_KEY, PATH, PI_CODING_AGENT_DIR, PYTHONPATH
  learner-1: role=learner enforced=True network=isolated probe=True forwarded=[<model>, <tool server>]
     env: HOME, LANG, MOCK_KEY, PATH, PI_CODING_AGENT_DIR, PI_OFFLINE, PI_TELEMETRY, PYTHONPATH
     program: <pi install>/pi   binds: /w/adapter, /w/material, /w/task-info, /w/history, /w/state, /w/scratch
  • Sandbox checklist, core change: isolate.py changes. Confirmed by post-pr.
  • Independent review (post-PR audit): one defect worth discussion.
    • Outage signals from pi are not fully recognized by UNAVAILABLE in cli.py. My suggestion is giving each CLI its own outage check inside its own subclass, and a new PR may fit this modification better and clearer.

veoery and others added 7 commits November 8, 2026 12:26
…-Lab#77

Infini-AI-Lab#77 (Claude subscription) reworked CliAgent's result: a CLI's `outcome()` reports `failed` for a session that
exited 0 without a result, and every caller reads it. Pi now implements `outcome()` (a last request that failed or
was aborted is `failed`) instead of the shared `exit_code` hook this branch had added, and takes its home from
`last_file`'s directory as Claude Code does, so the shared runner's `command()` keeps main's signature.

Pi takes no `login` (acting agent, isolate check, the Evolver's sign-in check); `isolation: container`, now Codex's
and Claude Code's, stays refused for Pi.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sume, RRSI, docs)

- A bound model's cost follows its price: Pi's own figure counted a model with no cost in models.json as free (an
  Anthropic reference recorded $0) and ignored a given `price`.
- A session that did not settle, aborted, or whose reply was cut at its output limit is `failed`, as Pi's text mode
  exits 1 for each.
- The MCP servers set Pi's per-request `timeout` (1800 s for the call's tools, 120 s for deployed ones, as Codex's
  tool_timeout_sec) instead of Pi's 60 s, which cut off Trainer tools that run rollouts.
- A compaction's summary request (`compaction_end.result.usage`) counts as a call.
- An inherited PI_CODING_AGENT_SESSION_DIR is emptied, so sessions stay in the home where a resume finds them.
- RRSI reads a log found below a trajectory with that host's reader (a nested agent-pi.jsonl gave no steps).
- Docs: the recorded cost, `evolver.base: pi` beside `evolver.model`, `pi` in config-fields, and that Pi speaks chat
  completions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inside a run (HA_ADAPTER_SANDBOX=1, every task-sandboxed acting run) the stdio tool server a deployed workspace
starts (`.codex/config.toml`, `.mcp.json`, `.pi/mcp.json`) builds a sandboxed Session, which loads no adapter in the
server's process; the server read `session.adapter.manifest` and crashed before MCP started, so the adapter's tools
never reached Codex, Claude Code or Pi. It now reads `session.manifest`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pi offers a model it does not know text only and drops a tool's image for it (a screenshot, a page read as an
image). `image_input: true` (agent.pi, evolver.pi) writes the bound model's `input: ["text", "image"]` into
models.json; the default stays text only, since a text-only model refuses images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@haizhongzheng haizhongzheng left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding Pi. We reviewed the PR (two independent reviews per round, each finding checked against the code
and, for Pi's behaviour, against Pi 1.5.0's source) and pushed four commits to this branch. All four are in the PR now.

2361862: merge main (#77) into the branch

#77 (Claude subscription) reworked how CliAgent reports a session: each CLI now implements outcome(), and
callers treat failed (exit 0 without a result) as a failure. So the merge:

  • Replaces the shared exit_code hook with Pi.outcome(). A failed Pi session now reads as exit code 0 with
    extra["failed"], not exit code 1 with cli_exit_code.
  • Has Pi.command() take its home from last_file's directory, as Claude Code does. The shared command() signature
    stays as on main, without the home keyword.
  • Makes Pi refuse login in the acting agent, the isolate.py check and the Evolver's sign-in check.
    isolation: container, which now runs Codex and Claude Code, stays refused for Pi.

47fa33c: review fixes

Each fix has a regression test that fails without it.

  • Cost. For a bound model, Pi's own cost figure was kept. Pi prices a model with no cost in models.json at
    0, so a model bound to Anthropic recorded $0, and a price given in the config was ignored. Now the cost of a
    bound model is computed from its price (Pi.adjust_usage).
  • Failed sessions. A session that did not settle, aborted, or whose reply was cut at its output limit
    (stopReason: length) was reported as done. It is now failed, matching Pi's text mode, which exits 1 for each
    of these.
  • Tool timeout. Pi's MCP servers default to a 60 s per-request timeout, which cut off Trainer tools that run
    rollouts. Both servers now set timeout, matching Codex's tool_timeout_sec: 1800 s for the call's tools and
    120 s for tools deployed with hermit deploy --host pi.
  • Compaction. The summary request's usage (compaction_end.result.usage) is now counted as a call.
  • Resume. An inherited PI_CODING_AGENT_SESSION_DIR moved sessions out of the private home, so a resume
    silently started over. It is now emptied.
  • RRSI. A nested agent-pi.jsonl was read with the Codex parser and gave 0 steps. It is now read by its host's
    reader.
  • Docs. Four corrections: how the recorded cost is computed; evolver.base: pi alongside evolver.model (without
    it the config is refused); pi in config-fields.md; and the backend guide's "the CLI agents do not [speak chat
    completions]".

68b42b2: tool server in a sandboxed run (also affects Codex and Claude Code)

In a task-sandboxed acting run, the stdio tool server a deployed workspace starts crashed at startup:
mcp_server.py read session.adapter.manifest, but a sandboxed Session has adapter = None. As a result the
adapter's tools never loaded. This predates the PR and hit Codex and Claude Code as well, but it also broke this
PR's Pi acting path, so it is fixed here (session.manifest), with an end-to-end test that starts the server
sandboxed and calls a tool.

1f207fd: image_input

Pi treats a model it does not know as text-only, so a bound vision model never received images that tools returned.
image_input: true (in agent.pi or evolver.pi) declares the model's input: ["text", "image"]. The default
stays text-only, because a text-only model rejects images.

We also updated the PR description to match: it no longer describes the exit_code hook and the home keyword,
and it has a short "After review" section.

CI passes on 1f207fd, the head with all four commits.

Noted, no change in this PR

  • Lost usage. A compaction that fails loses its usage, because Pi reports none for it.
  • Same on main. These also happen with Codex and Claude, so they are not specific to this PR:
    • the acting agent's retry loop keeps only the last attempt's usage;
    • extra_body is not passed to the CLI.
  • Rare cases. These need unusual inputs or setups:
    • a prompt starting with @;
    • system instructions longer than 120 KB in a sandbox;
    • session ids containing _ or .;
    • a workspace .pi/settings.json that sets sessionDir.

🤖 Generated with Claude Code

@haizhongzheng
haizhongzheng merged commit d5f0a64 into Infini-AI-Lab:main Nov 9, 2026
1 check passed
iamziyuzhao added a commit that referenced this pull request Nov 9, 2026
…Evolver)

DeepSeek Harness (`dsh`, npm @deepseek-ai/dsh 0.2.2-rc.1) beside codex, claude and pi, in the
same three layers as Pi (#89), on the runner's outcome() of #77.

- Layer 1: a <backend>/<model> reference becomes a route of dsh's pi-ai adapter
  (Binding.dsh_provider): the backend's URL in the first dialect it serves, the key by its
  variable's name (apiKeyEnv), the context window. dsh reports no cost: the backend's price
  applies.
- Layer 2: Dsh(CliAgent) runs `dsh --profile headless --json` in a private DSH_HOME, HOME
  and DSH_AGENTS_HOME, telemetry off, the caller's own DSH_* variables dropped. The
  session's settings are a patch layer of dsh's own rows (--patch): the bound model as the
  default model (effort as its reasoningEffort, image_input as Pi's), the call's tools as an
  MCP server offered directly (1800 s per call), web search and fetch off unless asked for,
  no title request, session log, plugin list, sub-agents or workflows. A server that needs
  no key gets the placeholder key through HERMIT_DSH_NO_KEY (a route that names none is
  refused by dsh). Approvals are never asked (--approval-policy never); permission_mode
  picks dsh's own sandbox. Resume by --session-id when the home keeps the session; the
  workspace's .dsh/skills are also the session's own (dsh reads project skills only at the
  nearest folder with a .git). outcome(): a run without a completed `final` has failed,
  whatever dsh's exit code (SIGTERM exits 0). Usage from step_end events; the trajectory
  from the stream, a command's exit code from dsh's end markers (a timeout or a stop:
  unknown; "(no output)": empty); every unfinished end is an error step. dsh's outage codes
  (RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, EMPTY_RESPONSE, STREAM_CLOSED) join UNAVAILABLE.
  Roles, a login and container isolation are refused.
- Layer 3: hermit/hosts/dsh.py (`hermit deploy --host dsh`): AGENTS.md, .dsh/skills and
  the adapter's tool server (120 s per call) as an insert row of .dsh/hermit.patch.yml,
  which the acting agent passes with --patch. The AgentEvolver, the loader and the config
  accept dsh; methods/dsh/ is the new Evolver (dsh's own sandbox off in the evolver
  sandbox, as Codex's is).
- Tests: tests/core/test_dsh_agent.py over streams recorded from dsh 0.2.2-rc.1
  (tests/fixtures/dsh) and, marked integration, the installed dsh: a session unsandboxed
  and in bubblewrap, the acting agent inside a git repository (the deployed skill offered,
  the adapter's own tool server answering), a keyless server, and --dump-config holding
  every row the session patches; the deploy host; selection entries.

Checked against dsh's source at the tag dsh-v0.2.2-rc.1 and with the installed dsh. First
built on #89's earlier head; rebuilt on main after #89 merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
haizhongzheng pushed a commit that referenced this pull request Nov 11, 2026
…Evolver) (#90)

* Add DeepSeek Harness (dsh) as a CLI Base Agent (acting agent and dsh Evolver)

DeepSeek Harness (`dsh`, npm @deepseek-ai/dsh 0.2.2-rc.1) beside codex, claude and pi, in the
same three layers as Pi (#89), on the runner's outcome() of #77.

- Layer 1: a <backend>/<model> reference becomes a route of dsh's pi-ai adapter
  (Binding.dsh_provider): the backend's URL in the first dialect it serves, the key by its
  variable's name (apiKeyEnv), the context window. dsh reports no cost: the backend's price
  applies.
- Layer 2: Dsh(CliAgent) runs `dsh --profile headless --json` in a private DSH_HOME, HOME
  and DSH_AGENTS_HOME, telemetry off, the caller's own DSH_* variables dropped. The
  session's settings are a patch layer of dsh's own rows (--patch): the bound model as the
  default model (effort as its reasoningEffort, image_input as Pi's), the call's tools as an
  MCP server offered directly (1800 s per call), web search and fetch off unless asked for,
  no title request, session log, plugin list, sub-agents or workflows. A server that needs
  no key gets the placeholder key through HERMIT_DSH_NO_KEY (a route that names none is
  refused by dsh). Approvals are never asked (--approval-policy never); permission_mode
  picks dsh's own sandbox. Resume by --session-id when the home keeps the session; the
  workspace's .dsh/skills are also the session's own (dsh reads project skills only at the
  nearest folder with a .git). outcome(): a run without a completed `final` has failed,
  whatever dsh's exit code (SIGTERM exits 0). Usage from step_end events; the trajectory
  from the stream, a command's exit code from dsh's end markers (a timeout or a stop:
  unknown; "(no output)": empty); every unfinished end is an error step. dsh's outage codes
  (RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, EMPTY_RESPONSE, STREAM_CLOSED) join UNAVAILABLE.
  Roles, a login and container isolation are refused.
- Layer 3: hermit/hosts/dsh.py (`hermit deploy --host dsh`): AGENTS.md, .dsh/skills and
  the adapter's tool server (120 s per call) as an insert row of .dsh/hermit.patch.yml,
  which the acting agent passes with --patch. The AgentEvolver, the loader and the config
  accept dsh; methods/dsh/ is the new Evolver (dsh's own sandbox off in the evolver
  sandbox, as Codex's is).
- Tests: tests/core/test_dsh_agent.py over streams recorded from dsh 0.2.2-rc.1
  (tests/fixtures/dsh) and, marked integration, the installed dsh: a session unsandboxed
  and in bubblewrap, the acting agent inside a git repository (the deployed skill offered,
  the adapter's own tool server answering), a keyless server, and --dump-config holding
  every row the session patches; the deploy host; selection entries.

Checked against dsh's source at the tag dsh-v0.2.2-rc.1 and with the installed dsh. First
built on #89's earlier head; rebuilt on main after #89 merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: fixes from review round 1 (two independent reviewers on 03fafff)

Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against
the installed dsh (new recorded streams in tests/fixtures/dsh); each has a test that fails
without it.

- Stream reading: a bash result dsh's --json cut at 8 KiB (`truncated`) lost its exit
  marker and read as exit 0; now unknown, and the text says it was cut. A command started
  in the background has no status yet (unknown). The end markers are anchored at the very
  end (\Z), as dsh's JavaScript reads them, so a command that printed a marker of its own
  and exited 0 is a clean exit.
- Usage: a step that answered counts as a call even when the provider reported no usage
  (dsh reports zeros), and so does a step dsh reports no usage for (an attempt without
  any); what dsh never reports (compaction, a step's partial usage) is documented.
  parse_log reads Pi's and dsh's streams with their own readers.
- outcome(): a session whose tools did not start (dsh writes the MCP row's error on stderr
  and runs on without them) has failed. A run that wrote only an `error` event (a usage
  error) or nothing (a working directory's .env that sets PATH, PYTHONPATH, ...) started
  no session, so the acting agent reports it instead of recording each task as a wrong
  answer: outcome() may now say `started` (shared runner; absent, as before).
- The workspace is the project wherever it lies: the session's layer sets the
  skill-filesystem row to the workspace's .dsh/skills alone and the agent-instructions row
  to the deployment's marker (.hermit-deploy.json) as the project root, so a workspace
  inside a git repository gets its own skills and none of the repository's skills or
  AGENTS.md (they outranked the adapter's). This replaces copying the skills to the home.
- `--add-dir` takes the real path of an existing directory, as dsh requires (a link failed
  every command and write); an effort without a model is refused; user_home() follows
  DSH_HOME.
- The acting agent passes agent.dsh.permission_mode on (it was dropped) and refuses a mode
  dsh has not.
- hermit deploy --host dsh: the patch layer also names the workspace's .dsh/skills (a
  workspace below a repository's root kept none), and a context past the 65,536 bytes dsh
  reads of a project's instructions is refused.
- The AHE template has a `dsh` wire (as Pi's: hooks run by hand hand the memory over); mwm
  and RRSI refuse a backend other than Codex or Claude Code at setup (pi and dsh were
  accepted and ran without their model).
- Docstrings say what dsh reads (.env files, the instruction chain) and sends (its own
  provider's ids).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: fixes from review round 2 (two independent reviewers on 7a076fb)

Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against the
installed dsh; each has a test that fails without it.

- A task's data is bound in the workspace, not linked (binds_mounts, as for Claude Code): dsh's
  glob and grep run ripgrep without --follow, so a linked folder was never listed or matched.
  dsh's own sandbox still runs nested in that mount namespace (integration test).
- Web search with a model of another backend: dsh's search is DeepSeek's own service (its
  Messages API on DEEPSEEK_API_KEY), so the session searched on a provider and key the run did
  not bind, outside its usage. Such a session gets web fetch only, as Codex's hosted search is
  off for a backend other than OpenAI's; a DeepSeek model or dsh's own provider keeps search.
- Usage: a step dsh reports no usage for counts as a call only if it produced something (a step
  whose requests all failed is none), and price_usage leaves calls of which no token was
  reported unpriced: their cost is unknown, not $0.00 (shared: a usage with calls and no tokens
  is no longer priced at 0 for any CLI).
- The instructions' project root: no root marker, so it is the working directory, the workspace.
  The deployment marker (.hermit-deploy.json, which every host's deployment writes) made a
  session without a deployment take a deployed folder above it for its root, and read its
  AGENTS.md.
- hermit deploy --host dsh: the tool server's row fails at a session's start when dsh cannot
  start it (failOnStartupError, as Codex's `required`), which the acting agent reads as a failed
  session; it was retried in the background while the session ran without the adapter's tools.
  The 65,536-byte check counts dsh's frame around AGENTS.md (304 bytes, a </system-reminder>
  one byte more): an AGENTS.md just under the cap was deployed and cut. Docstring: the command
  names its profile; the skills row replaces a skill-filesystem config of the user's own.
- dsh's effort levels and permission modes are checked as the run config loads (an unknown one
  failed every session, one task at a time), and an effort level dsh has not is refused in the
  session.
- The AHE learners' texts name a Pi or dsh acting agent's stream (agent-pi.jsonl,
  agent-dsh.jsonl) instead of agent-codex.jsonl (Pi's entry was missing on main too).
- Tests: a recorded stream of a session stopped by SIGTERM (stopped.jsonl: exit 0, no `final`,
  an `error` event) replaces a cut success stream, and is run through the acting agent and an
  AgentEvolver step; the deploy refuses at the exact limit and an MCP server name dsh refuses.

Correction to round 1's message: on 03fafff a dsh usage error was already an error record per
task (failed), not a wrong answer; `started` now makes the acting agent stop the run and say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: fixes from review round 3 (two independent reviewers on 9d87553)

Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against the
installed dsh; each has a test that fails without it.

- Usage: round 2's shared rule (a usage with calls and no tokens is not priced) is gone. It
  made Claude Code's results of refused requests (a 529, a 401: one turn, no tokens) cost
  unknown instead of $0.00, and the AHE and RRSI evolvers' merged usage then dropped the
  step's known spend from the budget. Usage gains `unreported_calls`, the calls whose tokens a
  CLI did not report; only dsh's reader sets it (a step that produced something without
  usage: the provider reported none, or dsh retried the step and so left its usage out), and
  price_usage leaves such a usage's cost unknown rather than pricing the part reported, which
  under rate limits may be far less.
- dsh's transcripts are plain JSON lines (the session layer sets session-persistence-jsonl to
  compression none, its root as dsh's own `!!js dshHomePath('sessions')`): they were zstd,
  which hermit.redact cannot scrub a key from. A scrubbed transcript still resumes.
- The dsh Evolver's sessions take no instruction file (instruction_files false: the
  agent-instructions budget 0): dsh adds a folder's AGENTS.md when its file tools touch it, so
  the learner was given the adapter's AGENTS.md, the acting agent's policy, as instructions.
  Codex's learner is given none.
- dsh's settings are checked as the run loads for the dsh Evolver too (an effort it has not
  failed every learner step, after its rollouts); evolver.dsh.permission_mode is refused (the
  Evolver runs dsh's own sandbox off), and YAML's bare `off`, a boolean, is `off`.
- Web search is kept only for a model reached at api.deepseek.com on DEEPSEEK_API_KEY, where
  dsh's search goes (a `deepseek` backend moved by DEEPSEEK_BASE_URL searched elsewhere).
- A run that ends CONTEXT_WINDOW_EXCEEDED with a backend that declares no context_window says
  to declare it (recorded: context_window.jsonl, vLLM's refusal of dsh's 128,000 output
  tokens; with the window declared dsh sizes the request and completes).
- A unit test with the fake dsh for the task data's binding, which CI runs where bubblewrap can
  bind (the only test needed the dsh CLI).
- Wording: the deployed tool server's row makes dsh report a server it cannot start and the
  acting agent fail the session (dsh itself goes on); a run dsh never started is each task's
  error record, not a stopped run (round 2's message said so); the deploy's size check is exact
  for a session at the workspace's root; the linked-data warning names dsh's glob and grep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: fixes from review round 4 (two independent reviewers on 2021944)

Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and against the installed dsh;
each has a test that fails without it.

- A step dsh retried after the provider refused a request lost its cache counts: the refused
  attempt reports zeros, pi-ai reports a count of 0 as none, and dsh sums a count into the
  step's only when every attempt reported it, while the step's totalTokens still holds them.
  The reader counts what the total holds beyond the counts reported as input, and the call as
  unreported (its cost unknown: those tokens were read from the cache or written to it, which
  are priced apart). Recorded: retried.jsonl (a 503, then 1000 prompt tokens, 900 cached:
  dsh reports inputTokens 100, totalTokens 1010). Round 3 said such a step's usage is left
  out; dsh does that only for an attempt that reports no usage at all (a stream that ended
  before its last event, an attempt a stop cut short).
- The dsh Evolver's settings are checked from its params as the Trainer has them (its method's
  own under the run's, the model bound), in login.evolver_problems, which train runs before its
  first step and hermit check now reports for an Evolver without a login. Round 3's check, as
  the run loaded, missed the dsh method named with its params (`backend: dsh` comes from the
  method's manifest) and refused an effort given with the run's own model. The acting agent's
  are still checked as the run loads; one function (cli.dsh_problems) holds both, and
  isolate.py's change is that call.
- dsh's own provider (a model of no binding, DeepSeek's API) takes the efforts off, low, high
  and max; another level failed every session (UNSUPPORTED_REASONING_EFFORT).
- evolver.dsh.instruction_files is refused, as evolver.dsh.permission_mode is: the Evolver
  dropped it.
- The caller's DEEPSEEK_SEARCH_BASE_URL is dropped with its DSH_* variables: without the
  sandbox it moved dsh's web search off DeepSeek's API.
- The context_window hint follows dsh's error whole (once), not its first 300 characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: fixes from review round 5 (two independent reviewers on f4b7b09)

Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and against the installed dsh;
each has a test that fails without it.

- A step dsh retried after a refused request is priced again, its dropped cache tokens counted
  as input. Round 4 made such a session's cost unknown, and a USD budget then counted none of
  its spend (rollouts and learner steps add no dollars for an unknown cost), under the very load
  that causes retries. Its tokens are known, and hermit prices a cache write as input: the price
  is the step's with the counts reported, or more if those tokens were read from the cache,
  never less. A call stays unreported, and its session's cost unknown, only when dsh reported
  none of its tokens.
- The caller's DEEPSEEK_BASE_URL is dropped as DEEPSEEK_SEARCH_BASE_URL is: without the sandbox
  it sent the requests of dsh's own provider, with DEEPSEEK_API_KEY, elsewhere. A variable the
  session is given (`env`) still passes.
- dsh's settings check takes what each agent uses: the Evolver's danger-full-access and
  instruction_files false are accepted (round 4 refused them); the acting agent's
  instruction_files false is refused (it was dropped; it would cut the adapter's context, which
  reaches dsh as its AGENTS.md).
- Tests pin the session's tool row (failOnStartupError, the 30-minute call timeout), every row
  DSH_OFF turns off, all six outage codes, and dsh's own provider's levels for the Evolver; the
  outage test is named for what it checks.
- Wording: effort off asks a bound model for no reasoning level, which leaves its server's
  default (dsh's own provider turns thinking off); the context-window hint keeps dsh's error
  line as far as it fits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: fixes from review round 6 (two independent reviewers on f1a92ea)

Each finding was checked against dsh's source at dsh-v0.2.2-rc.1 and the installed dsh; each
test added fails without what it checks. No behaviour changes: the fixes are tests and claims.

- Tests: a session's own `env` reaches dsh while the caller's DEEPSEEK_BASE_URL is dropped; the
  integration test of dsh's own sandbox shows a command's write outside the workspace refused
  (it passed with the sandbox off); a bash call that failed to run (a result dsh marks `error`)
  has no exit code; a linked session folder, or a session in a linked folder, is not resumed;
  a stopped session's usage is pinned (none).
- The reader's account: the last step of a session a stop cut short is not counted (dsh writes
  a step's usage at its end, after its tool calls ran, and a stopped session writes nothing
  more); the docstring said such a step was a call of unknown tokens. dsh's own adapter, not
  pi-ai, drops a cache count of 0. A retried step's price is never less than with its counts
  for a price whose cached input is no dearer than its input; on the Responses API the total
  is the server's own `total_tokens`, taken as given.
- dropped_env says why a sandbox's grant of DEEPSEEK_BASE_URL does not reach dsh (hermit's own
  is its deepseek backend's chat completions root, which dsh reads as its own provider's
  Messages root) and that `env` gives dsh its own; the Settings paragraph says so too.
- A stale test docstring: the Evolver's own permission mode and instruction_files are taken.

Not changed, disclosed: the acting agent's re-run after an outage keeps only its last
attempt's usage, for every CLI (shared code, a separate change).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: leave the Pi fixes to their own PRs

The Pi fixes this branch carried change Pi's behaviour on main, not dsh's: parse_log's Pi
reader, the AHE Evolver's texts for a Pi acting agent's stream, and mwm and RRSI refusing a Pi
learner. They go to their own PRs (fix/pi-stream-readers, fix/mwm-rrsi-backend-check). This
branch keeps dsh's entries only: parse_log reads a dsh stream as dsh's, the AHE texts name a
dsh acting agent's stream, and mwm and RRSI refuse dsh (the other PR's general check covers it
too, whichever merges first).

The integration test of dsh's own sandbox keeps the session's home under
~/.cache/hermit-tests, as the test checklist asks of a check under bubblewrap (it was under
/var/tmp).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: test which steps of dsh's stream count as calls

Review round 7 (five reviewers on 37aee8c, one lens each, every finding then reproduced by a
separate agent told to refute it) found 34 issues, all confirmed: one of medium severity, the
rest low. The medium one: no test guarded the reader's three rules for a step whose tokens dsh
did not report (a step that only called a tool is a call, so is one that only thought, and what
a step produced does not carry over to the next); each, undone, left every test passing while
it misstated a session's cost. The new test fails under each. The low ones are listed in the
PR description, not changed.

Also: a round-6 test docstring rewrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: test a command dsh moves to the background at its timeout

In a hermit dsh session a command still running at its timeout (dsh's default is 60 s) is
moved to a background job, unless dsh's job limit is reached: the session's layer leaves
tool-bash's promoteOnTimeout at its default, true. Its result ends `[still running after
<ms>ms; moved to background job <id>]` with no exit marker, and only the DSH_BACKGROUND check
keeps it from reading as exit 0. No test held that case: bash.jsonl was recorded with
promotion off. Review round 7 found this (rated medium by its reviewer, low by the agent that
reproduced it, as the code was right); the author asked for it fixed.

promoted.jsonl is recorded through make("dsh").run, as the other Base Agent streams are, and
the test now reads it, and also a command stopped from outside the call (`[stopped: <reason>]`,
which no headless session writes, so written as dsh's tool-bash renders it), with and without
a marker after it. Each check undone (the background check, the `stopped:` alternative, the
markers after a stop) fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: read a timeout marker at any timeout dsh prints (review round 8)

Review round 8 (two reviewers on f4792e0, one on the test and its fixture, one on the claims,
each finding then reproduced by a separate agent told to refute it) found:

- dsh prints a command's timeout as the model gave it, a JavaScript number (`1000.5`, `0.5`),
  and both timeout markers were matched only at a whole number of milliseconds: a command still
  running at such a timeout, moved to a background job, read as exit 0, and one killed at it that
  trapped the signal read as its trapped exit (rated medium). Both markers now match any number
  dsh prints. Recorded: promoted.jsonl's third call (1000.5 ms, moved to the background) and
  bash.jsonl's ninth (1000.5 ms, killed, exit 0 trapped).
- The test of a promoted command held only one with output before its marker; the marker alone,
  what a command that printed nothing yet gets, rests on DSH_BACKGROUND matching without a newline
  before it, which nothing held (rated medium). promoted.jsonl's second call is `sleep 5`.
- Wording (low, in f4792e0's own docstring): the stop marker comes before the signal or exit
  marker, not before every other marker (a sandbox denial's come first).

Each fix undone fails its test: the marker with a newline required before it, and either marker
at whole milliseconds only. bash.jsonl is recorded again by the same script, its first eight
results unchanged; its steps' count in the usage test follows (10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: wording from review round 9

Review round 9 (two reviewers on d25958a, one on the code and its tests, one on the claims, each
finding then reproduced by a separate agent told to refute it) found 6 issues (4 distinct), all
low. Three are wording this change's own text got wrong, corrected here (comments, a docstring
and the fixture README; no code or test changes):

- dsh prints the timeout it applied, not always the model's: its default (60 s) when the model
  gives none, and at most its cap (600 s).
- DSH_NO_OUTPUT's comment said `(no output)` comes before the marker of a command that printed
  nothing, which a command moved to the background does not get (its marker stands alone).

The fourth, the exponent form both timeout patterns match (`1e-7`, a timeout under 0.000001 ms)
held by no test, is listed in the PR description, not changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: an outage only by the run's own end; the web-search test on the built-ins alone (post-PR audit)

- Dsh.outcome read `unavailable` from every line of the stream that is no event, and dsh starts a deployed stdio
  tool server on its own stderr, which the stream keeps: outside the task sandbox, a tool's line with an outage's
  words (`Service Unavailable`) made a session that failed for another reason (`max-tokens`) an outage, run
  `retries` times and then an error of unknown cost. It is now read from the `final` event's error code
  (DSH_OUTAGES, Dsh.outage), and UNAVAILABLE is main's again.
- test_dsh_refuses_roles_and_a_container_and_searches_only_on_deepseeks_own_provider bound its DeepSeek models
  through the developer's own backends (HA_BACKENDS or ~/.config/hermit/backends.yaml, DEEPSEEK_BASE_URL,
  DS_BASE_URL), so it failed wherever those move `deepseek`: it binds them on the built-ins alone.
- The fixture README: tools_down.jsonl's port is one Node's fetch refuses, and the Base Agent's stream has that
  run's stderr lines first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: an overload dsh leaves unclassified is an outage by its words (review round 10)

- cacb4ae read an outage from the final event's code only, which narrowed a43698d's rule: a provider overloaded
  inside a 200 stream (the Messages API's `overloaded_error` event, a chat server's error chunk) has no status, and
  dsh ends the run with pi-ai's unclassified PI_AI_ERROR, which it does not retry; the task was then not run again.
  Such an end is now read by its words (UNAVAILABLE, as Pi's are), and only dsh's own end is read so, never a tool's
  stderr. Recorded through the Base Agent: overloaded.jsonl.
- A test docstring described parse_log as main had it before #49.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: guards and wording from review round 11 (post-PR audit of ace8fab)

- A kept home's patch layer is written as a file of its own, never through a link
  or into a pipe the last session left (the dsh Evolver's agent_home): the guard
  had no test.
- An end dsh leaves unclassified (PI_AI_ERROR) that is no outage, with a tool's
  outage words on dsh's stderr, runs once: ace8fab's new path had no test that
  its words come from the end alone.
- PI_AI_ERROR is the code of dsh's pi-ai adapter, not pi-ai's; outcome() and
  errors() no longer say the end's code alone tells an outage.
- test_deploy.py: the dsh test is named for what it holds, and its comments say
  what dsh does (a server that does not start is reported; the dsh acting agent
  fails the session; a project's root may be the working directory itself), as
  hermit/hosts/dsh.py now does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: documentation (the pages, the changelog, the AHE template's notes)

DeepSeek Harness joins the pages that name the CLI Base Agents, their settings, deployment, models and installation, in
the wording of Pi's and mini-swe-agent's entries; what is its own: its inner sandbox (permission_mode, and what it does
to environment.write and /tmp), its web search, its patch layer and how a session loads it, a tool server that does not
start, and the usage it leaves out. No code changed. convert's lists are untouched: dsh has no plugin format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: documentation, review round 13's corrections

Round 13 read f0d2091: 16 findings, all low. dsh's commands each have a /tmp of their own and its default sandbox refuses
a declared writable path (the sandbox guide); how to write a provider of one's own (model-backends); what a web fetch
reaches; which price the cost follows; what Landlock, a patch layer and the request left out of the usage are. No code
changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: documentation, two corrections from the read-back

The sandbox guide's two ways to reach a path under /tmp are for reading it (a mount is read-only); the requests that
summarize a session's history are uncounted each time, not once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* dsh: deploy retires stale skills as every host does since #97

hermit deploy --host dsh now calls check_skills before it writes
anything (a redeployment refuses to replace a skill folder no
deployment wrote, or to write through a linked skills folder) and
retire_skills after it copies the skills (a skill the last deployment
exported and this one does not moves to .retired-skills/). dsh copies
its skills before it writes its patch layer, whose skill-filesystem
row names them, so retire_skills follows copy_skills there. #97's
redeployment test gains the dsh case, which fails without the
retire_skills call; no test covers the check_skills call, as on main
for every host but Codex. #97's CHANGELOG entry names .dsh/skills.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
0