8000 Name a Pi or mini-swe-agent acting agent's event stream in the AHE Evolver's texts by iamziyuzhao · Pull Request #91 · Infini-AI-Lab/hermit · GitHub
[go: up one dir, main page]
Skip to content

Name a Pi or mini-swe-agent acting agent's event stream in the AHE Evolver's texts - #91

Merged
haizhongzheng merged 6 commits into
mainfrom
fix/pi-stream-readers
Nov 10, 2026
Merged

Name a Pi or mini-swe-agent acting agent's event stream in the AHE Evolver's texts#91
haizhongzheng merged 6 commits into
mainfrom
fix/pi-stream-readers

Conversation

@iamziyuzhao
@iamziyuzhao iamziyuzhao commented Nov 9, 2026 •
edited
Loading
Copy link
Copy Markdown
Collaborator

Change

The AHE Evolver's texts (START.md, DEBUGGER.md, the evolution query, the Debugger's question) name the acting
agent's event stream. debugger.STREAM_TEXTS had only Claude Code's wording, so with a Pi or mini-swe-agent acting
agent they named agent-codex.jsonl, which is not there (rollout_stream finds agent-pi.jsonl or
agent-mini-swe-agent.jsonl). They now name that agent's stream and the command that writes it (pi --mode json,
python -m hermit.agent.base.mini_swe_agent_run). The comparison table in docs/methods/ahe_adapter.md no longer says
the stream is only Codex's or Claude Code's, and claims the hook audit only for the hosts that run hooks.

records.parse_log is not changed here: #49 made it read a Pi stream as Pi's while this PR was open. The PR keeps a
test of that line, which had none.

Found while adding DeepSeek Harness (#90) and split out of it, so that #90 changes only dsh. #90 adds its own entry
after Claude Code's in STREAM_TEXTS; these come before it, so the two PRs merge in either order without a conflict.

Validation

At ae01e78 (main at f469f59 merged):

  • tests/methods/ahe/test_ahe_hosts.py::test_the_evolvers_texts_name_every_acting_agents_stream_it_reads: for every
    stream but Codex's (Claude Code's, Pi's, mini-swe-agent's), START.md, DEBUGGER.md and the evolution query name
    no Codex file, stream or command, START.md names that stream, and DEBUGGER.md names Pi's and mini-swe-agent's
    commands.
  • tests/core/test_pi_agent.py::test_a_pi_stream_is_read_back_by_its_name_or_by_its_events: parse_log on the
    recorded tests/fixtures/pi/success.jsonl with pi gives the stream's session and its 4 calls (4,000 input and 80
    output tokens).
  • Each test fails with what it covers undone, each undone in turn in a copy of the checkout: Pi's and mini-swe-agent's
    STREAM_TEXTS entries, each of their four replacements, and Pi's reader in parse_log (11 of 11).
  • Full suite, Linux container (Ubuntu 24.04) as root, --integration, split over six containers: 3 failed, 1325
    passed, 70 skipped. main at f469f59 split over five: 3 failed, 1324 passed, 70 skipped. The same three tests
    fail and the same tests skip. Two fail as root (test_isolate::test_a_rollout_sees_what_is_declared_and_nothing_else,
    test_ahe_hook_runner::test_logs_that_cannot_be_written_lose_the_line_not_the_call). The third
    (test_ahe_evolver::test_the_evolve_prompt_is_start_md_and_the_query) fails only in a split run, for lack of
    ~/.cache/hermit-tests, and passes with tests/methods/ahe in one pytest.
  • hermit check, train and eval on docs/_examples/first_training/run.yaml pass, and bash docs/build-pages.sh
    (Sphinx with -W) succeeds.
  • With Add DeepSeek Harness (dsh) as a CLI Base Agent (acting agent and dsh Evolver) #90 at 1e75c6f: main and the two PRs merge without a conflict in either order, to the same tree. That tree's
    full suite, split over six containers: 3 failed, 1374 passed, 70 skipped, the same three tests. The every-stream test
    passes there for dsh's stream too (its file name and no Codex wording; Add DeepSeek Harness (dsh) as a CLI Base Agent (acting agent and dsh Evolver) #90's own test holds its command).
  • Not run: an AHE step with a real Pi or mini-swe-agent acting agent. The test reads the texts as for_stream gives
    them to the learners, from the method's own files and build_evolution_query, not through a whole Evolver run.

Review

  • Tests and documentation cover the affected behavior, where applicable. Tests: see Validation. Documentation: the
    comparison table in docs/methods/ahe_adapter.md; no document names parse_log or STREAM_TEXTS.
  • Commands, defaults, and public interfaces are consistent. No signature changes; a stream without an entry keeps
    the Codex wording, as before.
  • Shared logs and examples contain no credentials or private data. The fixture is Add Pi as a CLI Base Agent (acting agent and pi Evolver) #89's recorded stream.
  • Documentation-only: N/A.
  • Core changes: none. The code change is the AHE method's debugger.py.
  • Implementation correctness and failures: a text substitution, which starts nothing and adds no failure path.
    The sandbox checklist does not apply: no process is started and no sandbox changes.
  • Real execution paths and tests: the AHE texts from the method's own files and build_evolution_query, not a
    whole Evolver run (see Validation). parse_log on a stream the real Pi recorded; it has no caller in a run
    (hermit.agent.evolver.agent re-exports it, and a Pi session goes through Pi.parse).
  • Test checklist: each test is in the group of its code (tests/methods/ahe/, tests/core/), named for the
    behaviour, and fails without what it covers. The added lines write no file and need no CLI or network (the Pi test's
    earlier lines copy the fixture into tmp_path). Each test takes well under a second.
  • This description: read back against the code and the run records; the audit checked its numbers at ae01e78.
  • Independent review (post-PR audit): a separate agent audited 63a781f against the general, test and sandbox
    checklists, including a Trainer step on hand-written Pi- and mini-swe-agent-shaped streams with stand-in learners,
    in which the right stream and command reached the learners. It then re-checked 0de3891 (the table row and this
    description) and ae01e78 (the entries' and the test's new places, the table row, and merging with Add DeepSeek Harness (dsh) as a CLI Base Agent (acting agent and dsh Evolver) #90 in either
    order, squash merges included): no high or medium finding in this PR's diff or description, or from merging the two.
    • High: none.
    • Medium, in two parts. The table row this PR had written claimed a hook audit for Pi and mini-swe-agent: fixed in
      0de3891. Left, on main before this PR and not changed by it: START.md describes middleware and MEMORY.md
      through session_start, and it and the evolution query name hook-calls.jsonl, without saying that neither host
      runs hooks; on mini-swe-agent the hook command its deployment lists does nothing (hook_hosts.HOSTS has no such
      host), so MEMORY.md never arrives; the template's EVOLVER.md does not mention mini-swe-agent. This PR renames a
      file in those sentences; changing what a learner is told to do on a host without hooks is left to a follow-up.
    • Low, fixed: the table row's reason for the missing timing breakdown (Pi's stream has timestamps), and three
      statements in this description.
    • Low, left: the new test holds that Codex's wording goes away and the file names, not that each replacement names
      the right agent or how rollout_stream chooses the stream (the audit's Trainer step exercised both once).
      With the follow-up above: EVOLVER.md's note on where the stream is names only Codex's and Claude Code's files,
      and docs/methods/ahe_adapter.md says each rollout's log holds hook-calls.jsonl and hook-state/. In the table
      row, "the one its CLI wrote" is loose for mini-swe-agent (the framework's own process writes that stream), and
      "one file per standard step" does not hold for a Harbor trial with only agent/codex.txt.

🤖 Generated with Claude Code

@iamziyuzhao @claude
Read a Pi acting agent's stream as Pi's in parse_log and the AHE Evol…
24bc867 
…ver's texts

records.parse_log read every stream but Codex's as Claude Code's, so a Pi stream gave no
session id and no usage; it now reads a pi stream with parse_pi. The AHE Evolver's texts named
agent-codex.jsonl for a Pi acting agent, whose stream is agent-pi.jsonl; debugger.STREAM_TEXTS
now has Pi's wording and command. Each test fails without its fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamziyuzhao iamziyuzhao mentioned this pull request Nov 9, 2026
Merged
10 of 12 tasks
iamziyuzhao and others added 3 commits November 10, 2026 00:24
@iamziyuzhao @claude
Merge main into fix/pi-stream-readers
814b540 
main's #49 already reads a Pi stream in parse_log (beside mini-swe-agent's), so its line is kept and this branch
no longer changes records.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamziyuzhao @claude
Name mini-swe-agent's event stream in the AHE Evolver's texts
4fc88da 
main added agent-mini-swe-agent.jsonl to the streams rollout_stream finds, and the learners' texts still named
agent-codex.jsonl for it: this branch's test (every stream but Codex's) fails once main is merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamziyuzhao @claude
Name Pi's and mini-swe-agent's streams in what the AHE learner sees
63a781f 
The comparison table in docs/methods/ahe_adapter.md listed only Codex's and Claude Code's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamziyuzhao iamziyuzhao changed the title Read a Pi acting agent's event stream as Pi's in parse_log and the AHE Evolver's texts Name a Pi or mini-swe-agent acting agent's event stream in the AHE Evolver's texts Nov 10, 2026
@iamziyuzhao @claude
Claim the hook audit and the missing timings only where they hold
0de3891 
Pi and mini-swe-agent run no hooks, and the timing breakdown comes from a Harbor trial's trajectory, not from
whether a stream has timestamps (Pi's has). Found by the post-PR audit of this table row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamziyuzhao
iamziyuzhao marked this pull request as ready for review November 10, 2026 06:06
@iamziyuzhao @claude
Keep this branch's lines apart from the DeepSeek Harness PR's
ae01e78 
Both PRs added to the end of STREAM_TEXTS and at one place in test_ahe_hosts.py, so whichever merged second
conflicted. These entries now come before Claude Code's and this test before its neighbour. The table row no longer
lists each stream, a list the DeepSeek Harness PR would have left short.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
haizhongzheng
haizhongzheng approved these changes Nov 10, 2026
View reviewed changes
@haizhongzheng haizhongzheng left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for splitting this out of #90. We reviewed ae01e78 with two independent reviewers, and found nothing that blocks the merge. We made no changes.

What we checked:

  • Coverage. Every Codex phrase in START.md, DEBUGGER.md (the Debugger's question) and build_evolution_query is one of the four STREAM_TEXTS patterns. So a Pi or mini-swe-agent run's texts name no Codex file, stream or command.
  • Commands. Both match the code: pi --mode json (hermit/agent/base/cli.py) and python -m hermit.agent.base.mini_swe_agent_run.
  • No change for the existing hosts. The Codex and Claude Code texts are unchanged from main, including the partial-pass and timeout queries.
  • The new test. It fails with either new entry removed.
  • The docs row. The hook claim matches hook_hosts.HOOK_HOSTS, and the timing claim matches timings_of (ATIF only).
  • Merging. The PR merges cleanly into main (f469f59), and together with #90 (1e75c6f).
  • Tests. python -m pytest --changed --base origin/main (core and ahe groups): 856 passed, 76 skipped, 0 failed.

Noted, no change (all already in the description):

  • "the one its CLI wrote" is loose for mini-swe-agent, whose stream the framework's own process writes.
  • The test checks that the Codex wording is gone and that the file is named. It does not check that each replacement names the right agent.
  • On main before this PR: for Pi and mini-swe-agent, the texts still describe middleware and hook-calls.jsonl, but neither host runs hooks. This is left for the follow-up.

🤖 Generated with Claude Code

@haizhongzheng
haizhongzheng merged commit 8a0aac2 into main Nov 10, 2026
1 check passed
@haizhongzheng
haizhongzheng deleted the fix/pi-stream-readers branch November 10, 2026 15:17
haizhongzheng added a commit to veoery/hermit that referenced this pull request Nov 10, 2026
@haizhongzheng @claude
opencode: tools from the first request, long prompts, extra_body, com…
80e04b1 
…paction usage

- The adapter's tools reach an OpenCode acting agent through the Base Agent's
  tool server: OpenCode 2.2 waits at most 2 s for an MCP server it starts
  (STARTUP_GRACE_MS) and the adapter's takes ~3 s, so the first request often
  went without them. hosts/opencode.prepare takes serve_tools; docs say so.
- The prompt is OpenCode's stdin, not an argument (E2BIG past 128 KiB).
- A binding's extra_body becomes the OpenCode model's options; the effort
  setting outranks its reasoning_effort.
- Usage counts compactions' model calls, which `run --format json` never
  streams: this call's share of the session's totals in OpenCode's database
  (read-only, immutable, nothing followed out of the home), a resumed
  session's earlier totals taken off (_this_call, now with `fields`).
- parse_log reads OpenCode's stream by its name.
- The AHE Evolver's texts name OpenCode's stream (agent-opencode.jsonl) as
  they name Pi's and mini-swe-agent's (Infini-AI-Lab#91).
- The deployer's docstring gives AGENTS.md discovery's real boundary.

Tests: the first request offers the adapter's tools (real binary); a prompt
past ARG_MAX; extra_body and effort; compaction usage from a stand-in
database and from the real binary (mid-run compaction, then a resume).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@haizhongzheng haizhongzheng mentioned this pull request Nov 10, 2026
Merged
8 of 11 tasks
haizhongzheng added a commit that referenced this pull request Nov 10, 2026
@veoery @claude @haizhongzheng
Add OpenCode as a CLI Base Agent (acting agent and opencode Evolver) (#…
1310de6 
…94)

* Add opencode as a CLI Base Agent (acting agent and Evolver)

* opencode integration bug fixed

* opencode: tools from the first request, long prompts, extra_body, compaction usage

- The adapter's tools reach an OpenCode acting agent through the Base Agent's
  tool server: OpenCode 2.2 waits at most 2 s for an MCP server it starts
  (STARTUP_GRACE_MS) and the adapter's takes ~3 s, so the first request often
  went without them. hosts/opencode.prepare takes serve_tools; docs say so.
- The prompt is OpenCode's stdin, not an argument (E2BIG past 128 KiB).
- A binding's extra_body becomes the OpenCode model's options; the effort
  setting outranks its reasoning_effort.
- Usage counts compactions' model calls, which `run --format json` never
  streams: this call's share of the session's totals in OpenCode's database
  (read-only, immutable, nothing followed out of the home), a resumed
  session's earlier totals taken off (_this_call, now with `fields`).
- parse_log reads OpenCode's stream by its name.
- The AHE Evolver's texts name OpenCode's stream (agent-opencode.jsonl) as
  they name Pi's and mini-swe-agent's (#91).
- The deployer's docstring gives AGENTS.md discovery's real boundary.

Tests: the first request offers the adapter's tools (real binary); a prompt
past ARG_MAX; extra_body and effort; compaction usage from a stand-in
database and from the real binary (mid-run compaction, then a resume).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Haizhong Zheng <hzzheng@umich.edu>
iamziyuzhao added a commit that referenced this pull request Nov 10, 2026
@iamziyuzhao @claude
Merge main (1b0d897) into the DeepSeek Harness branch
1f25839 
main gained #91 (the AHE Evolver's texts name a Pi or mini-swe-agent
acting agent's stream), #94 (OpenCode as a CLI Base Agent), #97
(hermit deploy retires skills the adapter no longer exports) and #99
(agent skills for this repository). #94 adds OpenCode at the places
this branch adds dsh, so 36 files conflicted. Each conflict keeps both
sides, OpenCode before dsh:

- CLIS, the readers (records.parse_log, trajectory's STREAM_HOSTS,
  READERS and MARKS, the AHE debugger's STREAM_TEXTS) and every list
  of CLIs name both, dsh last; cli.py's Dsh part follows OpenCode's;
- main's NO_LOGIN replaces this branch's tuples in login.py,
  isolate.py and acting/workspace.py, and gains "dsh";
- a comment that no longer fits 120 columns with both names is on
  its own line;
- docs: both names in each list. docs/methods/agent.md's dsh learner
  now "signs in as the pi learner does", since OpenCode's sentence
  stands between them; in docs/methods/ahe_adapter.md, #91's general
  wording ("the one its CLI wrote") replaces this branch's list of
  streams.

dsh does not call #97's check_skills and retire_skills yet; the next
commit does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@haizhongzheng haizhongzheng haizhongzheng approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@iamziyuzhao @haizhongzheng
0