Run every method on Pi (evolver.base: pi) by iamziyuzhao · Pull Request #92 · Infini-AI-Lab/hermit

iamziyuzhao · 2026-11-09T08:14:24Z

Change

evolver.base: pi with evolver.model: <backend>/<model> now runs every method's learner on Pi, on that model. Before,
AHE and Harness² refused it, EvoSafe failed each step, mwm ran without its bound model, RRSI's read-only roles had Pi's
default tools, and GEPA, SkillOpt, ACE, OPRO, ReHarness and Continual Harness ignored it.

Method	What runs on Pi
`pi`, Meta-Harness Adapter, ASPIRE	The learner's session, as before through `AgentEvolver`. Only the texts that named two CLIs change.
mwm	Each learner session, now with `provider` (Pi's `models.json` entry for the bound model) in `MwmEvolver._settings`.
RRSI Adapter	The proposer and every role. The analyst and the critic have no tools (`text_only`). A digester has `read` and `ls`, plus `grep` and `find` where the machine has `rg` and `fd`, which an offline Pi cannot fetch. A `roles.binding` becomes the roles' own provider. `roles.model` without a binding is refused at setup. A provider outage is retried as on the other CLIs, although Pi reports it with exit 0.
AHE Adapter	The Evolve Agent and the Agent Debugger. An effort `--thinking` does not take is refused at setup.
Harness²	The editor sessions (`FrameworkProposer._pi_cli`), which keep the bound model's key and count a failed session's usage. A run with no model reference, or with an effort Pi cannot send, is refused at setup, because upstream would take each failed session for the starting harness. A Pi acting agent's trajectory is rendered in upstream's grammar (`render_pi_events`).
EvoSafe	The proposer, and the Designer, Criticizer and Analyzer. None gets Pi's own tools; the TRAIN readers come over MCP. A session that failed is never taken for the role's answer (Pi exits 0), and an answer the provider declined is EvoSafe's safety refusal, not tried again. `proposer: pi` without `evolver.base: pi` is refused, with a message saying what to set.
GEPA, SkillOpt, ACE, OPRO, ReHarness, Continual Harness	These call the run's model, and their manifests now say so (`needs: [llm]`). `evolver.base: pi` makes that model Pi (`llm: {cli: pi, model: ...}`): one session with no tools per request.

mwm and RRSI still refuse, at setup, a backend they are not set up for (mini-swe-agent, opencode, ...), and name
the ones they take. This replaces this PR's first version, which refused Pi on those two.

Shared code (core changes, none in the sandbox):

Pi (hermit/agent/base/cli.py):
- text_only: --no-tools, and no MCP, extensions, skills, prompt templates or context files. A session that ran a
  tool has failed.
- allowed_tools: --tools, keeping the call's MCP tools. An empty list gives --no-builtin-tools.
- persist_session: false gives --no-session.
- A session's effort is what the model is sent. It outranks a backend's extra_body.reasoning_effort, which Pi
  would otherwise apply last. xhigh and max are mapped to themselves on an OpenAI-compatible API; for a model
  with no map, Pi sends both as high. On the Messages API, where Pi gives both the same thinking budget as high,
  they are refused. A level --thinking does not take is refused rather than ignored.
- A reply cut at the output limit is marked finish_reason: length.
- A request's temperature and top_p become the model's samplingParams on an OpenAI-compatible API.
- Pi exits 0 whatever happened, so the stream is read instead, and the last request's status decides.
  - A 429 or a 5xx is unavailable (retried), in each of Pi's forms: 503: {...}, 503 status code (no body),
    a proxy's 503 <html>..., the Messages API's 503 {...} and the Responses API's
    <provider> API error (503): .... So is an error with no status that Pi retried until it gave up (a dropped
    connection).
  - A 4xx that no retry can help is rejected, which the chat endpoint answers with 400. A 408 or a 409 is
    neither (502). Outages that ended in a 400 are rejected.
  - An answer the provider declined is rejected and model_refusal, as on Claude Code: Pi's stop reason
    refusal or sensitive (Messages API), content_filter (chat completions) or incomplete.content_filter
    (Responses API). EvoSafe takes it for its safety refusal and does not try the role again.
- command_timeout_s, below. It is in PI_PARAMS, the Pi settings that the acting agent, the agent Evolver, mwm,
  EvoSafe and Harness² pass on from agent.pi or evolver.pi.
llm: {cli: pi} (hermit/agent/base/endpoint.py, hermit/agent/model/backends.py):
- It needs a <backend>/<model> reference and isolation: sandbox, and takes no login or web search.
- A reply cut at Pi's output limit comes back with finish_reason: length, not as a whole answer.
- Pi applies no sampling field on the Messages API, so there a request that asks for one is refused, unless
  unapplied names that field.
config.layered:
- evolver.base: pi on a method whose manifest says needs: [llm] writes llm: {cli: pi, ...}, with
  evolver.model (else the run's model) and evolver.pi. Naming llm as well is refused.
- A custom Evolver's manifest is found from the config's folder, as the loader finds it.
- llm is a new needs entry (hermit/agent/evolver/loader.py).
A backend's max_output_tokens becomes Pi's maxTokens. Its extra_body goes with each chat-completions request
as the model's samplingParams, as the direct client sends it.

One change outside the learner. Pi's bash runs a command until it ends unless the model gives it a time limit.
In one real run, RRSI's proposer ran grep -r ... /, which held the step for the Trainer's whole hour
(call_timeout_min), and the step was lost. Every Pi session now gets a small extension in its home that gives a bash
call without a limit 600 s; the acting agent's sessions get it too. Set command_timeout_s in evolver.pi or
agent.pi to change it, or to 0 to turn it off. A limit the model gives is kept. For comparison, Claude Code's Bash
stops a command after 2 minutes by default.

What a user meets (docs/methods/agent.md#every-method-on-pi): Pi takes no login and has no web search. Pi's release
directory must itself be on PATH, because the sandboxes follow no link from /usr. Codex and Claude Code configs are
unchanged.

#90 (DeepSeek Harness) merged first and is merged in here (5d2adb4): mwm and RRSI keep this PR's check and refuse
dsh with the other backends they are not set up for.

Validation

On e3df125, the head now (current main with #90 merged in, plus the review fixes below). The run was on Linux
x86_64 (RHEL 9) as a non-root user.

python -m pytest --changed --base origin/main, which selects every group because shared code changed: 1449
passed, 127 skipped, 0 failed. The skips are for missing optional dependencies, CLIs or opt-ins: mini-swe-agent
29, agentdojo 19, --integration 17, pi CLI 15 (run below), Harbor 11, dsh CLI 11, metaworld 8, opencode 4,
clbench 4, Docker 4, OfficeQA 2, Harbor smoke 2, torch 1.
python -m pytest --integration on the Pi tests (the Pi Base Agent and the five methods' Pi files), with the real
Pi 1.5.0: 94 passed.

The rest of this section is from 677b36c.

The suite, the mutation guards, pre-commit, the docs build and the real runs below all ran on 677b36c, this PR's head,
which has current main (886c89d) merged in. They ran in a Linux container (Ubuntu 24.04, aarch64, as root) with
.[test,mini-swe-agent] and the gepa, skillopt, harness2 and rrsi groups, as CI installs. CI passes on
677b36c.

Every method on the real Pi and a real model.

Setup: Pi 1.5.0 (pi-linux-arm64.tar.gz, checked against SHA256SUMS), the model
fireworks/accounts/fireworks/models/deepseek-v4p1-flash, the sandbox enforced (sandbox.enforce: required), then
hermit check and hermit train.
Configs: each is the method's documented one with evolver: {base: pi, model: <reference>}. Each runs on the docs'
toy task (say a text in capitals; the judge's feedback says so), or on the smallest task that keeps the method's
contract.
Acting agent: Pi too, except for mwm, EvoSafe and ASPIRE. Their own acting pipelines need a benchmark, so these were
scripted.
Results: hermit check passed for all 15. The key's value is in none of the 909 files the runs wrote. No command in
the 110 session logs the runs kept was stopped by the new command limit. Total: USD 0.66.

Method	Learner steps	Mean score by step	What the run shows	USD
`pi`	done	0.0	adapter changed	0.024
GEPA	done	0.0	adapter changed (validation mean 1.0); 1 session through `llm`	0.004
OPRO	done	0.0	adapter changed; 2 sessions scored 2 new instructions at 0.5 and 1.0; its `temperature` taken with no `unapplied`	0.019
SkillOpt	done	0.0	1 session; its candidate skill was rejected on validation (0.0, as the current one)	0.003
ACE	done × 3	–, 0.0, 0.0	4 Reflector and Curator sessions; the playbook went from 0 to 7 bullets	0.017
ReHarness	done × 3	–, 0.0, 1.0	3 sessions; the second reflection retrieved and revised the first	0.005
Continual Harness	done × 2	0.0, 0.0	8 sessions; the adapter it wrote was deployed on Pi again	0.026
Meta-Harness Adapter	done × 2	0.0, 1.0	the proposer's candidate was scored on the next step (1.0, the seed 0.0) and became the best	0.034
AHE Adapter	done × 2	0.0, 1.0	Evolve Agent and Agent Debugger	0.127
RRSI Adapter	done, stop	0.0, 0.0	a whole round; 4 analyst and critic sessions ran no tool; 15 digesters ran only `read` and `ls` (no `rg` or `fd` in the image); a critic rejected one candidate as fitted to the task; the other scored 0.0, its rollouts taking up to 32 model calls a task, and the seed stayed	0.142
mwm	done × 3	–, 1.0, 1.0 (scripted judge)	the bootstrap and two reviews adopted	0.122
EvoSafe, `single`	done	1.0	defense accepted; attack success went from 1.0 to 0.0	0.008
EvoSafe, `roles`	done	1.0	Designer, Criticizer and Analyzer; 20 TRAIN reader calls over MCP, no other tool; defense accepted; attack success went from 1.0 to 0.0	0.028
Harness²	done	0.0	three editors changed the harness, and each answered `HELLO REEF`, where the base answered `hello reef`; the reflector added a library entry (kept in the Training State: Harness² never writes the adapter)	0.062
ASPIRE	done	0.0	the coordinator's session; nothing to promote, so no verifier	0.041

Notes on the table:

The EvoSafe runs used a backend with max_output_tokens: 64000; every other run used Pi's default.
mwm's two scripted rollouts report no cost (unknown_cost_calls: 2). Every learner call has its price.
Scores are the toy task's and vary between runs: on an earlier head, Meta-Harness's candidate did not beat the seed,
and RRSI lost its first step to the unlimited command described in Change.

Whole suite. The command was python -m pytest --integration -p no:cacheprovider -q -rfEs --durations=0, with the
test files split over 5 containers.

This head: 3 failed, 1427 passed, 89 skipped. main at 886c89d, split the same way: 3 failed, 1362 passed, 76
skipped.
The same 3 fail on both. Two fail as root: test_isolate::test_a_rollout_sees_what_is_declared_and_nothing_else and
test_ahe_hook_runner::test_logs_that_cannot_be_written_lose_the_line_not_the_call. The third,
test_ahe_evolver::test_the_evolve_prompt_is_start_md_and_the_query, needs ~/.cache/hermit-tests, which depends on
which tests ran before it in its container.
The 13 extra skips are the new integration cases that need Pi, which the suite's image does not have. They run
next.

With the real Pi, offline. python -m pytest --integration tests/core/test_pi_agent.py, in the image with Pi
1.5.0, against a scripted model on the loopback: 67 passed on 677b36c. They check:

what the model is offered in each limited session: nothing with text_only; only the call's MCP tools with
allowed_tools: []; the named tools and the call's with a list;
that a request's temperature and the backend's extra_body reach the model;
that the request's or the session's effort is sent over the backend's extra_body, xhigh and max as named;
that a command without a limit stops after command_timeout_s while one with a limit keeps it, with and without the
sandbox;
that Pi retries two 503s and gives up at a 400 (rejected), sends a 409 once (neither), and reports a content filter
as a refusal (rejected and model_refusal): the streams the unit tests replay.

As a non-root user, as CI runs. The PR's test files, tests/core/test_sandbox.py and the methods' Pi tests, with
the real Pi, on 677b36c: 503 passed and 1 failed,
test_claude_subscription::test_a_sandboxed_acting_session_gets_the_token_and_the_plan_not_the_login, which nests a
sandbox and fails the same way on main in this container, whose bwrap is setuid. CI had failed on two earlier
heads: mwm's Pi test gave its fake backend port 9, and a non-root sandbox cannot listen below 1024 (fixed in
8b67768).

Each change undone. 84 mutations were made in a copy of the checkout: each check removed or narrowed, each setting
dropped. A test fails for each of them, among them: the refusal check, the Responses API's refusal form, a refusal's
status read anyway, reading the status from every error rather than the last request's, only 502 to 504 as outages,
a 409 rejected, any error with no status taken for an outage, and EvoSafe taking a failed session's text for its
answer. The command-limit integration cases were also run with the limit removed, and with the model's own
limit overridden, and failed each time. python -m pytest --changed --base 886c89d selects the whole suite (1519
tests), every new test file with it. pre-commit run --all-files and bash docs/build.sh pass.

Not verified:

macOS.
The methods' real benchmarks (OfficeQA, AgentDyn, Robosuite, ...). Nothing here says how well a method learns on
Pi, only that it runs.
Another model or provider. What a real provider does with xhigh or max is up to it.
A Pi version other than 1.5.0.

Known limitations, each low:

A role's own binding loses the backend's developer_role.
A text-only Pi session keeps Pi's coding-assistant system prompt and a working directory of its own per call, so
prompt caching helps less.
A request's max_tokens is not applied: Pi caps a reply at 16384 tokens unless the backend's max_output_tokens
says otherwise. The reply lists it as unapplied.
No unit test drives the llm route through cli_sandbox end to end; the real runs above do.
EvoSafe and GEPA on opencode fail at each step, not at setup.
RRSI's generate retries a cut reply up to 6 times.
Meta-Harness and ASPIRE accept opencode.
evolver.base: codex, claude or opencode on a method that calls the run's model is still ignored, as before.
xhigh and max on the Messages API are refused by each session, not at setup (Harness² refuses them at setup).
Without evolver.model, Meta-Harness, RRSI and AHE copy Pi's own auth.json into the session's home.
effort: off sends no reasoning field, so the model's default applies.
A digester looks for fd, not Debian's fdfind.
A 429 for an exhausted quota, which Pi does not retry, is still unavailable, so it is retried.
A retried rollout loses its failed attempts' usage, as on Codex and Claude Code.
A served session can carry unavailable: true.
The failure table in docs/guides/claude-subscription.md does not list Pi's statuses.
A 501 or a 505 counts as unavailable, as any 5xx does, although Pi does not retry them.
A context overflow whose summary request fails during an outage reads as rejected, not unavailable.
No test fails if the rule for an error with no status keys on Pi's first retry rather than on its giving up.
EvoSafe still reads a failed Pi session's text for its refusal and login words. A reply cut at its output limit
that names a usage policy ends the role as a safety refusal, and one that names an invalid API key as a lost login,
without a retry; the login words are read before Pi's own model_refusal.
No test has an error message with no status that has a 3-digit number later on (PI_STATUS reads a status only at
the start, or in the Responses API's form).
docs/framework/evolver.md still lists only the Codex CLI for AHE and mwm, and docs/methods/rrsi_adapter.md does
not say that a Pi digester's grep and find need rg and fd.

Review

🤖 Generated with Claude Code

…e at setup mwm ran a Pi learner without its bound model (its session settings carry no provider), and RRSI's roles, kept read-only through Codex's own sandbox mode or Claude Code's tool settings, ran with all of Pi's tools. Both now refuse an evolver.params.backend other than codex or claude before the first step. Each test fails without its check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lowed list The tests held only that `pi` is refused on a hand-built context. mwm's now goes from the config through the Trainer, for evolver.base and for evolver.params.backend; RRSI's also holds that claude passes setup and that a second backend is refused. mwm's comment gave a reason that does not hold (its session texts name neither CLI): it now names the missing `provider`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pi was a Base Agent and an Evolver of its own, but the methods either refused it, ignored it or could not reach their model through it. Every method now runs on `evolver.base: pi`: - mwm, the RRSI Adapter, the AHE Adapter, Harness2 and EvoSafe run their own sessions on Pi, on the bound model. RRSI's roles stay read-only Pi's way: no tools for the analyst and the critic, read, grep, find and ls for a digester. EvoSafe's roles have none of Pi's own tools and keep the TRAIN readers. - GEPA, SkillOpt, ACE, OPRO, ReHarness and Continual Harness call the run's model, so `evolver.base: pi` makes that model Pi: `llm: {cli: pi}`, one tool-less session per call. - The pi Base Agent gains text_only, allowed_tools, persist_session and a check of its efforts. Each Evolver was run with the real Pi 1.5.0 and a real model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pi caps one reply at 16384 tokens for a model it does not know. A model that reasons at length was cut before it answered, and EvoSafe's whole-file answer failed in one of three real runs. A backend's max_output_tokens now becomes the model's maxTokens in Pi's models.json. Unset, Pi's default stands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- A reply cut at Pi's output limit comes back through the chat endpoint marked finish_reason: length, as from an API, and is no reply to Harness2's channel; Harness2 keeps a failed session's usage. - `evolver.base: pi` makes the run's model Pi only for a method whose manifest says it calls it (needs: [llm]); a custom agent Evolver keeps Pi as its CLI, as before. - Pi applies a request's temperature and top_p, and the backend's extra_body, as its model's samplingParams where the API is OpenAI-compatible; it takes effort max. - RRSI resumes a proposer session whose provider failed though Pi exited 0, and gives a digester Pi's grep and find only where rg and fd are installed. - Harness2 keeps the bound model's key variable; EvoSafe refuses `proposer: pi` with no model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…timeout_s Pi's bash tool has no default time limit, so one command that never ends (an RRSI proposer's `grep -r ... /` in a real run) held the learner's session until the Trainer's call timeout, an hour, and the step was lost. The Pi Base Agent now writes a small extension into the session's home (where Pi finds its own) that gives a bash call without a limit command_timeout_s (600 s; 0 for none); a limit the model gives is kept. The home's extensions folder is Hermit's alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CI runs the suite as a non-root user. The sandbox forwards a loopback model's port into the session's network namespace, where only root may listen below 1024, so the fake backend at 127.0.0.1:9 failed every mwm step there ("prepare_failed": the relay's bind was refused). The test now uses port 59999 and its assertion names the steps' summaries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… every session Fixes from the post-PR audit of 45acd88: - command_timeout_s could not be set from a run config: the acting agent, the agent Evolver (pi, Meta-Harness, ASPIRE, RRSI, AHE), mwm, EvoSafe and Harness² each passed a fixed list of Pi's settings. They now share PI_PARAMS. - A backend's extra_body.reasoning_effort replaced the session's or the request's effort, since Pi applies samplingParams last. xhigh and max were sent as high, since a bound model has no thinkingLevelMap. The session's effort now outranks extra_body's; xhigh and max are mapped to themselves on an OpenAI-compatible API and refused on the Messages API, where Pi gives them high's thinking budget. extra_body goes with chat-completions requests only. - On the Messages API, Pi reports a provider's status as "503 {...}": that is an outage too. A 4xx that no retry can help is `rejected` (the chat endpoint's 400), not a 502 the client retries. - Harness² on Pi refuses at setup a run with no model reference or an effort Pi cannot send: upstream would take each failed session for the starting harness. - tests: no sys.path insert in test_pi_agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… into the Pi branch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lingParams The test gave extra_body only a reasoning_effort, which a session's effort removes anyway, so sending extra_body on the Responses API went unnoticed (a mutation guard said so). It now holds a field of its own too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r forms Fixes from the post-PR audit of ac43960. Pi reports a provider's status in more forms than the two the outage pattern knew: "503 status code (no body)", a proxy's "503 <html>...", the Responses API's "<provider> API error (503): ...", and no status at all for a dropped connection, which Pi retries itself and then gives up. Such an outage was a failure, not one to retry (RRSI's proposer counted it toward max_failures). The Pi Base Agent now reads the status of the last request's error: 429 or 5xx is unavailable; a 4xx no retry can help is rejected, also on the Responses API; with no status, an error Pi gave up retrying is unavailable. A 400 after retried 503s is rejected, not an outage. Also a test for a request's sampling on the Responses API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lay what Pi sends From the post-PR audit of 25833aa (round 4): - A request whose answer the provider declined is `rejected` and `model_refusal`, as on Claude Code, and never `unavailable`: Pi's `rawStopReason` "refusal" or "sensitive" (Messages API), "content_filter" (chat completions) or "incomplete.content_filter" (Responses API). Such a request was answered and has no status; its words are the provider's. EvoSafe reads it as its safety refusal and does not try the role again. - An error with no status is `unavailable` only when Pi gave up its own retries (`auto_retry_end`, `success: false`); the shared UNAVAILABLE wording no longer decides. - The status tests replay the streams Pi 1.5.0 sends: a 4xx once with no retry event; a 429, a 5xx and a lost connection retried three times; outages that ended in a bad request, where the last request's status decides. New cases: 500, 529, 409, and 503, 503, then 400. - An integration test runs the installed Pi against a scripted model: 503, 503, then 400; a 409; a content filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

From the post-PR audit of e5490a4 (round 5): - Pi exits 0 however a session ended, so EvoSafe's exit-code gate let a failed session's text through: a reply the provider declined, one cut at its output limit, or one the provider stopped, became the role's answer whenever it parsed as the schema's JSON, and a declined one never raised SAFETY_REVIEW_REFUSAL. A session the Base Agent marks `failed` is now no answer; Codex sets no `failed`, and Claude Code exits non-zero on the same sessions. - Tests: those three endings with a reply that is the schema's JSON (the Pi stand's `stopped` finish reason); a finish reason Pi sends once and that is no refusal is neither `unavailable` nor `rejected`; the refusal whose words start with a number now uses one Pi does not retry, so the stream is what Pi sends. - Docs: the RRSI row of docs/framework/evolver.md and EvoSafe's requirements name Pi. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Conflicts: mwm's and RRSI's backend checks keep this PR's list (codex, claude, pi), which refuses dsh as #90 did; the tests add dsh to the refused backends. Docs list both Pi and dsh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…mappings - Harness²: a Codex or Pi editor session that times out or is stopped keeps its usage - Harness²: a Pi trajectory is rendered from Pi's events, every tool's result kept and a failed one marked [OBSERVATION ERROR] - mwm and RRSI accept backend claude-code again, as before their new checks - evolver.model: {ref, extra_body} binds on the Pi llm route Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

haizhongzheng

Dual review of 677b36c (this session and a separate Codex gpt-6.1-sol review, 3 rounds). I merged current main in (5d2adb4, #90 DeepSeek Harness) and pushed the fixes as e3df125.

Merge with #90. mwm and RRSI keep this PR's backend check, and dsh is refused there with the other backends they are not set up for. The tests that list refused backends now include dsh, and main's separate dsh-refusal test was folded into them. tests/selection.py, tests/README.md, agent-layers.md, model-backends.md and config-fields.md keep both sides.

Fixes in e3df125 (each with a test that fails on 677b36c):

Before	After	Test
Harness²'s `_reply` raised on a timed-out or stopped Codex/Pi editor session before adding its usage, so that spending left the step's usage and budget	usage is recorded first, then the error is raised	`test_harness2_on_pi.py::test_what_a_pi_session_spent_is_kept_when_it_times_out_or_is_stopped[timed_out, cancelled]`
`render_pi_events` kept only some tools' results and did not mark a failed tool	it reads Pi's raw events: every tool's result is kept, a failed one is `[OBSERVATION ERROR]`, an errored or aborted turn is `[ERROR]`	`test_a_pi_tool_that_failed_reads_as_an_error_whatever_the_tool`, updated `test_a_pi_trajectory_reads_in_upstreams_grammar`
mwm and RRSI's new check refused `backend: claude-code`, which they accepted before this PR	`claude-code` is in both allowlists	`test_mwm_evolver.py::test_a_learner_on_a_cli_mwm_is_set_up_for_passes_the_backend_check[codex, claude, claude-code, pi]`, and a `claude-code` setup in `test_rrsi_adapter.py`
`evolver.model: {ref, extra_body}` with `evolver.base: pi` was refused on the `llm` route (`_cli_model` used `is_ref`)	`as_binding` decides, so it binds as it does for mwm	the `{ref, extra_body}` case in `test_pi_agent.py::test_evolver_base_pi_is_the_run_model_of_a_method_that_calls_it_and_the_learner_of_one_that_has_a_cli`

Tests on e3df125: python -m pytest --changed --base origin/main (every group selected): 1449 passed, 127 skipped, 0 failed. The real Pi 1.5.0 --integration Pi tests: 94 passed.

Left as is: effort: off sending no reasoning control (already a known limitation); the Claude path's usage on failure and an llm-route failure's usage in hermit/tools.py, both of which predate this PR. Minor points: blank lines in rrsi_adapter/roles.py, and the RRSI exit-0 retry change.

🤖 Generated with Claude Code

- evolver.base: pi works: call_model takes pi, as Infini-AI-Lab#92 documents for every method. - setup() refuses a bad proposer, debugger, debugger_batch_size or gate_mode, or a missing learner model, before any TRAIN episode. - Each round must measure the whole TRAIN set (is_epoch_end) before the official selector compares it; an error before the round's records still persists the stop flag. - Case IDs follow upstream: `<id>::<condition>`, and a user task's clean case `task_<suite>-<user_task>::clean`, so upstream's clean-failure guidance and regression risks reach the learner. - Hook staging makes every file readable, so SHE's hooks start under umask 077. - CI installs --group she; run.yaml reads full-test-v1, names its llm and enables the semantic judge. - Docs: SHE in the Pi tables; the method index, shell-tool and launch statements corrected; results marked for reference only (the learner ran without Hermit's sandbox), as for Infini-AI-Lab#103. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Expand AgentDyn TEST and use Nemotron 3.5 Lightning in both EvoSafe recipes * Add official SHE evolution on AgentDyn with Lightning * SHE on AgentDyn: fixes from the dual review - evolver.base: pi works: call_model takes pi, as #92 documents for every method. - setup() refuses a bad proposer, debugger, debugger_batch_size or gate_mode, or a missing learner model, before any TRAIN episode. - Each round must measure the whole TRAIN set (is_epoch_end) before the official selector compares it; an error before the round's records still persists the stop flag. - Case IDs follow upstream: `<id>::<condition>`, and a user task's clean case `task_<suite>-<user_task>::clean`, so upstream's clean-failure guidance and regression risks reach the learner. - Hook staging makes every file readable, so SHE's hooks start under umask 077. - CI installs --group she; run.yaml reads full-test-v1, names its llm and enables the semantic judge. - Docs: SHE in the Pi tables; the method index, shell-tool and launch statements corrected; results marked for reference only (the learner ran without Hermit's sandbox), as for #103. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Haizhong Zheng <hzzheng@umich.edu> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

iamziyuzhao mentioned this pull request Nov 9, 2026

Add DeepSeek Harness (dsh) as a CLI Base Agent (acting agent and dsh Evolver) #90

Merged

10 of 12 tasks

iamziyuzhao marked this pull request as ready for review November 10, 2026 06:11

iamziyuzhao marked this pull request as draft November 10, 2026 06:40

iamziyuzhao and others added 4 commits November 10, 2026 02:11

Merge main into fix/mwm-rrsi-backend-check

9c64073

Merge main (OpenCode, the AHE texts' stream names) into the Pi branch

03ed729

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

iamziyuzhao changed the title ~~mwm and RRSI: refuse a learner backend other than Codex or Claude Code at setup~~ Run every method on Pi (evolver.base: pi) Nov 10, 2026

iamziyuzhao and others added 10 commits November 10, 2026 13:13

Merge main (deploy retires skills, agent skills) into the Pi branch

1de2b35

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge main (the sandbox runs a granted venv's python by its own name)…

ac43960

… into the Pi branch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

iamziyuzhao marked this pull request as ready for review November 11, 2026 02:29

haizhongzheng and others added 2 commits November 11, 2026 00:24

Merge main (#90 DeepSeek Harness) into PR 92

5d2adb4

Conflicts: mwm's and RRSI's backend checks keep this PR's list (codex, claude, pi), which refuses dsh as #90 did; the tests add dsh to the refused backends. Docs list both Pi and dsh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

haizhongzheng reviewed Nov 11, 2026

View reviewed changes

haizhongzheng merged commit 9c45a65 into main Nov 11, 2026
1 check passed

haizhongzheng deleted the fix/mwm-rrsi-backend-check branch November 11, 2026 07:57

iamziyuzhao mentioned this pull request Nov 12, 2026

Harbor foundation: every CLI in a task's container, and eval, split and stream runs #105

Open

8 of 11 tasks

haizhongzheng mentioned this pull request Nov 13, 2026

Add official SHE evolution on AgentDyn with Lightning #104

Merged

5 of 6 tasks

Navigation Menu

Repository navigation

Run every method on Pi (evolver.base: pi) - #92

Run every method on Pi (evolver.base: pi)#92
haizhongzheng merged 18 commits into
mainfrom
fix/mwm-rrsi-backend-check

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Uh oh!

Change

Validation

Review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants