8000 Harbor foundation: every CLI in a task's container, and eval, split and stream runs by iamziyuzhao · Pull Request #105 · Infini-AI-Lab/hermit · GitHub
[go: up one dir, main page]
Skip to content

Harbor foundation: every CLI in a task's container, and eval, split and stream runs - #105

Open
iamziyuzhao wants to merge 5 commits into
mainfrom
harbor-foundation
Open

Harbor foundation: every CLI in a task's container, and eval, split and stream runs#105
iamziyuzhao wants to merge 5 commits into
mainfrom
harbor-foundation

Conversation

@iamziyuzhao
@iamziyuzhao iamziyuzhao commented Nov 12, 2026 •
edited
Loading
Copy link
Copy Markdown
Collaborator

Change

This PR is the groundwork for running Harbor's benchmarks with Hermit's methods. After it, a benchmark from Harbor's
registry plugs in as an example folder: a utils/benchmark.yaml, from which python -m hermit.harbor.dataset prepare
makes the data and the split files. The folder's only code, when it needs any, is a patch of the downloaded tasks
(dataset.patch; LAB-Bench FigQA's removes the shuffle's seed, which gives the answer away). The config's run:
block then picks one of three ways to run it. This PR holds no benchmark folder. It covers what Terminal-Bench 2.0,
SWE-bench Verified, AlgoTune, Terminal-Bench Science and LAB-Bench FigQA need; a follow-up PR adds only their
folders. It builds on #56 (Harbor's task format as Hermit's one task protocol, a stream as a multi-step task). It also
covers the CLIs main added since: mini-swe-agent (#49), Pi (#89; every method on Pi, #92), OpenCode (#94) and the
DeepSeek Harness (dsh, #90) each run in a Harbor trial, as Codex and Claude Code now do.

The decisions it implements:

  • Hermit renders each CLI trial's launch with the CLI's own Base Agent, and one executor (harborside) runs it in the
    task's container. A Harbor trial is then one more session of that CLI: the same home, command, stream and reading
    of the outcome as Hermit's own sessions. The answer agent (cli: answer) makes its one model call from Harbor's
    process; Harbor's own agents run as Harbor installs them.
  • The adapter's tools are served from this machine, on the Docker bridge's gateway, to the session in the container.
  • A key or a token reaches the container by its variable's name only. Codex's ChatGPT sign-in is a file of its home
    there (mode 600).
  • A trial that is not the agent's result (infra) is unscored. It is never judged, and the Trainer's and hermit eval's means leave it out and count it. An Evolver that averages the records itself, as GEPA does, still takes it
    as a 0, as on main.
  • The judge scores the verifier's reward whenever the verifier gave one, a timeout or an agent error included, as
    Harbor's own job mean does.
  • A run is eval, split or stream. Separately, run.feedback sets what the learner is shown of each verdict:
    full, score or none.
  • Not in this PR (parked): splits that span benchmarks, and Codex run outside the task's container.

The first three commits follow these parts. The fourth holds the fixes of the reviews since the PR was opened (its
message lists them), and the fifth makes one of its tests run where Harbor's venv is not installed (CI); each part
below describes the code as it is with them.

1. Shared layers

What the Harbor runner needs of the layers every method shares, so that a trial in a task's container is one more
session of the CLI's Base Agent:

  • The hosts deploy for a caller that serves the adapter's tools itself (--no-serve-tools on every host; Pi's and
    dsh's prepare(serve_tools=False) names no server, and the acting agents' workspace now passes the flag on, so a
    container session of Hermit's own runs gets no server row it cannot start) and take one task's context
    (--context-file).
  • Every CLI Base Agent reads a stream its caller read elsewhere: outcome() with no stream file reads the stream's
    text as it reads the file, the reply included; a missing or empty stream is a session not served, never a TypeError.
  • A Codex session whose sign-in or key the provider refused is login_failed (the run stops) if its turn did not
    complete, read from its own notices: never one it recovered from, what its log quotes of the session's tool calls,
    or an event's text (U+0085, U+2028 and U+2029 end no line of a stream); unavailable comes from its last failure.
  • hermit.agent.base.trajectory, the AHE Debugger, Harness²'s bench and RRSI's render read a Harbor trial of any CLI
    (agent/agent-<backend>.jsonl, then Harbor's own output files, then its ATIF trajectory, which the Debugger takes
    before those files), and AHE's texts (EVOLVER.md's too) name that trial's files (the stream, hook audit and state
    under agent/; the record's instruction, as the Debugger's task0i.md; START.md's tree). Harness²'s bench takes a
    Claude Code trial's final message as the Base Agent reads its reply.
  • A Harbor acting agent's model is bound and checked in the acting CLI's dialect (agent (harbor, <cli>), an
    endpoint's too), the whole Binding spec kept, developer_role included, and its login is a contained session's
    (logins_of) unless claude_bin names this machine's Claude Code, whose version is then checked; the task set's
    benchmark file counts for both, as the runner merges it (relative to normalize_config's base, else the working
    directory). hermit.harbor.clis is the table of those CLIs.

tests/selection.py counts the recorded OpenCode and dsh streams as shared: the methods' tests read them too.

2. Run modes

How a run is done is the config's run: block, so that a benchmark plugs into each protocol by its split files and
nothing of its own:

  • mode: eval is a config for hermit eval alone: no train split, Evolver, task-info or budget, and hermit train
    refuses it; root there is the template's own adapter, and the tree it tests (trainer.tree_dir, with no template)
    is a run secret as a training run's is. split trains, then tests the final adapter on
    data.test. stream scores each row once, by the adapter the earlier rows taught, before the learner sees it:
    one epoch, no shuffle, the final adapter the last step's, run_agent only on rows already reached (no validation
    row may be a train row); its prequential score (overall, by halves, by step) goes to stream_summary.json and the
    summary. An Evolver whose protocol scores its own steps gives that score (Evolver.step_scores, None for a row it
    leaves unscored, left out and counted; Harness²: the deliverable it chose without the labels, 0 for a composed
    harness that delivered nothing, never the base harness's score), the rollouts' mean kept beside it.
  • feedback: full | score | none is what the learner is shown of each verdict, in the learning material (a README
    per policy), run_agent's results and the logs handed to it (the verifier's output and result files gone under
    score and none); a stream must name it, and hermit train refuses (hermit check reports) one below what the
    Evolver reads: Evolver.min_feedback, score by default (full for AHE Adapter and EvoSafeHarness; none for
    the CLI Evolvers, Harness², MWM, ReHarness and Continual Harness), or min_feedback_for(params). An Evolver that
    keeps the labels from its sessions itself (hides_labels: Harness²) gets the full records. The CLI Evolvers',
    Meta-Harness Adapter's and ASPIRE's prompts promise no more than the run shows.
  • orders runs a stream once per seed or order file (a task's rows kept together), combined in streams.json, each
    order to train checked before the first trains; checkpoints test the commit after the step a row ends on a split
    of their own, with run.state's environment and run.score's scores, after training or as each is reached (outside
    the training budget's hours); one inside a step is refused before training. hermit eval --checkpoints runs those
    without results, and one a halt cut short (a lost login, a usage limit), which is not kept as done.
  • A multi-step Harbor task's steps run in order: data.shuffle moves its rows as one, an order or split taken as
    given or an Evolver's next_tasks that names a step ahead of an earlier one is refused before any run takes it,
    and run_agent runs no training row that is a step.
  • A run the pipeline says is not the agent's result (RunOutput.extra["unscored"]: a Harbor infra trial) has status
    infra: never judged, null in steps.jsonl, left out of the Trainer's and hermit eval's means and counted (a 0 to an
    Evolver that averages the records itself, as GEPA does); eval.infra_retries runs those rows again, a multi-step
    task's steps and an unsupported_task excepted, until a halt. hermit eval --split tests another split; each repeat
    of eval.repeats > 1 hands the pipeline rows that name it (repeat); pass@k counts a Harbor task's rows as its
    attempts and a multi-step task's step (<task>:<step>) as a task; "solved" is the judge's passed when it is a bool
    (unknown when it is None), else a score of at least 1.
  • An acting agent of Harbor's own takes no adapter: hermit train refuses it, hermit check reports it, and
    hermit eval runs it with --adapter none only. A run: or eval: left empty takes its defaults, and so does a
    key left empty or left out of a whole-section override (eval={repeats: 3}).
  • The run's secrets include order files, checkpoint splits, the score hook and a Harbor runner's task and job
    directories.

3. Harbor

A Harbor trial ran only Codex well (Claude Code only as Harbor's own agent, which takes no adapter); the adapter's
tools, a per-task context and an adapter's notes never reached the container, and infra trials counted as zeros. Now:

  • One HermitXXX class per CLI (hermit.harbor.clis: codex, claude, pi, opencode, dsh, mini-swe-agent) over one executor
    (harborside/hermit.py). Hermit's side renders each trial's session with the CLI's own Base Agent
    (hermit.harbor.launch: the home, the command, the environment; host paths made the container's) against the adapter
    deployed as hermit deploy lays it out, and harborside uploads and runs it as the task's agent user: the prompt on
    stdin, the stream in agent/agent-.jsonl, the deployment record always there (so an adapter's hooks run on
    Codex too, HERMIT_HOOK_LOGS set), a session still running at Harbor's timeout killed with its process group. The steps
    after the session still run past that timeout; a copy that failed is dropped, never kept in part. A <cli>_bin serves
    only at the asked-for version and when the agent user may execute it. The runner reads the stream back with the same
    Base Agent: reply, served, the plan's or the provider's refusal, the usage at agent.params.price, else at the
    backend's price (Claude Code's own cost on Anthropic's API, as in Hermit's own sessions).
  • The adapter's tools are served from this machine on the Docker bridge's gateway (hermit.harbor.tools, every call
    logged); a per-task context is computed in the adapter sandbox (hermit.harbor.context); an adapter's notes come back
    after the session (memory/ in the log).
  • A key or a token reaches the container by its variable's name only (docker compose exec -e NAME; Claude Code's
    subscription token on a descriptor), never in a command line, a file, the plan or a log; Codex's ChatGPT sign-in is
    a file of its home there (mode 600). Codex and Claude Code run on the user's subscription
    (login: chatgpt | claude) in the plan's slots and holds, each trial noting the plan's usage (so MAX_USED holds
    trials back): a refused session again after the window, one the provider could not serve after a pause unless it
    ran out of the agent's time, a lost sign-in never, none once another trial halted the run; the trial's usage is
    every run's. Codex's refreshed sign-in comes back and becomes the shared copy. A task whose agent network is not
    public gets the model's and tools' hosts allowlisted and web search off; a CLI's trial (the answer agent's aside)
    needs web_search set.
    Harbor's own claude-code keeps Harbor's rule for its web tools unless web_search: false turns them off on every
    task, and bills the key this process has, never the subscription. On a bound model it is given the backend's
    Messages API root, and a model at Anthropic's own API by name (anthropic/<model>).
  • The answer-only agent (cli: answer) makes one model call through an endpoint the runner serves and writes the
    answer into the container (LAB-Bench's questions); a request still in flight when the trial ends is given up, its
    cost unknown, and on a model a CLI reaches on a login the trial waits for that plan first (a request the plan held
    past the agent's time is infra, usage_limit). A question or attachments it cannot read fail the launch. Its
    relay counts what a failed session used, which a CLI's endpoint reports in its error body.
  • The judge scores the verifier's reward whenever the verifier gave one, a timeout or an agent error included
    (Harbor's own mean), says "passed" at pass_reward (null for a score scale) and leaves out what verifier_redact
    matches. Not the agent's result, so unscored (infra): a verifier whose tools never downloaded or that hit a full
    disk, harborside's launch failing (HermitLaunchError, the CLI's install included), a session that never started or
    whose tools never came up, a lost sign-in, a Claude Code, Pi or OpenCode session or an answer request the provider
    refused for the run's configuration before serving any (its key, model, bill or a setting it does not take:
    model_rejected, no halt). A trial out of the agent's time is the agent's even after a provider notice, and so is a
    request refused for its input; the adapter's context over the CLI's limit beside the image's own is the adapter's 0,
    and so is a session the model's safeguards ended.
  • Startup refuses what every trial would: a login for another CLI, a login and a binding at once, a loopback model, a
    missing key or sign-in, a non-Docker environment for secrets by name, a plan the Base Agent cannot render or would
    refuse to run (mini-swe-agent's web search, a model class Hermit does not read), a harbor that is no executable
    file, mini-swe-agent without its package for a run that deploys adapters, the answer agent's attachments on a model a
    CLI reaches, and for Harbor's own claude-code CLAUDE_FORCE_OAUTH or the subscription's token as its only sign-in.
    A task Harbor runs for none of Hermit's agents (one with MCP servers, a skills folder or a Windows container) is
    infra, unsupported_task, and never handed to Harbor.
  • python -m hermit.harbor.dataset prepare makes an example's data from its benchmark file's dataset: block (the
    registry's tasks, rows, splits, smoke rows, a patch, the images' sizes with image_sizes: true, a lock). A download
    is moved into place only once whole, and what is in its way is refused first; a relative data: is the benchmark
    file's folder's. The tasks' source (the ref with repo, registry_url and registry_path) is recorded in the lock
    and in the download: another source's download is never reused, and the default data directory is the source's own.
    The lock holds no path of this machine (a registry file of this machine by its file name) and hashes only a mode's
    executable bit (one written before needs writing again when a file's mode was other than 644 or 755); an unreadable
    lock is refused unless --force.
  • Also, from the Terminal-Bench 2.1 runs: a host failure (Docker's, the host's /tmp full) is infra and no trial starts
    while /tmp is short; the trial of a task whose name is longer than 32 characters is found; a harbor run past its
    wait budget is killed and its trial's compose projects, the verifier's too, taken down by name alone (no compose
    file or COMPOSE_* variable read). A stream's step runs its own plan, its verifier read as a trial's; each
    hermit eval repeat and each run gets streams of its own, stopped when the run ends unless a hermit train may
    resume them. The trial cache keys the launch, the whole row, the task's files, the model's price and the adapter
    as deployed (its bytes and executable bits, not a umask's); a cached trial costs nothing again, and hermit check's
    probe never takes or keeps one. dsh's context limit counts the image's own file as dsh escapes it.
  • OfficeQA's Harbor judge reads the final message from HermitCodex's stream (agent-codex.jsonl): a task set built
    before needs building again.

Core changes. All the parts above change shared code:

  • The Base Agents (hermit/agent/base/: cli.py, endpoint.py, login.py, records.py, trajectory.py), the
    hosts' command lines (hermit/hosts/), the model binding (hermit/agent/model/backends.py), the chat client
    (hermit/agent/model/llm.py: a failed session's usage from a CLI endpoint's error body) and the acting agents'
    workspace.
  • The config (hermit/config.py, hermit/defaults.yaml: the run: block, eval.infra_retries, eval.infra_wait_s),
    the CLI, hermit/evaluate.py, hermit/settings_check.py and hermit/task.py.
  • The Trainer (hermit/trainer/: trainer.py, material.py, rollout.py, budget.py, card.py).
  • The Evolver base (hermit/agent/evolver/base.py): min_feedback, min_feedback_for(params) and step_scores.
    AgentEvolver and CommandEvolver (hermit/agent/evolver/agent.py) set min_feedback to none, and the Trainer
    reads hides_labels from an Evolver that sets it (Harness²'s).
  • The Harbor package (hermit/harbor/): the runner, the launch plan, the CLI table, the tool server, the per-task
    context, the dataset module and harborside.
  • hermit/isolate.py:
    • A declared environment path may not overlap the run's new secrets: order files, checkpoint splits, the score hook,
      a Harbor task set's tasks and jobs directories, and the tree an eval config names (trainer.tree_dir, with no
      template).
    • agent_pipeline gives a Harbor agent what its streams and startup checks need: the run's identity, whether it
      deploys adapters, and the model's CLI and login. It closes that agent with the pipeline, and hermit check's
      agent gets no trial cache.
  • conftest.py collects no benchmark's own tests from a prepared task set, and .gitignore ignores the task set's
    link (examples/*/utils/tasks).
  • hermit/sandbox.py, hermit/sandbox_relay.py and the sandbox's defaults are unchanged.

The run: block's defaults keep the old behavior: with no run: block, run.mode: null trains only and
run.feedback: null is full.

Changes an existing config meets:

  • A Codex session whose sign-in or key the provider refused is a lost sign-in: the codex acting agent and the agent
    Evolvers on Codex end the run, which hermit train resumes once the sign-in or key is restored. Before, only that
    session failed, and the run went on. So does a method's model call through Codex (a learner's llm: {cli: codex},
    as in OfficeQA's run_codex_luna.yaml configs: the endpoint answers 401 hermit_login_failed, and the call raises
    LoginFailed, which the methods' error handlers do not catch), and so do the Codex sessions of Harness², MWM, RRSI
    Adapter, the AHE Debugger and EvoSafeHarness. Before, such a call failed after the client's retries, and GEPA, OPRO,
    ACE, SkillOpt, ReHarness and Continual Harness skipped it and went on.
  • run_agent returns a result with status error for an id it does not run. Before, the whole call raised.
  • hermit eval's solved_once and pass_at_k take the judge's passed when it is a bool, before a score of 1.
  • Harbor: Codex on the ChatGPT sign-in is login: chatgpt, with the CLI's model name. The runner's codex_login
    (default true) is gone: a HermitCodex trial with a plain model name and no login is refused at startup, and Harbor's
    own codex agent is no longer given that sign-in.
  • Harbor: Harbor's own claude-code is no longer given the subscription's token nor CLAUDE_FORCE_OAUTH (Harbor
    would put the token on a command line). It bills the key this process has; a run that asks it for the subscription
    (CLAUDE_FORCE_OAUTH) or has only the token is refused at startup. The subscription runs with
    acting_agent.cli: claude and login: claude.
  • Harbor: Harbor's own claude-code on a <backend>/<model> reference was given the backend's chat base URL, to which
    Claude Code appends /v1/messages (its requests went to /v1/v1/messages). It is now given the Messages API root,
    and a model at Anthropic's own API by name (anthropic/<model>, with its key's variable).
  • Harbor: a trial of a CLI of Hermit's table (HermitCodex, the default, included; not the answer agent) needs
    web_search set.
  • Harbor: such a trial that passes a key or a token runs only on Harbor's local Docker environment.
  • A client of a CLI's chat endpoint (model: {cli: ...}) counts what a session that failed or that the provider
    refused used, which the endpoint reports in its error body: a run's usage and cost take those sessions in.
  • With eval.repeats above 1, each repeat hands the pipeline copies of the test rows that name it (repeat): a
    pipeline that records the whole row records another row in each repeat (AgentDyn's case_sha256 differs by
    repeat).
  • Whether the provider could not serve a Codex session is read from the session's last failure, never from a notice
    it recovered from: a session that recovered and then failed for its own reason is no longer run again as infra.
  • hermit eval --adapter root on a run whose tree does not exist yet starts it from the template; before, it failed.
  • An OfficeQA Harbor task set built before this PR needs building again. A dataset lock written before this PR needs
    writing again when any task file's mode was other than 644 or 755 (a umask other than 022 at download, or a file
    of a mode of its own).

Documentation:

  • The run block and the feedback policy: docs/framework/configuration.md, trainer.md, evolver.md,
    commands.md, custom-evolver.md, and the guides evaluation.md, runs.md and training.md. The docs' example
    Evolvers that read the judge's feedback (docs/_examples/) set min_feedback.
  • Harbor: docs/framework/task.md#benchmarks, examples/README.md, docs/guides/claude-subscription.md,
    model-backends.md, docs/get-started/installation.md, and OfficeQA's Harbor route (docs/recipes/officeqa.md,
    examples/officeqa/README.md).
  • The methods:
    • docs/framework/evolver.md and configuration.md list the run.feedback each Evolver needs.
    • docs/reference/compatibility.md says what run_agent's refusal of a multi-step task's steps does to each method.
    • The pages of ACE, AHE Adapte 8000 r, GEPA, Meta-Harness Adapter, OPRO, prompt-only, ReHarness, RRSI Adapter and
      SkillOpt say which run.feedback they need. Those of ACE, GEPA, OPRO, SkillOpt, Harness² and RRSI Adapter say
      what they do with a multi-step task, and Continual Harness's that a verifier's timeout is shown as one.
  • docs/reference/config-fields.md, docs/reference/compatibility.md, two tutorials
    (docs/tutorials/custom-evolver.md, your-own-task.md) and CHANGELOG.md.

Validation

On this PR's head 2ca476a, on main at 9c45a65. Linux x86_64 (RHEL 9) as a non-root user, Python 3.12.13, a
venv with .[test], mini-swe-agent 2.4.6 (the mini-swe-agent extra) and the gepa, skillopt, harness2 and
rrsi dependency groups (not ace's sentence-transformers).

  • The whole suite (tests/ and the examples' own tests), as python -m pytest collects it, run in 12 parallel shards
    of its files (each its own pytest process and temporary directory): 1881 passed, 84 skipped, 4 failed, in 176 s. The
    checks after this one, and the real-container runs, ran on 7f02e95, the head before the last commit, which changes
    one test only (it gives the runner a stand-in harbor, as the runner's own tests do, so that it runs on CI); there
    the whole suite gave the same.
    • The 4 failures are this machine's, and the same 4 fail on main here:
      tests/core/test_sandbox.py::test_a_linked_venv_runs_by_its_own_name_with_its_own_packages, and three tests in
      examples/clbench_database/utils/tests/test_parity_golden.py (ContractParity, SessionPromptParity), which
      compare against a local MetaWorld checkout.
    • The skips are for missing dependencies, CLIs or opt-ins: agentdojo 19, --integration 17, the pi CLI 15, the dsh
      CLI 11, MetaWorld 5, CL-Bench 4, the opencode CLI 4, a Docker opt-in 4, OfficeQA's data 2, the Harbor smoke 2 (run
      below), torch 1.
  • CI on 2ca476a: 1817 passed, 152 skipped. On 7f02e95 it failed the one test the last commit changes, which
    needed Harbor's venv to build the runner; 1816 others passed.
  • Each commit on its own:
    • At 0483ae0 (the head as the PR was opened), the whole suite gave 1857 passed, 84 skipped and the same 4
      failures.
    • At 8a7710e, the whole suite fails the same 4 tests and nothing else.
    • At 70340e6, the same holds. The whole suite ran on a first build of this commit and failed those 4, plus one
      test in each of tests/core/test_backends.py and tests/core/test_deploy.py that needs the third commit's code.
      Eleven of those two files' cases, those two among them, then moved to the third commit. The two files pass at
      70340e6 (71 tests), and the moved cases are the only difference between the two builds.
  • HERMIT_HARBOR_SMOKE=1 python -m pytest tests/core/test_harbor_smoke.py (Harbor's venv and Docker): 2 passed.
  • bash docs/build.sh (warnings are errors) and pre-commit run --all-files pass.
  • The offline walkthrough, hermit check, hermit train and hermit eval --adapter final on
    docs/_examples/first_training/run.yaml, passes; final scores 1.0 on its 2 test tasks.
  • The new test files, each run on its own: test_run_modes.py 70 tests in 101 s, test_harbor_launch.py 18 in 0.3 s,
    test_harbor_dataset.py 30 in 0.7 s, test_trajectory.py 22 in 0.2 s, test_harbor_judge.py 8 in 0.1 s. The PR's 25
    test files together take 11.1 min (909 passed, 4 skipped). These ran at the same time as the other checks above.

In real containers, on 7f02e95. hermit eval with agent.name: harbor, on this machine's Docker, on small
tasks made for the check: write 42 to /app/answer.txt (with an open network, and with network_mode: "no-network"), the same with an instruction that begins with /, call the adapter's MCP tool and write what it
returned, answer a question (the answer agent), and a three-step stream that appends a line per step.

Acting agent Model Tasks and adapter Score
Codex 0.164.0 on the ChatGPT login gpt-5.6-luna, effort low 2 tasks, open and closed network; the prompt-only root 1.0
Codex the same the MCP tool task, an adapter exporting the magic tool (one call, in tool-calls.jsonl) 1.0
Codex the same AHE's template (its hooks ran: 9 calls in agent/hook-calls.jsonl) 1.0
Codex the same Continual Harness's template (its notes came back: memory/NOTES.md in the trial's log) 1.0
Codex the same ASPIRE's template 1.0
Codex 0.167.2, installed by Harbor (no codex_bin) the same 1 task, the prompt-only root 1.0
Claude Code 2.1.322 on a subscription token claude-haiku-5-5, effort low 2 tasks, one whose instruction begins with / 1.0
Claude Code the same the MCP tool task (one call) 1.0
Claude Code the same AHE's template (8 hook calls) 1.0
Pi, dsh, mini-swe-agent a scripted model 1 task each 1.0
OpenCode a scripted model 1 task infra at the first run; 1.0 run again (see below)
OpenCode a scripted model the three-step stream 1.0 on each step
The answer agent a scripted model the question 1.0
  • The scripted model is an OpenAI-compatible server on the Docker bridge's gateway that answers by the request's
    shape: one shell call that writes the answer, then the answer. No real model was used for Pi, dsh, OpenCode,
    mini-swe-agent or the answer agent: this machine has no DeepSeek, Fireworks or other API key.
  • The OpenCode task's first run was infra (unscored, session_failed): OpenCode's own server (opencode serve --stdio) sent no request in the agent's 600 s and was stopped at Harbor's timeout. Run again on its own, it scored
    1.0 in 24 s. The same happened once on 0483ae0, and on the fourth commit's first build it scored 1.0 at the first
    run.
  • None of the secrets is in any of the 708 files these runs wrote: not the Claude Code token, not the scripted model's
    key, and not the ChatGPT sign-in's tokens. Codex's own session files name the ChatGPT account's ids. A trial on that
    login brings those files back into its log, and the learner is never shown them.

To reproduce: Harbor 0.25.0 in its own venv (~/.cache/hermit/harbor-venv), Docker with the default bridge, and for
each acting agent its sign-in: Codex's ChatGPT login (codex login) and its 0.164.0 build (the musl binary, mounted
with codex_bin; one run without it, installed by Harbor), a Claude Code token (claude setup-token, exported as
CLAUDE_CODE_OAUTH_TOKEN), and for the scripted model an OpenAI-compatible server on the bridge's gateway that
answers one shell call writing 42, then 42. The configs are hermit eval configs of this shape (Codex's; the
others change acting_agent.cli, model, login and the sign-in's settings):

data: {tasks: ../tasks.jsonl, test: ../splits/codex.txt}
judge: {path: hermit.harbor.judge}
template: prompt-only@1.0
run: {mode: eval}
agent:
  name: harbor
  concurrency: 3
  params:
    tasks_dir: ../tasks
    web_search: false
    agent_timeout_sec: 600
    disk_gate_gb: 5
    setup_timeout_sec: 900
    jobs_dir: ../jobs/codex
    acting_agent: {cli: codex}
    model: gpt-5.6-luna
    login: chatgpt
    effort: low
    codex_bin: ~/.codex/packages/standalone/releases/0.164.0-x86_64-unknown-linux-musl/bin/codex
trainer: {run_name: smoke-codex, output_dir: ../runs}

each run as hermit eval --config <file> --adapter <root, an adapter folder or a template> --split <one split>.

Not verified:

  • macOS.
  • dsh, Pi, OpenCode, mini-swe-agent and the answer agent on a real model (no key here; see above).
  • Codex on an API key: every Codex trial here ran on the ChatGPT login.
  • Harbor's own agents (acting_agent: {name}) in a container. For Harbor's own claude-code on a bound model, the
    path its requests take was checked with this machine's Claude Code (2.1.322) and a local listener: a base URL that
    ends in /v1 gives POST /v1/v1/messages, the Messages API root gives POST /v1/messages. The rest is the tests'.
  • hermit train through Harbor: only hermit eval ran in containers. The Trainer's Harbor path (the curated trial
    directories, what run.feedback strips from them, run_agent) is covered by the tests' stand-in trials.
  • OfficeQA's Harbor route with its changed judge: its own tests only (examples/officeqa/utils/tests/).
  • The five benchmarks themselves: their example folders, configs and runs come in the follow-up PR.

Review

  • Tests and documentation cover the affected behavior, where applicable.
    • Tests: see Validation and the test checklist below.
    • Documentation: listed in Change.
  • Commands, defaults, and public interfaces are consistent.
    • New:
      • the run: block (mode, feedback, orders, checkpoints, checkpoints_during, state, score);
      • eval.infra_retries and eval.infra_wait_s;
      • hermit eval --split and --checkpoints;
      • the hosts' --no-serve-tools and --context-file;
      • python -m hermit.harbor.dataset prepare and a benchmark file's dataset: block;
      • the Harbor agent's acting_agent.cli for each CLI, and its sign-in, network, judge, cache and disk-gate
        settings (docs/framework/task.md#benchmarks);
      • the Evolver's min_feedback, min_feedback_for(params), hides_labels and step_scores, and a pipeline's
        RunOutput.extra["unscored"].
    • The run: block's defaults keep the old behavior (Change).
    • Changes an existing config meets are listed in Change.
  • Shared logs and examples contain no credentials or private data.
    • The PR adds no logs or run records.
    • A prepared dataset's lock holds no path of this machine.
    • The real-container runs were checked for the secrets' values, the ChatGPT sign-in's tokens included (Validation).
  • Documentation-only: links the issue where it was discussed with the core team. (Not a documentation-only PR.)
  • Core changes: listed in Change.
  • Implementation correctness and failures:
    • A Harbor trial's verdict separates the agent's result from infra. The agent's result is scored by the
      verifier's reward, a timeout or an agent error included. Infra is unscored, counted and run again with
      eval.infra_retries: a host or Docker failure, a launch that failed, a session that never started, or a
      Claude Code, Pi, OpenCode or answer-agent request the provider refused for the run's configuration. A task Harbor
      runs for none of Hermit's agents is infra too, never started and not run again (unsupported_task).
    • A lost sign-in halts the run's trials: every later Harbor rollout returns at once as infra, with no container,
      and hermit eval runs none of them again. hermit train still runs its remaining steps on those rows, as on
      main. A lost sign-in in Hermit's own CLI sessions ends the run (LoginFailed).
    • Cancellation and timeouts:
      • a session still running at Harbor's timeout is killed with its process group;
      • the steps after a session still run;
      • a copy that failed is dropped, never kept in part;
      • a request still in flight when its trial ends is given up.
    • What every trial would fail on is refused at startup (Harbor's list in Change).
    • Ten review rounds traced these paths (Independent review, below). The last found no high or medium problem.
  • Real execution paths and tests:
    • The real-container runs in Validation go through hermit eval and Harbor.
    • test_run_modes.py goes through Config.load, the Trainer, hermit.evaluate, run_check and the CLI.
    • The harborside tests run the HermitXXX agents under Harbor's own interpreter.
    • The Base Agent, trajectory and method tests replay streams the real CLIs printed.
  • Test checklist:
    • Groups: the framework's tests are in tests/core/ (new: test_run_modes.py, test_trajectory.py,
      test_harbor_launch.py, test_harbor_dataset.py); a method's are in tests/methods/<name>/; the OfficeQA Harbor
      judge's are the example's own (examples/officeqa/utils/tests/test_harbor_judge.py). The recorded OpenCode and dsh
      streams, which the methods' tests now read too, are shared fixtures in tests/selection.py; .gitignore selects
      the core group (test_harbor_dataset.py checks what it ignores). tests/README.md says which tests read the
      recorded streams and how the Harbor tests skip. conftest.py collects no benchmark's own tests from a prepared
      task set (examples/*/utils/tasks).
    • Names state the behaviour as sentences. Each new file opens with a docstring naming the code under test and what
      it covers; all but test_harbor_launch.py's also name its stand-ins.
    • Each review fix's regression test was checked to fail with the fix undone.
    • Entry points: test_run_modes.py goes through Config.load, the Trainer, hermit.evaluate, run_check and the
      CLI; the harborside tests run the HermitXXX agents under Harbor's own interpreter (they skip without Harbor's
      venv); the trajectory, Base Agent and method tests replay streams the real CLIs printed (tests/fixtures/).
    • Isolation: tests write under tmp_path, use no sign-in, token or network, and start no container unless
      HERMIT_HARBOR_SMOKE=1 (tests/core/test_harbor_smoke.py, run in Validation).
    • pyflakes reports nothing new in the changed files but four more of one warning main gives 28 times:
      tests/methods/ahe/test_ahe_hosts.py imports its own tmp_path fixture (one outside /tmp), so each test
      that takes it is a "redefinition of unused 'tmp_path'", the PR's four new ones too. Five warnings main gives
      in those files are gone.
  • Sandbox checklist: not complete. It asks for the issue where a change to the sandbox itself was agreed with
    the core team, and none was opened for this PR's hermit/isolate.py changes (last item). The other items:
    • The CLI session of a trial runs in the task's container, as the task's agent user, with the task's network mode.
      That container is the boundary Harbor's tasks define, so the session does not go through hermit.sandbox.
    • The adapter's tools and a per-task context run in the strict adapter sandbox (hermit.harbor.tools,
      hermit.harbor.context): never in the runner's process, nor with its keys or network.
    • harbor run runs as a subprocess in Harbor's own venv, unconfined, as on main since Benchmarks: one task protocol (Harbor's format) for single tasks and streams #56. It drives Docker.
    • The learner's sandbox has the same binds, network and defaults. It now treats the run's new secrets as absent,
      and under run.feedback: score or none the learner gets no verdict files.
    • A Harbor task set's pipeline runs in the host process, and neither it nor a trial's container keeps a
      sandbox/<role>-<n>.json (a model only a CLI reaches would get one), so the real-container runs left none to link.
    • The sandbox itself: hermit/isolate.py changes, as described under Core changes. No issue was opened for this,
      so the box is left unchecked; the change is listed for the core team's review. hermit/sandbox.py and
      hermit/sandbox_relay.py are unchanged.
  • This description: read end to end against the code and the run records above.
  • Independent review (post-PR audit): run twice by a fresh Codex session (gpt-6.1-sol, high effort,
    read-only): on 0483ae0 once the PR was open, and on the fourth commit's first build (2389a10). The last round's
    fixes (in the fourth commit as pushed) were reviewed by the author's session only, so the box is left unchecked.
    • Before the PR was opened, the branch went through 7 rounds of dual review. Each round had two independent
      reviewers: a fresh Codex session (gpt-6.1-sol, high effort, read-only), and this session's own review, in
      which six reviewers (five in round 1) each took one area. From round 2 on, a verifier tried to reproduce each
      finding and a second rated it (a low finding its reviewer was not sure of had the first only); this session
      checked round 1's findings itself. Each reviewer's findings were written down before the other's were read.
    • Every confirmed high and medium finding of rounds 1 to 6 was fixed, and so were most low ones. Some low ones were
      resolved in the documentation only: a few that predate the PR, and three it introduced (an answer agent on a login
      whose request waits for one of the plan's session slots waits inside the agent's time; RRSI Adapter's smoke check
      cannot see a Harbor trial's status under run.feedback: score; verdict fields a pipeline carries under names of its
      own are not removed under score or none). Minor findings, and a few that predate the PR outside its scope, were
      recorded and left.
    • Each code fix got a regression test that failed before it. From round 2 on, a verifier other than its fixer
      checked each fix; round 1's fixes and this session's own follow-up fixes were checked by the next round's review.
      Rounds 1 to 6's fixes are folded into the first three commits; round 7's and later ones are the fourth.
    • New high or medium findings of the two reviewers, by round: round 1 (on the original fb1f678), 1 high and 25
      medium; round 2, 8 medium; round 3, 4 medium; round 4, 1 high and 2 medium; round 5, 3 medium; round 6, 1 medium;
      round 7 (on 0483ae0's content), 1 medium. The fixes' verifiers and this session's own checks found 6 more
      medium ones in round 2 and 1 in round 6, all fixed.
    • Round 8, the post-PR audit on 0483ae0: Codex found 2 medium problems (an eval config's tree was not kept from
      the learner; a download of the ref from another registry was taken for the benchmark's tasks) and 1 minor one.
      This session's own audit found 13, five of them rated medium by at least one verifier: a task set with a task
      that declares MCP servers, skills or a Windows container; Harbor's own claude-code's web rule; its sign-in;
      Harness²'s stream score for a composed harness that delivered nothing; a checkpoint test a halt cut short, kept as
      done. All of them, round 7's too, were fixed in the fourth commit.
    • Round 9, on that fix (2389a10): Codex found 3 medium problems (an endpoint model took the benchmark file's
      price; a loaded config's Anthropic model passed the subscription-only refusal of Harbor's own claude-code;
      Harbor's mini-swe-agent trials skipped mini-swe-agent's own checks) and 2 low ones (a registry file's path in the
      lock; this description). This session's review found 2 medium (the startup refusal took a whole task set for one
      task Harbor cannot run on Hermit's agents; Harbor's own claude-code on a bound model was sent to
      /v1/v1/messages, which main does too) and 2 low ones. All are fixed in the fourth commit.
    • Round 10, this session's review of round 9's fixes: 1 low finding (a model's own key variable, which Harbor's
      claude-code reads, counted as no bill), fixed. A last read of every change since 2389a10, the fifth commit's
      too, found nothing new. No high or medium finding is open.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP

iamziyuzhao and others added 5 commits November 12, 2026 11:54
@iamziyuzhao @claude
Shared layers: a Harbor trial of any CLI is deployed, bound and read …
70340e6 
…as Hermit's own sessions are

What the Harbor runner needs of the layers every method shares, so that a trial in a task's container is one more
session of the CLI's Base Agent:

- The hosts deploy for a caller that serves the adapter's tools itself (`--no-serve-tools` on every host; Pi's and
  dsh's `prepare(serve_tools=False)` names no server, and the acting agents' workspace now passes the flag on, so a
  container session of Hermit's own runs gets no server row it cannot start) and take one task's context
  (`--context-file`).
- Every CLI Base Agent reads a stream its caller read elsewhere: `outcome()` with no stream file reads the stream's
  text as it reads the file, the reply included; a missing or empty stream is a session not served, never a TypeError.
- A Codex session whose sign-in or key the provider refused is `login_failed` (the run stops) if its turn did not
  complete, read from its own notices: never one it recovered from, what its log quotes of the session's tool calls,
  or an event's text (U+0085, U+2028 and U+2029 end no line of a stream); `unavailable` comes from its last failure.
- hermit.agent.base.trajectory, the AHE Debugger, Harness²'s bench and RRSI's render read a Harbor trial of any CLI
  (`agent/agent-<backend>.jsonl`, then Harbor's own output files, then its ATIF trajectory, which the Debugger takes
  before those files), and AHE's texts (EVOLVER.md's too) name that trial's files (the stream, hook audit and state
  under agent/; the record's instruction, as the Debugger's `task0i.md`; START.md's tree). Harness²'s bench takes a
  Claude Code trial's final message as the Base Agent reads its reply.
- A Harbor acting agent's model is bound and checked in the acting CLI's dialect (`agent (harbor, <cli>)`, an
  endpoint's too), the whole Binding spec kept, developer_role included, and its login is a contained session's
  (logins_of) unless `claude_bin` names this machine's Claude Code, whose version is then checked; the task set's
  benchmark file counts for both, as the runner merges it (relative to `normalize_config`'s `base`, else the working
  directory). hermit.harbor.clis is the table of those CLIs.

tests/selection.py counts the recorded OpenCode and dsh streams as shared: the methods' tests read them too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP
@iamziyuzhao @claude
Run modes: eval, split and stream, what the learner is shown of a ver…
8a7710e 
…dict, orders and checkpoint tests

How a run is done is the config's `run:` block, so that a benchmark plugs into each protocol by its split files and
nothing of its own:

- `mode: eval` is a config for `hermit eval` alone: no train split, Evolver, task-info or budget, and `hermit train`
  refuses it; `root` there is the template's own adapter. `split` trains, then tests the final adapter on
  `data.test`. `stream` scores each row once, by the adapter the earlier rows taught, before the learner sees it:
  one epoch, no shuffle, the final adapter the last step's, run_agent only on rows already reached (no validation
  row may be a train row); its prequential score (overall, by halves, by step) goes to stream_summary.json and the
  summary. An Evolver whose protocol scores its own steps gives that score (Evolver.step_scores, None for a row it
  leaves unscored, left out and counted; Harness²: the deliverable it chose without the labels), the rollouts' mean
  kept beside it.
- `feedback: full | score | none` is what the learner is shown of each verdict, in the learning material (a README
  per policy), run_agent's results and the logs handed to it (the verifier's output and result files gone under
  score and none); a stream must name it, and `hermit train` refuses (`hermit check` reports) one below what the
  Evolver reads: `Evolver.min_feedback`, `score` by default (`full` for AHE Adapter, `none` for the CLI Evolvers
  and Harness²), or `min_feedback_for(params)`. An Evolver that keeps the labels from its sessions itself
  (`hides_labels`: Harness²) gets the full records. The CLI Evolvers', Meta-Harness Adapter's and ASPIRE's prompts
  promise no more than the run shows.
- `orders` runs a stream once per seed or order file (a task's rows kept together), combined in streams.json, each
  order to train checked before the first trains; `checkpoints` test the commit after the step a row ends on a split
  of their own, with run.state's environment and run.score's scores, after training or as each is reached (outside
  the training budget's hours); one inside a step is refused before training. `hermit eval --checkpoints` runs those
  without results.
- A multi-step Harbor task's steps run in order: `data.shuffle` moves its rows as one, an order or split taken as
  given or an Evolver's `next_tasks` that names a step ahead of an earlier one is refused before any run takes it,
  and run_agent runs no training row that is a step.
- A run the pipeline says is not the agent's result (`RunOutput.extra["unscored"]`: a Harbor infra trial) has status
  infra: never judged, null in steps.jsonl, left out of the Trainer's and hermit eval's means and counted (a 0 to an
  Evolver that averages the records itself, as GEPA does); `eval.infra_retries` runs those rows again, a multi-step
  task's steps excepted, until a halt. `hermit eval --split` tests another split; each repeat of `eval.repeats` > 1
  hands the pipeline rows that name it (`repeat`); pass@k counts a Harbor task's rows as its attempts and a
  multi-step task's step (`<task>:<step>`) as a task; "solved" is the judge's `passed` when it is a bool (unknown
  when it is None), else a score of at least 1.
- An acting agent of Harbor's own takes no adapter: `hermit train` refuses it, `hermit check` reports it, and
  `hermit eval` runs it with `--adapter none` only. A `run:` or `eval:` left empty takes its defaults, and so does a
  key left empty or left out of a whole-section override (`eval={repeats: 3}`).
- The run's secrets include order files, checkpoint splits, the score hook and a Harbor runner's task and job
  directories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP
@iamziyuzhao @claude
Harbor: every CLI in the task's container from one launch plan, with …
0483ae0 
…the adapter's tools, context and notes

A Harbor trial ran only Codex well (and Claude Code through code of its own); the adapter's tools, a per-task context
and an adapter's notes never reached the container, and infra trials counted as zeros. Now:

- One HermitXXX class per CLI (hermit.harbor.clis: codex, claude, pi, opencode, dsh, mini-swe-agent) over one executor
  (harborside/hermit.py). Hermit's side renders each trial's session with the CLI's own Base Agent
  (hermit.harbor.launch: the home, the command, the environment; host paths made the container's) against the adapter
  deployed as `hermit deploy` lays it out, and harborside uploads and runs it as the task's agent user: the prompt on
  stdin, the stream in agent/agent-<backend>.jsonl, the deployment record always there (so an adapter's hooks run on
  Codex too, HERMIT_HOOK_LOGS set), a session still running at Harbor's timeout killed with its process group. The
  steps after the session still run past that timeout; a copy that failed is dropped, never kept in part. A
  `<cli>_bin` serves only at the asked-for version and when the agent user may execute it. The runner reads the stream
  back with the same Base Agent: reply, served, the plan's or the provider's refusal, the usage at
  `agent.params.price`, else at the backend's price.
- The adapter's tools are served from this machine on the Docker bridge's gateway (hermit.harbor.tools, every call
  logged); a per-task context is computed in the adapter sandbox (hermit.harbor.context); an adapter's notes come back
  after the session (memory/ in the log).
- A key or a token reaches the container by its variable's name only (`docker compose exec -e NAME`; Claude Code's
  subscription token on a descriptor), never in a command line, a file, the plan or a log; Codex's ChatGPT sign-in is
  a file of its home there (mode 600). Codex and Claude Code run on the user's subscription
  (`login: chatgpt | claude`) in the plan's slots and holds, each trial noting the plan's usage (so `MAX_USED` holds
  trials back): a refused session again after the window, one the provider could not serve after a pause unless it
  ran out of the agent's time, a lost sign-in never, none once another trial halted the run; the trial's usage is
  every run's. Codex's refreshed sign-in comes back and becomes the shared copy. A task whose agent network is not
  public gets the model's and tools' hosts allowlisted and web search off; `web_search` must be set.
- The answer-only agent (`cli: answer`) makes one model call through an endpoint the runner serves and writes the
  answer into the container (LAB-Bench's questions); a request still in flight when the trial ends is given up, its
  cost unknown, and on a model a CLI reaches on a login the trial waits for that plan first (a request the plan held
  past the agent's time is infra, `usage_limit`). A question or attachments it cannot read fail the launch.
- The judge scores the verifier's reward whenever the verifier gave one, a timeout or an agent error included
  (Harbor's own mean), says "passed" at `pass_reward` (null for a score scale) and leaves out what `verifier_redact`
  matches. Not the agent's result, so unscored (infra): a verifier whose tools never downloaded or that hit a full
  disk, harborside's launch failing (HermitLaunchError, the CLI's install included), a session that never started or
  whose tools never came up, a lost sign-in, a Claude Code, Pi or OpenCode session or an answer request the provider
  refused for the run's configuration before serving any (its key, model, bill or a setting it does not take:
  `model_rejected`, no halt). A trial out of the agent's time is the agent's even after a provider notice, and so is a
  request refused for its input; the adapter's context over the CLI's limit beside the image's own is the adapter's 0,
  and so is a session the model's safeguards ended.
- Startup refuses what every trial would: a login for another CLI, a login and a binding at once, a loopback model, a
  missing key or sign-in, a non-Docker environment for secrets by name, a plan the Base Agent cannot render, a
  `harbor` that is no executable file, mini-swe-agent without its package for a run that deploys adapters, the answer
  agent's attachments on a model a CLI reaches.
- `python -m hermit.harbor.dataset prepare` makes an example's data from its benchmark file's `dataset:` block (the
  registry's tasks, rows, splits, smoke rows, a patch, the images' sizes with `image_sizes: true`, a lock). A download
  is moved into place only once whole, and what is in its way is refused first; a relative `data:` is the benchmark
  file's folder's. The lock holds no path of this machine and hashes only a mode's executable bit (one written under a
  umask other than 022 needs writing again); an unreadable lock is refused unless `--force`.
- Also, from the Terminal-Bench 2.1 runs: a host failure (Docker's, the host's /tmp full) is infra and no trial starts
  while /tmp is short; the trial of a task whose name is longer than 32 characters is found; a `harbor run` past its
  wait budget is killed and its trial's compose projects, the verifier's too, taken down by name alone (no compose
  file or `COMPOSE_*` variable read). A stream's step runs its own plan, its verifier read as a trial's; each
  `hermit eval` repeat and each run gets streams of its own, stopped when the run ends unless a `hermit train` may
  resume them. The trial cache keys the launch, the whole row, the task's files, the model's price and the adapter
  as deployed; a cached trial costs nothing again.
- OfficeQA's Harbor judge reads the final message from HermitCodex's stream (`agent-codex.jsonl`): a task set built
  before needs building again.

Checked in real containers on this machine: Codex on the ChatGPT login (tasks with an open and a closed network, the
adapter's MCP tool, AHE's hooks, Continual Harness's notes, ASPIRE's template) and Claude Code on a subscription token
(the MCP tool, AHE's hooks); Pi, OpenCode, dsh and mini-swe-agent on a scripted model; the answer agent; a three-step
stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP
@iamziyuzhao @claude
Review fixes: the PR's known limitations and the post-PR audit's find…
7f02e95 
…ings

Round 7's findings, listed as known limitations when the PR was opened:
- A client of a CLI's chat endpoint counts what a session that failed or
  that the provider refused used, which the endpoint reports in its error
  body, each failed attempt's too (ChatClient count_error_usage, set by
  endpoint_spec; the answer agent's relay takes it).
- harborside keeps the session's own error when Harbor's timeout lands in
  the steps after it (a launch failure or a non-zero exit stays infra, not
  the agent's timeout).
- CODEX_RECORD takes a record with span prefixes (RUST_LOG=codex_core=info).
- A split, order, checkpoint split or --split file that lists a step but
  not every earlier step of its stream is refused before any run, and so
  is a next_tasks request naming a step whose earlier step has neither run
  in the epoch nor come earlier in the request.
- The benchmark file's price applies to its model however the config
  writes it.
- Attempts are rollouts x repeats (CHANGELOG, evaluate.py); the
  troubleshooting row of hermit_login_failed covers Codex.
- The trial cache's adapter digest hashes whether a file is executable,
  not the bits a umask sets.
- hermit check's probe neither takes a trial from the trial cache nor
  keeps one.
- CHANGELOG (Breaking): the runner's codex_login is gone (login: chatgpt),
  a table CLI's trial needs web_search, secrets by name run on local
  Docker only.

The post-PR audit's findings (on 0483ae0):
- An eval config's tree (trainer.tree_dir, with no template) is a run
  secret.
- dataset prepare records the tasks' source (the ref with repo,
  registry_url and registry_path) in the lock and in the download, never
  reuses another source's download, compares the lock's source, and keys
  the default data directory by the source.
- A task Harbor runs for none of Hermit's agents (MCP servers, a skills
  folder, a Windows container) is an infra rollout (unsupported_task),
  never handed to Harbor; the task set's other tasks run.
- Harbor's own claude-code keeps Harbor's per-task web rule unless
  web_search is false; CLAUDE_FORCE_OAUTH, or the subscription's token as
  its only sign-in, is refused at startup (CHANGELOG, Breaking).
- Harness²'s stream score is 0 for a composed harness that delivered
  nothing (None when every run of it was infra), never H0's.
- A checkpoint test a halt cut short is not kept as done: its summary goes
  to halted.json, and hermit eval --checkpoints runs it again.
- dsh's context limit counts the image's own file as dsh escapes it.
- Docs: Claude Code's own cost is kept on Anthropic's API. CHANGELOG:
  hermit eval --adapter root starting a tree that does not exist yet, the
  hosts' --no-serve-tools and --context-file, and which locks written
  before need writing again.

The review of those fixes:
- A model the config gives as an endpoint mapping keeps its own price (or
  none) beside a benchmark file that prices its own model.
- Harbor's own claude-code is given a bound model's Messages API root
  (Claude Code appends /v1/messages; it was sent to /v1/v1/messages), and
  a model at Anthropic's own API by name with its key's variable, so the
  subscription-only refusal also holds for a loaded config's model; a bill
  in acting_agent.env or the model's own key variable counts.
- Harbor's mini-swe-agent trials refuse what MiniSweAgent.run refuses (web
  search, a model class Hermit does not read): CliAgent.check_settings,
  called by run and by the launch plan's render, so at startup too.
- dataset.lock names a registry file of this machine by its file name; the
  download's own record keeps the path.
- eval.infra_retries does not rerun an unsupported_task row.

Each code fix has a regression test that fails with the fix undone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP
@iamziyuzhao @claude
Tests: the endpoint-price test runs where Harbor's venv is not installed
2ca476a 
test_an_endpoint_models_price_is_its_own_never_the_benchmark_files_model
builds the Harbor runner, whose startup checks that the harbor command is
an executable file. It now gives the runner a stand-in, as the runner's
own tests do (fake_harbor), so it passes on CI, which installs no Harbor
venv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP
@iamziyuzhao iamziyuzhao mentioned this pull request Nov 12, 2026
Open
9 of 10 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@iamziyuzhao
0