Repository navigation
Benchmarks: one task protocol (Harbor's format) for single tasks and streams - #56
Merged
Merged
Conversation
8 tasks done
A task set in Harbor's format (task.toml, environment/, tests/test.sh writing the reward) runs as the acting pipeline through the built-in `harbor` agent: one rollout is one Harbor trial, started, installed, verified and read back by `harbor run` in Harbor's own venv, with the adapter deployed into the container as `hermit deploy` lays it out (context, skills, roles, hooks, the adapter's copy under hermit/<id>/). `hermit.harbor.judge` reads the reward and gives the verifier's output as feedback. `python -m hermit.harbor.dataset` downloads a dataset, indexes task directories into tasks.jsonl rows (per task, per rollout, per step), writes splits and a sha256 lock. A task with [[steps]] is a stream: one environment through many steps. A driver (harborside/driver.py, Harbor's MultiStepTrial fed one step at a time over a Unix socket) keeps the environment between the Trainer's steps so a learner acts between them; steps of one stream run in order, streams run concurrently, the last or an aborted step stops the environment, and `harbor run` evaluates the whole stream with a frozen adapter. The container-side half (hermit/harbor/harborside: HermitCodex, a scripted agent for model-free smoke tests, the driver) imports harbor and the standard library only. OfficeQA gets a Harbor packaging of the SkillOpt setting (utils/harbor/build_tasks.py, the exact-match judge inside tests/, gepa/run_codex_harbor.yaml), checked with the scripted agent; no result is claimed for it. Tests: a fake harbor command and a fake driver (tests/test_harbor_runner.py), the Harbor-side agents and the driver's protocol under the Harbor venv (tests/test_harbor_harborside.py), and a real-Docker smoke of a one-step task and a three-step stream (tests/test_harbor_smoke.py, HERMIT_HARBOR_SMOKE=1). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The task-set format of Harbor is the one way a container benchmark enters Hermit, so the docs section, the rule in examples/README.md, the changelog entry and the module docstrings say "benchmarks" and "Harbor's format" instead of "Harbor benchmarks"; the agent keeps its name, harbor, as the engine that runs them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FcJCJYNyvHmXqKo4LE96nN
The rule in examples/README.md and the Benchmarks section say so; the examples that still own their loop are listed as work to move onto it, and the limits are named as pieces still to build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FcJCJYNyvHmXqKo4LE96nN
Import the acting Agent from hermit.agent.acting, read templates from hermit/adapter/templates, move the Harbor tests to tests/core/, and list hermit/harbor/ as a shared layer (hermit/agent/acting/agent.py imports it).
haizhongzheng
force-pushed
the
harbor-interface
branch
from
November 5, 2026 07:05
cd65901 to
3bfeb39
Compare
haizhongzheng
approved these changes
Nov 5, 2026
Member
There was a problem hiding this comment.
Rebased onto main 77ec08e and ported to the agent layers and grouped tests; full suite has no new failures. Merging as the base for the sandbox refactor.
Merged
5 of 6 tasks
8 of 11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Naming: the protocol is the benchmark's, the engine is Harbor's. The docs section is "Benchmarks" (
docs/framework/task.md#benchmarks), the rule inexamples/README.mdsays every container benchmark comes as a task set in Harbor's format, andagent.name: harbornames the engine, ascodexnames Codex.Benchmarks enter Hermit through one protocol, the task-set format of Harbor;
harboris the agent that runs them. Concretely:agent.name: harborruns any task set in Harbor's format (task.toml,environment/,tests/test.shwriting the reward) as the acting pipeline, for single tasks and for streams, instead of each example driving Docker in its own way.hermit/harbor/(host side):runner.HarborAgentturns one rollout into one Harbor trial (job config,harbor runin Harbor's own venv, the trial'sresult.jsonread back: status, failure cause, reward, tokens, a curated log without session transcripts; the disk gate, the per-image cleanup modes and theHALTsentinel of the Terminal-Bench bridge, made generic).datasetdownloads a registry dataset, indexes task directories intotasks.jsonlrows (per task, per rollout#k, per step<task>:<step>), writes splits and a sha256 lock.stagingdeploys the adapter for the container exactly ashermit deploydoes (context, skills, roles, the whole adapter underhermit/<id>/, hooks rewritten to run from that copy).judge.scorereads the reward and gives the verifier's output as feedback. A bound backend reference (model: fireworks/...) hands Harbor the endpoint and key variable; a plain name uses the ChatGPT login for Codex.hermit/harbor/harborside/(Harbor's venv, harbor + stdlib only):codex.HermitCodex, Harbor's Codex with the deployed workspace uploaded into the task's workdir;scripted.ScriptedAgent, a shell script as the agent, to check a task set's packaging without a model;driver, Harbor'sMultiStepTrialfed one step at a time over a Unix socket.[[steps]](one environment through many steps, each with its instruction and verifier) is run one step per rollout: the driver keeps the environment between the Trainer's steps so a learner acts between them, steps of one stream run in order (a step whose turn has not come waits; a wrong order isstream_order, a dead driverstream_lost), different streams run concurrently, the last or an aborted step stops the environment, andharbor runevaluates the whole stream with a frozen adapter. This is the shape a CL-Bench order will take.examples/officeqa/utils/harbor/build_tasks.pywrites one Harbor task per question in the SkillOpt setting (SkillOpt's task text as the instruction, the exact-match judge insidetests/, the corpus bind-mounted at/app/docs/root_1);gepa/run_codex_harbor.yamlruns the Codex harness that way. No result is claimed for it: it is packaging, checked with the scripted agent, not a run.docs/framework/task.md"Harbor benchmarks", the recipe's "Through Harbor",config-fields, the rule inexamples/README.md(a container benchmark comes as a Harbor task set; no example drives Docker itself), CHANGELOG.Limits and follow-ups
sandbox.profiles.rollout.enabled: false(a driver started inside a bubblewrap worker would die with it); the same unconfined rollout every Docker-driving agent has today. Apresetthat grants the rollout role the Docker socket is a follow-up.context_toolis refused (as the CLI agents refuse it); the adapter's MCP tools are not served in the container.exa 8000 mples/terminal_bench_2/swebench_verified(the AHE branch) ontohermit/harbor; CL-Bench as multi-step tasks (needs a host-side chat agent whose tools run in the container, and a clbench image); an OfficeQA parity run through Harbor.