8000 Examples: five Harbor benchmarks with baselines and their papers' methods by iamziyuzhao · Pull Request #106 · Infini-AI-Lab/hermit · GitHub
[go: up one dir, main page]
Skip to content

Examples: five Harbor benchmarks with baselines and their papers' methods - #106

Open
iamziyuzhao wants to merge 3 commits into
harbor-foundationfrom
harbor-benchmark-examples
Open

Examples: five Harbor benchmarks with baselines and their papers' methods#106
iamziyuzhao wants to merge 3 commits into
harbor-foundationfrom
harbor-benchmark-examples

Conversation

@iamziyuzhao
@iamziyuzhao iamziyuzhao commented Nov 12, 2026 •
edited
Loading
Copy link
Copy Markdown
Collaborator

Change

#105, this PR's base, makes a Harbor task set the one way a benchmark enters Hermit, but packages no benchmark on it.
This PR adds five, each ready to hand to someone with a Docker host: the registry's task set in a
utils/benchmark.yaml (locked in utils/dataset.lock), a baseline config, and the method whose paper's protocol fits
the benchmark, in that protocol. Nothing under hermit/ changes. The way in is
examples/README.md#harbor-benchmarks: what a machine needs (Docker,
bubblewrap, Harbor 0.25.0 in its own venv, the CLI's sign-in), the commands, and a run sheet of every run with its
trials, each of which can go to another person.

Folder Tasks Baseline Method, in its paper's protocol
terminal_bench_2_0 89 terminal tasks, 1 h each the CLI with no adapter, each task's rollouts #3 and #4 (178 trials) AHE Adapter, the AHE paper's own protocol on this benchmark (ten iterations over every task twice, the best kept); its final adapter tested on #3 and #4
swebench_verified 500 issues no adapter, each issue once nothing trains: Terminal-Bench 2.0's final AHE adapter frozen, against AHE's seed, as the AHE and RRSI papers test a harness on SWE-bench Verified
algotune 154 speedup tasks no adapter, SkillOpt's 108 test tasks SkillOpt, its paper's protocol (split 2:1:7 with seed 42, 4 epochs, batch 40, minibatch 8, learning rate 4 to 2, strict gate, slow update, meta skill)
terminal_bench_science 70 science workflows, 8 h each no adapter, rollouts #3 and #4 (140) Terminal-Bench 2.0's AHE adapter frozen; AHE in Terminal-Bench 2.0's protocol
labbench 181 FigQA questions no adapter, SkillOpt's 127 test questions (and the 102 of them on a figure no training question has) SkillOpt, as for AlgoTune

Every config comes twice: Codex with GPT-5.6-Luna on a ChatGPT login and Claude Code with Claude Haiku 5.5 on a Claude
subscription, both at medium effort, acting and learning. Each README says why its method, with the paper's numbers,
compares its protocol with the paper's row by row, and has an empty results table.

Decisions:

  • The baseline is the same acting agent (CLI, model, login, effort) with no adapter, hermit eval --adapter none,
    on exactly the rows the method's final adapter is tested on, in the same trial cache; the method's starting point
    (--adapter root: AHE's seed, SkillOpt's empty skill) is tested on them too, so that what the start gives and what
    the search adds are read apart. tests/core/test_harbor_examples.py holds the configs to it.
  • Terminal-Bench 2.0's protocol. The AHE paper evolves and reports on all 89 tasks (k = 2 rollouts a task, ten
    iterations, the best by pass@1: in sample, its §4.1). The rows are <task>#1 to #4: the training steps run feat(tasks): package OfficeQA Pro paired GLM evaluation #1 and
    feat(evolvers): GEPA and SkillOpt as built-in Evolvers, with the SkillOpt paper's OfficeQA setting #2, every adapter is tested on feat(tasks/OfficeQA): OfficeQA Pro as a harnessadapter Environment (stacked on #2) #3 and examples: a folder per method, tasks shared under examples/tasks (old folders kept) #4, which no step ran, so the trial cache never hands a test a training trial;
    the paper's own in-sample number is the run's best_ever.json. An hour a task, as the paper (the tasks' own limits
    are 600-12,000 s). The rollback is the port's (rollback: auto: an edit the next step judges HARMFUL), where the
    paper's text reverts the edits whose predicted fixes do not materialize; the README's protocol table says so.
  • SWE-bench Verified keeps its instructions as Harbor publishes them: the issue as written, with no request to
    change /testbed. With no adapter Codex answered psf__requests-1142 in text and changed nothing (score 0); with
    AHE's seed, whose ADAPTER.md says to edit files and finish the work, it fixed it. Kept as published, the numbers
    compare with Harbor's; the seed is the arm the transfer is read against (its README, Known limits).
  • AlgoTune scores a trial 1 - 1/speedup (utils/judge.py): the mean m of those scores is 1 - 1/H for AlgoTune's
    harmonic mean H, so the Trainer's, SkillOpt's gate's and hermit eval's means move with AlgoTune's own score.
    SkillOpt's gate is its soft metric (a speedup has no pass); the benchmark file gives the containers 64 GB (the
    reference's timing was killed at the tasks' 16 GB), thread pools of the task's 8 CPUs, no core files, and the
    verifier 7,200 s, a verifier timeout being the agent's result (verifier_timeout: agent). A success, by which
    SkillOpt's analysts and its slow update sort the rollouts, is a solver faster than the reference, as the verifier's
    own test has it (success_score: 0.000001); at SkillOpt's default of 1.0 no speedup would count, and every rollout
    went to the failure analyst.
  • LAB-Bench FigQA's download is patched (utils/patches.py, run by prepare on every download and recorded in the
    lock): as published, each container saves the shuffle's seed in /app/.letter_mapping.json, where the agent reads
    it, and the right choice is first before the shuffle, so replaying the shuffle gives the answer, in all 181 tasks.
    The patch drops the seed (the verifier reads only the letters), and removes the verifier's fallback that runs
    /app/setup.py, a file the agent can rewrite, when the mapping is missing (the mapping is always written before
    the agent starts: a missing one now scores 0); Harbor's oracle still scores 1. It changes no task unless it knows
    every file (refused otherwise). The split is SkillOpt's, by question, and FigQA asks up to three questions about
    one figure: 25 of the 127 test questions share a figure with a training question, whose verifier output the
    learner reads, so utils/rows/test_new_figures.txt lists the other 102 (utils/figures.py) for a second score
    from the same trials.
  • SkillOpt's analysts get no Codex trace summary (codex_trace_to_optimizer: false, in both profiles' configs):
    the summary reads SkillOpt's own harness (task.md, SKILL.md, <answer>), and told the analyst that every
    Harbor rollout, right or wrong, gave no answer. SkillOpt's own Codex runs have it on; the protocol tables list it.
  • Smokes run each method on the benchmark's two smoke rows (<task>#0, which no other run takes) with their own
    trial cache; the AHE smokes set target_pass_rate: 1.01 so that the Debugger and the Evolve Agent run when both
    tasks pass, and the AlgoTune and Terminal-Bench-Science smokes cap a session at 1,800 s.

Files: the five folders under examples/; examples/README.md (the Harbor benchmarks section); CHANGELOG.md;
tests/core/test_harbor_examples.py; examples/algotune/utils/tests/test_judge.py;
examples/labbench/utils/tests/test_patches.py, test_figures.py.

This PR is stacked on #105 (base harbor-foundation); once #105 is merged it is retargeted to main.

Validation

All on one Linux host (Docker 29, bubblewrap, Harbor 0.25.0), on the branch's head.

  • Prepare, each benchmark from the registry: python -m hermit.harbor.dataset prepare examples/<folder>/utils/benchmark.yaml
    wrote the task links, tasks.jsonl, the splits and the locks committed here (LAB-Bench's lock records the patch:
    setup.py: 181 patched, 0 already patched; test.sh: 181 patched, 0 already patched, on a fresh download; the
    machine's earlier download, reused, matched it).

  • Oracle (Harbor's reference solutions, acting_agent: {name: oracle}, the two smoke rows): Terminal-Bench 2.0 1.0,
    SWE-bench Verified 1.0, Terminal-Bench-Science 1.0, LAB-Bench 1.0 (with both patches, re-locked from a fresh
    download), AlgoTune speedup 1x (the reference solver: score 0, the timing verifier ran).

  • Smokes (2 rows each; the scores say the pipeline ran, nothing about the methods):

    Benchmark Baseline (no adapter): Codex / Claude Code Method: Codex / Claude Code
    Terminal-Bench 2.0 0.5 / 1.0 AHE: 1.0 / 1.0, the Debugger and the Evolve Agent ran (450K / 840K tokens) and changed nothing, both tasks having passed
    SWE-bench Verified 0.5 / 1.0 (Codex answered psf__requests-1142 in text) transfer, the git archive export of Terminal-Bench 2.0's smoke: 1.0 / 1.0; AHE's seed as hermit/adapter/templates/ahe, Codex: 1.0
    AlgoTune H 1.56x (1 task of 2 sped up) / H 2.36x (2 of 2) SkillOpt, after the audit's fixes: H 1.50x / H 2.07x, both steps' skills accepted in both (validation 0.550, 0.579 / 0.639, 0.683), the success analyst run on the sped-up rollouts (before them: no rollout a success, H 1.54x / 2.76x)
    Terminal-Bench-Science 0 / 0 (one session ran out of the smoke's 1,800 s and was scored) AHE: 0 / 0.5; both steps at pass@1 0, so the final adapter is the seed; the learners changed ADAPTER.md, AGENTS.md and middleware.py (Claude Code: and added a skill); transfer, Codex: 0
    LAB-Bench FigQA 0.5 / 0.5; after the verifier patch, 1.0 / 0.5 SkillOpt, after the audit's fixes: 1.0 / 1.0 (Codex: the empty skill already 1.0 on validation, both candidates rejected; Claude Code: step 1's skill accepted); the analysts' prompts carry no Codex trace summary (2 of the 4 calls did before)

    hermit eval --adapter root on LAB-Bench's smoke (Codex) took both trials from the trial cache (the smoke tests on
    its training rows). No smoke had an infra trial or an error.

    hermit eval --adapter final --split on LAB-Bench's smoke ran its two trials again: once hermit train has tested
    the final adapter, its card holds the results (a new commit, the tag moved), so the trial cache no longer matches it
    (Harbor foundation: every CLI in a task's container, and eval, split and stream runs #105's behaviour, left as it is). The new-figure score is therefore read from each eval's results
    (python examples/labbench/utils/figures.py <eval dir>..., b6cf47b), not from a second eval. The adapter export
    (mkdir of a new folder, then git archive ... | tar -x -C) ran on Terminal-Bench 2.0's smoke tree, and mkdir
    refused the folder a second time.

  • Tests. The new tests (25 in the two examples' files, 58 in tests/core/test_harbor_examples.py, about a second
    in all) pass on b6cf47b. test_harbor_examples.py fails on each of these broken copies: a test row feat(evolvers): GEPA and SkillOpt as built-in Evolvers, with the SkillOpt paper's OfficeQA setting #2, a swapped
    split, a method config on other rows, another cache, another model; and, since the audit, Terminal-Bench 2.0's AHE
    trained on all four rollouts, a SkillOpt validation on the test split, a sandbox off or not enforced, a learner not
    sandboxed, web search on in a benchmark file or a config, the trace summary on, no success_score or one at 1x, a
    smoke row in the test, a new-figure row outside the test (12 copies). Every config also loads from an unprepared
    copy without skillopt, as in CI. On dcc8915, the tests CI selects (pytest --changed, 1,534): 1,469 passed, 61
    skipped, 4 failed; the 4 fail the same way on Harbor foundation: every CLI in a task's container, and eval, split and stream runs #105's head 2ca476a on this machine (the clbench parity tests read
    another metaworld checkout here; a venv layout test), and Harbor foundation: every CLI in a task's container, and eval, split and stream runs #105's CI passed them. On c780fde the whole suite in 16
    shards gave 1,938 passed, 84 skipped and the same 4 failed, and CI passed.

  • Secret scan of the committed files, of every run record above and of the smokes' trial folders, for the two
    Claude tokens and the ChatGPT sign-in's tokens, once every run had ended: none found (a live session's own
    auth.json, mode 600, exists only while it runs).

Not verified: none of the full runs (the results tables are empty; examples/README.md lists them); an API key in
place of a login, and the CLIs other than Codex and Claude Code; another host than this one; SWE-bench Verified's 500
images on a small Docker disk (they accumulate in Docker's build cache; the disk gate holds trials below 25 GB free).

Review

  • Tests and documentation cover the affected behavior, where applicable. Tests above; every folder has its README,
    examples/README.md the shared section, CHANGELOG the entry.
  • Commands, defaults, and public interfaces are consistent. No interface changes. Every command form the READMEs
    give ran here, on the smoke rows: prepare, hermit eval --adapter none with --split examples/<folder>/utils/splits/smoke.txt, hermit eval --adapter root, --adapter <a directory> (the git archive export of Terminal-Bench 2.0's smoke, and hermit/adapter/templates/ahe), hermit train of every smoke
    config, the adapter export, the oracle's command, utils/figures.py on an eval's folder; the long runs' configs
    only load (Config.load).
  • Shared logs and examples contain no credentials or private data. The secret scan above; runs/ and the task
    links are git-ignored.
  • Documentation-only: not applicable.

General checklist:

  • Core changes. None: no file under hermit/ changes.

  • Implementation correctness and failures. The configs go through the real entry points (hermit train,
    hermit eval) in the smokes, with their failure paths: the disk gate held trials at 24.7 GB free and released them;
    infra trials are retried (eval.infra_retries: 2); a verifier timeout on AlgoTune is the agent's result.

  • Real execution paths and tests. The smokes are the CLIs in Harbor's containers on real tasks, learners
    included; the tests read the committed files and load every config with Config.load, prepared or not.

  • Documentation. Five READMEs, five task-info/README.md (what the learners read: the task, the CLI, how the
    adapter is deployed; no answer or task-specific guidance), examples/README.md, CHANGELOG.

  • This description. Read against the files and the run records.

  • Independent review (post-PR audit). An agent that did not write the PR reviewed c780fde against the four
    checklists, the framework code the configs rely on and the papers' texts. No P1. Fixed in dcc8915 (and b6cf47b):

    • P2: AlgoTune's SkillOpt sorted every rollout as a failure (no speedup's score reaches the default success_score
      of 1.0): now a solver faster than the reference is a success.
    • P2: SkillOpt's Codex trace summary, written for SkillOpt's own harness, told the analyst that every Harbor
      rollout gave no answer: off in the SkillOpt configs.
    • P2: the tests held neither the training and validation rows apart from the test's nor the sandbox and web
      search, and 26 of their 40 skipped in CI: now they do, with no skip.
    • P2 (a question): LAB-Bench's test questions on a training question's figure; the 102 others are listed for a
      second score.
    • P3: LAB-Bench's verifier fallback to a script the agent can rewrite (patched, re-locked); the SWE oracle command
      writing over the baseline's results (its own run name); the adapter export mixing folders and covering Codex
      only (a fresh git-ignored folder per profile); claims against the papers (Meta-Harness's leaderboard figures,
      AHE's rollback, seed and efforts, RRSI's trials, GEPA's splits, NPUEval's score, SkillOpt's meta skill, SWE's
      runs per task); the SkillOpt trial counts (at most 364 and 408); Terminal-Bench 2.0's image pulls, the infra
      count, the learners' tokens and the senders' paths in the records (documented).

    Left as they are: the AHE rollback at the port's reading (HARMFUL only; rollback_verdicts: [HARMFUL, INEFFECTIVE]
    follows the paper's text), stated in Terminal-Bench 2.0's protocol table; the Codex AHE configs' allowed_hosts,
    which the trainer also adds for a ChatGPT login (as OfficeQA's AHE configs have it); the SkillOpt Claude learner's
    plan wait at its 12-hour default. Remaining: none of the full runs is done.

Test checklist: the tests are in tests/core/ (they read five examples' files: core runs for every change) and in
the examples' own utils/tests/, named as behaviours, each file opening with what it covers; they write only to
tmp_path, use no network, sign-in or CLI, and none skips: a config loads without its benchmark's download.

Method checklist: the templates (ahe@1.0, prompt-only@1.0) and Evolvers (ahe_adapter, skillopt) are the
existing ones, as #105 runs them; the protocols' differences from the papers are each README's protocol table (model,
the acting agent being a CLI the adapter is deployed into, concurrency 6 against the AHE paper's 96, no Explore Agent,
the port's rollback, SkillOpt's starting skill a placeholder with no instructions and its analysts without the Codex
trace summary); the learners see the training rows' trajectories and the task-info only, the judge and the
test rows never; every config keeps the sandbox on (sandbox.enabled: true, enforce at its default required) and
web_search: false for the acting agent and the learners. No experiment is reported: the smokes check the pipeline.

Sandbox checklist: every process runs through #105's paths (the learners in the evolver sandbox, the acting agent in
the task's container); this PR adds none.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP

iamziyuzhao and others added 3 commits November 12, 2026 18:47
@iamziyuzhao @claude
Examples: five Harbor benchmarks with baselines and their papers' met…
c780fde 
…hods

Terminal-Bench 2.0, SWE-bench Verified, AlgoTune, Terminal-Bench-Science and LAB-Bench FigQA through Harbor, each a
benchmark file (the registry's task set, locked), a baseline per profile (Codex with GPT-5.6-Luna on a ChatGPT login,
Claude Code with Claude Haiku 5.5 on a Claude subscription) and the method whose paper's protocol fits it: the AHE
Adapter on Terminal-Bench 2.0 in the AHE paper's in-sample protocol, trained on each task's rollouts #1 and #2 and
tested on #3 and #4; its final adapter, frozen, on SWE-bench Verified and Terminal-Bench-Science against AHE's seed;
SkillOpt in its paper's protocol on AlgoTune, a trial scored 1 - 1/speedup so that the mean gives AlgoTune's harmonic
mean, and on LAB-Bench FigQA, whose download is patched: the published tasks save the shuffle's seed where the agent
reads it, which gives the answer away. examples/README.md says what a machine needs and lists every run with its
trials, to be handed out; tests hold the protocols to the committed files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP
@iamziyuzhao @claude
Harbor benchmarks: the audit's fixes (SkillOpt's success and trace, L…
dcc8915 
…AB's verifier and figures, tests)

The post-PR audit of #106 found no P1; its three P2s and its P2 question, fixed:

- AlgoTune's SkillOpt counted every rollout a failure: a trial's score, 1 - 1/speedup, never reaches SkillOpt's
  success_score of 1.0, so its analysts and slow update saw failures only. success_score is now 0.000001: a solver
  faster than the reference, as the verifier's own test has it.
- SkillOpt's Codex trace summary, on by default, describes SkillOpt's own harness (task.md, SKILL.md, <answer>) and
  told the analyst that every Harbor rollout gave no answer: codex_trace_to_optimizer: false in the SkillOpt configs.
- tests/core/test_harbor_examples.py now holds the configs to training and validation rows disjoint from the test's,
  the smoke's rows apart from the run's, the learners sandboxed, web search off, and SkillOpt's success and trace
  settings, and loads every config with no skip (neither tasks.jsonl nor skillopt is needed): each of 12 broken
  copies fails it.
- LAB-Bench's split is by question, and FigQA asks up to three questions about a figure: 25 of the 127 test questions
  share a figure with a training question. utils/rows/test_new_figures.txt (written by utils/figures.py) lists the
  other 102 for a second score from the same trials.

And its P3s. LAB-Bench's verifier ran /app/setup.py, the agent's to rewrite, when the mapping was missing: the patch
(now patches.py:patch_tasks) removes that fallback too, re-locked from a fresh download. The SWE oracle command gets
its own run name. The final adapter's export goes to a fresh git-ignored folder per profile. The claims the audit
checked against the papers (Meta-Harness's leaderboard figures, AHE's rollback and efforts, RRSI's trials, GEPA's
splits, NPUEval's score, SkillOpt's meta skill) and the trial counts are corrected. The READMEs say how
Terminal-Bench 2.0's images are pulled, how infra trials count, and what the learners read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP
@iamziyuzhao @claude
LAB-Bench: the new-figure score from an eval's results, not a second …
b6cf47b 
…eval

`hermit eval --adapter final` after `hermit train` misses the trial cache: the training run's test rewrites the final
adapter's card with its results (a new commit, the tag moved), so the eval the README gave would run the 102
questions again (as it did on the smoke's two rows). utils/figures.py now takes the score on the 102 rows from an
eval's results.jsonl, as `hermit eval` takes a mean (infra trials left out and counted, rows it did not run
counted), and the README gives that command for the three arms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FcJCJanVqQuJyChDevJNYP

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@iamziyuzhao
0