Repository navigation
evolvers: add continual_harness (Continual Harness's label-free Refiner as a hermit Evolver) - #23
Draft
ChangyiYang wants to merge 6 commits into
Draft
ChangyiYang wants to merge 6 commits into
ChangyiYang wants to merge 6 commits into
Conversation
…ner as a hermit Evolver Ports the Refiner from Continual Harness (Karten et al. 2026; github.com/sethkarten/continual-harness) as a hermit Evolver, complementing the score-driven gepa/skillopt. Each step it reads the rollout trajectories, extracts failure signatures (tool-call errors, non-ok runs, the judge's feedback as a signal), and has the run's `llm` tool CRUD-rewrite the whole harness document — base policy (p) plus `## Sub-agents` (G), `## Skills` (K) and `## Memory` (M) sections — to fix them. It is label-free: needs: [], no run_agent scoring and no validation gate; it applies the refinement in place (only a check_spec guard), as CH's reset-free Refiner does. Self-contained: a new evolvers/continual_harness/ directory (evolver.yaml + evolver.py) the loader discovers by name (`evolver: name: continual_harness`); no changes to the loader, trainer, base or other evolvers. Verified: hermit check resolves and instantiates it; a synthetic step() refines ADAPTER.md on failure signals and leaves it unchanged when there are none. Also adds examples/officeqa/continual_harness/run.yaml showing how to run it on a prompt-only benchmark. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…efinement Adds a continual_harness adapter template whose learnable files mirror CH's harness state — policy.md (p), subagents.md (G), skills.md (K), memory.md (M) — rendered into ADAPTER.md, which is the agent's context (exports.context). Auto-discovered by hermit (new templates/ dir; no core edits). The evolver now does CH's four passes on that template: one CRUD pass per component (policy/sub-agents/skills/memory) via the run's llm tool, then re-renders ADAPTER.md from the parts. On a prompt-only adapter it falls back to the single holistic rewrite. Still label-free, with a check_spec guard that reverts a rewrite that fails to deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add interface.py to the continual_harness template so the harness is not only text the agent reads but things it invokes, using hermit's own export hooks and no other template's code: - exports.tools `harness(name, args)` — the pull channel: fetch skills K / memory M / sub-agents G / a list overview on demand (optional query filter). - exports.agents — each `## name` section of subagents.md becomes a delegable sub-agent G. interface.py reads o 8000 nly its own adapter files (Path(__file__).parent) and is marked fixed (code, never evolved). The Refiner's subagents/skills/memory intents now ask for `## <name>` sections so what it writes parses into callable entries. README documents push/pull/sub-agents. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s sharp A full OfficeQA direct-chat run surfaced that the label-free Refiner grows ADAPTER.md monotonically (887 -> 6.3k -> 10.8k -> ... -> 35.5k chars over six steps), then saturates the template's byte cap: every later rewrite exceeds it and is reverted by the check_spec guard, so the last ~10 steps are wasted and the oversized context makes every rollout slower and costlier. Quality still improved (0.484 -> 0.578, matching SkillOpt), but the growth is pure waste. _refine now reads the byte cap for the file it edits (structure.learnable max_bytes) and tells the Refiner the soft (85%) and hard limits, asking it to consolidate and prune rather than append; if a draft still exceeds the cap it gets one compaction pass. No manifest / no cap -> unchanged behaviour. In a re-run the harness grows far slower and stays well under the cap (12.8k at the step where the baseline had already saturated at 35.5k), so every step lands. Also add examples/officeqa/continual_harness/smoke.yaml (check -> train -> eval end-to-end on 4+4 questions), mirroring the gepa/skillopt smoke configs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add smoke_codex.yaml and run_codex.yaml for the continual_harness example: the direct-chat configs with agent.params.harness: codex, the rollout profile disabled (each Codex session is its own container), and no budget — mirroring the gepa/skillopt Codex configs. Validated on OfficeQA Full (172 test x3): the Continual Harness Evolver in the Codex harness improves accuracy 0.552 -> 0.589 (label-free), on par with GEPA (0.595), with the evolved ADAPTER.md staying lean (~11 KB) under the byte budget. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nt names, ADAPTER.md cap) + structured example
Addresses the must-fix items from the PR review, all verified with tests and a
real structured-template smoke run on OfficeQA:
- H1: _extract_block now takes the OUTERMOST fence (first opening -> last
closing) instead of the last two, so a refined document that itself contains
a nested ``` block (e.g. a skill with a shell example) is kept whole instead
of being truncated to the fragment after the inner fence. This path is used
by every prompt-only (flat) run.
- H3: interface.py _sections is now fenced-code aware — a `#` line inside a ```
block (a shell comment) is no longer parsed as a heading. _sections also
records the heading level.
- H2: interface.py agent_definitions slugifies each `## name` heading to the
`[a-z][a-z0-9-]{0,63}` a host requires (hosts raise on invalid role names,
and check_spec does not catch it), and only level-2 `##` sections become
sub-agents, so a `# Title` or leading text is not turned into a spurious
sub-agent. The evolver's sub-agent intent now asks for lowercase-hyphen slugs.
- M1: the ADAPTER.md byte cap is raised 80000 -> 200000 (>= the sum of the four
part caps). It is a derived artifact (policy + G + K + M); the old cap was
below the sum, so a grown harness failed check_spec and every step reverted,
silently freezing the Refiner.
- M2: add smoke_structured.yaml — a committed config on template
continual-harness@1.0 (no start_from -> the template seed) so the structured
4-pass CRUD, ADAPTER.md render, and the interface exports are exercised
end to end (previously only prompt-only configs were committed).
Structured smoke (OfficeQA, direct): check ok, both train steps run the 4-pass
refine with check passing, eval completes; the evolved sub-agents parse to valid
slugs (number-lookup, unit-normalizer) with no spurious entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
continual_harness— a new hermit Evolver that ports the Refiner from Continual Harness (Karten et al. 2026). It complements the score-drivengepa/skillopt: it is label-free.Each training step it:
learning_material/current/tasks/*.json);llmtool CRUD-rewrite the whole harness document — base policy (p) plus## Sub-agents(G),## Skills(K) and## Memory(M) sections — to fix the failures, codify what worked, and drop what's stale;needs: [], norun_agentscoring, no validation gate (only acheck_specguard so a malformed rewrite can't break later rollouts), as CH's reset-free Refiner does.That label-free, failure-signature-driven, whole-harness CRUD is the trait that distinguishes it from
gepa/skillopt(which need a judge score + validation gate). On a prompt-only adapter, G/K/M live as sections ofADAPTER.md(where a text-prompt agent reads them anyway).Boundary
Self-contained: only a new
hermit/algorithms/evolvers/continual_harness/directory (evolver.yaml+evolver.py). The loader discovers it by name (evolver: name: continual_harness) — no changes to the loader, trainer, baseEvolver, or any other evolver. Plus one additive demo config,examples/officeqa/continual_harness/run.yaml.Verified
hermit checkresolves + instantiates it:[ok] evolver — continual_harness (evolver.py:ContinualHarnessEvolver).step(): refinesADAPTER.mdfrom a tool-failure + judge feedback (onellmcall +check_specguard), and leaves the adapter unchanged when there are no failure signatures.Status
Draft — evolver built + unit-verified; real-run before/after is next (officeqa direct harness for a functional pass; a tool-heavy task — e.g. clbench_database or the Pokémon env — would exercise CH's failure-reflection best, since officeqa is label-rich and its failures are mostly wrong-numbers).
🤖 Generated with Claude Code