Repository navigation
Add OPRO: the opro Evolver on the prompt-only template and its OfficeQA example - #81
Open
ruiling-smartbear wants to merge 5 commits into
Open
ruiling-smartbear wants to merge 5 commits into
ruiling-smartbear wants to merge 5 commits into
Conversation
…QA example - hermit/agent/evolver/methods/opro/: OPRO (Yang et al., ICLR 2024; google-deepmind/opro at a76bdce), one OPRO step per training step, the authors' meta-prompt verbatim and their reply parsing and md5 de-duplication ported; the optimizer is the run's llm tool, candidates are scored with run_agent on every training task; final is the best on training or the best validation mean of the top top_k. - tests/methods/opro/, with its group in tests/selection.py and tests/README.md. - examples/officeqa/opro/: smoke and run configs in the GPT-5.6-Luna setting, and adapter/ with the measured run's final instruction and its card. - docs/methods/opro.md (flow chart, differences from the authors' code, what it is not); OfficeQA recipe section, OfficeQA README row, methods index, README method lists, CHANGELOG.
…differences table From the post-PR audit: - setup refuses a run without llm or with an unknown select, before the first rollouts; the step keeps its check for a resume. A Trainer test holds it. - The method page's table adds the Trainer's extra pass per step, the exemplar draw, polish_sentence on a whole document, where reruns are recorded and the setup check; "What it is not" counts the extra pass. - The recipe gives both evaluation times and no longer names the container image.
ruiling-smartbear
force-pushed
the
opro
branch
from
November 6, 2026 04:22
93d57a6 to
59ca92c
Compare
ruiling-smartbear
marked this pull request as ready for review
November 6, 2026 05:12
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
OPRO (Yang et al., ICLR 2024; google-deepmind/opro at
a76bdce) as an Evolver on the existingprompt-onlytemplate, with an OfficeQA example in the GPT-5.6-Luna setting. Each step, the optimizer (the run'sllm) reads the instructions scored so far in ascending order and three training problems, each with the agent's answer when judged right or the judge's feedback otherwise, and writesn_candidatesnew instructions; each is scored withrun_agenton every training task.finalizereturns the best on training, or withselect: validationthe best validation mean of thetop_k.Flow chart of one step
flowchart LR R[step rollouts:<br/>current instruction<br/>on all training tasks] --> H[history:<br/>instructions with<br/>training scores] H --> M[meta-prompt:<br/>best old ones, ascending<br/>+ 3 training problems] M --> L[llm writes<br/>n_candidates<br/>instructions] L --> D{new and passes<br/>template check?} D -- no --> S[skipped] D -- yes --> A[run_agent on all<br/>training tasks] A --> H H --> F[finalize: best on train,<br/>or best val of top_k]hermit/agent/evolver/methods/opro/): the authors' meta-prompt verbatim; their reply parsing and md5 de-duplication ported; the NOTICE names the source.examples/officeqa/opro/): the smoke and run configs, andadapter/with the measured run's final instruction and its card.tests/methods/opro/), registered as theoprogroup.docs/methods/opro.md, the recipe's OPRO section, the method lists, the OfficeQA row and CHANGELOG.Core and application-side changes: none.
Differences from the authors' code (the method page's table)
ADAPTER.md; the authors'polish_sentenceruns on the whole document (removes**)select: train, or the best validation mean of the toptop_kllm(a Codex session applies no temperature)ADAPTER.mdas the agent's instructions, then the taskinstructions_only, other exemplar criteria, the temperature schedule, few-shot evaluation of new instructions, the periodic eval-split scores, the GSM8K digit filterEvery difference is acceptable for this reproduction; none is open.
Validation
python -m pytest tests/methods/opro tests/core/test_selection.py tests/core/test_packaging.py: 43 passed. They run every stage with a scripted model, check the meta-prompt against a stored output of the authors'gen_meta_prompt, and cover the failure and budget paths and the whole loop through theTrainer.mainon this machine (no bubblewrap,/proc);test_an_interrupted_caller_stops_the_sessionfailed once under load and passed alone five times out of five.final0.436 against the initial skill's 0.355 on the 172 test questions, +8.1 [+1.7, +14.5] paired (24 questions better, 10 worse).Experiment: protocol, settings, result and records
Protocol: train / test. The Evolver learns on the 50 training questions, selects on the 24 validation questions, and the frozen adapter is evaluated on the 172 test questions (exact match). The optimizer sees the judge's feedback, which names the expected answer.
Settings:
gpt-5.6-lunaat medium effort acting and as the optimizer on a ChatGPT login (one Codex session per optimizer call); the Codex agent, a container per session (3,600 s), four side by side; 3 steps of 4 candidates; one pass per evaluation; SkillOpt's split at79124b3(shuffle_seed: 42). One training, on the method code ofe7d668f(later commits move two checks intosetupand touch docs and tests). Training 2.4 h and 124.5 M tokens; evaluations 30 min (final) and 57 min (the initial skill).final(candidate 2 of 13, validation 0.625)Paired per question, 95% bootstrap; 0 errors in either evaluation. The gain is the answer's form: long or Markdown answers fall from 55 to 13, while answers containing the reference value are 118 and 112. The
finalinstruction and its card are inexamples/officeqa/opro/adapter/; the run records are local (examples/officeqa/opro/runs/luna-opro).A first evaluation of
final(0.419) lost 10 answers to the shared-layer fault fixed in #73; it was discarded and both evaluations were run again on that fix. The training met the fault once, and that rollout was run again.Checked in the records: the 9 exemplars' right / wrong labels match the judge; a fresh judge agrees with all 150 training answers and both evaluations; no validation or test question appears in a meta-prompt; every evaluation session deployed the selected instruction.
Left out and not verified: the table's "Not ported" row, the no-adapter baseline, the GLM 5.3 Flash column, repeated trainings, and Linux.
Review
tests/methods/opro/, selected bypytest --changed, stand-ins only for the model and the agent, no network or CLI).evolver.yamlparameter is used; a run withoutllmor with an unknownselectstops insetup, before its first rollouts.adapter/holds the instruction, its card and the template's files, with no local paths.Trainer.trainruns the bootstrap outside its budgettry.Trainer; the measured run went throughhermit trainandhermit eval, which read the adapterfinalizereturned.prompt-onlytemplate; a newoproEvolver; the differences and their kinds in the table under Change.finalize, the test split only byhermit eval; the judge runs outside the session.ask(ctx.agent)(onecodexBase Agent session per call), rollouts throughrun_agent; no method-specific execution, timeout or usage code.e7d668f) confirmed the verbatim meta-prompt, the numbers, the exemplar labels and the splits, and its findings are fixed in59ca92c(left open: the 55→13 answer-form count is a heuristic reading; its own gave 62→21); the second (59ca92c) found no defects, and its two wording points are fixed in938e554.