Repository navigation
evals: record each prompt, skill and tool version, and name what changed - #16
Merged
Merged
Conversation
Every saved run now carries a short hash of each part the model is shown: the system, memory and summary prompts, every skill (description, tools, body) and every tool definition (description, parameters). They are read from a sandbox runtime built like run_case's, with a stand-in model client, so no API key is needed. A regression report lists the parts whose text changed and, for each case that got worse, the changed parts it used in either run: the skills it read, the tools it called, the prompts sent with every request, and the summary prompt only after compaction. A skill the baseline read and the new run skipped counts. This is where to look first, not a verdict. build_runtime takes an optional llm, which run_case now passes through instead of swapping the agent loop's client afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A regression report could say which case and stage got worse, but not which prompt or skill changed. The only version it had was the repo's git sha.
Now every saved run records a short hash of each part the model is shown:
prompt:system,prompt:memory,prompt:summaryskill:<name>: description, tools and bodytool:<name>: description and parameters. This goes one step past "skills and prompts", because tool definitions are also model-facing text.compareandrun --baselinethen print:改过的部件: …: the parts whose text changed, was added or was removed.先查: …: the changed parts that case used in either run. These are the skills it read, the tools it called, the prompts sent with every request, and the summary prompt only after compaction. A skill the baseline read and the new run skipped counts, because a changed description often means the model stops reading it.ContextBuilderwrites (they are not a part).This is a reading order, not a verdict. Every skill description and tool definition is in every request, so a part a case never used can still move it.
How
sandbox.model_facing_parts(config)reads the parts from a sandbox runtime built likerun_case's. It uses a stand-in model client, so no API key is needed, and no model is called. Temp paths and times do not leak into the hashes (tested).build_runtime(llm=...)is new.run_casepasses its injected client through it instead of swappingagent_loop.llmafterwards, so the summarizer gets the same client.summarize_eventsnow countscompactions.Verified
MINIBOT_HOME,run --local; the new code does not depend on the mode), on the two reminders cases with--repeat 2:skills/reminders.md's description to something unrelated, then ran again with--baseline. The file was restored before committing.改过的部件: skill:reminders.reminders-recover-app-not-runningwent 2/2 → 1/2 with→ skill_selection: 没有读取任何 skilland先查: skill:reminders(这个用例用到、又改过的部件). One of the two new trials still read the skill (skill_selection 100% → 50%), so this run does not isolate the "read only in the baseline" path; a unit test covers it.Not in this PR
🤖 Generated with Claude Code