8000 evals: record each prompt, skill and tool version, and name what changed by Jiminyang1 · Pull Request #16 · Jiminyang1/minibot · GitHub
[go: up one dir, main page]

Skip to content

evals: record each prompt, skill and tool version, and name what changed - #16

Merged
Jiminyang1 merged 1 commit into
mainfrom
feat/eval-component-versions
Nov 18, 2026
Merged

Jiminyang1 merged 1 commit into
mainfrom
feat/eval-component-versions

Conversation

@Jiminyang1
@Jiminyang1 Jiminyang1 commented Nov 18, 2026 •
Copy link
Copy Markdown
Owner

What

A regression report could say which case and stage got worse, but not which prompt or skill changed. The only version it had was the repo's git sha.

Now every saved run records a short hash of each part the model is shown:

  • prompt:system, prompt:memory, prompt:summary
  • skill:<name>: description, tools and body
  • tool:<name>: description and parameters. This goes one step past "skills and prompts", because tool definitions are also model-facing text.

compare and run --baseline then print:

  • 改过的部件: …: the parts whose text changed, was added or was removed.
  • For each case that got worse, 先查: …: the changed parts that case used in either run. These are the skills it read, the tools it called, the prompts sent with every request, and the summary prompt only after compaction. A skill the baseline read and the new run skipped counts, because a changed description often means the model stops reading it.
  • When nothing recorded changed but git did, the report says the change may come from code, including the section headings ContextBuilder writes (they are not a part).
  • Runs saved before this change get a caution, not an error.

This is a reading order, not a verdict. Every skill description and tool definition is in every request, so a part a case never used can still move it.

How

  • sandbox.model_facing_parts(config) reads the parts from a sandbox runtime built like run_case's. It uses a stand-in model client, so no API key is needed, and no model is called. Temp paths and times do not leak into the hashes (tested).
  • build_runtime(llm=...) is new. run_case passes its injected client through it instead of swapping agent_loop.llm afterwards, so the summarizer gets the same client.
  • summarize_events now counts compactions.

Verified

  • Unit tests: 492 pass, 3 skipped (the same skips as main). Ruff is clean.
  • Real model (deepseek-v4-pro, scratch MINIBOT_HOME, run --local; the new code does not depend on the mode), on the two reminders cases with --repeat 2:
    1. Baseline run: both cases pass 2/2.
    2. I changed skills/reminders.md's description to something unrelated, then ran again with --baseline. The file was restored before committing.
    3. The report printed 改过的部件: skill:reminders. reminders-recover-app-not-running went 2/2 → 1/2 with → skill_selection: 没有读取任何 skill and 先查: skill:reminders(这个用例用到、又改过的部件). One of the two new trials still read the skill (skill_selection 100% → 50%), so this run does not isolate the "read only in the baseline" path; a unit test covers it.

Not in this PR

  • The hashes are not attached to Langfuse traces or experiment metadata.
  • There is no automatic swap-and-rerun to confirm a suspect.

🤖 Generated with Claude Code

Every saved run now carries a short hash of each part the model is
shown: the system, memory and summary prompts, every skill (description,
tools, body) and every tool definition (description, parameters). They
are read from a sandbox runtime built like run_case's, with a stand-in
model client, so no API key is needed.

A regression report lists the parts whose text changed and, for each
case that got worse, the changed parts it used in either run: the skills
it read, the tools it called, the prompts sent with every request, and
the summary prompt only after compaction. A skill the baseline read and
the new run skipped counts. This is where to look first, not a verdict.

build_runtime takes an optional llm, which run_case now passes through
instead of swapping the agent loop's client afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jiminyang1
Jiminyang1 merged commit c8ddf15 into main Nov 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0