Repository navigation
evals: the answer judge can be pinned apart from the tested model - #15
Merged
Merged
Conversation
The judge that grades the answer stage was always the tested model (build_judge used config.model). Comparing two models then also compared two judges: switching MINIBOT_MODEL switched the grader, and the answer scores of the two runs did not mean the same thing. - MINIBOT_EVAL_JUDGE_MODEL picks the judge. Unset, it is the tested model, as before. A separate judge uses its own catalog limits, not the tested model's MINIBOT_* overrides, and an unknown judge fails with a message that names the judge setting. - Every run records judge_model (also in the Langfuse experiment metadata) and prints it in the run header. A run saved before this was graded by its tested model, and the comparison reads it that way. - When the two sides were graded by different judges, the comparison warns and marks the answer stage as not directly comparable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
build_judgealways usedconfig.model, so the model under test also graded its own answers. Comparing two models (MINIBOT_MODEL=Abaseline, thenMINIBOT_MODEL=B --baseline) therefore swapped the judge too. The answer-stage scores of the two runs came from different graders, and the report did not say so. The Langfuse regression-testing guide also recommends pinning the judge model.What changed
MINIBOT_EVAL_JUDGE_MODEL(evals/runner.py::judge_model) picks the judge. When it is unset, the judge is the tested model, as before.MINIBOT_*limit overrides describe the tested model, so they don't apply to it.judge_modelgoes into every saved run and into the Langfuse experiment metadata. The run header now reads评测 NAME(模型 …,评审 …,lesson …).answerline "(评审不同,不能直接比)". Runs saved before this change were graded by their tested model, and the comparison reads them that way..env.example(new "Golden evals" section), the config tables in both READMEs, anddocs/architecture.md§11.Limits
OPENAI_BASE_URL,MINIBOT_LLM_PROVIDER) as the tested model. A judge from another provider works only when those settings fit both models. This is not new; this PR just doesn't change it.tokensmetric.Tests
tests/test_eval_baseline.py::JudgeModelTests: the default and the pinned judge; a pinned judge ignores the tested model's overrides while the default keeps them; the unknown-judge message; saved runs record the judge; comparisons with a pinned judge and with an old run whose judge is inferred.ruff check .is clean.Verified with the real model (2026-11-17)
MINIBOT_EVAL_JUDGE_MODEL=deepseek-v4-flash evals run --local --case chat-concept-no-tools --baseline bl-base, with the tested modeldeepseek-v4-pro. The header showed the judge. Flash graded the answer (1.0, with a one-line reason). The saved run hasjudge_model: deepseek-v4-flash. The comparison against an older run (judge inferred asdeepseek-v4-pro) printed the judge warning and marked the answer line.🤖 Generated with Claude Code