8000 evals: the answer judge can be pinned apart from the tested model by Jiminyang1 · Pull Request #15 · Jiminyang1/minibot · GitHub
[go: up one dir, main page]

Skip to content

evals: the answer judge can be pinned apart from the tested model - #15

Merged
Jiminyang1 merged 1 commit into
mainfrom
feat/eval-judge-model
Nov 17, 2026
Merged

Jiminyang1 merged 1 commit into
mainfrom
feat/eval-judge-model

Conversation

@Jiminyang1
Copy link
Copy Markdown
Owner

Why

build_judge always used config.model, so the model under test also graded its own answers. Comparing two models (MINIBOT_MODEL=A baseline, then MINIBOT_MODEL=B --baseline) therefore swapped the judge too. The answer-stage scores of the two runs came from different graders, and the report did not say so. The Langfuse regression-testing guide also recommends pinning the judge model.

What changed

  • MINIBOT_EVAL_JUDGE_MODEL (evals/runner.py::judge_model) picks the judge. When it is unset, the judge is the tested model, as before.
    • A separate judge resolves its limits from the built-in catalog. The MINIBOT_* limit overrides describe the tested model, so they don't apply to it.
    • An unknown judge fails with a message that names the judge setting, not the tested model's settings.
  • Recorded: judge_model goes into every saved run and into the Langfuse experiment metadata. The run header now reads 评测 NAME(模型 …,评审 …,lesson …).
  • Compared: when the two sides used different judges, the report warns and marks the answer line "(评审不同,不能直接比)". Runs saved before this change were graded by their tested model, and the comparison reads them that way.
  • Docs: .env.example (new "Golden evals" section), the config tables in both READMEs, and docs/architecture.md §11.

Limits

  • The judge uses the same provider settings (API key, OPENAI_BASE_URL, MINIBOT_LLM_PROVIDER) as the tested model. A judge from another provider works only when those settings fit both models. This is not new; this PR just doesn't change it.
  • The judge's own token usage is still not counted in a run's tokens metric.

Tests

  • 6 new tests in tests/test_eval_baseline.py::JudgeModelTests: the default and the pinned judge; a pinned judge ignores the tested model's overrides while the default keeps them; the unknown-judge message; saved runs record the judge; comparisons with a pinned judge and with an old run whose judge is inferred.
  • Full suite OK (3 skipped). ruff check . is clean.

Verified with the real model (2026-11-17)

MINIBOT_EVAL_JUDGE_MODEL=deepseek-v4-flash evals run --local --case chat-concept-no-tools --baseline bl-base, with the tested model deepseek-v4-pro. The header showed the judge. Flash graded the answer (1.0, with a one-line reason). The saved run has judge_model: deepseek-v4-flash. The comparison against an older run (judge inferred as deepseek-v4-pro) printed the judge warning and marked the answer line.

🤖 Generated with Claude Code

The judge that grades the answer stage was always the tested model
(build_judge used config.model). Comparing two models then also compared
two judges: switching MINIBOT_MODEL switched the grader, and the answer
scores of the two runs did not mean the same thing.

- MINIBOT_EVAL_JUDGE_MODEL picks the judge. Unset, it is the tested
  model, as before. A separate judge uses its own catalog limits, not the
  tested model's MINIBOT_* overrides, and an unknown judge fails with a
  message that names the judge setting.
- Every run records judge_model (also in the Langfuse experiment
  metadata) and prints it in the run header. A run saved before this was
  graded by its tested model, and the comparison reads it that way.
- When the two sides were graded by different judges, the comparison
  warns and marks the answer stage as not directly comparable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jiminyang1
Jiminyang1 merged commit a77e6e7 into main Nov 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0