Repository navigation
Request official evaluation scripts/configs for reproducing MiniCPM5-2B benchmark results #376
Description
Hello MiniCPM Team,
I am trying to reproduce the benchmark results of MiniCPM5-2B for academic research.
I have successfully deployed the released MiniCPM5-2B model following the official instructions. However, I could not find the complete evaluation pipeline used to obtain the benchmark results reported in the model card.
Could you please share the official evaluation scripts and configurations, including:
- Benchmark evaluation scripts (MATH-500, MMLU/MMLU-Pro, AIME, HMMT, etc.);
- Inference settings:
- prompt/chat template;
- whether
enable_thinking=Trueis used; - generation parameters (
max_new_tokens, temperature, top_p, do_sample, stop criteria, etc.);
- Answer extraction and grading scripts;
- Dataset preprocessing and evaluation commands.
I especially want to reproduce the reported MATH-500 score (94.6), and I would like to confirm whether the official evaluation uses any reasoning budget, length control, or special stopping strategy for the thinking mode.
Any scripts or configuration files would be greatly appreciated.
Thank you very much.
<
604B
div role="toolbar" aria-label="Reactions" class="d-flex gap-1 flex-wrap">
Activity
Yilisameen commentedon Sep 20, 2026
Hello,
Thanks for your interest in reproducing our results. Here are the details.
Prompt template. For math benchmarks, we use a zero-shot, single-turn prompt with no few-shot examples:
Thinking mode.
enable_thinking=True, passed viachat_template_kwargsthrough an OpenAI-compatible endpoint.Generation parameters.
On your reasoning-budget question: no reasoning budget, no length control, and no special stopping strategy. The model is allowed to think to completion. This matters for reproduction — if you cap
max_new_tokensat a typical value like 4096 or 8192, a meaningful fraction of problems will have their reasoning truncated, and a truncated, unclosed<think>block is treated as an empty answer scored 0, with no partial credit.Sampling protocol — this differs by benchmark. Our reported figures don't all come from a single sample per problem:
MATH-500 and MMLU / MMLU-Pro: one sample per problem.
AIME and HMMT: 16 samples per problem, averaged.
GPQA-Diamond: 5 samples per problem, averaged.
For other benchmarks and scoring details, our configurations broadly follows the AA-Index methodology, which is publicly documented.