You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The GPQA-Diamond score is based on the result officially released by Artificial Analysis.
For the score differences on LiveCodeBench and AIME 2026, our initial assessment is that they fall within the expected range of evaluation variance.
Regarding the relatively larger discrepancies on IFBench and IFEval, we are currently conducting a further review of the evaluation process and results. We will keep you updated once we have any further findings.
Hi, we'd like to follow up on our previous update regarding the IFBench and IFEval results for gemma-4-E2B-it.
During our review, we identified some compatibility issues between SGLang and the Gemma 4 model family, particularly in reasoning content parsing, which affected the previous evaluation results.
After re-running the evaluations with an updated inference setup, we have updated the scores for gemma-4-E2B-it as follows:
IFBench: 37.33
IFEval: 83.55
We will also update the corresponding results on the MiniCPM page.
2. Hi, we'd like to follow up on our previous update regarding the IFBench and IFEval results for gemma-4-E2B-it.
During our review, we identified some compatibility issues between SGLang and the Gemma 4 model family, particularly in reasoning content parsing, which affected the previous evaluation results.
After re-running the evaluations with an updated inference setup, we have updated the scores for gemma-4-E2B-it as follows:
IFBench: 37.33
IFEval: 83.55
We will also update the corresponding results on the MiniCPM page.
Thank you for your patience and feedback!
Does the benchmark (IFBench and IFEval) use eval_results_loose/eval_results_strict and prompt-level/instruction-level?
Activity
Yilisameen commentedon Sep 24, 2026
Thank you for your feedback.
The GPQA-Diamond score is based on the result officially released by Artificial Analysis.
For the score differences on LiveCodeBench and AIME 2026, our initial assessment is that they fall within the expected range of evaluation variance.
Regarding the relatively larger discrepancies on IFBench and IFEval, we are currently conducting a further review of the evaluation process and results. We will keep you updated once we have any further findings.
Yilisameen commentedon Oct 27, 2026
Hi, we'd like to follow up on our previous update regarding the IFBench and IFEval results for gemma-4-E2B-it.
During our review, we identified some compatibility issues between SGLang and the Gemma 4 model family, particularly in reasoning content parsing, which affected the previous evaluation results.
After re-running the evaluations with an updated inference setup, we have updated the scores for gemma-4-E2B-it as follows:
IFBench: 37.33
IFEval: 83.55
We will also update the corresponding results on the MiniCPM page.
Thank you for your patience and feedback!
hedes1992 commentedon Nov 2, 2026
Does the benchmark (IFBench and IFEval) use eval_results_loose/eval_results_strict and prompt-level/instruction-level?
Yilisameen commentedon Nov 3, 2026
For IFBench, we report Prompt-level Loose Accuracy, which is aligned with the AA Index v4.1 methodology.
For IFEval, we report Prompt-level Strict Accuracy.