8000 how to get the evaluation result of gemma4? · Issue #382 · OpenBMB/MiniCPM · GitHub
[go: up one dir, main page]

Skip to content

how to get the evaluation result of gemma4? #382

Open

Description

@hedes1992

I find that the evaluation result of gemme4 is different from the paper, Could you please share how to evaluate gemma4?
paper:

Image

here:

Image

from leader board

Activity

  1. Yilisameen commented on Sep 24, 2026

    @Yilisameen

    Thank you for your feedback.

    The GPQA-Diamond score is based on the result officially released by Artificial Analysis.

    For the score differences on LiveCodeBench and AIME 2026, our initial assessment is that they fall within the expected range of evaluation variance.

    Regarding the relatively larger discrepancies on IFBench and IFEval, we are currently conducting a further review of the evaluation process and results. We will keep you updated once we have any further findings.

  2. Yilisameen commented on Oct 27, 2026

    @Yilisameen

    Hi, we'd like to follow up on our previous update regarding the IFBench and IFEval results for gemma-4-E2B-it.

    During our review, we identified some compatibility issues between SGLang and the Gemma 4 model family, particularly in reasoning content parsing, which affected the previous evaluation results.

    After re-running the evaluations with an updated inference setup, we have updated the scores for gemma-4-E2B-it as follows:
    IFBench: 37.33
    IFEval: 83.55

    We will also update the corresponding results on the MiniCPM page.

    Thank you for your patience and feedback!

  3. hedes1992 commented on Nov 2, 2026

    @hedes1992
    Author

    2. Hi, we'd like to follow up on our previous update regarding the IFBench and IFEval results for gemma-4-E2B-it.
    During our review, we identified some compatibility issues between SGLang and the Gemma 4 model family, particularly in reasoning content parsing, which affected the previous evaluation results.
    After re-running the evaluations with an updated inference setup, we have updated the scores for gemma-4-E2B-it as follows:
    IFBench: 37.33
    IFEval: 83.55
    We will also update the corresponding results on the MiniCPM page.
    Thank you for your patience and feedback!

    Does the benchmark (IFBench and IFEval) use eval_results_loose/eval_results_strict and prompt-level/instruction-level?

  4. Yilisameen commented on Nov 3, 2026

    @Yilisameen

    For IFBench, we report Prompt-level Loose Accuracy, which is aligned with the AA Index v4.1 methodology.

    For IFEval, we report Prompt-level Strict Accuracy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0