8000 Partial replication failure on ClueWeb12-B13 for ax models · Issue #592 · castorini/anserini · GitHub
[go: up one dir, main page]
Skip to content

Partial replication failure on ClueWeb12-B13 for ax models #592

New issue
New issue
Closed
Closed
Partial replication failure on ClueWeb12-B13 for ax models#592

Description

@lintool
lintool
opened on Apr 15, 2019

According to #590 @matthew-z wasn't able to replicate results with ax models on ClueWeb12-B13. We should look into this...

Activity

  1. matthew-z commented on Apr 17, 2019

    @matthew-z
    matthew-z
    on Apr 17, 2019
    Contributor

    Somehow the AX results matched after indexing again with oracle JVM (1.8.0_201)....

  2. lintool commented on Apr 17, 2019

    @lintool
    lintool
    on Apr 17, 2019
    MemberAuthor

    Interesting.

    Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM. This would explain the differences in ax results, which rely on sampling docs.

    @matthew-z Just to be clear, with Oracle JVM you are able to replicate our figures exactly, while with OpenJDK you come close? (If so, how close is "close" in the latter case?)

  3. matthew-z commented on Apr 18, 2019

    @matthew-z
    matthew-z
    on Apr 18, 2019
    Contributor

    Did you mean no-AX algorithms? They matched exactly.

    E.g., 201-250, Bm25, NDCG@20, it got 0.12862 and expected 0.1286

    201-250, QL, NDCG@20, it got 0.11067 and expected 0.1107

  4. lintool commented on Apr 18, 2019

    @lintool
    lintool
    on Apr 18, 2019
    MemberAuthor

    Okay, thanks! I'll send a PR to capture this discussion and ask you to review.

  5. lintool commented on Apr 18, 2019

    @lintool
    lintool
    on Apr 18, 2019
    MemberAuthor

    @matthew-z please look at #594

  6. lintool commented on Apr 18, 2019

    @lintool
    lintool
    on Apr 18, 2019
    MemberAuthor

    Resolved with #594, closing.

  7. lintool
    closed this as completedon Apr 18, 2019

  8. Peilin-Yang commented on May 21, 2019

    @Peilin-Yang
    Peilin-Yang
    on May 21, 2019
    Contributor

    Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM.

    I am not sure, but this is unlikely IMO.
    I am going to test it again by myself

  9. lintool commented on Oct 17, 2019

    @lintool
    lintool
    on Oct 17, 2019
    MemberAuthor

    Whatever the issue, Oracle vs. openJDK isn't the issue. All regressions run fine on damiano:

    [jimmylin@damiano ~]$ java --version
openjdk 11.0.4 2019-07-16 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11-LTS, mixed mode, sharing)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions
      0