Judge Commons images in Seed Search holistically by daoudclarke · Pull Request #615 · mwmbl/mwmbl

daoudclarke · 2026-11-09T18:05:17Z

Stack

PR 5 of 7 in the #610 evaluation stack: #611 ← #612 ← #613 ← #616 ← #615 ← #617 ← #614. Merge in that order.

What changed

mwmbl/rankeval/seed_images/holistic_arms.py builds the arms on the 295 en-gb queries, which nothing was tuned on, and writes their holistic-judge batches under devdata/seed_images/holistic/. Commands: composite, arms, displace (the A2-displace control), vision (20 comparisons with thumbnails) and splits (local-layer split, vision against text, A2-displace against A2).
label_batches.save_thumbnail now takes a path.
seed-search-images-eval.md gets the "Holistic comparison" section and its recommendation.

The arms:

A0: production's order, Jev composite + Staan rank.
A2, the recommended design: when Jev need ≥ 1.63, a strip of up to 4 Jev-safe images with Jev relevance ≥ 1.5, highest first, as one extra entry after the top web result. It displaces nothing. 80 queries are gated in and 59 get a strip, of 213 images in all.
A2-displace: A2 with the strip in place of the tenth web result, so the list stays at ten. Its batches drop the clause saying a strip adds to the results.
A1, no gate: the two most relevant Jev-safe images join the web results as single results, merged by Jev composite (relevance − 0.5), cut back to ten. 191 images go into 118 queries' top tens, mostly at positions 7 to 10.

Results

Haiku 4.5 judged each comparison in both orders. Only queries whose lists differ are compared. Preference runs from −3 to 3, and a positive value favours the image arm.

Comparison	Queries	Preference [95% CI]	Images / A0 / tie	Orders agree
A2 vs A0	59	+0.84 [+0.64, +1.01]	85% / 8% / 7%	93%
A2-displace vs A0	59	+0.75 [+0.55, +0.92]	80% / 12% / 8%	80%
A1 vs A0	118	+0.12 [−0.03, +0.29]	47% / 38% / 15%	73%
A2 vs A0, strip passed every local layer	26	+0.85 [+0.63, +1.02]	85% / 4% / 12%	88%
A2 vs A0, strip flagged by a local layer	33	+0.83 [+0.52, +1.11]	85% / 12% / 3%	97%
A2 vs A0, vision subset, text only	20	+0.82 [+0.57, +1.02]	85% / 5% / 10%	90%
A2 vs A0, vision subset, with thumbnails	20	+1.27 [+0.93, +1.57]	90% / 5% / 5%	85%

Caveats:

The judge sees no cost for an added strip. A2 adds an entry and displaces nothing, and the prompt said a strip "adds to the web results". Of the 104 verdicts for A2, 93 are "slight" and none "strong". Read A2 as "the strips are mostly relevant", not "a better page". A2-displace makes the strip pay for itself and keeps nearly all of the win: 0.09 lower than A2 per query, 95% interval [−0.22, +0.03], same direction on 49 of 59 queries, correlation 0.79.
The results are conditional. Only the 59 of 295 queries (20%) that get a strip are compared. Over all en-gb queries, the mean preference is about +0.17 for A2 and +0.15 for A2-displace.
A1 differs from A2 in several ways at once: no gate, no minimum relevance, single results rather than a strip, low positions, and displacing web results. On the 44 queries both arms change, A1 scores +0.28 and A2 +0.82. A1 is a weak test of single images as such.
Wrong-sense strips are the remaining failure. All five of A0's wins over A2 are strips showing the wrong sense ("boone helm", "tom wagner", "isak", "versailles series"). A2-displace loses 7: those five, plus "kharg island" and "minecraft movie".
Local safety layers don't pick out worse strips: flagged and passed strips score the same.
Vision: with thumbnails the judges favour A2 more (+1.27 against +0.82 on the same 20), agreeing in direction on 19 of 20. The text-only judgment doesn't overstate A2.

Recommendation: ship option 2, the gated strip after the top web result. Don't merge single images into the web ranking as built in A1.

How verified

make check passes. 33 tests pass across the holistic-judge and seed-images test files, including test/test_seed_images_holistic.py (new).
Every judge output was checked for missing or duplicate ids and for templated verdicts. The thumbnails were deleted after judging.
A code-review subagent and a holistic-review subagent reviewed the change. Fixes that followed:
- The write-up now says what A2's win measures, before the table, and gives the all-query mean.
- The A2-displace control arm was added and judged.
- A1's result no longer rules out single images in general.
- The vision check counted a tie as agreeing with a loss. It now compares signs.
- The test imported the shared test image with the test. prefix, which fails in CI.
Coverage: mwmbl/rankeval/seed_images/* is omitted from coverage (set in Add a harness to evaluate images in Seed Search #611), because it is offline evaluation code.

Decisions already made

Only images that pass the Jev NSFW check (score below 1.0) are shown or indexed. There is no additional pixel check behind it.

Nothing outside mwmbl/rankeval imports this code, so production is unchanged.

Part of #610

🤖 Generated with Claude Code

holistic_arms builds, on the en-gb queries, A2 (the recommended gated strip at position 2, displacing nothing) and A1 (no gate, the two most relevant Jev-safe images merged into the web order by Jev composite), and writes holistic judge batches for each against production's order, plus a 20-comparison vision check with the strips' thumbnails. The holistic judge learns to render image strips and single images, tells the judge about them only in batches that show them, and names each batch's output path. Part of #610 No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

On the 295 en-gb queries the gated strip (option 2) beats production's order, +0.84 [+0.64, +1.01] over 59 queries, and judges with the thumbnails favour it more strongly. Merging single images into the web order (option 1) is no better than production, +0.12 [-0.03, +0.29]. holistic_arms gains a splits command for the local-layer and vision breakdowns, and the write-up gets the results and a recommendation. Part of #610 No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… the vision check A2 adds a strip without displacing a web result, and the judges' notes mostly say it adds relevant pictures to the same results, so the write-up now says so before the table, gives the all-query mean, and asks for a control that pays for the strip. A1 changes five things at once, so its result no longer rules out single images in general. The vision check counted a tie as agreeing with a loss; it now compares signs. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The holistic judge sees no cost to A2's strip, so A2's win may mostly measure added content. A2-displace is A2 with the strip taking the tenth web result's place, so the list stays at ten. The judge is not told such a strip adds to the results; the prompt is otherwise the same. Part of #610 No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…sult A2-displace, the gated strip in place of the tenth web result, beats production's order by +0.75 [+0.55, +0.92] on the 59 gated-in queries, against A2's +0.84, about +0.15 over all en-gb queries. The splits command compares the two per query, and the write-up and its recommendation take the control into account. Part of #610 No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…listic

No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

In CI, `test` resolves to Python's own test package. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

codecov · 2026-11-09T18:43:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

blueheron786 · 2026-11-09T21:26:39Z

I don't understand this enough to review and comment. I will say, I think en-gb might be too localized. en-us instead? Do we need a localized area/language?

blueheron786 · 2026-11-09T21:34:11Z

Review: mwmbl/mwmbl#615

Initial assessment: The experimental design is thoughtful, and the results support testing a gated image strip rather than merging individual images into the web ranking.

However, I'd pay particular attention to the statistical interpretation and whether the evaluation justifies the recommendation.

The most important distinction is that A2 improves judged relevance on queries receiving images, but it doesn't necessarily demonstrate an improvement to overall search quality.
Findings

[P2] The screen-space conclusion is stronger than the experiment supports

File: mwmbl/rankeval/seed-search-images-eval.md, lines 308–310.

The recommendation says:

Screen space is therefore no reason to hold the strip back.

I don't think the experiment establishes this.

A2-displace tests whether replacing the tenth web result with an image strip improves the judged list. It doesn't test:

The vertical space occupied by the strip.
How far users must scroll to reach subsequent results.
Whether images distract from the primary search task.
Whether users interact with the strip or simply ignore it.

The text-based judge also cannot realistically model these effects.

Suggested change: Qualify the conclusion. The experiment establishes that the strip can outperform the tenth web result in relevance, not that its UI cost is negligible.
2. [P2] The evaluation is still relatively narrow

File: seed-search-images-eval.md, lines 226–261.

The experiment uses 295 en-gb queries, with only 59 receiving an image strip.

The reported conditional improvement is +0.75 for A2-displace, but the estimated improvement across all queries is approximately +0.15.

That's useful evidence, but I'd be careful about generalising it.

In particular:

Would the same threshold work for en-us?
Would image availability differ significantly across languages?
Would the results hold for other query distributions?
Are there seasonal or topic-specific effects?

Suggested change: Explicitly describe the recommendation as provisional pending evaluation on a broader query set.

I wouldn't block this PR on that, since it is an offline experiment rather than a production change.
3. [P2] Wrong-sense images deserve a more explicit mitigation

File: holistic_arms.py, lines 101–105.

The current implementation selects images based on Jev relevance and sorts them by score.
python

shown = [
image for image, relevance in scored
if relevance >= gate["min_jev_relevance"]
][: gate["num_images"]]

The evaluation itself identifies wrong-sense images as the remaining failure mode.

Examples include:

Boone Helm
Tom Wagner
Isak
Versailles Series

This is particularly important because the image strip appears immediately after the first web result.

A wrong-sense image in that position could be more misleading than an irrelevant image near the bottom of the page.

Suggested change: Consider testing whether image relevance combined with agreement with the top web result improves disambiguation.

I wouldn't necessarily add another filter before merging this evaluation, but I'd make this an explicit follow-up experiment.

…nto seed-images-eval-holistic

…notes Says why the holistic comparison uses the en-gb held-out queries, that the measured runs used the judge's earlier description cut, and adds two notes for production: drop a strip image the Wikipedia card already shows, and render Commons text escaped. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

daoudclarke · 2026-11-09T21:40:14Z

I don't understand this enough to review and comment. I will say, I think en-gb might be too localized. en-us instead? Do we need a localized area/language?

We are using en-gb for now - we will add more locales at a later point.

…stic No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rd notes Says this comparison uses the Combined eval's en-gb run (it also has en-us lists that could check the locale claim), names commons_text as what now normalises Commons text, records how the card's lead image is matched, and points the wrong-sense failure at giving Jev the top web results. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

daoudclarke · 2026-11-10T06:08:32Z

Reply from Claude: Replying to the en-gb question above (6751890856). These are the held-out queries from the earlier Combined Search evaluations (combined-search-haiku-eval.md). They come from UK autocomplete, and this comparison uses that evaluation's en-gb run, judged for a UK searcher. Reusing them gives a held-out set (all image settings were tuned on the 849 train queries), and the stored web lists need no new paid Staan calls.

The conclusions probably carry over to en-us. Commons search ignores locale, the gate lets in 27% of these queries against 28% of the train queries, and the strip's wins and losses come from relevant versus wrong-sense (homonym) pictures, not UK-specific content. That evaluation also has en-us lists for the same queries, so this could be checked. We don't need a localised strip. The doc's holistic section now says this.

daoudclarke and others added 6 commits November 9, 2026 18:04

Merge branch 'seed-images-eval-judge-images' into seed-images-eval-ho…

1b7f930

…listic

daoudclarke mentioned this pull request Nov 9, 2026

Let the holistic judge show Commons images #616

Merged

daoudclarke changed the base branch from seed-images-eval-gate to seed-images-eval-judge-images November 9, 2026 18:26

daoudclarke and others added 2 commits November 9, 2026 18:31

Merge seed-images-eval-judge-images

6044df6

No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Import the shared test image without the test package prefix

68e41c4

In CI, `test` resolves to Python's own test package. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

daoudclarke added this pull request to stack #618 November 9, 2026 19:03

daoudclarke and others added 2 commits November 9, 2026 21:35

Merge remote-tracking branch 'origin/seed-images-eval-judge-images' i…

a6fc217

…nto seed-images-eval-holistic

daoudclarke and others added 3 commits November 9, 2026 21:41

Merge origin/seed-images-eval-judge-images into seed-images-eval-holi…

b464444

…stic No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge origin/seed-images-eval-judge-images into seed-images-eval-holi…

95b8861

…stic No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

daoudclarke merged commit 0b161ab into wikicommons_image_search Nov 10, 2026
5 checks passed

daoudclarke deleted the seed-images-eval-holistic branch November 10, 2026 06:13

Navigation Menu

Repository navigation

Judge Commons images in Seed Search holistically - #615

Judge Commons images in Seed Search holistically#615
daoudclarke merged 13 commits into
seed-images-eval-judge-imagesfrom
seed-images-eval-holistic

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Uh oh!

Stack

What changed

Results

How verified

Decisions already made

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants