8000 Seed Search images eval: indexing approved images for standard search by daoudclarke · Pull Request #617 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]
Skip to content

Seed Search images eval: indexing approved images for standard search - #617

Merged
daoudclarke merged 7 commits into
seed-images-eval-holisticfrom
seed-images-eval-indexing
Nov 10, 2026
Merged

Seed Search images eval: indexing approved images for standard search#617
daoudclarke merged 7 commits into
seed-images-eval-holisticfrom
seed-images-eval-indexing

Conversation

@daoudclarke
@daoudclarke daoudclarke commented Nov 9, 2026 •
edited
Loading
Copy link
Copy Markdown
Contributor

Stack

PR 6 of 7 in the #610 evaluation stack: #611 ← #612 ← #613 ← #616 ← #615 ← #617 ← #614. Merge in that order.

What changed

  • mwmbl/rankeval/seed_images/indexing.py indexes the 11,652 Jev-safe images (NSFW below 1.0) of the train queries as pages of their own, through index_documents, under their own terms, never the query that found them:

    • URL: the Commons file page (…/wiki/File:X.jpg);
    • title: the file name without its extension;
    • extract: the description, with categories as a variant; cut to the crawler's 65 and 155 characters.

    It writes them into copies of the en-gb lookup index and the dev index, measures eviction, searches the 295 en-gb queries with standard search's ranker (MMRRanker(LTRRanker)), scores each shown image with Jev relevance and builds holistic-judge batches. search() refuses to run without PYTHONHASHSEED=0: with a random seed, tied results swap and change 34 of 295 top tens.

  • The "Indexing" and "Summary" sections of mwmbl/rankeval/seed-search-images-eval.md.

  • test/test_seed_images_indexing.py (new).

Results

The en-gb test index has 65,536 pages to production's 102M, so full writes evict far more than production would. Same-term arms write an image only to terms the index already holds, the competition production would see. Full-write arms are an upper bound on eviction. The dev index is too thin to be a fair test.

Eviction:

Index, arm Image term docs stored Existing docs evicted Evicted by images of the same term
en-gb, description, full write 241,964 3,087 of 39,315 (7.9%), 579 terms 11
en-gb, + categories, full write 263,382 4,013 of 39,665 (10.1%), 599 terms 15
en-gb, description, same term 109, on 59 terms 5 5
en-gb, + categories, same term 74, on 28 terms 6 5
dev, description, full write 87,466 85,162 of 90,690 (94%) 129

What standard search shows:

Index, arm Top tens changed Changed with no image shown Queries with images Images Jev rel ≥ 1.5 Jev rel < 1 On gate-rejected queries Web results displaced (by images)
en-gb, description, full 106 92 14 31 12 19 30 219 (35)
en-gb, + categories, full 112 97 15 30 11 18 28 258 (36)
en-gb, description, same term 11 5 6 14 12 2 13 19 (14)
en-gb, + categories, same term 10 4 6 14 12 2 13 18 (14)
dev, description, full 40 13 27 31 10 15 23 26 (2)

Holistic, en-gb same-term arm only (the closest to production):

Comparison Queries Preference for X (95% CI) X / Y / tie Orders agree
Same term vs original 11 −0.27 [−0.55, +0.00] 9% / 36% / 55% 64%
  • The LTR was NOT retrained on image pages. These runs use today's LTR, which has never seen an image page. Retraining it with image results is expected to fix the ranking losses (a store photo at Referer is sent to selected URLs #1 for "matalan", junk images for "italy vs bosnia" and "cast of wednesday").
  • n = 11 is tiny. This is a direction, not a measurement. No query wins in both orders, and six of the eleven are ties.
  • The gate doesn't apply: 13 of the 14 same-term images are on queries Jev need would reject. Nothing caps images per list ("euro 2025" got 6).
  • Categories add terms for only 4,962 of 11,652 images, mostly licence and maintenance text ("self-published work", "taken with", "cc-zero"), and improve nothing. Drop them.

Recommendation: retrain the LTR with image results. Index the file name and description only. Cap images at one or two per list. Decide whether standard search shows images on gate-rejected queries. Check thin terms against a production-sized index before rollout.

How verified

  • make check passes. test/test_seed_images_indexing.py covers the document text, the same-term filter and the top-ten diff.
  • Every run is seeded with PYTHONHASHSEED=0.
  • A code-review subagent reviewed the change. Later fixes: search() now refuses to run without the fixed hash seed; the Summary no longer keeps a pixel layer behind Jev; and the write-up states that the LTR was never trained on image pages, so it no longer recommends against serving indexed images.
  • Coverage: mwmbl/rankeval/seed_images/* is omitted from coverage (set in Add a harness to evaluate images in Seed Search #611), because it is offline evaluation code.

Decisions already made

  • Only images that pass the Jev NSFW check (score below 1.0) are shown or indexed. There is no additional pixel check behind it.

Nothing outside mwmbl/rankeval imports this code, so production is unchanged.

Part of #610

🤖 Generated with Claude Code

daoudclarke and others added 3 commits November 9, 2026 18:54
@daoudclarke @claude
Measure indexing Seed Search's Commons images for standard search
ae600e2 
Index the 11,652 Jev-approved images of the train queries into copies of
the en-gb lookup index and the dev index, as pages of their own (file name,
description, optionally categories), and search the 295 en-gb queries with
standard search's ranker on each copy.

With writes limited to the queried terms' own pages, as production's page
count makes likely, 6 queries show 14 images, 12 of them relevant, and 5
existing documents are evicted. Categories add mostly licence and
maintenance terms and help nothing, so index the description only.
Holistic judge batches are built; the verdicts are pending.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke @claude
Judge indexed Commons images in standard search and summarise the eval
66c5d10 
The same-term arm, the closest to production, judges slightly worse than
today's standard search: -0.27 [-0.55, +0.00] over only 11 queries, with
no query winning in both orders. Recommend not serving indexed images yet;
if they are indexed, use the file name and description only, cap them per
list and decide on gate-rejected queries. Add a closing summary across the
evaluation's phases.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke @claude
Trim the indexing eval to the 500-line limit
d446efd 
Fold the category-term table into prose, simplify the command line and
inline the relevance file's path. Results are unchanged.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke
daoudclarke added this pull request to stack #618 November 9, 2026 19:03
@daoudclarke @claude
Gate images on the Jev check alone and require a fixed hash seed for …
cd351cc 
…search

The summary kept a pixel layer behind Jev; the decision is that an image is
shown or indexed only if Jev passes it, with no further check. State that and
the residual risk. search() now refuses to run without PYTHONHASHSEED=0, which
the top-ten counts depend on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No functional changes to the crawler.
@daoudclarke daoudclarke mentioned this pull request Nov 9, 2026
Merged
@daoudclarke @claude
Say the indexing result used an LTR never trained on image pages
b36e79b 
Retraining it with image results is expected to fix the ranking losses, so
the recommendation no longer says not to serve indexed images, and
retraining plus a re-evaluation is on the production list.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 9, 2026
Merged
Merged
Merged
Merged
Merged
@codecov
codecov Bot commented Nov 9, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@daoudclarke @claude
Merge origin/seed-images-eval-holistic into seed-images-eval-indexing
b654560 
Kept both sides of the eval doc: the holistic "Before shipping" list
stays at the end of the holistic comparison, ahead of the Indexing section.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke
daoudclarke commented Nov 9, 2026
Copy link
Copy Markdown
Contributor Author

Looks fine to me

@daoudclarke @claude
Merge origin/seed-images-eval-holistic into seed-images-eval-indexing
86fa6c7 
No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke
daoudclarke merged commit 0b161ab into wikicommons_image_search Nov 10, 2026
6 checks passed
@daoudclarke
daoudclarke deleted the seed-images-eval-indexing branch November 10, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daoudclarke
0