Seed images: evaluate retraining the LTR with image pages, and a per-list image cap by daoudclarke · Pull Request #626 · mwmbl/mwmbl

daoudclarke · 2026-11-10T17:04:23Z

From Claude:

Stack: PR 8 of the #610 plan, the evaluation half. It builds on #621 (seed-images-ltr-data). It ships no model and changes no production ranking code. The model file and the per-list image cap go in a separate production PR.

What changed

mwmbl/rankeval/seed_images/ltr_retrain.py runs AGENTS.md's two evaluation steps over Seed images: LTR training rows for Commons image pages #621's dataset.
- cv cross-validates four arms over the 849 LLM-labelled queries, in five folds by query, on pass-3 NDCG@10. The arms are today's recipe and today's recipe plus the image rows, each trained with the overall ≥ 6 label and with a relevance ≥ 2 label. Every arm is scored on the whole test fold, on the fold with its image rows removed, and with relevance as the gain.
- train, search and batches produce the holistic run. search puts the en-gb queries through standard search's ranker (LTR + MMR), over production's own pages with images overlaid the way Seed images: LTR training rows for Commons image pages #621 overlays them, and with an optional per-list image cap.
Results and a recommendation are in seed-search-images-eval.md, in the new section "Retraining the LTR with image pages", and in the Summary.

Why

Standard search's LTR has never seen an image page. Before images are indexed, we need to know whether retraining it with image rows helps, whether those rows cost web ranking anything, and how many images one list should show.

Results

Cross-validation. Each difference is paired by query against today, which is the deployed recipe retrained in each fold. The deployed model itself was trained on all of these queries, so its own row is in-sample.

Arm	NDCG@10	Web rows only	Relevance as gain	Human pair acc.	Images in top 10	Image at #1
today	0.762	0.764	0.774	0.797	1.24	1.2%
+ images, overall	+0.004 ± 0.002	+0.002 ± 0.001	+0.004 ± 0.002	0.797	0.44	0%
today, relevance	+0.003 ± 0.002	+0.002 ± 0.002	+0.009 ± 0.002	0.816	0.66	0.7%
+ images, relevance	+0.006 ± 0.002	+0.004 ± 0.002	+0.013 ± 0.002	0.812	0.55	0%

The image rows cost web ranking nothing measurable.
Training on the image rows demotes images. The ethos lift didn't teach the model "Commons page = good".
No is_image feature is needed, because the existing URL and domain features already separate image pages. Adding one would have changed the Rust feature set.

Holistic (Haiku 4.5, both orders, en-gb queries, production-sized pages).

X	Y	Queries	Pref. for X (95% CI)
today's model, images	today's model, no images	45	+0.42 [+0.13, +0.69]
retrained, images, no cap	retrained, no images	43	+0.43 [+0.15, +0.67]
retrained, images, cap 1	retrained, no images	43	+0.59 [+0.40, +0.79]
retrained, images, cap 2	retrained, no images	43	+0.64 [+0.42, +0.87]
retrained, images, cap 1	today's model, images	245	−0.01 [−0.14, +0.12]

With only the train queries' images, as in the training rows, images reach just 3 of the 295 en-gb queries. So the holistic run also overlays the en-gb queries' own Jev-safe images. That is an upper bound: it is the case of a query someone already searched in Seed Search.
The cap matters on the few queries that get many images. Without a cap, "callerton parkway metro" shows nine station photos and loses at −2.5; with a cap of 2 it scores +1.5. On the 7 queries where the two caps differ, cap 2 beats cap 1 by +0.36.
Against today's model, the retrained model ties. That run doesn't separate the model from the cap, so the case for the retrain rests on the cross-validation. The holistic run shows only that it is no worse.

Recommendation: the images_relevance model with at most 2 images per list.

Flag for the production PR

A crawled Commons file page that Jev never checked has the same URL pattern as an approved image (https://commons.wikimedia.org/wiki/File:…), so the model and the cap treat it as an image. Any learned boost for images must therefore ship together with PR 9's serving-time store lookup, which drops every Commons file-page URL that isn't approved in the store. The lookup should canonicalise the URL first, covering http, mobile, index.php?title=File: and encoded variants. It will also drop the crawled Commons file pages that are served today.

Verification

make check passes. make test passes (1977 passed). The new tests are in test/test_seed_images_ltr_retrain.py.
60 Claude Haiku 4.5 judge batches were graded, each validated against its batch's ids before merging. Four of them came back through the orchestrator after a usage-limit stop.
Reviews by subagents:
- Code review: no high-severity bugs; the doc's figures were checked against the data. Fixed:
  - The demotion claim now pairs each image arm with its own no-image twin.
  - Documents with no term are now handled as TinyIndex.retrieve handles them.
  - For a repeated image, the first copy is kept, as in the training rows.
  - A stale lists file was deleted.
- Trust and safety, plus holistic review. Fixed in the doc:
  - "Never first" was overclaimed, because the holistic run shows 5 images at Referer is sent to selected URLs #1.
  - The doc now states that the comparison against today's model doesn't separate model from cap.
  - The doc now calls the en-gb overlay an upper bound.
  - The doc now explains the store lookup and its canonicalisation.

Raised, not fixed:

The relevance label also drops ethos from the web rows' target. No ethos loss shows: the overall-gain NDCG rises, and the holistic judge, which sees ethos, ties. An arm that uses the overall label for web rows and the relevance label for image rows wasn't tested.
Today's model with a cap of 2 wasn't judged against the retrained model with a cap of 2. Doing that would isolate the effect of the model.
Before images are indexed, the production side needs some further protections. A crafted upload's file name is a strong ranking signal. The options are a minimum file age, a relevance minimum, or a per-uploader limit. Descriptions also need a check for spam or SEO text, and approved images need re-checking after approval.
devdata/seed_images/ is untracked but not gitignored, and it holds data keyed by query.
The set of indexed images derives from what users searched, so a rare query could leave a matching photo in the public index.

Part of #610

🤖 Generated with Claude Code

…image cap ltr_retrain.py cross-validates four arms over the LLM-labelled queries (today's recipe and the image rows, each with an overall and a relevance label) and scores each on an image-free test pool too, then runs standard search's ranker on the en-gb queries over production's pages with the images overlaid, with and without a cap, for the holistic judge. The relevance-labelled image arm is best on cross-validation, costs web ranking nothing and never ranks an image first; a cap of 2 beats no cap. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pair each image arm with its own no-image twin, say the retrain-vs-today run doesn't separate model from cap, call the en-gb overlay an upper bound, and spell out what the serving-time store lookup must cover. Match TinyIndex.retrieve on term-less documents and keep the first copy of a repeated image, as the training rows do. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

codecov · 2026-11-10T17:15:17Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…per list The model is #626's images_relevance arm: the deployed recipe plus the image rows, on the relevance >= 2 label (sha256 ea9ab57c...aa83). Cross-validation put it at +0.006 pass-3 NDCG@10 over today's recipe, and the holistic judge preferred a cap of 2 images per list to a cap of 1 or none. ImageCapRanker drops every Commons file page after the first MAX_IMAGES_PER_RESULT_LIST from the final order, after the LTR and MMR, so web results move up. Combined Search's ranker gets the same cap for consistency. Merge with or after PR 9's serving-time store lookup: the model scores Commons file pages higher than today's, and only that lookup keeps unchecked crawled ones out. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

daoudclarke and others added 2 commits November 10, 2026 16:56

daoudclarke mentioned this pull request Nov 11, 2026

Seed images: ship the LTR retrained with image pages, and cap images at 2 per list #636

Open

Navigation Menu

Repository navigation

Seed images: evaluate retraining the LTR with image pages, and a per-list image cap - #626

Seed images: evaluate retraining the LTR with image pages, and a per-list image cap#626
daoudclarke wants to merge 2 commits into
seed-images-ltr-datafrom
seed-images-ltr-retrain

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

What changed

Why

Results

Flag for the production PR

Verification

Uh oh!

Codecov Report

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant