8000 Seed images: ship the LTR retrained with image pages, and cap images at 2 per list by daoudclarke · Pull Request #636 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]
Skip to content

Seed images: ship the LTR retrained with image pages, and cap images at 2 per list - #636

Open
daoudclarke wants to merge 3 commits into
seed-images-prod-standard-displayfrom
seed-images-prod-ltr
Open

Seed images: ship the LTR retrained with image pages, and cap images at 2 per list#636
daoudclarke wants to merge 3 commits into
seed-images-prod-standard-displayfrom
seed-images-prod-ltr

Conversation

@daoudclarke
@daoudclarke daoudclarke commented Nov 11, 2026 •
edited
Loading
Copy link
Copy Markdown
Contributor

From Claude:

Stack: PR 8 of the #610 plan, the production half. It builds on #637 (seed-images-prod-standard-display, PR 9), which builds on #635. The evidence is the evaluation PR #626.

Merge order: merge this after #637, which it is stacked on. The new model gives Commons file pages a higher score than today's model does. #637's serving-time store lookup (ServedImagesRanker) is what stops a crawled Commons file page that the NSFW judge never checked from getting that boost: it drops every Commons File: URL that isn't approved in the store. Without it, a crawled, unchecked Commons page could rank higher than it does today.

What changed

  • The retrained standard-search model. mwmbl/resources/model.xgb, which RUST_MODEL_PATH points at, is now Seed images: evaluate retraining the LTR with image pages, and a per-list image cap #626's images_relevance model.
    • sha256: ea9ab57c784fdac735327d0b321795afa7507bd49d7ece7846bc91caeaacaa83 (it was b6f81657…5d84).
    • Recipe: the deployed recipe (commit 76b58d2: scale_pos_weight 0.1, reg_lambda 2, 100 rounds, curation pairs at weight 0.5, weak negatives at 0.25), trained on every LLM-labelled query plus Seed images: LTR training rows for Commons image pages #621's Commons image rows, with the label relevance ≥ 2 in place of overall ≥ 6. The feature set is the same 50 features, with no is_image feature, so the Rust extension is unchanged.
    • Reproduce it on the eval branch with: DATABASE_URL="postgres://daoud@" DJANGO_SETTINGS_MODULE=mwmbl.settings_dev uv run python -m mwmbl.rankeval.seed_images.ltr_retrain train images_relevance
    • Super Search also scores with this model (search_setup.ltr_model), so its ranking moves too.
  • A per-list cap on Commons images. The new mwmbl/tinysearchengine/image_cap.py adds cap_images and an ImageCapRanker decorator.
    • It counts any URL that commons.commons_file_key recognises as a Commons file page, which covers the mobile site, index.php?title= and Image: URLs. Other Commons pages, such as categories, don't count.
    • It keeps the first MAX_IMAGES_PER_RESULT_LIST files (a new setting, 2) and drops the rest, so the web results below them move up. A second URL for a file already kept is dropped as well, so one picture can't take both slots.
    • In standard search it wraps Show indexed Commons files in search only as the image store vouches for them #637's ServedImagesRanker, which wraps MMRRanker: ImageCapRanker(ServedImagesRanker(MMRRanker(LTRRanker(…))), cap). So it runs after the LTR, MMR and the store's check, and a file the store doesn't vouch for is dropped before it can use a slot. It keeps the ImageDocuments that the store's check returns as they are. No server code cuts the list to ten; callers take the top ten from it, so the cap applies before that cut. It covers the v1 and v2 search APIs, the web page and Super Search's mwmbl_index source. Super Search's merged final list gets the same cap too, after its MMR and Show indexed Commons files in search only as the image store vouches for them #637's check. /raw is left uncapped.
    • Combined Search gets the same cap around its own ranker (ServedImagesRanker(JevRanker(…))), for consistency. The cap there runs before the strip's images are removed from the web results. So a web list can show at most 2 images, and fewer when the strip already shows them. The cap was not evaluated for Combined Search. Seed images: evaluate retraining the LTR with image pages, and a per-list image cap #626 evaluated it with standard search's ranker only, and Combined Search keeps its own, unchanged model. It is there for consistency only.

Why

#626's recommendation was the images_relevance model with at most 2 images per list.

  • Cross-validation, pass-3 NDCG@10, paired against today's recipe retrained per fold: +0.006 ± 0.002 on the whole fold, and +0.004 ± 0.002 on web rows only, so the image rows cost web ranking nothing. Training on the image rows also demotes images: 0.55 images in the top ten, and none at Referer is sent to selected URLs #1, against 1.24 and 1.2% for today's recipe.
  • Holistic (Haiku 4.5, both orders): retrained with images and cap 2 against no images scored +0.64 [+0.42, +0.87]. Cap 1 scored +0.59 and no cap +0.43. On the 7 queries where the two caps differ, cap 2 beat cap 1 by +0.36. Without a cap, one query showed nine station photos and lost at −2.5.

Checks

  • The model loads with the current Rust extension. It has 50 features and 100 trees, like today's. test_the_configured_model_loads_and_scores loads RUST_MODEL_PATH and scores documents through RustXGBPipeline.
  • Dev queries. Standard search's ranker (LTR + MMR) over the dev index gave the same top results with both models for python, rust and linux kernel, and close variants for bbc news and london. In a synthetic three-document pool for "eiffel tower", the new model still ranks the Commons photo last. It scores it 0.54 against today's 0.12, while Wikipedia gets 0.91 and the official site 0.81. That rise is why this needs PR 9.
  • Crawler. The crawler doesn't load the model: no ranking module is among the modules mwmbl.crawl imports. The release check still counts mwmbl/resources/ and settings_common.py as crawler files, so the commit says No functional changes to the crawler. rather than raising CRAWLER_VERSION.

Verification

  • make check passes, and make test passes on the rebased branch (1991 passed, 10 skipped).
  • New tests in test/test_image_cap.py:
    • ordering: later images are dropped and web results move up
    • more than 2 images, a list within the cap, and a list with no images
    • which URLs count as images (mobile, index.php, Image:), while a category page doesn't
    • a second URL for a file already kept is dropped without using a slot
    • the decorator caps both search and search_retrieved and passes the other calls through
    • after MMR: the cap keeps the two images that MMR placed highest
    • both served rankers are capped at the setting, outside ServedImagesRanker
    • the configured model path loads
  • New tests in test/test_image_serving.py:
    • through the v2 search API, an unstored Commons file ranked first is dropped before the cap counts it, so the two stored files below it are shown and a third is capped
    • Super Search's final list keeps 2 stored files out of 3
  • The tests that stub the inner ranker (test_combined_search.py, test_image_serving.py) now reach one level further in.
  • Review by a subagent (code, trust and safety, holistic). It found no correctness bugs and confirmed that every caller of the served rankers is capped, apart from /raw.
    • Fixed: two URLs for the same file no longer take two slots.
    • Fixed: the body now says that the Combined Search cap is unevaluated.
    • Fixed after rebasing onto Show indexed Commons files in search only as the image store vouches for them #637: the cap runs outside the store's check, and Super Search's merged list is capped.
    • Raised, not fixed:
      • No is_image feature. The model's view of image pages comes from generic text, URL and domain features, so a non-Commons page with the same profile, such as an SEO page with a keyword-stuffed URL slug, might share some of the lift. The web-only cross-validation (+0.004) shows no loss. A look at score changes on a spam sample would settle it.
      • Pages the cap doesn't recognise. File: pages on other wikis (en.wikipedia.org/wiki/File:…), and Special:FilePath and Media: URLs, aren't counted as images.
      • Order in Combined Search. The cap runs before strip duplicates are removed, so its web list can show fewer than 2 images. Removing them first would keep 2.

Part of #610

🤖 Generated with Claude Code

@codecov
codecov Bot commented Nov 11, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@daoudclarke daoudclarke mentioned this pull request Nov 11, 2026
Open
@daoudclarke
daoudclarke changed the base branch from seed-images-prod-index to seed-images-prod-standard-display November 11, 2026 11:01
@daoudclarke
daoudclarke added this pull request to stack #632 November 11, 2026 11:10
daoudclarke and others added 3 commits November 11, 2026 11:14
@daoudclarke @claude
Ship the LTR retrained with Commons image pages, and cap images at 2 …
c3fb1ab 
…per list

The model is #626's images_relevance arm: the deployed recipe plus the image rows, on the
relevance >= 2 label (sha256 ea9ab57c...aa83). Cross-validation put it at +0.006 pass-3
NDCG@10 over today's recipe, and the holistic judge preferred a cap of 2 images per list
to a cap of 1 or none.

ImageCapRanker drops every Commons file page after the first MAX_IMAGES_PER_RESULT_LIST
from the final order, after the LTR and MMR, so web results move up. Combined Search's
ranker gets the same cap for consistency.

Merge with or after PR 9's serving-time store lookup: the model scores Commons file pages
higher than today's, and only that lookup keeps unchecked crawled ones out.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke @claude
Don't let two URLs for one Commons file take two image slots
0fa4d35 
No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke @claude
Cap images outside the store's check, and in Super Search's final list
3b75bff 
The cap now wraps ServedImagesRanker, so a Commons file the store doesn't vouch for is dropped
before it can take one of the two slots. Super Search's merged list gets the same cap after
its MMR. Tests that stub the inner ranker reach one level further in.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daoudclarke
0