Repository navigation
Seed images: LTR training rows for Commons image pages - #621
Open
daoudclarke wants to merge 3 commits into
Open
daoudclarke wants to merge 3 commits into
daoudclarke wants to merge 3 commits into
Conversation
Writes the eval's Jev-safe images into production's index pages in memory (read whole from api.mwmbl.org, merged with the indexer's own write order and page size), finds the ones the LLM-labelled queries would retrieve, and grades them with the pass-3 prompt, with graded anchors per query to check the scale. The dataset with the image rows is tagged with its own pool, so llm_experiment can train with and without them; it gains --llm-dataset to read it. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the reviews: overlay only the train queries' images, as indexing.py does, so an en-gb evaluation never ranks images its own queries found; count only the overlaid images as image rows, since a Commons file page production already holds is a web row; fail a merge that grades one id twice differently, which a description addressing the judge could cause; and report every axis for the images and the anchors. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From Claude:
Stack: PR 7 of the #610 plan, on
wikicommons_image_search(independent track). PR 8 (seed-images-ltr-retrain) builds on it.What changed
mwmbl/rankeval/seed_images/ltr_data.pybuilds LTR training rows for Commons image pages:overlaywrites the train queries' 11,652 Jev-safe images (NSFW < 1.0) into production's own pages. Each image is indexed the wayindexing.pydoes it: the file page URL, the file name as title, the description as extract, no categories. The pages are read whole from api.mwmbl.org (/api/v1/search/page/{n}, cached, eval User-Agent). The merge uses the indexer's owncombine_documentsandpack_index_pageat production's 102.4M pages and 4096 bytes. So an image is kept only where it would really fit, and it competes with every term sharing its page. Each LLM-labelled query then takes the overlaid images its terms and bigrams retrieve, less the ones the majority-terms rule would score 0.batches,merge: pass-3 grading with the exactJUDGE_PROMPT, in the sameid || rel || ethos || overallformat, through a resumable checkpoint keyed by (query, url). Each query's block shuffles in two graded web anchors, to check the scale. A merge fails if one id gets two different grades.buildwrites the LLM dataset plus the image rows, taggedpools=commons_image.llm_experiment.py --llm-dataset PATH. To train with images, point it at the new dataset. To train without them, also pass--exclude-pool commons_image. The query split is unchanged (checked).seed-search-images-eval.md.Why
Standard search's LTR has never seen an image page. The eval found it putting a store photo at #1 for "matalan". PR 8 retrains it with these rows.
Results (run for real, full set)
Surprises:
Verification
make checkandmake testpass (1971 passed).indexing.pydoes. The en-gb queries' images stay out, so PR 8's en-gb evaluation isn't optimistic.Raised, not fixed (for PR 8 / the user):
--exclude-pool, the test split keeps the image rows, so NDCG@10 is not comparable with earlier numbers on the old dataset. PR 8 should also score both models on an image-free test pool, and decide with the holistic judge.Part of #610
🤖 Generated with Claude Code