Show indexed Commons files in search only as the image store vouches for them by daoudclarke · Pull Request #637 · mwmbl/mwmbl

daoudclarke · 2026-11-11T08:34:51Z

From Claude:

Stack: builds on #635 (seed-images-prod-index), which builds on #634 and #633. PR 9 of the #610 plan. #636 (seed-images-prod-ltr, the 2-images-per-list cap) will be rebased on top of this PR so that its ImageCapRanker wraps ServedImagesRanker.

What changed

ServedImagesRanker (mwmbl/imagesearch/serving.py) is a decorator. It wraps the outermost served ranker: search_setup.ranker (around MMR) and combined_ranker (around Jev). Any result whose URL is a Commons File: page, in any spelling commons_file_key accepts, is looked up in CommonsImage. The lookup is one batched in_bulk query per search.
- A result is kept only if the row exists, revoked_at is null and last_checked_at is within the new IMAGE_SERVE_MAX_CHECK_AGE (48 hours). Otherwise it is dropped, and the results below it move up.
- A file reached by two URLs (the desktop and mobile sites, say) is shown once, at its highest rank.
- A kept file becomes an ImageDocument, which carries the image fields from the store row to the formatting step. Formatting needs no second query, and the image survives any decorator that keeps document objects, such as Seed images: ship the LTR retrained with image pages, and cap images at 2 per list #636's cap.
- A kept file is shown with the title and extract the NSFW judge read: the stored caption or cleaned file name, and the stored description. The index's text for that URL is not used, because a crawler may have written it.
The ranker layer, not formatting. The check is in the ranker so that Seed images: ship the LTR retrained with image pages, and cap images at 2 per list #636's cap counts only files that will actually be shown. An unvetted Commons page can't take one of the two slots.
SearchHit.image is a new nullable object: thumbnail_url, thumbnail_width, thumbnail_height, license_name, license_url, attribution. Served files have engine: "commons"; every other hit has image: null.
- The change is additive and backwards compatible. A client that switches exhaustively on engine will meet the new value commons.
Every display path is covered: /api/v2/search/ and /api/v1/search/ (v1 only drops files and has no image field), the HTML search page, and Combined Search's web results. The strip de-dup from Show Seed Search's images on Combined Search with images=true #631 stays.
Super Search also checks each source's results as they arrive, and the merged final list (which includes pages crawled for their links), with serving.served.
Threads: the lookup runs on the thread that ranks. Combined Search and Super Search's index source now rank with sync_to_async instead of asyncio.to_thread. Under to_thread, a worker thread's database connection is never closed at the end of a request, and it would break for good after a database restart. Ranking still overlaps the Staan index write and the image phase, which run in to_thread.
API descriptions (v2 and Combined Search):
- what commons means, and that the only safety check is an LLM reading the text, with no pixel check
- load thumbnails with referrerpolicy="no-referrer"
- licence and attribution must be shown with the image
- text is not HTML-safe and must be escaped
- license_url is set by the uploader: link it with rel="noopener noreferrer nofollow"
commons_file_key now checks for the substring "wikimedia.org" before parsing. It returns exactly what it did before, because every Commons hostname contains that string and a URL contains its hostname in some case. A non-Commons URL now costs about 0.2 µs instead of about 1.6 µs.

Why 48 hours

Times written from the search path are rounded down to midnight, so a file found at 23:59 already looks 24 hours old.
A daily recheck_images run adds up to 24 hours more.
So 48 hours is the smallest window in which a daily recheck keeps every approved file servable. It also bounds how long a file swapped on Commons under unchanged text can be shown before the recheck sees the new SHA-1 and revokes it.
A file the recheck falls behind on is hidden, which fails safe. The schedule therefore has to cover the whole store within 48 hours. At IMAGE_STORE_RECHECK_LIMIT = 2000 per run, a daily run covers up to about 4,000 files. A larger store needs more frequent runs or a higher limit.
No schedule is wired up here, because switching on is PR 10.

User-facing change: crawled Commons file pages disappear

This is not gated on IMAGE_SEARCH_ENABLED. With an empty store, which is production today, every Commons File: result is dropped, including those the crawler found on its own. The judge never saw them, and they could be anything. Category, gallery and other Commons pages are unaffected.

How much this affects today:

Dev index (devdata/index-v2.tinysearch): 0 of 90,690 documents are Commons file pages.
Production index: from 128,734 cached /api/v1/search/raw responses (devdata/request_cache), 4,228,266 retrieved documents, of which 9 are Commons file pages, in 7 term lookups (e.g. Lyme_Regis_beach_04.JPG for "regis beach", Leeds_UK_location_map.svg for "uk location").

So the change in today's results is negligible.

Cost on the hot path

Measured with the serving code on dev-index URLs and local Postgres. Times are p50 / p95.

Results	Commons file pages	Store rows	Time
30	0	–	5 µs / 5 µs
100	0	–	17 µs / 17 µs
500	0	–	91 µs / 96 µs
103	3	5,000	0.75 ms / 0.78 ms

When no result is a Commons file page there is no query at all; this is tested with django_assert_num_queries(0). Before the substring check, 500 results took 2.4 ms.

Verification

make check passes.
make test passes: 1980 passed, 10 skipped.
New test/test_image_serving.py:
- a stored file is served (checked 1 h and 47 h ago) with its image and its judged text
- revoked (and stale at 49 h) files are dropped, and the results below move up
- a missing file is dropped with the store empty; a Category page and a non-Commons URL that contains wikimedia.org/wiki/File: are kept
- one file reached by two URLs is shown once
- the lookup is exactly one query, and none without Commons results
- /api/v2/search/ schema: engine, image, score and number_of_results after a drop
- v1 and the HTML search page drop unvetted files
- Super Search's pipeline drops unvetted files
Combined Search tests:
- a stored file among the web results gets image and engine: "commons"
- an unvetted or revoked one (including a Commons page returned by Staan) is dropped
- the existing strip de-dup test still passes
- the stubs now patch ranking inside ServedImagesRanker, so the check runs in every Combined Search test
Key test: a mixed-case Commons host still resolves.

Review

One subagent did a combined code, security, trust-and-safety and holistic review of the first version.

Fixed:

The text shown came from the index, not the judge. A crawler could put any title or extract under a vetted file URL. The title and extract are now taken from the store row.
Test gaps: the 47 h boundary, the fast path's negative case, the HTML search page and Super Search are now tested.
The settings comment overstated the swap bound. It now says the bound depends on the recheck's SHA-1 check.

The design also changed after review. At the orchestrator's request the filter moved from formatting into the ranker chain, so that #636's cap counts only files that will be shown, and it now covers Super Search.

Not changed, for you:

The recheck has to keep up with the store. Otherwise approved files silently vanish from results (fail safe). Track the share of stale rows, or scale IMAGE_STORE_RECHECK_LIMIT and the schedule with the store.
Pixel swaps between rechecks. Thumbnail URLs are keyed by name, so a re-upload changes the pixels behind a served thumbnail at once. Until the next recheck (at most 48 h), only Commons' own patrolling stands in the way. This is the known no-pixel-check trust model.
Paths not covered:
- Ranker.complete's go: suggestions can still name a Commons file URL (text only).
- The crawler's raw-results endpoint (/api/v1/crawler, get_raw_results) and /api/v1/search/raw return index items as they are. They are index-sync endpoints, not display endpoints.
The front end must actually escape attribution and text, and use no-referrer. A front-end PR is needed before thumbnails are shown on mwmbl.org; the HTML search page shows none yet.

Part of #610

🤖 Generated with Claude Code

codecov · 2026-11-11T08:43:43Z

Codecov Report

❌ Patch coverage is 97.36842% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
mwmbl/imagesearch/serving.py	97.87%	1 Missing ⚠️
...nysearchengine/super_search_sources/mwmbl_index.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

… for them ServedImagesRanker wraps both served rankers (inside anything that counts images, such as the coming per-list cap): a result that is a Commons File: page is looked up in CommonsImage with one batched query, and kept only if unrevoked and rechecked within IMAGE_SERVE_MAX_CHECK_AGE (48h: a daily recheck plus the search path's day rounding); otherwise it is dropped and the results below move up. No query when no result is a Commons file page. A kept file becomes an ImageDocument with the judged title and description and the store's thumbnail and licence, which /api/v2/search/ and Combined Search return as a nullable `image` with `engine: "commons"`. Super Search checks its sources and merged list too. Combined Search and Super Search's index source rank with sync_to_async so the lookup runs on a thread Django cleans up. commons_file_key checks for "wikimedia.org" before parsing. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

daoudclarke mentioned this pull request Nov 11, 2026

Seed images: ship the LTR retrained with image pages, and cap images at 2 per list #636

Open

daoudclarke added this pull request to stack #632 November 11, 2026 11:10

daoudclarke force-pushed the seed-images-prod-standard-display branch from 215b168 to 35e19e7 Compare November 11, 2026 11:33

Navigation Menu

Repository navigation

Show indexed Commons files in search only as the image store vouches for them - #637

Show indexed Commons files in search only as the image store vouches for them#637
daoudclarke wants to merge 1 commit into
seed-images-prod-indexfrom
seed-images-prod-standard-display

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

What changed

Why 48 hours

User-facing change: crawled Commons file pages disappear

Cost on the hot path

Verification

Review

Uh oh!

Codecov Report

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant