Repository navigation
Show indexed Commons files as images in standard search, from the index alone - #650
Open
daoudclarke wants to merge 2 commits into
Open
daoudclarke wants to merge 2 commits into
daoudclarke wants to merge 2 commits into
Conversation
…ex alone A document whose source is a COMMONS_* member gets a nullable image object in the v2 and Combined Search results: a thumbnail URL derived from the file name, the licence from the source, and the file page. No database read, no request. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…DB mark on test No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From Claude:
What changed
A result whose
sourceis aDocumentSource.COMMONS_*member (set by the indexing PR A; the enum is from #648) now has a nullableimageobject in/api/v2/search/and Combined Search web results:thumbnail_url,license_name,license_url,page_url(the file page, which names the author). Every other hit hasimage: null. No width or height: the front end uses a fixed box. Additive and backwards compatible; v1 and the HTML search page are unchanged.mwmbl/imagesearch/display.pyderives everything from the Document alone: the licence fromsource(LICENCE_NAMES_AND_URLS), the file name from the URL (commons_file_key), and the thumbnail URL from the name:https://upload.wikimedia.org/wikipedia/commons/thumb/{h[0]}/{h[:2]}/{name}/330px-{name},h = md5(name). SVGs get.pngappended. Names over 160 bytes use330px-thumbnail.{ext}(found from the data, see below). Formats we cannot derive confidently (tif, pdf, djvu, video, long SVG names) getimage: nulland stay plain text results.source; md5 and URL parsing run only for Commons results.commonsmeans, that safety is an LLM reading the file's text only (no pixel check), load thumbnails withreferrerpolicy="no-referrer", show the licence and file-page link with the image, escape text, linklicense_urlwithrel="noopener noreferrer nofollow".None) and Staan results (sourceSTAAN/STAAN_INDEXED) for Commons URLs stay plain text with no image.Why
Issue #610. This replaces #637 8000 , whose design read Postgres on every search. Here standard search reads nothing and makes no request: the result is a pure function of the indexed Document. Tested with
django_assert_num_queries(0).It shows nothing until PR A's indexing is switched on (no document has a Commons source before then). A front-end PR is needed to render the thumbnails.
Verification
thumbnail_urlof every Commons file in the Seed Search candidates (devdata/seed_images/candidates): 16,570 distinct files. 16,167 have a/thumb/path, and all 16,167 (100%) match on directory, escaped name and thumb file name (everything but the width). The first pass matched 98.7%; the misses were all names over 160 bytes, which Commons namesthumbnail.{ext}(lower-cased).make checkpasses. The changed areas pass (98 tests: thumbnails, format, Combined Search, OpenAPI). The fullmake testcould not finish here because several concurrent sessions were sharing the one test database and Redis; CI is the full gate.Review (code, security, trust and safety, holistic)
Fixed: a stemless name such as
File:Pngno longer gets a thumbnail; API docs now saycommonsresults of unsupported formats haveimage: null; a test needed the DB mark. Checked: nothing but PR A's indexing can set a COMMONS_* source (crawled documents have none, Staan sets STAAN), so a crawler or Staan cannot get an image shown; no XSS surface added (image fields are constants or a quoted name). Left for decision: the browser loads thumbnails from upload.wikimedia.org, so Wikimedia sees the viewer's IP (the front end should say so).Part of #610
🤖 Generated with Claude Code