8000 Write approved Commons files to the search index, behind IMAGE_INDEXING_ENABLED by daoudclarke · Pull Request #635 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]
Skip to content

Write approved Commons files to the search index, behind IMAGE_INDEXING_ENABLED - #635

Open
daoudclarke wants to merge 2 commits into
seed-images-prod-recheckfrom
seed-images-prod-index
Open

Write approved Commons files to the search index, behind IMAGE_INDEXING_ENABLED#635
daoudclarke wants to merge 2 commits into
seed-images-prod-recheckfrom
seed-images-prod-index

Conversation

@daoudclarke
@daoudclarke daoudclarke commented Nov 11, 2026 •
edited
Loading
Copy link
Copy Markdown
Contributor

From Claude:

Stack: builds on #634 (seed-images-prod-recheck), which builds on #633. PR 6 of the #610 plan.

What changed

  • index_images management command (mwmbl/imagesearch/indexing.py, mwmbl/management/commands/index_images.py). It writes approved files from the CommonsImage store to the main index as ordinary documents:
    • URL: the canonical store key, https://commons.wikimedia.org/wiki/File:….
    • Title: the caption, or failing that the file name with no extension, underscores turned to spaces and whitespace collapsed. When the file name is a camera's (IMG_1234) and there is a description, the title is the description's opening words, cut at a word boundary to the title length.
    • Extract: the description. Categories are left out.
    • Lengths: both are cut to the crawler's limits (NUM_TITLE_CHARS 65, NUM_EXTRACT_CHARS 155), the same way crawler/app.py cuts them.
    • Path: the documents go through index_documents, the path crawled pages take. That means they are indexed under their own terms and never under any query, with the same heuristic write priority, the same cleaning and the normal blacklist check.
  • Skipping files with no real text. A file is skipped only when it has no caption, no description, and a file name that is only a camera, phone or date name. Camera names recognised: IMG_1234, IMG_E1234, DSC_0045, P1010001, IMG-20230101-WA0001, PXL_…~2, DSC01234-Enhanced-NR, SANY0001, 20230101_123456, Screenshot 2023-01-01 at 12.34.56.
    • Short numbers are kept, so P-51, 747, 1984 and WP 40 count as real text. A camera prefix needs at least 3 digits after it; a bare number needs at least 6.
  • Write once, re-write on change. A new field CommonsImage.indexed_text_hash (migration 0054) records the text_hash last taken, whether that file was written or skipped.
    • A run takes non-revoked rows where indexed_text_hash != text_hash, oldest first, up to IMAGE_INDEX_LIMIT (1000).
    • After a recheck changes a file's text, the file is written again.
    • A row whose text changed or that was revoked during the run is not marked as done, so the next run takes it again.
    • The command reports how many files the index stored, how many it dropped (pages already full of higher-priority documents, or the blacklist), and how many were skipped.
  • Minimum upload age. Only files uploaded at least IMAGE_INDEX_MIN_UPLOAD_AGE (30 days) ago are written, against spam. A younger file isn't marked, so a later run takes it. Commons deals with copyright violations and obvious spam by speedy deletion, usually within days. Deletion requests are slower: one Phabricator analysis found about 38% closed within a week and most of the rest within a month, with a tail past three months. The recheck already revokes a file once it is nominated, so 30 days covers the fast path and nominations cover the slow one.
  • IMAGE_INDEXING_ENABLED (env, default off) gates the command. It is separate from IMAGE_SEARCH_ENABLED.

Why a command and not the store's thread

  • A command run is bounded and can be put on a schedule.
  • The index's page locks and disk I/O stay out of the web processes.
  • It also picks up text changes that recheck_images finds, which no search ever sees.

The privacy decision says image writes should not be batched or delayed. The command is chosen for the operational reasons above, not for privacy, and it batches writes only as far as the schedule does. It is easy to run often. No schedule is wired up here: switching indexing on is PR 10.

Revocation

  • Revoked rows are never written.
  • The index can't delete a document, so a file revoked after it was written stays in the index. Revocation takes effect at serving time instead: PR 9 drops results that are not in the store or are revoked.
  • The settings comment says not to turn on IMAGE_INDEXING_ENABLED before then.
  • If a file's text changes, its old copy stays under the terms only that old text had. A re-crawled page behaves the same way.

Verification

  • make check passes. make test: 1967 passed, 10 skipped.
  • test/test_image_indexing.py (43 tests) covers:
    • Document construction: title from the caption or the cleaned file name, the description as extract, last_crawled, and the crawler's lengths.
    • The no-real-text filter: 18 camera/date names with no other text, 10 real names including the short-number cases, a caption that rescues a camera file name, and a camera file titled by its description's opening words.
    • The upload age: a file uploaded too recently is left unmarked and written once it is old enough.
    • Idempotence: written once, under its own terms; the second run writes nothing; a skipped row is marked.
    • Re-writing after a text change.
    • Revoked rows are skipped.
    • Mid-run changes: a row changed or revoked during the run is left unmarked.
    • Blacklist: a blacklisted file is counted as dropped.
    • The limit bounds a run.
    • The command: writes to DATA_PATH/INDEX_NAME, and writes nothing while switched off.

Review

One subagent did a combined code, security, trust-and-safety and holistic review.

Fixed:

  • The regex wrongly flagged short numbers and the WP, SAM and P prefixes as camera names. It also missed SANY, VID, IMG_E, -HDR, -Enhanced-NR, ~2, and screenshot or WhatsApp names with "at".
  • The indexed count included documents the index trimmed. It now reports stored and dropped from index_documents' result.
  • A row revoked during a run was marked as done.
  • The setting was checked twice. Only the command checks it now.
  • The setting's comment now tells an operator not to enable it before PR 9.

Decided by the user since: a camera-named file with a description is kept and titled from the description; a 30-day minimum upload age.

Not changed, for the user to decide:

  • Privacy side channel. No query is stored, but the set of indexed files follows from what people searched for. A Commons file on a sensitive topic appearing in public search says that someone's search fetched it, though not who or when. This is now stated in the module docstring. The plan's privacy decision ruled out batching or delaying writes for this reason. The reviewer suggests settling it before switching on, for example by indexing a file only once Seed Search has seen it on k distinct days.

  • Uploader spam. An uploader controls the caption and description, which become the title and extract. The NSFW judge doesn't look for spam. What limits this:

    • the crawler's length cuts
    • the 7-day minimum revision age
    • Commons patrolling
    • the URL is always a Commons page, so no outbound link is planted

    The reviewer suggests:

    • a minimum upload age (done since, by user decision: 30 days), or requiring the file to be used on Wikipedia
    • deny rules on the index text
    • a distinct DocumentSource, so these documents can be filtered or purged as a group

  • Eviction cost. Up to 1000 low-prior documents a run compete for term pages. Measure before switching on.

  • Category-only edits. text_hash covers categories, so a category-only edit re-writes an unchanged document. This is harmless.

Part of #610

🤖 Generated with Claude Code

@daoudclarke @claude
Write approved Commons files to the search index, behind IMAGE_INDEXI…
eaad237 
…NG_ENABLED

The index_images command takes up to IMAGE_INDEX_LIMIT approved files whose text the
index hasn't taken (CommonsImage.indexed_text_hash != text_hash) and writes each through
index_documents as an ordinary page: canonical Commons URL, caption or cleaned file name
as title, description as extract, cut to the crawler's lengths, under its own terms with
the crawled pages' write priority and blacklist. Files with no real text (no caption or
description, or a camera/date file name) are skipped. Revoked files are never written;
one revoked or changed mid-run isn't marked as taken. Off by default, and separate from
IMAGE_SEARCH_ENABLED.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov
codecov Bot commented Nov 11, 2026
Copy link
Copy Markdown
8000

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This was referenced Nov 11, 2026
Open
Open
@daoudclarke
daoudclarke added this pull request to stack #632 November 11, 2026 11:10
@daoudclarke @claude
Index camera-named files by their description, and only files a month…
b3e5e37 
… old

A file whose name is a camera's (IMG_1234) but which has a description is now
indexed, titled by the description's opening words cut at a word boundary; only a
file with no caption, no description and a camera's name is skipped. Files uploaded
less than IMAGE_INDEX_MIN_UPLOAD_AGE (30 days) ago are left for a later run, so
Commons has time to delete spam and copyright violations before they reach the index.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daoudclarke
0