8000 Recheck stored Commons files against Commons and the NSFW judge by daoudclarke · Pull Request #634 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]
Skip to content

Recheck stored Commons files against Commons and the NSFW judge - #634

Open
daoudclarke wants to merge 1 commit into
seed-images-prod-storefrom
seed-images-prod-recheck
Open

Recheck stored Commons files against Commons and the NSFW judge#634
daoudclarke wants to merge 1 commit into
seed-images-prod-storefrom
seed-images-prod-recheck

Conversation

@daoudclarke
@daoudclarke daoudclarke commented Nov 11, 2026
Copy link
Copy Markdown
Contributor

From Claude:

Stack: builds on #633 (seed-images-prod-store), which builds on #631. PR 5 of the #610 plan, part 2 of 2: the recheck.

What changed

  • recheck_images management command (mwmbl/imagesearch/recheck.py). It rechecks the files checked longest ago, up to IMAGE_STORE_RECHECK_LIMIT (2000) a run, so it can run on a schedule.
    • Files are fetched by title in batches of 10 (commons.fetch_files), through the existing client with the shared back-off honoured. A back-off or refusal stops the run, leaving the unfinished batch first in line for the next one.
    • Revoked when the file is:
      • deleted (gone), or moved (its descriptionurl canonicalises to another key; the new name gets its own row when Seed Search finds it)
      • missing the metadata the checks need (unusable)
      • nominated for deletion or a copyvio (nominated)
      • in IMAGE_SEARCH_DENIED_FILES (denied)
      • flagged by the deny rules (rules)
      • changed but not yet settled for IMAGE_SEARCH_MIN_REVISION_AGE (unsettled)
    • Judged again when the SHA-1 or the judged text changed, the prompt/model fingerprint is stale, or the revocation said nothing about the content. The file is revoked (nsfw) unless it still passes.
    • Restored only on a fresh pass of changed content, except that unsettled, unusable and unjudged are lifted by any fresh pass. Operator revocations are never rechecked.
    • Unchanged approved files get fresh licence, thumbnail and dates without a judge call.
  • Locking: each batch is re-read and locked (select_for_update) after the Commons request, so a revocation made during a run, by an operator or by a search, is never written over.
  • Judge failures: a judge failure or a malformed answer revokes that batch as unjudged. It is retried next time, and one bad file can no longer wedge every later run.
  • Supporting changes:
    • commons.FILE_PARAMS is shared by search and title fetches.
    • parse_page accepts pages without a search rank.
    • seed_images.is_eligible is split into is_settled, is_denied and is_nominated, so the recheck applies exactly Seed Search's checks.

Why

Approval is a one-time judgment of text that anyone can edit. The store needs to follow deletions, deletion nominations, the denylist, changes to the file or its text, and changes to the NSFW question. Only then can PR 6 index from it and PR 9 drop revoked files at serve time.

Verification

  • make check passes; make test passes: 1924 passed, 10 skipped.
  • New tests (test/test_image_recheck.py, with Commons and the judge mocked) cover every path:
    • fetch_files: one request for all titles, missing pages left out, unusable pages returned as None
    • unchanged files: updated with no judge call
    • one revocation test per reason: gone, moved, unusable, nominated, rules, unsettled, denied
    • changed SHA-1 or text: re-judged, revoked on a fail, approved afresh on a pass
    • stale fingerprint: rescored, or revoked
    • content revoked as NSFW is not re-judged while it is unchanged
    • a judge failure revokes the batch, and the next pass restores it
    • an unchanged revoked file stays revoked
    • changed content restores a revoked file
    • an operator revocation made during a run survives, and operator rows are never fetched
    • the oldest-first limit and batching
    • a back-off stops the run and leaves the batch untouched
  • Review: the combined code, security, trust-and-safety and holistic review on Store every Commons file Seed Search judges safe #633 covered this code too, and its recheck findings are fixed here:
    • Re-judging of failed content: fixed.
    • Stale snapshot writes: fixed with per-batch locking and re-reading.
    • Judge or 4xx failures wedging the run: judge failures now fail closed and recoverably; batches are smaller to avoid 414s.
    • Flaky Commons answers revoked for good: they are now unusable and recoverable, and only missing counts as gone.
    • Moved files: revoked as gone.

Not done (raised for the user)

  • The recheck shares Commons' back-off with live Seed Search. A 429 during a run pauses images for users too, briefly; it runs serially.
  • Files revoked for gone, denied or rules stay revoked while unchanged, even if the denylist or rules change. An operator restores one by deleting its row.
  • Swap risk: a file re-uploaded with the same text is re-judged on text only once it has settled. There is no pixel check, by design. PR 9 should serve only recently rechecked rows, and the schedule should cover the whole store within a bounded time.
  • No schedule is added here. Run uv run manage.py recheck_images from cron, or wrap it in a background task when indexing is switched on.

Part of #610

🤖 Generated with Claude Code

@daoudclarke @claude
Recheck stored Commons files against Commons and the NSFW judge
a8b1808 
recheck_images re-fetches the files checked longest ago, ten titles a request with the
shared back-off honoured, up to IMAGE_STORE_RECHECK_LIMIT a run. It revokes deleted,
moved, nominated, denied and rule-flagged files; when the SHA-1 or judged text changed,
the NSFW fingerprint is stale or a revocation said nothing about the content, it asks
the judge again and revokes unless the file still passes. Each batch is locked and
re-read, so a revocation made during a run is never written over, and a judge failure
revokes the batch recoverably rather than stopping every later run.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov
codecov Bot commented Nov 11, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This was referenced Nov 11, 2026
Open
Open
@daoudclarke
daoudclarke added this pull request to stack #632 November 11, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daoudclarke
0