Repository navigation
Carry the Commons licence in DocumentSource and block single URLs - #648
Merged
Merged
Conversation
The licence of an indexed Commons file is one of eleven new DocumentSource members, so the result page can credit it without a database read or another request. A moderator can block one URL (BlockedUrl, admin): the URL's hash goes into the existing blacklist snapshot, and retrieval, indexing and the background purge all check it, so a blocked file is dropped, removed and not re-added. commons_file_key moves to imagesearch.commons and normalises one canonical URL. No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The standalone crawler publishes its own blacklist snapshot with no database, so build_snapshot raised ImproperlyConfigured there and the crawler smoke test failed. Guard the read with HAS_DATABASE, as get_curated_domains does, and test that a snapshot builds without one. No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An unparseable URL no longer raises in retrieval, indexing or the web-result filter; the admin refuses a bare domain, an over-long URL and a second spelling of a blocked file; a fragment no longer evades a block; and titles are matched in NFC. The Seed Search strip now skips blocked files, and a web result is no longer dropped when a shown image has no key. Licence coverage is stated correctly (ten licences, 94%). No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
test_blocked_urls imported make_image from another test module, which resolved locally but not in CI, where `test` is not a package, so collection failed for the whole run. No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From Claude:
Alternative to #633. Same goal, without the
CommonsImagetable. Opened separately so the two can be compared; #633 is untouched. PR 5 of the #610 plan. Only one of #633 and this PR should merge (both add migration 0053).Why
#633 stores every safe file in Postgres, with a revoke/lift state machine and a recheck command (#634). That was built for swapped-file vandalism, and a sample of Commons upload logs doesn't show it as common: about 1.3–1.8k overwrites a day across about 150M files (29 and 16 on the two days sampled were reverts). In a 200-event sample, 80% were by the original uploader, mostly bot imports (maps, charts, Lingua Libre audio); I saw no NSFW swaps, but 200 events can't rule out a rate of around 1%. Standard search also can't read Postgres or make extra requests (up to 5M requests a day), so a table can't gate display anyway. This PR keeps what the later PRs need on that path, in the index and the existing blacklist.
What changed
DocumentSource. Eleven new members, from 100: public domain, CC0, CC BY and CC BY-SA at 2.0, 2.5, 3.0 and 4.0, andCOMMONS_OTHER. Across 17,607 seed candidates there are 62 distinct licence names, and the ten listed ones cover 94.3%. The rest go toCOMMONS_OTHER. Wire label is"commons".mwmbl/imagesearch/licences.pymaps Commons'LicenseShortNameto a member. It adds no field to the index. Nothing sets the source on a Commons document yet (that is PR 6), so this half is unused until then.BlockedUrl(admin) holds an exact URL, keyed by canonical page URL for Commons files, so every Commons spelling is caught; any other URL is keyed by itself minus its fragment. Its hash goes into the existing blacklist snapshot with aurl:prefix, so there is no new Redis key and no format change. Retrieval (find_blacklisted_urls), indexing (filter_blacklisted_documents), the purge task and the Seed Search strip all check it, so a blocked file is dropped from results, removed from the index, and not re-added. Saving or deleting a block schedules a snapshot rebuild; a block is not instant (a few minutes plus each worker's next check). Standard search still doesn't touch Postgres.commons_file_keymoves tomwmbl/imagesearch/commons.pyand normalises first-letter case,Image:, underscores/spaces, the mobile host,/w/index.php?title=URLs and NFC. A URL that can't be parsed gives no key rather than an error.HAS_DATABASEis set. The crawler's snapshot therefore carries no URL blocks.Known gaps
Datei:),Special:FilePathURLs and non-Commons URLs that differ by scheme, query string or trailing slash are different URLs and are not blocked. A bare domain is refused by the admin form.CommonsImagehave nothing to build on if this replaces Store every Commons file Seed Search judges safe #633. The newDocumentSourcevalues must be deployed to every worker before PR 6 writes any, becauseDocumentraises on an unknown source.DocumentSourceenum on therust_crawlbranch isn't mirrored; it isn't onmain.Verification
make checkpasses. Fullmake testpasses: 1909 passed, 10 skipped. CI's crawler smoke job failed on the first push (the crawler has no database); fixed in ef7f803.test/test_blocked_urls.py(key spellings, fragments, NFC, unparseable URLs, snapshot lookups, domain not blocked, retrieval, indexing, purge, the Seed Search strip, admin validation, rebuild scheduling, building without a database) andtest/test_licences.py.Nonekey in the web-result filter, all fixed here; and a few low items left as known gaps above. Trust and safety and holistic found the strip not honouring blocks (fixed), the licence-coverage figure being stated wrong (fixed), and the gaps above.Part of #610
🤖 Generated with Claude Code