Repository navigation
Obey every robots.txt group and check robots on redirects - #668
Merged
Merged
Conversation
urllib.robotparser on Python 3.11 keeps only the first "User-agent: *" group, takes the first matching rule rather than the longest, and ignores * and $ in paths. A site whose first group was "Allow: /" had all its other rules ignored, which is what was reported in #333. Parse with Protego, which follows RFC 9309, and drop a UTF-8 byte order mark before parsing. fetch() also followed redirects without checking robots.txt, so a redirect could take the crawler into a disallowed path. crawl_url now has each redirect target checked and reports RobotsDenied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4m1aHvu1Fh6Y7JdQaa
From review of the robots.txt fix: - Validate a redirect target before looking up its robots.txt, so a redirect to ftp://victim/ cannot cache victim's robots.txt as unavailable and stop us crawling it for an hour. - Match robots.txt groups against "mwmbl" rather than the full user agent: Protego finds a group's name anywhere in the string, so "User-agent: github" or a word in the contact details matched us. - Strip a byte order mark from cache entries written before the fix. - Round a fractional Crawl-delay up rather than down. - Require the Protego version that was tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4m1aHvu1Fh6Y7JdQaa
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #333
What changed
Protego replaces
urllib.robotparser. On Python 3.11, robotparser:User-agent: *group. A site whose first group isAllow: /(proycon's, and cl0ck-byte's Cloudflare-plus-own layout) had all its other rules ignored, honeypot paths included.*and$in paths.Protego follows RFC 9309 on all three.
robots.txt groups are matched on
mwmblonly. Protego looks for a group's name anywhere in the user agent it's given. With the full UA, groups likeUser-agent: githubor a word from the contact details matched us.Each redirect target is checked against robots.txt.
fetch()used to follow redirects without checking, so a redirect could lead us into a disallowed path. Each hop now goes throughvalidate_urlbefore its robots.txt is looked up. Without that, a redirect toftp://victim/could cache victim's robots.txt as unavailable and block crawling of that site for an hour.A UTF-8 BOM is stripped. robots.txt is decoded as
utf-8-sig, and the BOM is also stripped from cache entries written before this change.Fractional
Crawl-delayvalues are rounded up.CRAWLER_VERSIONis now 1.1.3.Verification
mwmblgroup overriding the*group;make testpasses: 2206 passed, 10 skipped.ruffis clean.tyreports the same 1293 diagnostics as onmain.Not addressed
Generated by Claude Code