8000 Obey every robots.txt group and check robots on redirects by daoudclarke · Pull Request #668 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Obey every robots.txt group and check robots on redirects - #668

Merged
daoudclarke merged 2 commits into
mainfrom
claude/wizardly-mccarthy-wke2iw
Nov 16, 2026
Merged

daoudclarke merged 2 commits into
mainfrom
claude/wizardly-mccarthy-wke2iw

Conversation

@daoudclarke
Copy link
Copy Markdown
Contributor

Closes #333

What changed

  • Protego replaces urllib.robotparser. On Python 3.11, robotparser:

    • keeps only the first User-agent: * group. A site whose first group is Allow: / (proycon's, and cl0ck-byte's Cloudflare-plus-own layout) had all its other rules ignored, honeypot paths included.
    • applies the first matching rule instead of the longest.
    • ignores * and $ in paths.

    Protego follows RFC 9309 on all three.

  • robots.txt groups are matched on mwmbl only. Protego looks for a group's name anywhere in the user agent it's given. With the full UA, groups like User-agent: github or a word from the contact details matched us.

  • Each redirect target is checked against robots.txt. fetch() used to follow redirects without checking, so a redirect could lead us into a disallowed path. Each hop now goes through validate_url before its robots.txt is looked up. Without that, a redirect to ftp://victim/ could cache victim's robots.txt as unavailable and block crawling of that site for an hour.

  • A UTF-8 BOM is stripped. robots.txt is decoded as utf-8-sig, and the BOM is also stripped from cache entries written before this change.

  • Fractional Crawl-delay values are rounded up.

  • CRAWLER_VERSION is now 1.1.3.

Verification

  • New tests cover:
    • proycon's robots.txt (trimmed);
    • a mwmbl group overriding the * group;
    • longest-match;
    • BOM, both when fetched and in an old cache entry;
    • UA-token matching;
    • a redirect into a disallowed path;
    • a redirect to a non-http scheme;
    • an allowed cross-host redirect.
  • Gates: make test passes: 2206 passed, 10 skipped. ruff is clean. ty reports the same 1293 diagnostics as on main.
  • Reviews: a subagent ran a code and security review, and its material findings are fixed in the second commit. A trust-and-safety check found that the change only makes the crawler more conservative. A site can stop us crawling only itself, so it gives no new way to influence results.

Not addressed

  • More requests on long redirect chains. A redirect to a new host now costs an extra robots.txt fetch, so a hostile 5-hop chain now costs about 36 requests per URL, against 6 before. Possible follow-ups: a per-crawl request budget, or a size cap on robots.txt.
  • Old volunteer crawlers. They keep the old behaviour until they update. Consider raising the server's minimum crawler version.

Generated by Claude Code

urllib.robotparser on Python 3.11 keeps only the first "User-agent: *"
group, takes the first matching rule rather than the longest, and
ignores * and $ in paths. A site whose first group was "Allow: /" had
all its other rules ignored, which is what was reported in #333. Parse
with Protego, which follows RFC 9309, and drop a UTF-8 byte order mark
before parsing.

fetch() also followed redirects without checking robots.txt, so a
redirect could take the crawler into a disallowed path. crawl_url now
has each redirect target checked and reports RobotsDenied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4m1aHvu1Fh6Y7JdQaa
From review of the robots.txt fix:
- Validate a redirect target before looking up its robots.txt, so a
  redirect to ftp://victim/ cannot cache victim's robots.txt as
  unavailable and stop us crawling it for an hour.
- Match robots.txt groups against "mwmbl" rather than the full user
  agent: Protego finds a group's name anywhere in the string, so
  "User-agent: github" or a word in the contact details matched us.
- Strip a byte order mark from cache entries written before the fix.
- Round a fractional Crawl-delay up rather than down.
- Require the Protego version that was tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4m1aHvu1Fh6Y7JdQaa
@codecov
codecov Bot commented Nov 16, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@daoudclarke
daoudclarke merged commit 22ad690 into main Nov 16, 2026
7 checks passed
@daoudclarke
daoudclarke deleted the claude/wizardly-mccarthy-wke2iw branch November 16, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Respect robots.txts

2 participants

0