Repository navigation
Share one blacklist snapshot across crawler processes - #566
Merged
daoudclarke merged 3 commits intoOct 30, 2026
Merged
Conversation
Problem: on a 4GB / 2 CPU machine the kernel OOM-killed crawler processes
with CRAWLER_WORKERS of 2 to 4, even at CRAWL_THREADS=10, while
CRAWLER_WORKERS=1 with CRAWL_THREADS=80 ran fine. Memory scaled with the
number of processes, not threads.
Cause: every worker built its own remote blacklist providers (HaGeZi
tif_medium, ~390k domains, and the Block List Project adult list, ~950k
domains) and parsed them into Python sets of str, ~100MB+ per list:
- RedisURLQueue.__init__ called get_default_blacklist_provider(), so
each worker downloaded and held both lists for its whole life.
- record_urls_in_database() built another provider on every batch,
re-downloading and re-parsing both lists each time, with a transient
peak well above the final size (response text, split lines and the
set alive together).
That is roughly 300MB held plus a larger transient per worker, times the
workers, plus the indexer and parent.
Fix: the crawler now reads the hashed snapshot the server already uses
(mwmbl.indexer.blacklist_snapshot), ~11MB per process instead of ~300MB.
- Crawler.refresh_blacklist_snapshot() downloads and parses the lists
in a short-lived subprocess and publishes the snapshot to the
crawler's own Redis, so the parsing memory is returned to the OS when
it exits. run() and run_once() do this before any worker starts.
- Each process lazily loads the snapshot after the fork through a
SnapshotBlacklist and re-checks Redis every
BLACKLIST_SNAPSHOT_CHECK_SECONDS.
- Crawler.new_blacklist_provider() wraps it in a
CombinedBlacklistProvider with the curated-domains exemption, so
approved domains still override the lists as before. A fresh wrapper
is made per batch because it resolves the exemption once and would
otherwise never see a later approval.
- The run() loop republishes every BLACKLIST_SNAPSHOT_REFRESH_SECONDS
(6h). A failed refresh is logged and the current snapshot kept.
- record_urls_in_database() takes the provider as an argument instead
of building one per batch.
- SnapshotBlacklist now subclasses BlacklistProvider (typing only).
Behaviour changes:
- A crawler started without network access now fails at startup rather
than retrying every 10s, since the first snapshot must exist before
workers start.
- Blocklist updates reach the crawler within 6 hours, not per batch.
- Not touched: analyse/update_urls.py calls record_urls_in_database()
with the old signature, but it was already broken (stale import,
wrong queue type).
Tests: added coverage for a failed periodic refresh not stopping the
supervisor, a failed publisher raising, and the queue and batches sharing
one snapshot; updated the supervisor, --once and update_urls tests for
the new refresh step and signature. make check passes (ty is at the
existing 1123-diagnostic baseline) and the 8 related test files (125
tests) pass. Not run on the XA11 itself, and the full suite was not run.
-- GitHub Copilot (Claude Sonnet 5.5)
…shot Follow-up to d45c8be from the review of that commit. - release_check.py fails a pull request that changes crawler modules (mwmbl/crawl.py, update_urls.py, blacklist_snapshot.py) without raising CRAWLER_VERSION, and the bump is what publishes the release to volunteers. - mwmbl/crawler/README.md now says memory scales with CRAWLER_WORKERS rather than CRAWL_THREADS, and describes the blocklist snapshot: downloaded once at startup (the crawler exits if it cannot be), refreshed every 6 hours, and kept if a refresh fails. -- GitHub Copilot (Claude Sonnet 5.5)
refresh_blacklist_snapshot() joined the publisher subprocess with no timeout, so a hang there would stop the supervisor from restarting dead workers. It now waits SNAPSHOT_PUBLISH_TIMEOUT_SECONDS (600: two downloads of up to 60s each, plus parsing and hashing), kills the process and raises, which run() already logs and survives; at startup it is fatal as before. Raised in the review of d45c8be. Test added for the hang case. -- GitHub Copilot (Claude Sonnet 5.5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Each crawler worker now reads one shared blacklist snapshot (~11 MB hash array in Redis) instead of building its own sets of ~1.3M domain strings (~300 MB each).
Why
On a 4 GB / 2 CPU machine the kernel OOM-killed crawler processes with
CRAWLER_WORKERSof 2 to 4 (even atCRAWL_THREADS=10), while 1 worker with 80 threads ran fine. Memory scaled with processes, not threads:RedisURLQueue.__init__calledget_default_blacklist_provider(), so each worker downloaded and held the HaGeZitif_mediumand Block List Project adult lists for its whole life.record_urls_in_database()built another provider on every batch, re-downloading and re-parsing both lists each time, with a transient peak well above the final size.How
Crawler.refresh_blacklist_snapshot()downloads and parses the lists in a short-lived subprocess and publishes the existingblacklist_snapshotto the crawler's own Redis, so the memory is returned to the OS on exit.run()andrun_once()do this before any worker starts. It is bounded bySNAPSHOT_PUBLISH_TIMEOUT_SECONDS(600).SnapshotBlacklist, wrapped in aCombinedBlacklistProviderso approved (curated) domains still override the lists. The wrapper is rebuilt per batch because it resolves the exemption once.run()republishes everyBLACKLIST_SNAPSHOT_REFRESH_SECONDS(6h). A failed refresh is logged and the current snapshot kept.record_urls_in_database()takes the provider as an argument.SnapshotBlacklistsubclassesBlacklistProvider(typing only).CRAWLER_VERSION1.0.0 -> 1.0.1 (required byrelease_check.py); crawler README documents the memory behaviour.Behaviour changes
Verification
make checkpasses;tyis at the existing 1123-diagnostic baseline.-k "release or version or crawl_supervisor". New tests cover a failed periodic refresh, a failed publisher, a hung publisher, and the queue and batches sharing one snapshot. The full suite was not run, and it has not been tried on the XA11.Reviews (subagents)
printfindings were not applicable (fork start method; pre-existing code). The join timeout it asked for is included.build_snapshot). Not changed here.Follow-ups for a maintainer
build_snapshotwould publish an empty or truncated blocklist without complaint (affects the server too); a minimum-size check would fix it.BLACKLIST_SNAPSHOT_REFRESH_SECONDSwould narrow the staleness window.-- GitHub Copilot (Claude Sonnet 5.5)