Repository navigation
Cut crawler memory by moving wiki scores out of Python dicts - #580
Merged
Merged
Conversation
rank.py loaded wiki_stats.json and document_counts.json into dicts at import, about 150MB in every crawler process. The document counts were only used by get_tf_idf_features, which nothing calls, so they are gone. The wiki scores now come from mwmbl_rank, where build.rs turns the JSON into a sorted table embedded in the library's read-only data: no heap, shared between processes, and a lookup is a binary search (~170ns). One process after scoring a document: 290MB RSS before, 147MB after. Main process plus three forked workers: 473MB PSS before, 331MB after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4o4Ebr1WH6ifqqfqNp
…docs - build.rs asserts the counts have a positive max and checks the u32 casts, instead of silently falling back to zero scores or truncating. - The serde_json build-dependency gets float_roundtrip, like the runtime one. - idf.rs was never called (TF-IDF features are not in the model), so it goes too. - Dockerfile comment and analyse/wiki_stats.py now say the table is built by build.rs, and that mwmbl_rank needs rebuilding after regenerating the stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4o4Ebr1WH6ifqqfqNp
This was referenced Oct 31, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step towards a crawler that fits on small machines (e.g. a Raspberry Pi).
Why
At import,
mwmbl/tinysearchengine/rank.pyloadedwiki_stats.json(14.6MB) anddocument_counts.json(14MB) into Python dicts. That cost about 150MB in every crawler process, and the crawler runs several: main, 2 crawl workers, the indexer and the blacklist-snapshot process. Because of reference counting, forked processes gradually copy those pages. It was the biggest single cost. Redis, the memory-mapped index and the bloom filters cost far less.Changes
document_counts.json:rank.pyno longer loads it. Its only Python user wasget_tf_idf_features, whose one call was commented out, so the function is deleted.wiki_stats.json:get_wiki_scorenow callsmwmbl_rank.wiki_score. A newmwmbl_rank/build.rsturns the JSON at build time into a sorted table embedded withinclude_bytes!.wiki.rsbinary-searches it, so it needs no heap, all processes share it, and the OS can drop it from memory when short. This also replaces the RustHashMapthe server's ranking model used, which cost about 60MB once loaded.CRAWLER_VERSIONis bumped to 1.0.4.test/test_wiki_score.py: scores matchcount / max_countfrom the JSON exactly, including non-ASCII titles.Measurements
Python 3.11,
mwmbl.settings_crawler, measured after scoring documents:rankand scoringThe fork saving is smaller than the single-process one because forked processes already shared some of those pages.
Testing
make checkpasses.test_wiki_score.pyfailed in that run on titles containing/, which a URL's last segment can never match (true before this change too). The test now skips those titles, and it passes withtest_rust_features.py.cargo test wikipasses (4 tests), including a check that every key in the table is found.Local pitfall:
uv run(including insidemake check) can reinstall an old cached build ofmwmbl_rankover a freshmaturin develop. Rebuild withuv run --no-sync maturin developand run the tests with--no-sync. CI builds from scratch, so it isn't affected.Next steps (not in this PR)
rank, mostly pandas andcompleter.pyloading its CSV.HAS_DATABASE=False.Generated by Claude Code