Repository navigation
draft: Rust crawler - #629
Merged
Merged
Conversation
No functional changes to the crawler.
No functional changes to the crawler.
No functional changes to the crawler.
No functional changes to the crawler.
Lama-Thematique
marked this pull request as draft
November 11, 2026 01:17
No functional changes to the crawler.
daoudclarke
pushed a commit
that referenced
this pull request
Nov 11, 2026
The page read, write and pack code moves from mwmbl_rank/src/index.rs into rust/mwmbl_core, so the Rust crawler can share the page format with the server. mwmbl_rank keeps only the PyO3 layer. These files are taken unchanged from #629 (rust_crawl), along with that branch's clippy cleanups to mwmbl_rank and the removal of the stray rust-xgboost gitlink, which had no .gitmodules entry. The crate's Cargo.toml and lib.rs follow in the next commit, so this one does not build on its own. No functional changes to the crawler.
daoudclarke
pushed a commit
that referenced
this pull request
Nov 11, 2026
The page read, write and pack code moves from mwmbl_rank/src/index.rs into rust/mwmbl_core, so the Rust crawler can share the page format with the server. mwmbl_rank keeps only the PyO3 layer. These files are taken unchanged from #629 (rust_crawl), along with that branch's clippy cleanups to mwmbl_rank and the removal of the stray rust-xgboost gitlink, which had no .gitmodules entry. The crate's Cargo.toml and lib.rs follow in the next commit, so this one does not build on its own. No functional changes to the crawler.
daoudclarke
pushed a commit
that referenced
this pull request
Nov 11, 2026
The commits before this one are #629's (rust_crawl) Rust work as written. This keeps what the server uses, index.rs unchanged, and removes the rest, which stays on rust_crawl until it has tests checking it against the Python: the crawler binary, the blacklist and indexer modules, the copied HN domains table, and their dependencies (tokio, reqwest, redis, anyhow). So the server's extension links only serde_json and zstd, at mwmbl_rank's 0.13. Rust tests for the page format: round trips, term filtering, packing to the longest prefix that fits, and each way a page can fail to read. `make rust-core` runs them with rustfmt and clippy, and CI runs it in the test job. The crawler links mwmbl_core through mwmbl_rank, so a change to the page format there needs a crawler release like a change to mwmbl_rank does: add it to CRAWLER_DIRECTORIES. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4qHPfdk1QZ2ZSYHnn7
daoudclarke
pushed a commit
that referenced
this pull request
Nov 11, 2026
The commits before this one are #629's (rust_crawl) Rust work as written. This keeps what the server uses, index.rs unchanged, and removes the rest, which stays on rust_crawl until it has tests checking it against the Python: the crawler binary, the blacklist and indexer modules, the copied HN domains table, and their dependencies (tokio, reqwest, redis, anyhow). So the server's extension links only serde_json and zstd, at mwmbl_rank's 0.13. Rust tests for the page format: round trips, term filtering, packing to the longest prefix that fits, and each way a page can fail to read. `make rust-core` runs them with rustfmt and clippy, and CI runs it in the test job. The crawler links mwmbl_core through mwmbl_rank, so a change to the page format there needs a crawler release like a change to mwmbl_rank does: add it to CRAWLER_DIRECTORIES. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4qHPfdk1QZ2ZSYHnn7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.