Repository navigation
Move index page I/O into a shared rust/mwmbl_core crate - #639
Closed
daoudclarke wants to merge 7 commits into
Closed
daoudclarke wants to merge 7 commits into
daoudclarke wants to merge 7 commits into
Conversation
No functional changes to the crawler.
No functional changes to the crawler.
No functional changes to the crawler.
No functional changes to the crawler.
No functional changes to the crawler.
daoudclarke
force-pushed
the
claude/funny-fermi-gi5gi0
branch
2 times, most recently
from
November 11, 2026 16:10
294d21e to
d5c668a
Compare
The commits before this one are #629's (rust_crawl) Rust work as written. This keeps what the server uses, index.rs unchanged, and removes the rest, which stays on rust_crawl until it has tests checking it against the Python: the crawler binary, the blacklist and indexer modules, the copied HN domains table, and their dependencies (tokio, reqwest, redis, anyhow). So the server's extension links only serde_json and zstd, at mwmbl_rank's 0.13. Rust tests for the page format: round trips, term filtering, packing to the longest prefix that fits, and each way a page can fail to read. `make rust-core` runs them with rustfmt and clippy, and CI runs it in the test job. The crawler links mwmbl_core through mwmbl_rank, so a change to the page format there needs a crawler release like a change to mwmbl_rank does: add it to CRAWLER_DIRECTORIES. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4qHPfdk1QZ2ZSYHnn7
daoudclarke
force-pushed
the
claude/funny-fermi-gi5gi0
branch
2 times, most recently
from
November 11, 2026 16:14
ce189e2 to
dca2970
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
|
It wouyld be nice if it kept the rest of the rust code so i don't have to reintroduce again all my commits and code. |
Contributor
Author
|
Following @Lama-Thematique's suggestion |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #629.
What changed
The server-side half of Lama's Rust crawler work in #629, so the shared page format can land on main without waiting for the crawler.
History: the first six commits are Lama's commits from
rust_crawl, cherry-picked unchanged (their merges of main are left out). The last commit is mine. It keeps what the server uses and removes the crawler-only parts, which stay onrust_crawl. The overall diff is only what's listed below.rust/mwmbl_core: the new crate fromrust_crawl, holding onlyindex.rs, which is byte-for-byte the same as on that branch.mwmbl_rank/src/index.rskeeps only the PyO3 layer and calls into it. Themwmbl_rankclippy cleanups, the Dockerfile lines and the removal of the strayrust-xgboostgitlink (no.gitmodulesentry) also come from draft: Rust crawler #629.mwmbl_coredepends only onserde_jsonandzstd0.13, matchingmwmbl_rank. The crawler binary, the blacklist and indexer modules and the HN domains table stay onrust_crawl, along with tokio, reqwest, redis and anyhow, so the server's extension doesn't link them. There's one zstd inmwmbl_rank/Cargo.lock, not two.rust/mwmbl_core/tests/index.rs): round trips, neighbouring pages left intact, term filtering (own term, null, truncated tuple, non-string), packing to the longest prefix that fits, and the ways a read can fail (zeroed page, short file, oversized frame header, bad JSON, a syscall error).make rust-corerunscargo fmt --check,clippy -D warningsandcargo teston the crate. It runs in the test job.rust/mwmbl_core/toCRAWLER_DIRECTORIES. The crawler links the crate throughmwmbl_rank, so a change to the page format needs a crawler release.Why
We're merging the Rust work in steps: Rust code goes to main once the server uses it, or once tests check it against the Python. The page I/O is already what the server runs. The crawler-only modules wait until they have tests that compare them with the Python.
For
rust_crawlThese are cherry-picks rather than Lama's original commits, so merging main into
rust_crawlwon't delete the crawler code that main doesn't have. It should only conflict inrust/mwmbl_core/Cargo.toml,lib.rsand the lockfiles. To resolve, keep that branch's module list and dependencies, withzstdat 0.13.Verification
make rust-core: fmt and clippy are clean, and 11 Rust tests pass.uv run maturin develop, then the index and rank tests (test_index_page_writes,test_index_page_api,test_indexer,test_copy_index,test_rust_*,test_rank*): 111 passed.make test: 1883 passed.make check: passed.Size: 579 added lines excluding lockfiles. 251 of those are
index.rsmoved frommwmbl_rank, which loses 229.🤖 Generated with Claude Code
https://claude.ai/code/session_01DTbu4qHPfdk1QZ2ZSYHnn7