8000 Run the crawler benchmark with only Docker and python3 by blueheron786 · Pull Request #674 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Run the crawler benchmark with only Docker and python3 - #674

Open
blueheron786 wants to merge 6 commits into
crawler-benchmark-fastfrom
crawler-benchmark-standalone
Open

blueheron786 wants to merge 6 commits into
crawler-benchmark-fastfrom
crawler-benchmark-standalone

Conversation

@blueheron786
Copy link
Copy Markdown
Collaborator

Fixes #672. Builds on #669 (based on its branch).

What changed

  • mwmbl/crawler/benchmark.py is now standard-library only for the compose runner, and runs on Python 3.9+ (macOS's system python3), so it needs no uv, Rust or build. Ruff lints it as py39.
  • The process runner (psutil, redis, python-dotenv) moves to mwmbl/crawler/benchmark_process.py, imported only when --runner process is chosen.
  • deploy/crawler/benchmark.sh checks for python3/docker compose, downloads benchmark.py and runs it, so a volunteer can run, in their crawler directory:
    curl -fsSL https://raw.githubusercontent.com/mwmbl/mwmbl/main/deploy/crawler/benchmark.sh | sh
  • make optimize uses the system python3.
  • Compose runner: --env-file is passed to docker compose instead of loaded with python-dotenv; a missing compose file gets a clear error.

Docs: mwmbl/crawler/README.md, and the book (blueheron786/book branch benchmark-no-build).

Verification

  • make check, test/test_crawler_benchmark.py (56 passed), including a new test that runs the file with python -S (no site-packages).
  • Ran benchmark.py --help and the missing-compose-file path under a real Python 3.9 with -S.
  • Ran benchmark.sh with MWMBL_BENCHMARK_URL=file://...; shellcheck and dash -n clean; temp file cleaned up.
  • Not run end-to-end against a live compose crawler.

Reviewed by subagents (code, security, trust & safety, holistic): no blockers; fixed truncated-download guard, trap handling, py39 lint. Open: memory check uses host RAM (not the Docker Desktop VM's) on macOS — pre-existing.

The compose runner now needs nothing but the standard library, so
benchmark.py runs on its own with any python3 3.9+, with no uv, Rust or
build. deploy/crawler/benchmark.sh downloads it and runs it in the
crawler's directory, so a volunteer can run:

    curl -fsSL https://raw.githubusercontent.com/mwmbl/mwmbl/main/deploy/crawler/benchmark.sh | sh

The process runner, which needs psutil, redis and python-dotenv, moves to
benchmark_process.py and is imported only when chosen. make optimize uses
the system python3.

Fixes #672

No functional changes to the crawler.
…mark.py for 3.9

Fixes #672

No functional changes to the crawler.
@codecov
codecov Bot commented Nov 17, 2026 •
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.95181% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mwmbl/crawler/benchmark_process.py 87.30% 8 Missing ⚠️
mwmbl/crawler/benchmark.py 90.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@blueheron786
blueheron786 added this pull request to stack #675 November 17, 2026 03:25
The process runner now runs the crawler with the Python the benchmark runs
under, with the checkout on PYTHONPATH, so it no longer fails with 'No module
named mwmbl'. It explains a missing checkout or package instead of a
traceback, and deploy/crawler/benchmark-requirements.txt lists what it imports.

Fixes #672

No functional changes to the crawler.
The script always downloaded benchmark.py from main, so a checkout ran main's
older benchmark, which needs psutil, requests and an installed mwmbl. In a
checkout it now runs the file beside it and downloads only when piped from curl.

Fixes #672

No functional changes to the crawler.
A user outside the docker group got a failure from deep inside the benchmark.
The script now checks 'docker info' first, and tells them to join the docker
group, or that the daemon is not running.

Fixes #672

No functional changes to the crawler.
Remove --exhaustive and the climbing search: the benchmark now tries every
combination of the process and thread counts.

Fixes #672

No functional changes to the crawler.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimization is Hard to Use

1 participant

0