8000 Make the crawler benchmark run in minutes and read .env by blueheron786 · Pull Request #669 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Make the crawler benchmark run in minutes and read .env - #669

Open
blueheron786 wants to merge 3 commits into
mainfrom
crawler-benchmark-fast
Open

blueheron786 wants to merge 3 commits into
mainfrom
crawler-benchmark-fast

Conversation

@blueheron786
@blueheron786 blueheron786 commented Nov 16, 2026 •
Copy link
Copy Markdown
Collaborator

What changed

  • make optimize (alias make crawler) runs mwmbl benchmark.
  • The benchmark now scores each configuration by fetches per minute, read from the crawler's own Redis counter (crawl-domain-overlap fetches, which every worker already increments as a batch finishes). It no longer asks the server how many new URLs were indexed. The default crawler still needs an API key to run.
  • The measurement window opens at the first batch to finish after warmup and closes at the first to finish after --duration, so it is never cut through a batch.
  • The default search climbs: threads go up for a process count until more of them gain under 10%, and process counts go up until doubling them gains under 10% at the lowest thread count. --exhaustive tries all combinations.
  • Defaults are 10s warmup and 30s measurement. The container stop grace is 2s instead of 10s, since the crawler is PID 1 with no SIGTERM handler.
  • .env beside docker-compose.yml (or --env-file) is loaded into the environment; variables already set win. Before, the benchmark's own process never read .env, which is why MWMBL_API_KEY had to be exported.
  • The Compose runner checks required settings before stopping the crawler, reports how to inspect container logs on failure, and exits nonzero if no configuration completes.
  • .gitignore ignores benchmark-*.csv. The default CSV is now benchmark-results-fetches-<date>.csv because the columns changed.

Why

Scoring by what the server indexed took 35 minutes per configuration (about 700 minutes for the 20-configuration grid), because submissions reach the server long after the fetches behind them.

How it was verified

  • test/test_crawler_benchmark.py: 53 tests, including startup validation and failure reporting. make check passed.
  • make test: 2,216 passed, 10 skipped, 4 deselected, with an isolated Redis, PostgreSQL test database and temporary filesystem.
  • Ran uv run mwmbl benchmark against a real docker crawler: 8 configurations in 8m20s (about 85x faster). The crawler was left stopped, as it started.
  • Reviewed with a standard code review and a security review. A follow-up review found and fixed Compose profile handling in startup validation.

Caveats

  • This measures fetch throughput, not ho 8000 w many pages were new or worth indexing. Differences under about 10% are noise at these durations, and more threads always fetch more, so the README says to prefer the cheaper configuration when two are close.
  • The server's device-results endpoint is no longer used by the benchmark; it is left in place.
  • Trust and safety: a local developer tool that touches neither search results nor the index.

No functional changes to the crawler.

blueheron786 and others added 2 commits November 16, 2026 16:55
Measure fetches a minute from the crawler's own Redis counter instead of
what the server indexed, search by climbing threads and processes until
they stop helping, add make optimize, and load .env beside the compose file.

No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov
codecov Bot commented Nov 16, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.nply.github.com>
@daoudclarke
Copy link
Copy Markdown
Contributor

I don't think it makes sense to run the benchmark for such a short time. Each process is started staggered so I don't think you'll even have all processes running or have done a sync before you stop the benchmark!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0