Repository navigation
Release crawler 1.0.0 and publish on every version bump - #561
Merged
Merged
Conversation
- Raise CRAWLER_VERSION to 1.0.0. - crawler-release.yml now runs when version.py changes on main: it publishes mwmbl/crawler:X.Y.Z, :X.Y and :stable, then pushes the crawler-vX.Y.Z tag. A version that is already tagged is skipped. - CI fails a pull request that changes the crawler without raising CRAWLER_VERSION. The crawler's code is every mwmbl module loaded by importing mwmbl.crawl, plus the Rust extension, the lockfile, pyproject.toml and Dockerfile.crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
- Drop docker-publish.yml, which rebuilt mwmbl/crawler:latest on every merge to main. The release workflow now pushes :latest with each release, so setups pulling the untagged image move with releases. - Pin docker-compose.yml to mwmbl/crawler:stable. - Point the stats page at the book's Docker Compose guide instead of the archived crawler-script repo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Count mwmbl/resources/ and the justext stoplists as crawler code: modules the crawler loads read them at import time, and they ship in the image. - Say in the failure message that a rollback ships as a new, higher version. - Let the module-listing subprocess's stderr through, so an import error shows in CI. - Note in crawler-release.yml that a version queued behind two others is skipped. - Update a comment that still described the tag-triggered release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers each outcome of main() and the git helper, which codecov/patch flagged as untested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
|
What kind of versioning system are we planning to use? I was leaning towards semantic versioning, which is why I suggested a version |
Contributor
Author
|
Semantic versioning is great |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
CRAWLER_VERSIONis now1.0.0.crawler-release.ymlnow runs on pushes tomainthat changemwmbl/crawler/version.py, instead of on a pushed tag. It builds and pushesmwmbl/crawler:X.Y.Z,:X.Y,:stableand:latest, then tags the commitcrawler-vX.Y.Z. If that tag already exists (the version didn't change, or was reverted to an older one), it does nothing. Because the workflow only runs frommain, the old "is the tag on main" check isn't needed. Releases share a concurrency group, so two merges can't race on:stable.python -m mwmbl.crawler.release_check HEAD^1, fails when crawler code changed butCRAWLER_VERSIONdidn't go up. "Crawler code" means everymwmblmodule loaded by importingmwmbl.crawl(the image's entry point), plusmwmbl_rank/,uv.lock,pyproject.tomlandDockerfile.crawler. The import runs in a subprocess with a throwawayHOME, so its ~400MB index never lands in~/.mwmbl. Modules the crawler only imports inside functions aren't seen.:latestfollows releases: deleteddocker-publish.yml, which rebuilt:lateston every merge to main. Now that every crawler change has to bump the version, that rebuild was redundant. Setups that pull the untagged image (including existing compose installs) get each release and nothing in between.docker-compose.ymlnow pinsmwmbl/crawler:stable.mwmbl/crawler-scriptrepo. It now links to the book's Docker Compose guide, the same one the README uses.Why
Without this, a crawler change could merge and never reach volunteers on
:stable, because someone had to remember to bump the version and push a tag by hand.Note: the imported set includes shared modules such as
settings_common.py,models.pyandtinysearchengine/, and alsouv.lock. Changing any of these means a crawler version bump too. That's on purpose: they change what the image runs.Verification
test/test_crawler_release_check.pypasses, and so doesmake check.main. A commit that touchedmwmbl/crawl.pywithout a bump failed and listed the file. The same change with the bump to 1.0.0 passed.crawler-v1.0.0.🤖 Generated with Claude Code