Repository navigation
#557: Crawler version spread for staff, and versioned crawler releases - #558
Merged
Merged
Conversation
Record, per day and per crawler version, how many crawlers submitted and how many results they sent, and serve the last seven days to staff at GET /api/v1/platform/crawler-versions. A crawler is a user and a device name, not an API key, since one key can serve several crawlers and one user can hold several keys. Move CRAWLER_VERSION into mwmbl/crawler/version.py, which imports nothing, and add a workflow that publishes mwmbl/crawler:X.Y.Z, :X.Y and :stable for a crawler-vX.Y.Z tag, refusing a tag that does not match it.
This was referenced Oct 27, 2026
Review fixes: - A client-supplied version containing a digit int() rejects (e.g. "1.²") made the staff endpoint raise a 500 for as long as the counters lived. Only X.Y.Z versions of ASCII digits are now given their own row; anything else is counted as "unknown", which also stops a client minting a row and Redis keys per request. - Cut the device name to MAX_ERROR_TAG_LENGTH before it goes in the Redis set. - Read the per-version crawler counts in one pipelined round trip. - Only publish a crawler-vX.Y.Z tag whose commit is on main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
This is looking good. Can you clarify the deploy/release procedure? |
Collaborator
Author
|
As usual, this (back-end) before the front end. If you push a tag in the format of |
This was referenced Oct 31, 2026
daoudclarke
pushed a commit
that referenced
this pull request
Oct 31, 2026
Part of #557. The version-spread dashboard (#558) and tagged crawler releases are merged, so the server can now tell old crawlers to upgrade: POST /api/v1/crawler/results returns 426 Upgrade Required when the submitted crawler_version is below the new MIN_CRAWLER_VERSION setting (default 0.0.0, which accepts every crawler, including ones too old to report a version). The crawler waits out an hour on a 426 instead of crash-looping, giving an external updater time to replace the process. CRAWLER_VERSION is raised to 1.0.2 because this changes what the crawler runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #557.
Front-end: mwmbl/front-end#99
What
GET /api/v1/platform/crawler-versions(403 for anyone who isn't staff).CRAWLER_VERSIONintomwmbl/crawler/version.py, which imports nothing.crawler-release.yml: pushing acrawler-vX.Y.Ztag publishesmwmbl/crawler:X.Y.Z,:X.Yand:stable, and refuses a tag that doesn't matchCRAWLER_VERSION.:lateststill follows main.Why
The epic asks for "what spread of versions users are actually crawling" before any crawler is told to update. A crawler is counted as a user plus a device name, not an API key: one key can serve several crawlers and one user can hold several keys. Counts live in Redis with the same 30 day expiry as the other daily stats, so there is no migration.
How it was verified
make checkpasses.tyreports 8 more warnings than before, all of the kinds the repo already has (Django and redis stubs,ninja_jwttoken types).test/test_crawler_versions.pycover the counting, the per-user device identity, version ordering, the 7-day window, truncation, expiry, the real/resultsendpoint, dry runs not being counted, and the staff gate on the new endpoint.Screenshot
The page in the front-end PR, fed by this endpoint, with sample data:
After merging
Nothing is tagged yet. To publish the first release:
git tag crawler-v0.2.0 && git push upstream crawler-v0.2.0. Docker Hub secrets are needed, and the release workflow has not been run.