Repository navigation
Add a crawl-key endpoint for a device's results - #664
Merged
Merged
Conversation
GET /api/v1/crawler/device-results returns how many results the named device has submitted, in total and since a given time, together with the server's clock. It authenticates with the crawl-scoped key the crawler already holds and only ever covers that key's owner's devices. The devices page reads these counts through JWT-only endpoints, so a crawler (or a tool driving one, such as the benchmark to come) could not read its own device's results without the account password. Summing the per-submission rows since a time gives exactly the interval asked for, unlike the rolling 24-hour figure, and timing it with the server's clock means the caller's own clock does not matter. The existing JWT endpoints are unchanged. No functional changes to the crawler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A `since` of nan, inf or a huge value reached datetime.fromtimestamp and raised ValueError or OverflowError, answering 500. Bound it in the query parameter so it is a 422, like any other malformed input. No functional changes to the crawler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
record_device truncates a device name to the column's length, so a longer one was saved truncated and device-results, which looked it up as sent, reported it unknown for ever. Truncate the lookup the same way, and cover the inclusive `since` boundary. No functional changes to the crawler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
Review resultsReviewed independently by three subagents (code review, security review, trust-and-safety/holistic) before asking anyone to look. Fixed in this PR
Security review: no other findings. Checked and clear: key scope ( Acknowledged, not changed
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
GET /api/v1/crawler/device-results?device_name=…&since=<unix timestamp>returns, for the named device of the key's owner:known: whether the account has a device of that nameresults_total: results it has submitted since it was first countedresults_since: results submitted at or aftersince(0 whensinceis omitted)crawler_version: from its latest submissionserver_time: the server's clock, to use as a latersinceIt takes a crawl-scoped key in
X-API-Key, as/resultsand/errorsdo, and only covers that key's owner's devices. An unknown name answers 200 withknown: false, since "this crawler has not submitted yet" is a normal state for a caller to see.Why
The devices page reads these counts through JWT-only endpoints, so a crawler, or a tool driving one, cannot read its own device's results without the account password. The crawler benchmark (follow-up PRs) needs the contribution of one device over one interval. The existing figures cannot give that:
last 24his a rolling window shared by everything that ran in it, and subtracting two totals races with submissions in flight. Summing the storedDeviceSubmissionrows fromsincegives exactly the interval, and using the server's clock for both ends means the caller's clock is irrelevant.The JWT endpoints and the key scopes are unchanged; no existing endpoint accepts a new kind of credential.
Notes for review
nowinpost_results), not the end, so a slow request that straddlessinceis credited to the earlier side.DeviceSubmissionrows are pruned after 30 days, sosinceolder than that undercounts.results_totalis unaffected.Verification
make checkmake test-file FILE=test/test_device_results.py: new tests cover the windowed count, total-only, unknown device, another user's device not being visible, and 401 for no key or a non-crawl key.mwmbl_devicesubmissionrows: identical (529 and 950).Part of the crawler benchmark work (no tracking issue yet). Followed by two PRs: the benchmark itself, then its runners and CLI.