8000 Add a crawl-key endpoint for a device's results by blueheron786 · Pull Request #664 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Add a crawl-key endpoint for a device's results - #664

Merged
daoudclarke merged 3 commits into
mainfrom
device-results-endpoint
Nov 16, 2026
Merged

daoudclarke merged 3 commits into
mainfrom
device-results-endpoint

Conversation

@blueheron786
Copy link
Copy Markdown
Collaborator

What

GET /api/v1/crawler/device-results?device_name=…&since=<unix timestamp> returns, for the named device of the key's owner:

  • known: whether the account has a device of that name
  • results_total: results it has submitted since it was first counted
  • results_since: results submitted at or after since (0 when since is omitted)
  • crawler_version: from its latest submission
  • server_time: the server's clock, to use as a later since

It takes a crawl-scoped key in X-API-Key, as /results and /errors do, and only covers that key's owner's devices. An unknown name answers 200 with known: false, since "this crawler has not submitted yet" is a normal state for a caller to see.

Why

The devices page reads these counts through JWT-only endpoints, so a crawler, or a tool driving one, cannot read its own device's results without the account password. The crawler benchmark (follow-up PRs) needs the contribution of one device over one interval. The existing figures cannot give that: last 24h is a rolling window shared by everything that ran in it, and subtracting two totals races with submissions in flight. Summing the stored DeviceSubmission rows from since gives exactly the interval, and using the server's clock for both ends means the caller's clock is irrelevant.

The JWT endpoints and the key scopes are unchanged; no existing endpoint accepts a new kind of credential.

Notes for review

  • Submissions are timed from the start of their request (now in post_results), not the end, so a slow request that straddles since is credited to the earlier side.
  • DeviceSubmission rows are pruned after 30 days, so since older than that undercounts. results_total is unaffected.

Verification

  • make check
  • make test-file FILE=test/test_device_results.py: new tests cover the windowed count, total-only, unknown device, another user's device not being visible, and 401 for no key or a non-crawl key.
  • Ran a real crawler against a local server and compared this endpoint's windowed counts with a direct sum over the mwmbl_devicesubmission rows: identical (529 and 950).

Part of the crawler benchmark work (no tracking issue yet). Followed by two PRs: the benchmark itself, then its runners and CLI.

GET /api/v1/crawler/device-results returns how many results the named
device has submitted, in total and since a given time, together with the
server's clock. It authenticates with the crawl-scoped key the crawler
already holds and only ever covers that key's owner's devices.

The devices page reads these counts through JWT-only endpoints, so a
crawler (or a tool driving one, such as the benchmark to come) could not
read its own device's results without the account password. Summing the
per-submission rows since a time gives exactly the interval asked for,
unlike the rolling 24-hour figure, and timing it with the server's clock
means the caller's own clock does not matter.

The existing JWT endpoints are unchanged.

No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
blueheron786 and others added 2 commits November 15, 2026 21:13
A `since` of nan, inf or a huge value reached datetime.fromtimestamp and
raised ValueError or OverflowError, answering 500. Bound it in the query
parameter so it is a 422, like any other malformed input.

No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
record_device truncates a device name to the column's length, so a longer
one was saved truncated and device-results, which looked it up as sent,
reported it unknown for ever. Truncate the lookup the same way, and cover
the inclusive `since` boundary.

No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@blueheron786
Copy link
Copy Markdown
Collaborator Author

Review results

Reviewed independently by three subagents (code review, security review, trust-and-safety/holistic) before asking anyone to look.

Fixed in this PR

Finding Severity Fix
since of nan, inf or a huge value reached datetime.fromtimestamp and answered 500 (reported by security and code review) Low Bounded in the query parameter (0 <= since <= 4e9), so it is a 422. Tested with nan, inf, -1, 1e20.
A device name longer than the column was saved truncated by record_device, but this endpoint looked it up as sent, so it was reported unknown for ever Low The lookup truncates the same way. Tested.
No test for the inclusive since boundary Test gap Added.

Security review: no other findings. Checked and clear: key scope (_get_crawl_api_key, as /results and /errors), cross-user reads (the lookup is filtered by the key's owner, and another user's device and an unknown name give the same known: false), injection (ORM only), and no new credential type on any existing endpoint.

Acknowledged, not changed

  • Attribution at a window edge. A submission is stamped with the time its request arrived (now in post_results), and recorded after indexing. A request still in flight at since is credited to the earlier side, and the endpoint has no upper bound, so a later one can be included after server_time. This is the existing timestamp the hourly/daily figures use; changing it would change them. The effect is bounded by one request's duration against a window of minutes. The benchmark README now says so rather than claiming exactness.
  • Retention. Rows are pruned on the next non-empty submission, so a dormant device can keep rows past 30 days. That is existing behaviour and results_total is unaffected.
  • What the key can see. A crawl key reveals the names, versions and counts of its owner's devices, including ones hidden in the UI. It is the owner's own data, but worth stating.
  • Score as a proxy. results_since counts URLs new to the index, not their quality. The endpoint exposes figures the owner could already see on the devices page; it adds no way to submit anything. Submitting junk was already possible and is handled by the existing validation and moderation, not by this endpoint.

@codecov
codecov Bot commented Nov 16, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@blueheron786
blueheron786 added this pull request to stack #666 November 16, 2026 13:25
@daoudclarke
daoudclarke merged commit 77ad624 into main Nov 16, 2026
7 checks passed
@daoudclarke
daoudclarke deleted the device-results-endpoint branch November 16, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0