8000 Add the crawler agreement overview admin page by daoudclarke · Pull Request #579 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]
Skip to content

Add the crawler agreement overview admin page - #579

Open
daoudclarke wants to merge 2 commits into
mainfrom
crawler-agreement-overview
Open

Add the crawler agreement overview admin page#579
daoudclarke wants to merge 2 commits into
mainfrom
crawler-agreement-overview

Conversation

@daoudclarke
@daoudclarke daoudclarke commented Oct 31, 2026
Copy link
Copy Markdown
Contributor

Part of #533 (item 2): the overview artboard of the design canvas, as an admin page at /admin/crawler-agreement/, linked from the admin index.

What it shows

  • Totals for the window (7, 30 or 90 days).
  • Who disagrees with whom: median extract similarity for each pair of the 18 busiest crawlers. A pair with fewer than 20 shared URLs is hatched. That is a sample-size floor, not a bar for behaviour.
  • Breadth against agreement: partners (log scale) against median extract similarity as submitter, with the network's middle half shaded.
  • The odd bad item: the funnel plot of each crawler's rate of landing in the network's lowest-agreeing 5% against its volume. The allowance is 2 and 3 binomial standard errors around the typical crawler's rate.
  • Every crawler against the network: each column's network band, median and the crawler's value, a daily sparkline, and an off-network score. Rows link to /admin/worst-agreements/.
  • The selected crawler over time: its daily median against the network's middle half, with version changes marked.

Selecting a crawler (any dot or name) is a ?crawler=<user id> link, so the page needs no JavaScript.

How

  • agreement_rollups gains a daily rollup per submitter: its median extract similarity and the version of its last submission that day. It is cached with the rest.
  • mwmbl/crawler/agreement_panel.py turns the rollups into what each chart draws, and is unit-tested without a database.
  • Ranking: a crawler's off-network score is its largest distance from the network median, in units of the network's interquartile range, across the table's columns and the funnel's standard errors. There is no fixed threshold. A column where the network has no spread adds nothing to the ranking.

Differences from the issue and canvas

  • Server-drawn rather than Chart.js. The pair grid, band strips and funnel aren't Chart.js chart types, and the admin pages don't load the front-end bundle. This follows the provider-costs page instead.
  • "Agreement, gap under a day" replaces the canvas's "Extract sim, gap under a day", because median_recent_score is the lower of title and extract similarity.
  • The drill-down artboard is not here. Its new parts are items 3 and 4.
  • About 1,050 added lines, above the issue's ~500, mostly the template and tests.

Testing

  • test/test_agreement_panel.py: ranking (the magpie, linnet and dunlin cases), selection fallback, pair-grid hatching, version-change markers, crawlers that met no one, the empty window, and the view's responses and bad parameters.
  • test/test_agreement_rollups.py: the daily rollup.
  • Rendered with synthetic crawlers modelled on the issue's five sample cases. All five rank at the top, and wren's step lines up with its version marker.

🤖 Generated with Claude Code

@daoudclarke @claude
Add the crawler agreement overview admin page
ad768a0 
Lays out the agreement rollups at /admin/crawler-agreement/ as the
design canvas in #533 draws them: the pair grid, breadth against
agreement, the tail-rate funnel, every crawler against the network's
middle half, and the selected crawler's daily trend with version
changes. Crawlers are ranked by distance from the network in its own
interquartile range, with no fixed cut-offs.

The rollups gain a daily median per submitter and the version it ran,
for the trend, the sparklines and the version column.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov
codecov Bot commented Oct 31, 2026 •
edited
Loading
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.30769% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mwmbl/admin_views.py 84.61% 4 Missing ⚠️
mwmbl/crawler/agreement_panel.py 98.65% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@daoudclarke @claude
Address reviews of the crawler agreement overview
6e5ca5b 
- Mark only versions not seen before, so an operator alternating
  versions across machines doesn't draw spurious changes.
- Keep a day's version when its last comparison sent none.
- Say why a selected crawler's trend is empty.
- Caveat that the version is the crawler's own claim.
- Reject non-ASCII digits in ?crawler= with a 400 rather than a 500.
- Pin the daily rollup test's times so it can't cross midnight.
- Tell an all-empty pair apart from one with too few shared URLs.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daoudclarke
0