8000 Widgets (#317) PR 5: tannerd.net provider and the widget indexing command by daoudclarke · Pull Request #657 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Widgets (#317) PR 5: tannerd.net provider and the widget indexing command - #657

Merged
daoudclarke merged 2 commits into
claude/issue-317-pr4from
claude/issue-317-pr5
Nov 15, 2026
Merged

daoudclarke merged 2 commits into
claude/issue-317-pr4from
claude/issue-317-pr5

Conversation

@daoudclarke
Copy link
Copy Markdown
Contributor

What changed

  • mwmbl/widgets/providers/tannerd-*.json: five reviewed specs for tannerd.net calculators (percentage change, days between dates, salary to hourly, trip cost, electricity cost). Written from the real OpenAPI (/api/openapi.json) and real sample calls; each spec's query patterns were also run against the live API once, and all five answered 200 with the expected fields. tannerd.net's robots.txt allows everything but /admin/.
  • mwmbl/widgets/openapi_import.py: draft_spec(openapi, path, sample) drafts a spec from an OpenAPI GET operation (query parameters become inputs, examples become defaults, scalars under result become outputs), plus a small CLI. Drafts are for a person to finish (triggers and query patterns).
  • manage.py index_widgets: checks robots.txt for each spec's page and API (an unreadable robots.txt counts as blocked), deletes the widget documents of blocked sites and of specs that changed or went (keys remembered in DATA_PATH/widget-keys.json), then writes one document per allowed spec through index_widget_documents.
  • test/test_widgets_tannerd.py: no network. Saved real answers, trigger and non-trigger queries, ambiguous values, sync/purge, robots, the command, and the importer.

Why / decisions

  • Trigger tokens are conservative: every spec needs a number or date token plus 2-3 specific words (for example {mpg, trip, cost, :n}), so bare "trip cost" or "salary" queries do not fire. Patterns refuse ambiguous values (1,000, 60k, 18 cents per kwh, monthly salary) and the widget is then hidden.
  • page_url is https://tannerd.net/tools/<slug>/?ref=mwmbl, with no placeholders. The endpoint takes a value from the page URL before the query, so a share_url with example values filled in would make the widget always show the example. The ref parameter keeps the URL from equalling the real page's URL. Downside: the link back does not carry the user's values.
  • Writing removes by (key, URL) on pages under the widget keys only, rather than purge_documents, which removes by URL across the token pages and could delete a crawled page with the same URL.
  • Deviations from the plan: unit price compare is left out (four positional values cannot be extracted reliably); "days until" needs today's date, so the spec is "days between" two dates.

Verification

make check passes; test_widgets_tannerd, test_widgets_lookup, test_widgets_api and test_index_batches pass (102 tests) 8000 . 500 added lines.

Part of #317. Stacked on #317 PR 4 (claude/issue-317-pr4).


Generated by Claude Code

Five reviewed calculator specs for tannerd.net (percentage change, days between
dates, salary to hourly, trip cost, electricity cost), an OpenAPI importer that
drafts a spec from an operation and one sample answer, and `manage.py
index_widgets`, which skips (and removes the documents of) sites whose robots.txt
does not allow Mwmbl, and removes the documents of specs that changed or went.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
Remove an unused local that failed make check, and save the widget keys that a
failed index_widgets run may have written so the next run can remove them.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
@codecov
codecov Bot commented Nov 13, 2026
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.72566% with 15 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mwmbl/widgets/openapi_import.py 75.55% 11 Missing ⚠️
mwmbl/management/commands/index_widgets.py 94.11% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@daoudclarke
daoudclarke merged commit 71dcf5a into main Nov 15, 2026
7 checks passed
@daoudclarke
daoudclarke deleted the claude/issue-317-pr5 branch November 15, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0