8000 Widgets (#317) PR 4: The widget endpoint by daoudclarke · Pull Request #656 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Widgets (#317) PR 4: The widget endpoint - #656

Merged
daoudclarke merged 3 commits into
claude/issue-317-pr3from
claude/issue-317-pr4
Nov 15, 2026
Merged

daoudclarke merged 3 commits into
claude/issue-317-pr3from
claude/issue-317-pr4

Conversation

@daoudclarke
Copy link
Copy Markdown
Contributor

What changed

  • GET /api/v2/widgets/?url=&q= (mwmbl/widgets/api.py): matches url to a spec's page template (else 404), fills inputs from the URL, then the query, then the spec defaults, builds the API URL, and returns the stub plus display, fields ({label, value, unit}, plain text), inputs and share_url.
  • mwmbl/widgets/fetch.py, the SSRF-guarded fetch: https GET only, host must equal the spec's api_host. DNS is resolved once, every address must be public (reuses crawler.ssrf.ip_is_safe, renamed from the private _ip_is_safe), and the connection is pinned to the checked IP (Host header and SNI keep the real name, so TLS still verifies the certificate). No redirects, 2 s timeout, 64 KB cap on decoded bytes, identity encoding.
  • Inputs are validated by declared type (number, date, choice, text) and percent-encoded; invalid input returns 422. Output values must be a string, number or boolean found by the spec's path, with control characters stripped and a length cap. Nested JSON is dropped.
  • robots.txt for Mwmbl's user agent via the existing crawler.retrieve.robots_allowed (Redis cached for a day). A blocked or unavailable robots.txt returns 404.
  • Redis answer cache keyed on spec and API URL for the spec's TTL, with a SET NX lock so concurrent misses share one upstream call. Per-IP rate limit (30/min, using search_callers.client_network) plus a per-spec upstream-call limit to bound cache-busting inputs.
  • The router is registered on the v2 API in mwmbl/api.py. New WIDGET_* settings are in settings_common.py.

Verification

make check passes. test/test_widgets_api.py has 24 tests using pytest-httpx and redis_cache. They cover internal and mixed addresses, IP pinning, redirects, oversize and non-JSON answers, other hosts, percent-encoding, invalid input, caching, lock sharing across threads, rate limiting, and robots. The widget, SSRF and arithmetic tests together give 127 passed.

Notes

  • robots.txt is fetched by the crawler's existing fetch, which validates but does not pin the connection. It only ever targets a spec's fixed host.
  • The stub title is the provider's name; the per-instance titles live on the indexed documents.

Part of #317. Stacked on PR 3 (claude/issue-317-pr3).


Generated by Claude Code

GET /api/v2/widgets/ fills a widget stub's values from the provider's API:
https GET to the spec's API host only, connection pinned to a checked public
address, no redirects, 2 s timeout, 64 KB cap. Inputs are cleaned by type and
percent-encoded; outputs are plain strings. Answers are cached in Redis with a
SET NX lock, robots.txt is checked for Mwmbl's user agent, and callers are
rate limited per address.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
…ON (#317)

The provider fetch only had a per-operation timeout, so a server dribbling
response headers could hold a worker far past 2 s. A connection wrapper now
bounds every connect, read and write by one deadline. The size cap is applied
to the undecoded body so a compressed answer cannot expand past it, and deeply
nested JSON is a FetchError (502) rather than a RecursionError (500).

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
@codecov
codecov Bot commented Nov 13, 2026
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.46288% with 31 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mwmbl/widgets/fetch.py 77.77% 18 Missing ⚠️
mwmbl/widgets/api.py 90.51% 13 Missing ⚠️

📢 Thoughts on this report? Let us know!

@daoudclarke
daoudclarke merged commit 71dcf5a into main Nov 15, 2026
7 checks passed
@daoudclarke
daoudclarke deleted the claude/issue-317-pr4 branch November 15, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0