Repository navigation
Widgets (#317) PR 4: The widget endpoint - #656
Merged
Merged
Conversation
daoudclarke
force-pushed
the
claude/issue-317-pr4
branch
from
November 13, 2026 22:06
c777943 to
8a80da9
Compare
GET /api/v2/widgets/ fills a widget stub's values from the provider's API: https GET to the spec's API host only, connection pinned to a checked public address, no redirects, 2 s timeout, 64 KB cap. Inputs are cleaned by type and percent-encoded; outputs are plain strings. Answers are cached in Redis with a SET NX lock, robots.txt is checked for Mwmbl's user agent, and callers are rate limited per address. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
…ON (#317) The provider fetch only had a per-operation timeout, so a server dribbling response headers could hold a worker far past 2 s. A connection wrapper now bounds every connect, read and write by one deadline. The size cap is applied to the undecoded body so a compressed answer cannot expand past it, and deeply nested JSON is a FetchError (502) rather than a RecursionError (500). No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
daoudclarke
force-pushed
the
claude/issue-317-pr4
branch
from
November 13, 2026 22:40
8a80da9 to
681df11
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
GET /api/v2/widgets/?url=&q=(mwmbl/widgets/api.py): matchesurlto a spec's page template (else 404), fills inputs from the URL, then the query, then the spec defaults, builds the API URL, and returns the stub plusdisplay,fields({label, value, unit}, plain text),inputsandshare_url.mwmbl/widgets/fetch.py, the SSRF-guarded fetch: https GET only, host must equal the spec'sapi_host. DNS is resolved once, every address must be public (reusescrawler.ssrf.ip_is_safe, renamed from the private_ip_is_safe), and the connection is pinned to the checked IP (Host header and SNI keep the real name, so TLS still verifies the certificate). No redirects, 2 s timeout, 64 KB cap on decoded bytes, identity encoding.crawler.retrieve.robots_allowed(Redis cached for a day). A blocked or unavailable robots.txt returns 404.SET NXlock so concurrent misses share one upstream call. Per-IP rate limit (30/min, usingsearch_callers.client_network) plus a per-spec upstream-call limit to bound cache-busting inputs.mwmbl/api.py. NewWIDGET_*settings are insettings_common.py.Verification
make checkpasses.test/test_widgets_api.pyhas 24 tests usingpytest-httpxandredis_cache. They cover internal and mixed addresses, IP pinning, redirects, oversize and non-JSON answers, other hosts, percent-encoding, invalid input, caching, lock sharing across threads, rate limiting, and robots. The widget, SSRF and arithmetic tests together give 127 passed.Notes
fetch, which validates but does not pin the connection. It only ever targets a spec's fixed host.titleis the provider's name; the per-instance titles live on the indexed documents.Part of #317. Stacked on PR 3 (
claude/issue-317-pr3).Generated by Claude Code