Repository navigation
Widgets (#317) PR 3: Provider specs and widget stubs in search - #655
Merged
Merged
Conversation
WidgetSpec describes a provider (page and API URL templates, the one allowed https API host, typed inputs, output paths, display, trigger tokens, cache TTL) and is loaded from mwmbl/widgets/providers/*.json. find_widgets runs before ranking: it stops after tokenizing when no trigger word is present, otherwise reads only keys that contain a trigger word (at most 21), keeps WIDGET documents whose URL matches a spec's page template and whose tokens are all in the query, and returns at most one per spec. Ordinary retrieval no longer returns WIDGET documents. No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
…317) Specs are read when the v2 search is registered, so a broken one stops the app starting instead of failing the first search. A spec's hosts must be names, not IP addresses or localhost. Drops the find_widgets wrapper and a one-use helper. No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
…317) The production ranker is MMRRanker wrapping LTRRanker, which has no tiny_index, so every non-arithmetic v2 search raised AttributeError. The test stubs had the attribute, which hid it. No functional changes to the crawler. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
daoudclarke
force-pushed
the
claude/issue-317-pr3
branch
from
November 13, 2026 22:40
d376e5b to
ce2202f
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
mwmbl/widgets/spec.py: pydanticWidgetSpec(site/source_name, page URL template, API URL template plus the single allowed API host, typed inputs with query patterns and defaults, output fields as JSON paths with label/unit,displayin fields/form/forecast, trigger word and tokens, cache TTL) andload_specs(dir)formwmbl/widgets/providers/*.json. Validation: https only, host must be a plain lower case name equal toapi_host(page host equal tosite), no placeholders in hosts, placeholders must be declared inputs, input patterns must compile with exactly one group. No real provider is shipped; tests load specs from a tmp dir.mwmbl/widgets/lookup.py:WidgetStubandfind_widgets(q, tiny_index). No trigger word means it returns after tokenizing, with no index read. Otherwise it reads only keys that contain a trigger word (at most 21; newtrigger_query_keysinkeys.py, so a trigger word or anchor is never dropped by the 6 token cap). It keeps onlyDocumentSource.WIDGETdocuments whose URL matches a spec's page template (so a document cannot point at a host its spec does not allow) and whose tokens (the document's extract, space separated) plus the spec's trigger tokens are all in the query. It returns at most one per spec: the match with most trigger tokens, ties to the first stored.search.py:SearchResponse.widgetsis now a discriminated union ofcalculationandwidgetstubs (type,title,url,source_name);find_widgetsruns beforeranker.search.rank.py: ordinary retrieval drops WIDGET documents (the tokenizer keepswidget:calculatorwhole, so typing a key would otherwise return the document).Why
Part of #317. Search only decides which widget applies, cheaply, with no outbound call and no database query. Specs are reviewed data, so PRs 4 to 6 can reuse
WidgetSpecunchanged.How verified
make checkpasses.test/test_widgets_lookup.py(spec validation, loader, lookup cases, no keys read without a trigger word, 21 read cap, search with no network and 0 DB queries, ordinary retrieval excludes widgets) plus the arithmetic, tokens, rank, combined search, openapi and index_batches 8000 tests pass.Notes
search.pywiring had not been committed, so it is pushed to that branch as its own commit.extractholds its trigger tokens separated by spaces, and itsurlmust match the spec's page template.Part of #317
Generated by Claude Code