8000 Widgets (#317) PR 3: Provider specs and widget stubs in search by daoudclarke · Pull Request #655 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Widgets (#317) PR 3: Provider specs and widget stubs in search - #655

Merged
daoudclarke merged 3 commits into
claude/issue-317-pr2from
claude/issue-317-pr3
Nov 15, 2026
Merged

daoudclarke merged 3 commits into
claude/issue-317-pr2from
claude/issue-317-pr3

Conversation

@daoudclarke
Copy link
Copy Markdown
Contributor

What changed

  • mwmbl/widgets/spec.py: pydantic WidgetSpec (site/source_name, page URL template, API URL template plus the single allowed API host, typed inputs with query patterns and defaults, output fields as JSON paths with label/unit, display in fields/form/forecast, trigger word and tokens, cache TTL) and load_specs(dir) for mwmbl/widgets/providers/*.json. Validation: https only, host must be a plain lower case name equal to api_host (page host equal to site), no placeholders in hosts, placeholders must be declared inputs, input patterns must compile with exactly one group. No real provider is shipped; tests load specs from a tmp dir.
  • mwmbl/widgets/lookup.py: WidgetStub and find_widgets(q, tiny_index). No trigger word means it returns after tokenizing, with no index read. Otherwise it reads only keys that contain a trigger word (at most 21; new trigger_query_keys in keys.py, so a trigger word or anchor is never dropped by the 6 token cap). It keeps only DocumentSource.WIDGET documents whose URL matches a spec's page template (so a document cannot point at a host its spec does not allow) and whose tokens (the document's extract, space separated) plus the spec's trigger tokens are all in the query. It returns at most one per spec: the match with most trigger tokens, ties to the first stored.
  • search.py: SearchResponse.widgets is now a discriminated union of calculation and widget stubs (type, title, url, source_name); find_widgets runs before ranker.search.
  • rank.py: ordinary retrieval drops WIDGET documents (the tokenizer keeps widget:calculator whole, so typing a key would otherwise return the document).

Why

Part of #317. Search only decides which widget applies, cheaply, with no outbound call and no database query. Specs are reviewed data, so PRs 4 to 6 can reuse WidgetSpec unchanged.

How verified

make check passes. test/test_widgets_lookup.py (spec validation, loader, lookup cases, no keys read without a trigger word, 21 read cap, search with no network and 0 DB queries, ordinary retrieval excludes widgets) plus the arithmetic, tokens, rank, combined search, openapi and index_batches 8000 tests pass.

Notes

  • Stacked on PR 2. PR 2's search.py wiring had not been committed, so it is pushed to that branch as its own commit.
  • Contract for PR 5: a widget document's extract holds its trigger tokens separated by spaces, and its url must match the spec's page template.
  • Added lines are about 509, slightly over the 500 guideline.

Part of #317


Generated by Claude Code

@daoudclarke
daoudclarke added this pull request to stack #659 November 13, 2026 20:47
WidgetSpec describes a provider (page and API URL templates, the one allowed https
API host, typed inputs, output paths, display, trigger tokens, cache TTL) and is
loaded from mwmbl/widgets/providers/*.json. find_widgets runs before ranking: it
stops after tokenizing when no trigger word is present, otherwise reads only keys
that contain a trigger word (at most 21), keeps WIDGET documents whose URL matches
a spec's page template and whose tokens are all in the query, and returns at most
one per spec. Ordinary retrieval no longer returns WIDGET documents.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
…317)

Specs are read when the v2 search is registered, so a broken one stops the app
starting instead of failing the first search. A spec's hosts must be names, not IP
addresses or localhost. Drops the find_widgets wrapper and a one-use helper.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
…317)

The production ranker is MMRRanker wrapping LTRRanker, which has no tiny_index, so every
non-arithmetic v2 search raised AttributeError. The test stubs had the attribute, which hid it.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
@codecov
codecov Bot commented Nov 13, 2026
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.82540% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mwmbl/widgets/spec.py 95.20% 6 Missing ⚠️

📢 Thoughts on this report? Let us know!

@daoudclarke
daoudclarke merged commit 71dcf5a into main Nov 15, 2026
7 checks passed
@daoudclarke
daoudclarke deleted the claude/issue-317-pr3 branch November 15, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0