8000 Widgets (#317) PR 6: Open-Meteo weather provider by daoudclarke · Pull Request #658 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Widgets (#317) PR 6: Open-Meteo weather provider - #658

Merged
daoudclarke merged 3 commits into
claude/issue-317-pr5from
claude/issue-317-pr6
Nov 15, 2026
Merged

daoudclarke merged 3 commits into
claude/issue-317-pr5from
claude/issue-317-pr6

Conversation

@daoudclarke
Copy link
Copy Markdown
Contributor

What

  • Two provider specs on Open-Meteo: open-meteo-weather (trigger weather) and open-meteo-weather-forecast (trigger forecast). Each shows current conditions plus the next 3 days (display: forecast), cached 30 minutes. The API URL asks for forecast_days=4, and the outputs are bounded to now plus days 1-3. Source credit is "Open-Meteo.com" (CC BY 4.0), linked via share_url.
  • OutputField.transform: "wmo_code" turns WMO weather codes into words (mwmbl/widgets/wmo.py). An unknown code is dropped.
  • mwmbl/widgets/places.py reads GeoNames cities15000.txt and countryInfo.txt. They are downloaded by the operator and never committed.
    • index_widgets --places CITIES COUNTRY_INFO writes one document per place per place-based spec, most populous first.
    • Titles are " weather, " and " forecast, ". Same-name places in one country get "()" added, and are dropped if that is still not unique.
    • Page URLs are https://open-meteo.com/en/docs?latitude=..&longitude=..&ref=mwmbl (&ref=mwmbl-forecast for the forecast spec). The ref keeps them distinct from a real crawled open-meteo.com URL.
    • Trigger tokens are the trigger word plus the ASCII name's tokens. The key anchor is the name's rarest token across all places.
  • sync_widgets takes the places. Place documents sync and purge under the same widget-keys.json keys as the other widgets. Titles and URLs must be unique, and robots is checked once per spec rather than once per document.
  • WidgetFinder treats specs that share an API URL as one widget, so "york weather forecast" shows one widget. Ties go to the weather spec.
  • site_allows_mwmbl fills the page URL placeholders before checking robots.
  • A leftover unused local in test/test_widgets_tannerd.py was failing make check, so I removed it.

Needs a decision: Open-Meteo API robots.txt

https://api.open-meteo.com/robots.txt is User-agent: * / Disallow: /. open-meteo.com itself allows everything. If the API robots check stayed, the widget would never be indexed and the endpoint would always return 404. I added WidgetSpec.check_api_robots (default true) and set it to false only for these two specs, on the reasoning that the documented public API is meant for programmatic use and the robots file is aimed at crawlers. The spec's page (open-meteo.com) is still robots-checked. Please confirm you are happy with this, or drop the spec if not.

Open-Meteo's free API is for non-commercial use with attribution, which fits a non-profit. Its free limits are 600 calls/min and 10,000/day per IP, and the 30-minute cache plus the per-spec rate limit keep us well inside them.

Known limitation

Place tokens come from GeoNames' ASCII name, so "zurich weather" works but "zürich weather" does not.

Verification

  • make check passes (ty is non-gating).
  • test/test_widgets_weather.py (new) covers the specs, place reading and title disambiguation, and WMO mapping. It also covers the 3-day bound, lookup for weather, forecast and combined queries, sync purging of removed or blocked places, and the command with its robots calls.
  • The widget tests (weather, tannerd, lookup, api, tokens) pass: 106 tests.
  • Tests use a small inline TSV and a hand-trimmed API answer. The live API returned 429 from this shared IP, so no live calls were made.

Part of #317. Stacked on PR 5 (claude/issue-317-pr5).


Generated by Claude Code

Two specs (weather, forecast) on Open-Meteo, current conditions plus three
days with WMO codes in words, cached for 30 minutes. index_widgets --places
reads GeoNames cities15000 and countryInfo.txt (downloaded, never committed)
and writes one document per place per spec, synced under the same keys as
the other widgets so stale places are removed. The finder treats specs on one
API as one widget.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
GeoNames names it "New York City", so "weather in new york" showed York, UK.
A trailing "city" is dropped from names of three or more words.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
A plain run used to remove every indexed place, because it synced the place specs with no
places. It now syncs only the specs without page-URL placeholders. Place keys are kept in
widget-place-keys.json; --places syncs the places again and --no-places removes them all.

No functional changes to the crawler.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTbu4ortjqC5Z7Kw9377LE
@codecov
codecov Bot commented Nov 13, 2026
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.63866% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mwmbl/management/commands/index_widgets.py 91.11% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@daoudclarke
daoudclarke merged commit 71dcf5a into main Nov 15, 2026
7 checks passed
@daoudclarke
daoudclarke deleted the claude/issue-317-pr6 branch November 15, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0