Repository navigation
Add OpenRouter provider for automatic tool naming - #146
Conversation
TOOL_LABEL_PROVIDER only supported "ollama" or off. Adds an "openrouter"
option using the existing OPENROUTER_API_KEY / OPENROUTER_LABEL_MODEL
settings and the OpenRouter URL already used for gemini mask generation,
for setups without a local Ollama server or GPU headroom to run one.
OPENROUTER_LABEL_MODEL accepts a comma-separated model list, tried in
order with a couple of quick retries per model on HTTP 429 before
falling through to the next. Free-tier (":free") models share a 20
req/min pool per model across all OpenRouter users, so a popular model
can rate-limit under third-party load unrelated to the caller's own
account -- a single congested model choice shouldn't fail naming for
an entire trace.
Updated docs/tool-naming.md and .env.example to document the new
provider and the fallback-list behavior.
There was a problem hiding this comment.
thanks for this, provider fallthrough is a nice idea. one blocker and a few suggestions inline.
heads up: the backend unit suite is red on this branch, 1 failed / 259 passed (test_create_tool_namer_supports_only_ollama_and_fallback raises TypeError from the new required kw-only fields). ci didn't catch it because the backend check is ruff only, there's no pytest job. I'll add one.
also:
- no test changes in the PR: the factory gains an openrouter branch, so that test is stale as well as broken. fix/rename it and add a provider="openrouter" dispatch assertion. mocked units for the retry/fallthrough would be nice to have, not blocking
- docs/tool-naming.md: worth a line noting tool crop images transit openrouter, same as the remote tracer docs do
- tiny nit: unrelated blank line removal after the logger line in tool_namer.py
| timeout_seconds: float = 30.0 | ||
| max_crop_px: int = 512 | ||
| openrouter_api_key: str | None = field(kw_only=True) | ||
| openrouter_model: str = field(kw_only=True) |
There was a problem hiding this comment.
these two need defaults. with kw_only and no default they're required, so ToolNamerConfig(provider=...) in test_tool_namer.py raises TypeError and the suite fails. suggest = None and = "google/gemini-2.0-flash-001" to match config.py, then kw_only can probably go too
There was a problem hiding this comment.
I was unsure, about putting a default or None here, since i generally want to avoid duplication, and since in the config.py is already an etry like this, i could just take these values with settings.openrouter_api_key or settings.openrouter_label_model accordingly
| break | ||
| if attempt < max_attempts_per_model - 1: | ||
| retry_after = response.headers.get("retry-after") | ||
| delay = float(retry_after) if retry_after else 2.0 * (attempt + 1) |
There was a problem hiding this comment.
float(retry_after) will blow up on the http-date form of retry-after (legal, and openrouter is behind cloudflare), which skips the remaining fallback models for that polygon. worth a try/except around the parse
There was a problem hiding this comment.
try/except with delay=2. * (attempt +1)
| "openrouter 429 on %s, retrying in %.1fs (attempt %d/%d)", | ||
| model, delay, attempt + 1, max_attempts_per_model, | ||
| ) | ||
| await asyncio.sleep(delay) |
There was a problem hiding this comment.
this sleep is server-controlled and uncapped, and sits outside both the httpx timeout and TOOL_LABEL_TIMEOUT_SECONDS, so a big retry-after stalls the whole trace request. suggest capping it to a few seconds
There was a problem hiding this comment.
will cap to self.config.timeout_seconds
| logger.warning("openrouter 429 on all configured models: %s", models) | ||
| return None | ||
|
|
||
| response.raise_for_status() |
There was a problem hiding this comment.
only 429 falls through. a typo'd or delisted model id 404s here and skips the rest of the list, and it's re-attempted for every polygon. deliberate? if the goal is resilience i'd fall through on these too
There was a problem hiding this comment.
will capture everything >= 400 and continue;
will use a specific logging for 429 and retry another model
|
|
||
| response.raise_for_status() | ||
| result = response.json() | ||
| raw = result["choices"][0]["message"]["content"] |
There was a problem hiding this comment.
reasoning models can return null content (the docs example recommends one), and None here becomes an AttributeError in parse_label_response. it degrades to the generic label rather than crashing, but the ollama path's .get chain is the safer pattern to copy
|
|
||
| from app.config import settings | ||
| from app.models.schemas import Point, Polygon | ||
| from app.services.ai_tracer import OPENROUTER_URL |
There was a problem hiding this comment.
suggestion: OPENROUTER_URL could live in config.py rather than importing from ai_tracer, config already holds the other openrouter settings
There was a problem hiding this comment.
will move it to the config.py and also change the ai_tracer
|
thanks @1Richter |
…o Settings - ToolNamerConfig.openrouter_api_key/openrouter_model now default from settings directly instead of required kw_only fields, fixing the TypeError in test_create_tool_namer_supports_only_ollama_and_fallback (blocker). - Retry-After parsing wrapped in try/except (HTTP-date form no longer crashes) and capped to TOOL_LABEL_TIMEOUT_SECONDS so a large value can't stall the whole trace request. - Fallthrough to the next model now triggers on any error status, not just 429, so a typo'd/delisted model doesn't dead-end the list. - Response content read via a null-safe chain; missing/null content degrades to "no name" instead of raising in parse_label_response. - OpenRouterToolNamer split into name() (model fallthrough) and _post_with_retry() (single-model retry), replacing the nested loop. - OPENROUTER_URL moved out of ai_tracer.py into config.py as a Settings field (openrouter_url), alongside the other openrouter_* settings, and consumed via settings.openrouter_url in both ai_tracer.py and tool_namer.py. - Renamed/extended the factory test to cover the openrouter branch, and added mocked tests for 429 fallthrough and missing-content handling. - docs/tool-naming.md: noted that tool crops transit OpenRouter, same as the remote tracer docs. - Dropped the fork-residue docstring line and the unrelated blank-line removal.
|
Thanks for the thorough review — pushed a follow-up commit addressing all of it:
Tested locally end-to-end again against a running deployment (both single-model and CSV fallback list) after these changes. |
|
thanks for this @1Richter |
Summary
TOOL_LABEL_PROVIDERcurrently only supportsollamaor off (create_tool_namerintool_namer.py). This adds anopenrouteroption for setups that don't want to run a local Ollama server (or don't have GPU headroom for one), reusing the OpenRouter URL/API-key/model settings already present inapp/config.pyand used elsewhere for gemini mask generation.OpenRouterToolNamer, wired intocreate_tool_namer()forprovider == "openrouter".OPENROUTER_LABEL_MODELnow accepts a comma-separated priority list of models, e.g.:Retry-After) onHTTP 429before falling through to the next one in the list. This is aimed at OpenRouter's free-tier (:free) models, which share a 20 req/min pool per model, across all OpenRouter users — a popular free model can 429 under load that has nothing to do with the caller's own account or credit balance. A single model (no comma) behaves exactly as before.docs/tool-naming.mdand.env.exampleto document the new provider and the fallback-list behavior.Tested end-to-end against a self-hosted deployment: traced tools with
TOOL_LABEL_PROVIDER=openrouterand a two-model fallback list, including forcing a 429 on the first model and confirming fallthrough to the second.Test plan
TOOL_LABEL_PROVIDER=openrouter+ single model names traced tools correctlyTOOL_LABEL_PROVIDER=openrouter+ comma-separated model list falls through to the second model on 429TOOL_LABEL_PROVIDER=ollamastill works unaffected (openrouter fields arekw_only, always populated byfrom_settings()but unused byOllamaToolNamer)TOOL_LABEL_PROVIDER=none/unset still falls back to generictool Nlabels