8000 Add OpenRouter provider for automatic tool naming by 1Richter · Pull Request #146 · tracefinity/tracefinity · GitHub
[go: up one dir, main page]

Skip to content

Add OpenRouter provider for automatic tool naming - #146

Merged
jasonmadigan merged 2 commits into
tracefinity:mainfrom
1Richter:feat/openrouter-tool-namer
Jul 22, 2026
Merged

jasonmadigan merged 2 commits into
tracefinity:mainfrom
1Richter:feat/openrouter-tool-namer

Conversation

@1Richter
@1Richter 1Richter commented Jul 21, 2026 •
Copy link
Copy Markdown
Contributor

Summary

TOOL_LABEL_PROVIDER currently only supports ollama or off (create_tool_namer in tool_namer.py). This adds an openrouter option for setups that don't want to run a local Ollama server (or don't have GPU headroom for one), reusing the OpenRouter URL/API-key/model settings already present in app/config.py and used elsewhere for gemini mask generation.

  • New OpenRouterToolNamer, wired into create_tool_namer() for provider == "openrouter".
  • OPENROUTER_LABEL_MODEL now accepts a comma-separated priority list of models, e.g.:
    OPENROUTER_LABEL_MODEL=google/gemma-4-31b-it:free,nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free
    
    Each model gets a couple of quick retries (honoring Retry-After) on HTTP 429 before falling through to the next one in the list. This is aimed at OpenRouter's free-tier (:free) models, which share a 20 req/min pool per model, across all OpenRouter users — a popular free model can 429 under load that has nothing to do with the caller's own account or credit balance. A single model (no comma) behaves exactly as before.
  • Updated docs/tool-naming.md and .env.example to document the new provider and the fallback-list behavior.

Tested end-to-end against a self-hosted deployment: traced tools with TOOL_LABEL_PROVIDER=openrouter and a two-model fallback list, including forcing a 429 on the first model and confirming fallthrough to the second.

Test plan

  • TOOL_LABEL_PROVIDER=openrouter + single model names traced tools correctly
  • TOOL_LABEL_PROVIDER=openrouter + comma-separated model list falls through to the second model on 429
  • TOOL_LABEL_PROVIDER=ollama still works unaffected (openrouter fields are kw_only, always populated by from_settings() but unused by OllamaToolNamer)
  • TOOL_LABEL_PROVIDER=none/unset still falls back to generic tool N labels

TOOL_LABEL_PROVIDER only supported "ollama" or off. Adds an "openrouter"
option using the existing OPENROUTER_API_KEY / OPENROUTER_LABEL_MODEL
settings and the OpenRouter URL already used for gemini mask generation,
for setups without a local Ollama server or GPU headroom to run one.

OPENROUTER_LABEL_MODEL accepts a comma-separated model list, tried in
order with a couple of quick retries per model on HTTP 429 before
falling through to the next. Free-tier (":free") models share a 20
req/min pool per model across all OpenRouter users, so a popular model
can rate-limit under third-party load unrelated to the caller's own
account -- a single congested model choice shouldn't fail naming for
an entire trace.

Updated docs/tool-naming.md and .env.example to document the new
provider and the fallback-list behavior.
@jasonmadigan jasonmadigan left a comment •
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for this, provider fallthrough is a nice idea. one blocker and a few suggestions inline.

heads up: the backend unit suite is red on this branch, 1 failed / 259 passed (test_create_tool_namer_supports_only_ollama_and_fallback raises TypeError from the new required kw-only fields). ci didn't catch it because the backend check is ruff only, there's no pytest job. I'll add one.

also:

  • no test changes in the PR: the factory gains an openrouter branch, so that test is stale as well as broken. fix/rename it and add a provider="openrouter" dispatch assertion. mocked units for the retry/fallthrough would be nice to have, not blocking
  • docs/tool-naming.md: worth a line noting tool crop images transit openrouter, same as the remote tracer docs do
  • tiny nit: unrelated blank line removal after the logger line in tool_namer.py

Comment thread backend/app/services/tool_namer.py Outdated
timeout_seconds: float = 30.0
max_crop_px: int = 512
openrouter_api_key: str | None = field(kw_only=True)
openrouter_model: str = field(kw_only=True)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these two need defaults. with kw_only and no default they're required, so ToolNamerConfig(provider=...) in test_tool_namer.py raises TypeError and the suite fails. suggest = None and = "google/gemini-2.0-flash-001" to match config.py, then kw_only can probably go too

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was unsure, about putting a default or None here, since i generally want to avoid duplication, and since in the config.py is already an etry like this, i could just take these values with settings.openrouter_api_key or settings.openrouter_label_model accordingly

Comment thread backend/app/services/tool_namer.py Outdated
break
if attempt < max_attempts_per_model - 1:
retry_after = response.headers.get("retry-after")
delay = float(retry_after) if retry_after else 2.0 * (attempt + 1)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

float(retry_after) will blow up on the http-date form of retry-after (legal, and openrouter is behind cloudflare), which skips the remaining fallback models for that polygon. worth a try/except around the parse

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try/except with delay=2. * (attempt +1)

Comment thread backend/app/services/tool_namer.py Outdated
"openrouter 429 on %s, retrying in %.1fs (attempt %d/%d)",
model, delay, attempt + 1, max_attempts_per_model,
)
await asyncio.sleep(delay)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this sleep is server-controlled and uncapped, and sits outside both the httpx timeout and TOOL_LABEL_TIMEOUT_SECONDS, so a big retry-after stalls the whole trace request. suggest capping it to a few seconds

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will cap to self.config.timeout_seconds

Comment thread backend/app/services/tool_namer.py Outdated
logger.warning("openrouter 429 on all configured models: %s", models)
return None

response.raise_for_status()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only 429 falls through. a typo'd or delisted model id 404s here and skips the rest of the list, and it's re-attempted for every polygon. deliberate? if the goal is resilience i'd fall through on these too

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will capture everything >= 400 and continue;
will use a specific logging for 429 and retry another model

Comment thread backend/app/services/tool_namer.py Outdated

response.raise_for_status()
result = response.json()
raw = result["choices"][0]["message"]["content"]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reasoning models can return null content (the docs example recommends one), and None here becomes an AttributeError in parse_label_response. it degrades to the generic label rather than crashing, but the ollama path's .get chain is the safer pattern to copy

Comment thread backend/app/services/tool_namer.py Outdated

from app.config import settings
from app.models.schemas import Point, Polygon
from app.services.ai_tracer import OPENROUTER_URL
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: OPENROUTER_URL could live in config.py rather than importing from ai_tracer, config already holds the other openrouter settings

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will move it to the config.py and also change the ai_tracer

Comment thread backend/app/services/tool_namer.py Resolved
@jasonmadigan
Copy link
Copy Markdown
Collaborator

thanks @1Richter

…o Settings

- ToolNamerConfig.openrouter_api_key/openrouter_model now default from
  settings directly instead of required kw_only fields, fixing the
  TypeError in test_create_tool_namer_supports_only_ollama_and_fallback
  (blocker).
- Retry-After parsing wrapped in try/except (HTTP-date form no longer
  crashes) and capped to TOOL_LABEL_TIMEOUT_SECONDS so a large value
  can't stall the whole trace request.
- Fallthrough to the next model now triggers on any error status, not
  just 429, so a typo'd/delisted model doesn't dead-end the list.
- Response content read via a null-safe chain; missing/null content
  degrades to "no name" instead of raising in parse_label_response.
- OpenRouterToolNamer split into name() (model fallthrough) and
  _post_with_retry() (single-model retry), replacing the nested loop.
- OPENROUTER_URL moved out of ai_tracer.py into config.py as a Settings
  field (openrouter_url), alongside the other openrouter_* settings,
  and consumed via settings.openrouter_url in both ai_tracer.py and
  tool_namer.py.
- Renamed/extended the factory test to cover the openrouter branch, and
  added mocked tests for 429 fallthrough and missing-content handling.
- docs/tool-naming.md: noted that tool crops transit OpenRouter, same
  as the remote tracer docs.
- Dropped the fork-residue docstring line and the unrelated blank-line
  removal.
@1Richter
Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review — pushed a follow-up commit addressing all of it:

  • Blocker fixed: openrouter_api_key/openrouter_model now default straight from settings (= settings.openrouter_api_key, etc.) instead of required kw_only fields, so ToolNamerConfig(provider=...) works again without duplicating the literal defaults from config.py.
  • Retry-After parsing wrapped in try/except (HTTP-date form no longer raises) and capped to TOOL_LABEL_TIMEOUT_SECONDS so it can't stall the whole trace request.
  • Fallthrough now triggers on any error status, not just 429 — a typo'd/delisted model id no longer dead-ends the rest of the list.
  • Response content read via a null-safe chain (choices → message → content, each with a fallback); missing/null content now degrades to "no name" instead of raising in parse_label_response.
  • Split OpenRouterToolNamer.name() into model-fallthrough vs. a _post_with_retry() helper for the single-model retry — flattens what was a nested loop.
  • Moved OPENROUTER_URL out of ai_tracer.py into config.py as a Settings field (openrouter_url), same as the other openrouter_* settings, and updated both call sites to use it.
  • Renamed/extended the factory test to cover the openrouter branch, added mocked tests for 429 fallthrough and missing-content handling.
  • Added the doc line noting tool crops transit OpenRouter, dropped the fork-residue docstring line and the unrelated blank-line removal.

Tested locally end-to-end again against a running deployment (both single-model and CSV fallback list) after these changes.

@jasonmadigan
jasonmadigan merged commit 216512e into tracefinity:main Jul 22, 2026
4 checks passed
@jasonmadigan
Copy link
Copy Markdown
Collaborator

thanks for this @1Richter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0