Repository navigation
evals: an empty Langfuse page no longer ends a trace fetch - #11
Merged
Merged
Conversation
Since Langfuse's adaptive scan, GET /api/public/v2/observations can return an empty page that still has a cursor: the server ran out of its 45 s budget, and more rows remain. promote's fetch_observations and the traces listing stopped on the first empty page, so promote could draft a case from part of a trace and traces could list nothing, with no warning. - observation_pages follows the cursor until the server sends none. A cursor seen twice also ends it, so a bad server cannot loop forever. - Every page request waits 60 s; the SDK default of 5 s is shorter than one server scan. - Rows repeated across pages are dropped (the server does not dedupe). - The traces listing fills its limit from short pages, and also pages through the tool errors it uses for the failure flags. Same fix as langfuse-python#2097 / #2096 (SDK 5.2), for the 4.16 SDK we pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On 2026-11-16 Langfuse removed the v3 dataset-run endpoints (
GET .../datasets/{name}/runsandGET /dataset-run-itemsnow return 410). I checked whether this breaks our Langfuse paths. It does not:evals run --case chat-concept-no-toolsprobe-410-v4shows up underapi.experiments, linked to theminibot-goldendataset, with its scores. The SDK logged no errors.evals syncevals traces/traces --failedpromote(fetch_observations+draft_case, no write)While comparing against SDK 5.2, I found a real bug in the same code. Since Langfuse's adaptive scan,
GET /api/public/v2/observationscan return an empty page that still has a cursor. This happens when the server runs out of its 45 s scan budget while more rows remain (see langfuse/langfuse-python#2097).fetch_observationsandlist_tracesstopped on the first empty page. Sopromotecould draft a case from part of a trace, andtracescould list nothing, with no warning.What changed (
evals/promote.py)observation_pageshelper. It follows the cursor until the server sends none. A cursor seen twice also ends paging, so a bad server cannot loop forever.list_tracesfills its limit from short pages. It also pages through the tool errors it uses for the failure flags. Before, the non---failedlisting read only one page.Tests
Six new tests in
tests/test_evals_promote.py::ObservationPagingTests. All six fail onmainand pass here.uv run --project minibot python -m unittest discover -s minibot/tests -t .: 443 passed, 3 skipped.uv run ruff check .: clean.traces,traces --failed, and apromotedry run against the probe trace all work.Not verified
POST /dataset-run-itemsinsiderun_experiment. That endpoint worked today, but it belongs to the deprecated v3 model. SDK 5.2 no longer calls it, and the APIs we use differ only by added parameters. That upgrade belongs in a separate PR.🤖 Generated with Claude Code