8000 evals: an empty Langfuse page no longer ends a trace fetch by Jiminyang1 · Pull Request #11 · Jiminyang1/minibot · GitHub
[go: up one dir, main page]

Skip to content

evals: an empty Langfuse page no longer ends a trace fetch - #11

Merged
Jiminyang1 merged 1 commit into
mainfrom
fix/langfuse-empty-pages
Nov 17, 2026
Merged

Jiminyang1 merged 1 commit into
mainfrom
fix/langfuse-empty-pages

Conversation

@Jiminyang1
Copy link
Copy Markdown
Owner

Why

On 2026-11-16 Langfuse removed the v3 dataset-run endpoints (GET .../datasets/{name}/runs and GET /dataset-run-items now return 410). I checked whether this breaks our Langfuse paths. It does not:

Path Result on 2026-11-17 (SDK 4.16.0)
evals run --case chat-concept-no-tools Passed. The experiment probe-410-v4 shows up under api.experiments, linked to the minibot-golden dataset, with its scores. The SDK logged no errors.
evals sync 10 cases synced.
evals traces / traces --failed Listed.
promote (fetch_observations + draft_case, no write) Drafted a case from the probe trace.

While comparing against SDK 5.2, I found a real bug in the same code. Since Langfuse's adaptive scan, GET /api/public/v2/observations can return an empty page that still has a cursor. This happens when the server runs out of its 45 s scan budget while more rows remain (see langfuse/langfuse-python#2097). fetch_observations and list_traces stopped on the first empty page. So promote could draft a case from part of a trace, and traces could list nothing, with no warning.

What changed (evals/promote.py)

  • New observation_pages helper. It follows the cursor until the server sends none. A cursor seen twice also ends paging, so a bad server cannot loop forever.
  • Every page request waits 60 s. The SDK default is 5 s, which is shorter than one server scan (fix(batch-evaluation): use a 60 s timeout for observation page fetches langfuse/langfuse-python#2096).
  • Rows repeated across pages are dropped, because the server does not dedupe.
  • list_traces fills its limit from short pages. It also pages through the tool errors it uses for the failure flags. Before, the non---failed listing read only one page.

Tests

Six new tests in tests/test_evals_promote.py::ObservationPagingTests. All six fail on main and pass here.

  • uv run --project minibot python -m unittest discover -s minibot/tests -t .: 443 passed, 3 skipped.
  • uv run ruff check .: clean.
  • Real Langfuse: traces, traces --failed, and a promote dry run against the probe trace all work.

Not verified

  • I did not see an empty page with a cursor from the live server. The behaviour comes from the upstream PR description. Our project is small, so a scan probably never runs out of time today.
  • I did not upgrade to SDK 5.x. 4.16 still sends POST /dataset-run-items inside run_experiment. That endpoint worked today, but it belongs to the deprecated v3 model. SDK 5.2 no longer calls it, and the APIs we use differ only by added parameters. That upgrade belongs in a separate PR.

🤖 Generated with Claude Code

Since Langfuse's adaptive scan, GET /api/public/v2/observations can
return an empty page that still has a cursor: the server ran out of its
45 s budget, and more rows remain. promote's fetch_observations and the
traces listing stopped on the first empty page, so promote could draft a
case from part of a trace and traces could list nothing, with no warning.

- observation_pages follows the cursor until the server sends none. A
  cursor seen twice also ends it, so a bad server cannot loop forever.
- Every page request waits 60 s; the SDK default of 5 s is shorter than
  one server scan.
- Rows repeated across pages are dropped (the server does not dedupe).
- The traces listing fills its limit from short pages, and also pages
  through the tool errors it uses for the failure flags.

Same fix as langfuse-python#2097 / #2096 (SDK 5.2), for the 4.16 SDK we
pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jiminyang1
Jiminyang1 merged commit 4af77c8 into main Nov 17, 2026
4 checks passed
Jiminyang1 added a commit that referenced this pull request Nov 17, 2026
Brings in #7–#11. #11 already contains this branch's base commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0