8000 deps: move to the Langfuse Python SDK 5.2 by Jiminyang1 · Pull Request #12 · Jiminyang1/minibot · GitHub
[go: up one dir, main page]

Skip to content

deps: move to the Langfuse Python SDK 5.2 - #12

Merged
Jiminyang1 merged 2 commits into
mainfrom
chore/langfuse-sdk-5
Nov 17, 2026
Merged

Jiminyang1 merged 2 commits into
mainfrom
chore/langfuse-sdk-5

Conversation

@Jiminyang1
@Jiminyang1 Jiminyang1 commented Nov 17, 2026 •
Copy link
Copy Markdown
Owner

Why

SDK 4.16 still sends POST /api/public/dataset-run-items inside run_experiment. That endpoint belongs to the v3 dataset-run model. Langfuse removed the v3 read endpoints on 2026-11-16. SDK 5 records experiments through OpenTelemetry only, so this PR moves MiniBot off the deprecated write path before it goes away too.

This branch was cut from #11's branch, because both PRs touch evals/promote.py. #11 has now merged, and I merged origin/main (#7–#11) into this branch. The diff against main is only this PR's change (commit feec3f1).

What changed

  • pyproject.toml: langfuse>=4.16,<5 → langfuse>=5.2,<6. uv.lock resolves to 5.2.0, and no other package changes. The lock revision field also changed from 3 to 5, because a newer uv wrote it.
  • No call-site changes were needed.
  • evals/promote.py: a docstring note only. I kept the local observation_pages and did not switch to the SDK's new iter_observation_pages. That helper has the same stop rule (only a missing cursor ends paging) and the same 60 s timeout. It has no repeated-cursor guard, no per-page dedupe and no page bound (max_items counts rows, so it does not limit empty pages). Wrapping it would also make evals: an empty Langfuse page no longer ends a trace fetch #11's fakes test a fake loop instead of ours.
  • langfuse_tracing.py: the 8000 docstring now says "SDK v4 and later" instead of "v4".

v5 changes I checked

GitHub has no release notes for v5.0.0. The full change list is in the description of langfuse/langfuse-python#1991. I also diffed the 4.16.0 and 5.2.0 sources: span.py (start_observation, update, create_event, end), attributes.py, propagation.py, datasets.py, experiment.py, get_client.py, resource_manager.py, and the client.py hunks for __init__, flush, run_experiment and iter_observation_pages. get_dataset, create_dataset, create_dataset_item and get_trace_url have no diff hunk, so for those I relied on the live runs. These changes affect MiniBot:

Change Effect on MiniBot
run_experiment no longer calls POST dataset-run-items. It links the experiment to the dataset through span attributes. This is the reason for the upgrade. Verified below.
Metadata values, strings included, are JSON-encoded on write, and the server decodes them. None values are skipped. promote reads tool_call_id, approved, run_id and other values back. I wrote a trace with SDK 5.2 and read it back: every value has the same shape as a v4-written trace (no extra quotes; bools and ints keep their types).
The SDK derives the experiment id from the project, the dataset and the run name: 16 hex characters, where v4 used the server's dataset-run UUID. Runs with the same run_name share one experiment. MiniBot does not store or read experiment ids. The default run name is <model>-<YYYYmmdd-HHMM>-<git sha>, so two runs in the same minute on the same commit share one experiment. Use --name to keep them apart. --repeat (#10) names each trial NAME-tN, so every trial stays its own experiment.
Scores go to POST /api/public/scores in batches instead of as ingestion events. A score sent after shutdown() is dropped with a warning. run_experiment calls flush(), not shutdown(). The live run below has all its scores.
ExperimentResult.dataset_run_id / dataset_run_url are now deprecated aliases of experiment_id / experiment_url. MiniBot reads neither of them.
Removed: api.legacy, api.sessions, api.dataset_run_items, get_dataset_runs, set_trace_io, and others. MiniBot calls none of them.

Every call site I checked behaves the same: Langfuse(), get_client, propagate_attributes, start_observation, start_as_current_observation, update, create_event, end, flush, shutdown, get_dataset, create_dataset, create_dataset_item, run_experiment, Evaluation, get_trace_url and api.observations.get_many.

Verification

All checks ran on 2026-11-17 against Langfuse Cloud (US) with SDK 5.2.0, and MINIBOT_HOME pointed at a temporary directory.

  • uv run ruff check .: clean.
  • After merging main (session: one terminal event per run, also on Ctrl+C and subscriber errors #7–evals: an empty Langfuse page no longer ends a trace fetch #11): unit tests 460 passed, 3 skipped. Ruff is clean, and uv sync --locked succeeds. I also repeated the real run on this tree (sdk5-rebased-20261117-1451, experiment 469f810132622c1d): dataset id set, 7 item scores, and traces --limit 3 lists it. The other checks below ran before the merge, on evals: an empty Langfuse page no longer ends a trace fetch #11 plus this change. The merge brought no new SDK calls.
  • Unit tests before the merge: 443 passed, 3 skipped. This worktree's folder is not named minibot, so I ran the documented command from a temporary parent with a minibot symlink, and checked that minibot.__file__ pointed at this worktree and that langfuse was 5.2.0. The tracing tests use a fake client, so they do not test the SDK itself. The real runs below do.
  • evals run --case chat-concept-no-tools --name sdk5-check-20261117-1441: all stages passed. No SDK warnings, with -W default on. In api.experiments.list, experiment ef92238028406aef has the minibot-golden dataset id, its run metadata and 1 item. In api.experiments.list_items, that item has all 7 scores (skill_selection, tool_selection, answer, end_to_end, model_calls, tokens, latency_s). get_trace_url returned a project trace link.
  • 4.16 baseline: evals: an empty Langfuse page no longer ends a trace fetch #11's probe-410-v4 experiment (written by SDK 4.16 today) has the same shape: dataset id set, 0 run-level scores (MiniBot passes no run evaluators), 7 item scores, and experimentDatasetId: null on the item. So the null item field comes from the server and is not a v5 change.
  • evals traces --limit 3: listed the new 5.2 trace and two older 4.16 traces.
  • draft_case (called in-process, so the dataset is not synced) works on the new 5.2 trace and on an older 4.16 trace with a denied tool call.
  • Tool and approval metadata: the no-tools case writes neither, so I sent one synthetic run through the real LangfuseFold. It had a tool call, an approval and a denial, under the trace name minibot.sdk5-check (trace de21cbce79e1800c73968ab00f9c36cc, still in the project). On read-back, both approval events have the tool as their parent, and draft_case drafts approval: deny, tools_required: [exec] and the exec args check.
  • Golden pytest, one case: 4 passed, 2 skipped (no tool_args or outcome expectations). The golden-pytest:chat-concept-no-tools span reached Langfuse with the turn under it.

Not verified

  • The other 9 golden cases. I ran only chat-concept-no-tools.
  • The full promote CLI. I called draft_case directly, because the CLI also syncs the live dataset.
  • A real TUI or server session with tracing on. Only the eval sandbox and the synthetic fold run used the real client.
  • The Langfuse UI. After a run, evals run still prints "Datasets → minibot-golden → Runs". That path may be out of date now that runs are experiments. v5 returns result.experiment_url, so printing that link would be a small follow-up.

Seen in passing (not changed)

  • python -m minibot.evals pytest cannot collect tests in a worktree whose folder is not named minibot: run_pytest resolves the real path, and the relative imports in test_golden.py then fail. This happens on main too. I ran pytest on the symlinked path instead.

🤖 Generated with Claude Code

Jiminyang1 and others added 2 commits November 17, 2026 14:46
Langfuse removed the v3 dataset-run read endpoints on 2026-11-16.
SDK 4.16 still writes each experiment item with
POST /api/public/dataset-run-items, which belongs to that model.
SDK 5 records experiments through OpenTelemetry only.

No call site changes. The v5 behaviour changes that touch MiniBot:

- Metadata values are JSON-encoded on write and decoded by the server.
  Read back through the v2 observations API, tool_call_id, approved,
  requires_approval, run_id and turn_index keep their v4 shape, so
  promote needs no change.
- run_experiment derives the experiment id from the dataset and the
  run name. Runs with the same name now share one experiment.

promote keeps its own observation_pages: SDK 5.2's
iter_observation_pages has the same stop rule and timeout, but no
repeated-cursor guard and no page bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #7–#11. #11 already contains this branch's base commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jiminyang1
Jiminyang1 merged commit 823c055 into main Nov 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0