Repository navigation
deps: move to the Langfuse Python SDK 5.2 - #12
Merged
Merged
Conversation
Langfuse removed the v3 dataset-run read endpoints on 2026-11-16. SDK 4.16 still writes each experiment item with POST /api/public/dataset-run-items, which belongs to that model. SDK 5 records experiments through OpenTelemetry only. No call site changes. The v5 behaviour changes that touch MiniBot: - Metadata values are JSON-encoded on write and decoded by the server. Read back through the v2 observations API, tool_call_id, approved, requires_approval, run_id and turn_index keep their v4 shape, so promote needs no change. - run_experiment derives the experiment id from the dataset and the run name. Runs with the same name now share one experiment. promote keeps its own observation_pages: SDK 5.2's iter_observation_pages has the same stop rule and timeout, but no repeated-cursor guard and no page bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
SDK 4.16 still sends
POST /api/public/dataset-run-itemsinsiderun_experiment. That endpoint belongs to the v3 dataset-run model. Langfuse removed the v3 read endpoints on 2026-11-16. SDK 5 records experiments through OpenTelemetry only, so this PR moves MiniBot off the deprecated write path before it goes away too.What changed
pyproject.toml:langfuse>=4.16,<5→langfuse>=5.2,<6.uv.lockresolves to 5.2.0, and no other package changes. The lockrevisionfield also changed from 3 to 5, because a neweruvwrote it.evals/promote.py: a docstring note only. I kept the localobservation_pagesand did not switch to the SDK's newiter_observation_pages. That helper has the same stop rule (only a missing cursor ends paging) and the same 60 s timeout. It has no repeated-cursor guard, no per-page dedupe and no page bound (max_itemscounts rows, so it does not limit empty pages). Wrapping it would also make evals: an empty Langfuse page no longer ends a trace fetch #11's fakes test a fake loop instead of ours.langfuse_tracing.py: the 8000 docstring now says "SDK v4 and later" instead of "v4".v5 changes I checked
GitHub has no release notes for
v5.0.0. The full change list is in the description of langfuse/langfuse-python#1991. I also diffed the 4.16.0 and 5.2.0 sources:span.py(start_observation,update,create_event,end),attributes.py,propagation.py,datasets.py,experiment.py,get_client.py,resource_manager.py, and theclient.pyhunks for__init__,flush,run_experimentanditer_observation_pages.get_dataset,create_dataset,create_dataset_itemandget_trace_urlhave no diff hunk, so for those I relied on the live runs. These changes affect MiniBot:run_experimentno longer callsPOST dataset-run-items. It links the experiment to the dataset through span attributes.Nonevalues are skipped.promotereadstool_call_id,approved,run_idand other values back. I wrote a trace with SDK 5.2 and read it back: every value has the same shape as a v4-written trace (no extra quotes; bools and ints keep their types).run_nameshare one experiment.<model>-<YYYYmmdd-HHMM>-<git sha>, so two runs in the same minute on the same commit share one experiment. Use--nameto keep them apart.--repeat(#10) names each trialNAME-tN, so every trial stays its own experiment.POST /api/public/scoresin batches instead of as ingestion events. A score sent aftershutdown()is dropped with a warning.run_experimentcallsflush(), notshutdown(). The live run below has all its scores.ExperimentResult.dataset_run_id/dataset_run_urlare now deprecated aliases ofexperiment_id/experiment_url.api.legacy,api.sessions,api.dataset_run_items,get_dataset_runs,set_trace_io, and others.Every call site I checked behaves the same:
Langfuse(),get_client,propagate_attributes,start_observation,start_as_current_observation,update,create_event,end,flush,shutdown,get_dataset,create_dataset,create_dataset_item,run_experiment,Evaluation,get_trace_urlandapi.observations.get_many.Verification
All checks ran on 2026-11-17 against Langfuse Cloud (US) with SDK 5.2.0, and
MINIBOT_HOMEpointed at a temporary directory.uv run ruff check .: clean.main(session: one terminal event per run, also on Ctrl+C and subscriber errors #7–evals: an empty Langfuse page no longer ends a trace fetch #11): unit tests 460 passed, 3 skipped. Ruff is clean, anduv sync --lockedsucceeds. I also repeated the real run on this tree (sdk5-rebased-20261117-1451, experiment469f810132622c1d): dataset id set, 7 item scores, andtraces --limit 3lists it. The other checks below ran before the merge, on evals: an empty Langfuse page no longer ends a trace fetch #11 plus this change. The merge brought no new SDK calls.minibot, so I ran the documented command from a temporary parent with aminibotsymlink, and checked thatminibot.__file__pointed at this worktree and thatlangfusewas 5.2.0. The tracing tests use a fake client, so they do not test the SDK itself. The real runs below do.evals run --case chat-concept-no-tools --name sdk5-check-20261117-1441: all stages passed. No SDK warnings, with-W defaulton. Inapi.experiments.list, experimentef92238028406aefhas theminibot-goldendataset id, its run metadata and 1 item. Inapi.experiments.list_items, that item has all 7 scores (skill_selection,tool_selection,answer,end_to_end,model_calls,tokens,latency_s).get_trace_urlreturned a project trace link.probe-410-v4experiment (written by SDK 4.16 today) has the same shape: dataset id set, 0 run-level scores (MiniBot passes no run evaluators), 7 item scores, andexperimentDatasetId: nullon the item. So the null item field comes from the server and is not a v5 change.evals traces --limit 3: listed the new 5.2 trace and two older 4.16 traces.draft_case(called in-process, so the dataset is not synced) works on the new 5.2 trace and on an older 4.16 trace with a denied tool call.LangfuseFold. It had a tool call, an approval and a denial, under the trace nameminibot.sdk5-check(tracede21cbce79e1800c73968ab00f9c36cc, still in the project). On read-back, both approval events have the tool as their parent, anddraft_casedraftsapproval: deny,tools_required: [exec]and theexecargs check.tool_argsoroutcomeexpectations). Thegolden-pytest:chat-concept-no-toolsspan reached Langfuse with the turn under it.Not verified
chat-concept-no-tools.promoteCLI. I calleddraft_casedirectly, because the CLI also syncs the live dataset.evals runstill prints "Datasets → minibot-golden → Runs". That path may be out of date now that runs are experiments. v5 returnsresult.experiment_url, so printing that link would be a small follow-up.Seen in passing (not changed)
python -m minibot.evals pytestcannot collect tests in a worktree whose folder is not namedminibot:run_pytestresolves the real path, and the relative imports intest_golden.pythen fail. This happens onmaintoo. I ran pytest on the symlinked path instead.🤖 Generated with Claude Code