8000 feat(llm): stream SageMaker and Bedrock answers as they are written by adeelahmad · Pull Request #672 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content

feat(llm): stream SageMaker and Bedrock answers as they are written - #672

Merged
adeelahmad merged 1 commit into
main
Oct 19, 2026
Merged

adeelahmad merged 1 commit into
mainfrom
claude/project-thread-azm2mb

Conversation

@adeelahmad
Copy link
Copy Markdown
Owner

Description

Before: with an Amazon SageMaker endpoint or Amazon Bedrock as the LLM provider, a chat's answer arrived all at once at the end. Docs listed both streams under Refine later.

After: the answer streams as it is written. SageMaker uses InvokeEndpointWithResponseStream and Bedrock uses ConverseStream. An error the model hits mid-answer says so ("stopped mid-answer: ModelStreamError: …"). Keys whose IAM policy predates this (only sagemaker:InvokeEndpoint or bedrock:InvokeModel) keep working: the stream's 403 falls back to the whole answer, as before.

How:

  • SageMaker: a streamed request goes to /invocations-response-stream with X-Amzn-SageMaker-Accept: text/event-stream. The answer's AWS event stream is decoded with botocore's own EventStreamBuffer, and its PayloadPart bytes are the container's server-sent events, read like any server's.
  • Bedrock: converse_stream events become OpenAI chat completion chunks (bedrock.Streamed).
  • Tests: they speak the real wire format. tests/aws_eventstream.py encodes messages with the prelude, headers and CRCs that botocore checks. The SageMaker request is checked against botocore's model of the operation. Bedrock's goes through botocore's own HTTP path, with no Stubber.

Motivation and Context

This is part of Adeel's auto scaling ask: SageMaker and Bedrock units serve Lens's chat, which should feel like any other model server.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (ruff, eslint, prettier).
  • I added or updated tests. In both test_llm_sagemaker.py and test_llm_bedrock.py: test_a_streamed_answer_…, test_a_model_error_mid_stream_says_so and test_keys_that_may_only_invoke_get_the_answer_whole.
  • Documentation updated (docs/configuration.md, docs/scaling.md IAM lists and Refine later, docs/requirements.md).
  • Changelog fragment in changelog.d/llm-aws-streams.md.

Done questions (AGENTS.md)

  • Goal: merges on green CI.
  • Agents: n/a. The same chat path serves the assistant, MCP and the API.
  • Zero setup: nothing new to set up. Old IAM policies keep working, with whole answers.
  • Fewest clicks: n/a.
  • Configurable: n/a, no new setting.
  • Light: botocore stays the optional extra it was; no new dependency.
  • Private and secure: the same signed calls to the same runtime. The secret is never sent.
  • Cost: the same model calls and the same ledger rows, including usage from the stream's metadata.
  • User decides: existing defaults are unchanged. Keys without the stream permission behave as before.
  • Upgrades: no stored-data change. Old IAM policies are covered by the fallback.
  • Honest: docs say what streams and what each permission does.
  • Tested and documented.
  • Open: n/a, no new dependency.
  • Reported in the thread.
  • Reuse: botocore's event stream decoder and the existing server-sent event reader in llm._stream.

Generated by Claude Code

9E98
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ALjNbLobvPe7VwvDL8j7V3
@adeelahmad
adeelahmad merged commit b65a5f7 into main Oct 19, 2026
12 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-azm2mb branch October 19, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0