8000 feat(llm): stream SageMaker and Bedrock answers as they are written by adeelahmad · Pull Request #672 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
8000
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog.d/llm-aws-streams.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **SageMaker and Bedrock answers stream.** With an Amazon SageMaker endpoint or Amazon Bedrock as the LLM provider, a
chat's answer now comes as it is written (InvokeEndpointWithResponseStream and ConverseStream) instead of all at the
end, and an error the model hits mid-answer says so. Keys also allowed `sagemaker:InvokeEndpointWithResponseStream`
or `bedrock:InvokeModelWithResponseStream` stream; keys without them get the answer whole, as before ([LLM provider](docs/configuration.md)).
6 changes: 4 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,8 +208,10 @@ makes such endpoints). So do Amazon Bedrock's models: the address is
`https://bedrock-runtime.<region>.amazonaws.com`, the model a Bedrock model id, inference profile
(`us.amazon.nova-lite-v1:0`) or provisioned throughput ARN, and the key an AWS access key the same way. Lens sends its
requests to Bedrock's Converse API, with the system prompt, images, tools and tool results it carries, asks for JSON
in the system prompt, and gets each answer whole (not streamed yet). Embeddings and speech need a server of their own
then: they never borrow the AWS key. The keys need `bedrock:InvokeModel` on the model. Picking a provider changes nothing until you save; from then on, what Lens asks its model
in the system prompt; a chat's answer streams as it is written (ConverseStream). Embeddings and speech need a server
of their own then: they never borrow the AWS key. The keys need `bedrock:InvokeModel` and
`bedrock:InvokeModelWithResponseStream` on the model. A SageMaker endpoint's chat answers stream the same way
(InvokeEndpointWithResponseStream), as the container writes them. Keys allowed only `bedrock:InvokeModel` or `sagemaker:InvokeEndpoint` still work: their answers come whole. Picking a provider changes nothing until you save; from then on, what Lens asks its model
(summaries, chat, descriptions, with the text they need) goes to that provider.

## Chat models
Expand Down
4 changes: 2 additions & 2 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,8 +332,8 @@ request, to another check that proves the same thing; loosening a requirement or
| When AWS takes a spot instance back, the scaler says why and launches another | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_describe_says_when_aws_took_a_spot_instance_back`, `test_the_scaler_replaces_a_spot_unit_aws_took_back`, `test_terminate_check_and_listing` |
| RunPod: GPU pods serving a model with vLLM, each with an API key of its own sealed on the unit, running once vLLM answers, deleted when stopped; no mesh needed | [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_a_runpod_group_serves_a_model_on_gpus_it_rents`, `test_runpod_says_why_it_made_no_pod` |
| Hugging Face: Inference Endpoints serving a model with vLLM on hardware picked from Hugging Face's list (price, cloud, GPUs), each with a key of its own as an endpoint secret, never the Hugging Face token; requests and answers held to Hugging Face's published API document; units that never come up pause the group after three | [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_a_hugging_face_group_serves_a_model_on_an_inference_endpoint`, `test_endpoints_that_never_come_up_pause_the_group`, `test_lens_reads_only_fields_hugging_face_answers_with` |
| Amazon SageMaker: real-time endpoints serving a model with AWS's vLLM container, each a model, an endpoint configuration and an endpoint tagged with the unit and all three deleted when it goes, priced from the AWS Price List; Lens's model calls to it are signed with AWS Signature Version 4, never carry the secret key, and embeddings and speech never borrow it | [test_scaling_sagemaker_moto.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_sagemaker_moto.py) `test_a_unit_is_a_model_a_configuration_and_an_endpoint_and_goes_as_all_three`, `test_an_endpoint_that_fails_says_sagemakers_reason`, `test_without_a_price_or_with_a_bad_setting_nothing_is_made`, `test_a_sagemaker_group_serves_lenss_model_calls_signed_until_it_scales_in`; [test_llm_sagemaker.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_llm_sagemaker.py) `test_a_chat_goes_to_invoke_endpoint_signed_and_never_carries_the_secret`, `test_a_streamed_answer_reads_as_one_from_any_server`, `test_an_endpoint_serves_its_one_model_and_is_never_asked_for_a_list`, `test_a_key_that_isnt_an_aws_key_says_what_it_should_be`, `test_other_servers_still_get_a_bearer_token_at_chat_completions`, `test_embeddings_and_speech_never_borrow_the_aws_key` |
| Amazon Bedrock as the LLM provider: chat, tool calls and their results, images, JSON answers and summaries go to Bedrock's Converse API signed with the AWS key, Bedrock's answers and refusals read as an OpenAI-compatible server's (a context too long, tools a model doesn't take), and embeddings and speech never borrow the key; each request and answer is held to botocore's model of the API | [test_llm_bedrock.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_llm_bedrock.py) `test_a_chat_is_a_converse_call_and_reads_as_an_openai_reply`, `test_tool_calls_and_their_results_go_as_converse_turns`, `test_a_json_answer_asks_for_the_schema_in_the_system_prompt`, `test_a_summary_goes_to_bedrock_too`, `test_a_streamed_answer_comes_whole`, `test_bedrocks_refusals_read_as_lenss`, `test_bedrock_lends_its_key_to_nothing_and_lists_the_model_it_was_given` |
| Amazon SageMaker: real-time endpoints serving a model with AWS's vLLM container, each a model, an endpoint configuration and an endpoint tagged with the unit and all three deleted when it goes, priced from the AWS Price List; Lens's model calls to it are signed with AWS Signature Version 4, never carry the secret key, stream as the container writes (InvokeEndpointWithResponseStream, read with botocore's own event stream decoder; whole when the key may only invoke), and embeddings and speech never borrow it | [test_scaling_sagemaker_moto.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_sagemaker_moto.py) `test_a_unit_is_a_model_a_configuration_and_an_endpoint_and_goes_as_all_three`, `test_an_endpoint_that_fails_says_sagemakers_reason`, `test_without_a_price_or_with_a_bad_setting_nothing_is_made`, `test_a_sagemaker_group_serves_lenss_model_calls_signed_until_it_scales_in`; [test_llm_sagemaker.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_llm_sagemaker.py) `test_a_chat_goes_to_invoke_endpoint_signed_and_never_carries_the_secret`, `test_a_streamed_answer_comes_as_the_container_writes_it`, `test_a_model_error_mid_stream_says_so`, `test_keys_that_may_only_invoke_get_the_answer_whole`, `test_an_endpoint_serves_its_one_model_and_is_never_asked_for_a_list`, `test_a_key_that_isnt_an_aws_key_says_what_it_should_be`, `test_other_servers_still_get_a_bearer_token_at_chat_completions`, `test_embeddings_and_speech_never_borrow_the_aws_key` |
| Amazon Bedrock as the LLM provider: chat, tool calls and their results, images, JSON answers and summaries go to Bedrock's Converse API signed with the AWS key, Bedrock's answers and refusals read as an OpenAI-compatible server's (a context too long, tools a model doesn't take), chat answers stream as they are written (ConverseStream; whole when the key may only invoke), and embeddings and speech never borrow the key; each request and answer is held to botocore's model of the API | [test_llm_bedrock.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_llm_bedrock.py) `test_a_chat_is_a_converse_call_and_reads_as_an_openai_reply`, `test_tool_calls_and_their_results_go_as_converse_turns`, `test_a_json_answer_asks_for_the_schema_in_the_system_prompt`, `test_a_summary_goes_to_bedrock_too`, `test_a_streamed_answer_is_converse_stream_as_it_is_written`, `test_a_model_error_mid_stream_says_so`, `test_keys_that_may_only_invoke_get_the_answer_whole`, `test_bedrocks_refusals_read_as_lenss`, `test_bedrock_lends_its_key_to_nothing_and_lists_the_model_it_was_given` |
| Amazon Bedrock provisioned throughput: made with no commitment, named and tagged with the unit, running once in service, failed ones gone with Bedrock's reason, found by name across pages and deleted when it goes; nothing launches without the admin's price; while it serves, Lens's model calls go to it signed; every request and answer held to AWS's model of Bedrock's API and its name patterns | [test_scaling_bedrock.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_bedrock.py) `test_a_unit_is_a_provisioned_throughput_held_until_it_goes`, `test_one_that_fails_says_bedrocks_reason_and_none_launch_without_a_price`, `test_a_long_name_keeps_its_number_and_bedrocks_pattern`, `test_a_bedrock_group_serves_lenss_model_calls_until_it_scales_in` |
| A running model unit is Lens's LLM provider until none runs, then the provider from before comes back; a provider a person picks meanwhile stays | [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_a_running_model_unit_serves_lenss_model_calls_until_none_runs`, `test_a_provider_a_person_picks_while_a_unit_serves_stays` |
| Provider fakes take only the requests the real API would, answer only what it could, and Lens reads only fields the provider documents; AWS runs from launch to scale-in on moto | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_lens_reads_only_fields_the_clouds_answer_with`; [test_scaling_models.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_models.py) `test_lens_reads_only_fields_runpod_answers_with`; [test_scaling_aws_moto.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws_moto.py) `test_a_spot_instance_launches_tagged_hardened_and_is_found_by_its_tag`, `test_an_instance_someone_stopped_is_terminated_so_its_disk_stops_costing`, `test_the_scaler_runs_a_group_on_moto_from_launch_to_scale_in` |
Expand Down
10 changes: 5 additions & 5 deletions docs/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ AWS's vLLM container is published by account `763104351884` in most regions (`vl
others. The keys need `sagemaker:ListEndpoints`, `sagemaker:DescribeEndpoint`, `sagemaker:CreateModel`,
`sagemaker:CreateEndpointConfig`, `sagemaker:CreateEndpoint`, `sagemaker:DeleteEndpoint`,
`sagemaker:DeleteEndpointConfig`, `sagemaker:DeleteModel`, `sagemaker:AddTags`, `sagemaker:InvokeEndpoint`,
`sagemaker:InvokeEndpointWithResponseStream`,
`iam:PassRole` on the execution role, and `pricing:GetProducts` for prices you don't give yourself. The execution role
is one SageMaker may assume (the `AmazonSageMakerFullAccess` managed policy, or one that only pulls from ECR and
writes logs).
Expand All @@ -197,8 +198,8 @@ doesn't list provisioned throughput in the AWS Price List, so the provider's pri
pricing page, and no unit launches without it. Which models take provisioned throughput, and in which regions, is
Bedrock's to say; a launch it refuses fails with its message. The keys need
`bedrock:CreateProvisionedModelThroughput`, `bedrock:GetProvisionedModelThroughput`,
`bedrock:ListProvisionedModelThroughputs`, `bedrock:DeleteProvisionedModelThroughput`, `bedrock:TagResource` and
`bedrock:InvokeModel`.
`bedrock:ListProvisionedModelThroughputs`, `bedrock:DeleteProvisionedModelThroughput`, `bedrock:TagResource`,
`bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream`.

A unit that never comes up, whether it doesn't join within `join_minutes` or its provider reports it failed, counts
as a failed launch: three in a row pause the group (with the reason in its `paused_why`), and a unit that comes up
Expand Down Expand Up @@ -373,9 +374,8 @@ which only that yes in the web app does.
## Refine later

Spreading model calls across a group's units, and serving only some kinds of call (chat, decisions); more model
hosts (Lambda instances serving a model); Bedrock's price per unit-hour from AWS, its streamed answers
(`ConverseStream`) and Bedrock API keys as the provider's key; SageMaker's streamed invocations (`InvokeEndpointWithResponseStream`), so a SageMaker answer
streams as it is written; Hugging Face's scale to zero and private endpoints; RunPod network volumes, so a model is downloaded once; Proxmox LXC containers; a live euro rate;
hosts (Lambda instances serving a model); Bedrock's price per unit-hour from AWS and Bedrock API keys as the
provider's key; Hugging Face's scale to zero and private endpoints; RunPod network volumes, so a model is downloaded once; Proxmox LXC containers; a live euro rate;
preemptible and spot instances at clouds other than AWS and RunPod; draining an AWS spot unit on its two-minute notice
(the node agent reading the instance's metadata) and handing its job back at once; following the spot price as it
changes; AWS sign-in with an instance role or IAM Identity Center instead of keys; mixed instance types per group;
Expand Down
46 changes: 45 additions & 1 deletion fastapi_backend/app/domain/bedrock.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
extra, fetched on first use), which signs them with AWS Signature Version 4, so the key itself is never sent.

Converse takes the parts of a conversation Lens sends: system prompts, text and images, tool definitions, the
assistant's tool calls and their results. A JSON Schema asked for (`response_format`) goes into the system prompt."""
assistant's tool calls and their results. A JSON Schema asked for (`response_format`) goes into the system prompt. A
streamed answer is ConverseStream's, read as an OpenAI-compatible server's server-sent events as it is written."""

from __future__ import annotations

Expand Down Expand Up @@ -126,6 +127,49 @@ def add(role, blocks):
return params


class Streamed:
"""ConverseStream's events as an OpenAI-compatible server's streamed reply: `data:` lines of chat completion
chunks, then `data: [DONE]`, for llm.py to read as it reads any server's. An error event mid-answer is an OSError,
which llm.py reports as the server stopping mid-answer."""

headers = {"Content-Type": "text/event-stream"}

def __init__(self, events, model):
self.events, self.model = events, model

def _line(self, chunk):
return b"data: " + json.dumps({"object": "chat.completion.chunk", "model": self.model, **chunk}).encode() + b"\n"

def __iter__(self):
from botocore.exceptions import BotoCoreError, ClientError

try:
for e in self.events:
if "contentBlockDelta" in e:
text = ((e["contentBlockDelta"].get("delta") or {}).get("text")) or ""
if text:
yield self._line({"choices": [{"index": 0, "delta": {"content": text}, "finish_reason": None}]})
elif "messageStop" in e:
why = STOP.get(e["messageStop"].get("stopReason"), "stop")
yield self._line({"choices": [{"index": 0, "delta": {}, "finish_reason": why}]})
elif "metadata" in e:
u = e["metadata"].get("usage") or {}
usage = {
"prompt_tokens": u.get("inputTokens"),
"completion_tokens": u.get("outputTokens"),
"total_tokens": u.get("totalTokens"),
}
yield self._line({"choices": [], "usage": usage})
except (ClientError, BotoCoreError) as e: # an exception event, or the stream cut off
raise OSError(str(e)[:300]) from None
yield b"data: [DONE]\n"

def close(self):
close = getattr(self.events, "close", None)
if close:
close()


def openai_answer(out, model):
"""OpenAI's chat reply for Converse's answer."""
content = ((out.get("output") or {}).get("message") or {}).get("content") or []
Expand Down
Loading
Loading
0