Repository navigation
Fixed-length streamed upstream response body is dropped with StreamResponseBody #239
Description
Summary
When Apinto proxies an OpenAI-compatible chat completion endpoint that returns a normal JSON response with a fixed Content-Length, the client can receive HTTP 200 with an empty body through Apinto even though the upstream returned a valid JSON body.
I reproduced this with APIPark using Apinto gateway v0.22.19 and a KServe/vLLM OpenAI-compatible endpoint.
Reproduction
Upstream KServe endpoint, called directly from the Apinto container:
wget -S -O - \
--header="Content-Type: application/json" \
--post-data='{"model":"smollm2","messages":[{"role":"user","content":"Say hello!"}],"stream":false}' \
http://host.docker.internal:8080/openai/v1/chat/completionsDirect upstream response:
HTTP/1.1 200 OK
server: uvicorn
content-length: 668
content-type: application/jsonThe body is a valid OpenAI-compatible chat.completion JSON object.
The same model through APIPark/Apinto originally returned:
HTTP/1.1 200 OK
Server: uvicorn
Content-Type: application/json
Content-Length: 0
X-Ai-Provider: kserve-smollm2
X-Ai-Model: smollm2No JSON body was returned to the client.
Expected Behavior
For stream:false, Apinto should forward the upstream JSON response body:
{
"id": "...",
"object": "chat.completion",
"model": "smollm2",
"choices": [...]
}Actual Behavior
Apinto returns status and headers, but the body becomes empty for this fixed-length response path.
Root Cause
Apinto enables fasthttp response streaming:
StreamResponseBody: trueIn node/http-context/context.go, Apinto handles unknown-length body streams:
if response.IsBodyStream() && response.Header.ContentLength() < 0 {
...
}But the normal path then calls:
response.CopyTo(ctx.response.Response)fasthttp Response.CopyTo explicitly does not copy body streams:
// CopyTo copies resp contents to dst except of body stream.
func (resp *Response) CopyTo(dst *Response) { ... }So for a response that has a fixed Content-Length and is represented internally as a streamed response, Apinto can copy the headers but not the body.
Additional Issues Found While Testing A Local Fix
While building a local patched Apinto gateway, I hit two adjacent issues:
ai-convert/message.gocan panic for custom/local model names whentiktoken.GetEncoding(...)returns nil and the error is ignored.ai-convert/openai.gosetscontent-encoding: utf-8. UTF-8 is a character set, not an HTTP content encoding. Later body reads can treat this as an encoding and clear the response body. The safer behavior is to delete/avoid that header and leave charset handling toContent-Type.
Working Local Patch Shape
This local patch made APIPark return the KServe non-stream JSON body successfully while keeping SSE streaming working.
diff --git a/node/http-context/context.go b/node/http-context/context.go
@@
response.CopyTo(ctx.response.Response)
+ if response.Header.ContentLength() > 0 && len(ctx.response.Response.Body()) == 0 {
+ body := response.Body()
+ if len(body) > 0 {
+ ctx.response.SetBody(body)
+ }
+ }
+ ctx.response.Response.StreamBody = false
+ ctx.response.Response.ImmediateHeaderFlush = false
agent.responseBody.Write(ctx.response.Response.Body())diff --git a/ai-convert/message.go b/ai-convert/message.go
@@
if err != nil {
tkm, _ = tiktoken.GetEncoding(tiktoken.MODEL_CL100K_BASE)
}
+ if tkm == nil {
+ return 0
+ }diff --git a/ai-convert/openai.go b/ai-convert/openai.go
@@
- httpContext.Response().SetHeader("content-encoding", "utf-8")
+ httpContext.Response().DelHeader("content-encoding")
httpContext.Response().SetBody(body)Verification With Patched Gateway
Non-stream request through APIPark/Apinto:
curl -m 60 -i -sS -X POST http://localhost:8099/localmodel/chat/completions \
-H "Authorization: <consumer-api-key>" \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "Say hello in one word."}],
"max_tokens": 8,
"temperature": 0.2,
"stream": false
}'Patched response:
HTTP/1.1 200 OK
Server: uvicorn
Content-Type: application/json
Content-Length: 606
X-Ai-Provider: kserve-smollm2
X-Ai-Model: smollm2Body was returned successfully as OpenAI-compatible JSON.
Streaming still works:
curl -m 30 -N -sS -X POST http://localhost:8099/localmodel/chat/completions \
-H "Authorization: <consumer-api-key>" \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "Say hello in one sentence."}],
"max_tokens": 12,
"temperature": 0.2,
"stream": true
}'Response:
data: {"id":"...","object":"chat.completion.chunk",...}
data: [DONE]
Related Issues Checked
I checked existing issues:
- bug: 启用应用后，GET和DELETE请求的body会置为null #228: request body issue for GET/DELETE; not the same response-body bug.
- bug: 网关异常 #219: broader streaming stability issue; related area, but not this fixed-length response path.
- bug: 网关HTTP响应一直处于pending状态 #230: response/logging issue reported as fixed in v0.22.19; not this bug.
- bug: 附带额外参数应用上线后，文件上传请求报错 opied 0 bytes from body stream instead of 7094 bytes #233: request body stream upload issue; related stream handling area, but not this response-body bug.
Activity