phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-12-18 11:29:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix(inference): AttributeError in streaming response cleanup (#4236)

Browse source 
This PR fixes issue #3185 
The code calls `await event_gen.aclose()` but OpenAI's `AsyncStream`
doesn't have an `aclose()` method - it has `close()` (which is async).
when clients cancel streaming requests, the server tries to clean up
with:

```python
await event_gen.aclose()  # ❌ AsyncStream doesn't have aclose()!
```

But `AsyncStream` has never had a public `aclose()` method. The error
message literally tells us:

```
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
                                                                            ^^^^^^^^
```

## Verification
* Reproduction script
[`reproduce_issue_3185.sh`](https://gist.github.com/r-bit-rry/dea4f8fbb81c446f5db50ea7abd6379b)
can be used to verify the fix.
*   Manual checks, validation against original OpenAI library code
This commit is contained in:
Roy Belio 2025-12-14 13:51:09 +01:00 • committed by GitHub
parent dfb9f6743a
commit c574db5f1d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 213 additions and 30 deletions
Download patch file Download diff file Expand all files Collapse all files

2
src/llama_stack/providers/remote/inference/bedrock/bedrock.py
View file

@ -69,7 +69,7 @@ class BedrockInferenceAdapter(OpenAIMixin):
async def openai_completion(
self,
params: OpenAICompletionRequestWithExtraBody,
) -> OpenAICompletion:
) -> OpenAICompletion | AsyncIterator[OpenAICompletion]:
"""Bedrock's OpenAI-compatible API does not support the /v1/completions endpoint."""
raise NotImplementedError(
"Bedrock's OpenAI-compatible API does not support /v1/completions endpoint. "