mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-12-21 07:22:25 +00:00
fix(inference): AttributeError in streaming response cleanup (#4236)
This PR fixes issue #3185 The code calls `await event_gen.aclose()` but OpenAI's `AsyncStream` doesn't have an `aclose()` method - it has `close()` (which is async). when clients cancel streaming requests, the server tries to clean up with: ```python await event_gen.aclose() # ❌ AsyncStream doesn't have aclose()! ``` But `AsyncStream` has never had a public `aclose()` method. The error message literally tells us: ``` AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'? ^^^^^^^^ ``` ## Verification * Reproduction script [`reproduce_issue_3185.sh`](https://gist.github.com/r-bit-rry/dea4f8fbb81c446f5db50ea7abd6379b) can be used to verify the fix. * Manual checks, validation against original OpenAI library code
This commit is contained in:
parent
dfb9f6743a
commit
c574db5f1d
14 changed files with 213 additions and 30 deletions
|
|
@ -1020,11 +1020,11 @@ class InferenceProvider(Protocol):
|
|||
async def openai_completion(
|
||||
self,
|
||||
params: Annotated[OpenAICompletionRequestWithExtraBody, Body(...)],
|
||||
) -> OpenAICompletion:
|
||||
) -> OpenAICompletion | AsyncIterator[OpenAICompletion]:
|
||||
"""Create completion.
|
||||
|
||||
Generate an OpenAI-compatible completion for the given prompt using the specified model.
|
||||
:returns: An OpenAICompletion.
|
||||
:returns: An OpenAICompletion. When streaming, returns Server-Sent Events (SSE) with OpenAICompletion chunks.
|
||||
"""
|
||||
...
|
||||
|
||||
|
|
@ -1036,7 +1036,7 @@ class InferenceProvider(Protocol):
|
|||
"""Create chat completions.
|
||||
|
||||
Generate an OpenAI-compatible chat completion for the given messages using the specified model.
|
||||
:returns: An OpenAIChatCompletion.
|
||||
:returns: An OpenAIChatCompletion. When streaming, returns Server-Sent Events (SSE) with OpenAIChatCompletionChunk objects.
|
||||
"""
|
||||
...
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue