fix(inference): AttributeError in streaming response cleanup (#4236)

This PR fixes issue #3185 
The code calls `await event_gen.aclose()` but OpenAI's `AsyncStream`
doesn't have an `aclose()` method - it has `close()` (which is async).
when clients cancel streaming requests, the server tries to clean up
with:

```python
await event_gen.aclose()  #  AsyncStream doesn't have aclose()!
```

But `AsyncStream` has never had a public `aclose()` method. The error
message literally tells us:

```
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
                                                                            ^^^^^^^^
```

## Verification
* Reproduction script
[`reproduce_issue_3185.sh`](https://gist.github.com/r-bit-rry/dea4f8fbb81c446f5db50ea7abd6379b)
can be used to verify the fix.
*   Manual checks, validation against original OpenAI library code
This commit is contained in:
Roy Belio 2025-12-14 13:51:09 +01:00 committed by GitHub
parent dfb9f6743a
commit c574db5f1d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 213 additions and 30 deletions

View file

@ -1020,11 +1020,11 @@ class InferenceProvider(Protocol):
async def openai_completion(
self,
params: Annotated[OpenAICompletionRequestWithExtraBody, Body(...)],
) -> OpenAICompletion:
) -> OpenAICompletion | AsyncIterator[OpenAICompletion]:
"""Create completion.
Generate an OpenAI-compatible completion for the given prompt using the specified model.
:returns: An OpenAICompletion.
:returns: An OpenAICompletion. When streaming, returns Server-Sent Events (SSE) with OpenAICompletion chunks.
"""
...
@ -1036,7 +1036,7 @@ class InferenceProvider(Protocol):
"""Create chat completions.
Generate an OpenAI-compatible chat completion for the given messages using the specified model.
:returns: An OpenAIChatCompletion.
:returns: An OpenAIChatCompletion. When streaming, returns Server-Sent Events (SSE) with OpenAIChatCompletionChunk objects.
"""
...