mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-10-04 20:14:13 +00:00
ddf3f1735a
# What does this PR do? Refs: https://github.com/llamastack/llama-stack/issues/3420 When telemetry is enabled the router uncondionally expects the usage attribute to be availble and fails if it is not present. Usage is not currently being requested by litellm_openai_mixin.py for streaming requests when using the responses API which means that providers like vertexai fail if telemetry is enabled and streaming is used. This is part of the required fix. Other part is in liteLLM, will plan to submit PR for that soon. ## Test Plan I applied this change along with the change for litellm in a llama stack deployment and validated that I could make streaming requests through the responses API to a gemini model and they would succeed instead of failing due to the missing usage attribute when telemetry is enabled. Signed-off-by: Michael Dawson <midawson@redhat.com> |
||
|---|---|---|
| .. | ||
| bedrock | ||
| common | ||
| datasetio | ||
| inference | ||
| kvstore | ||
| memory | ||
| responses | ||
| scoring | ||
| sqlstore | ||
| telemetry | ||
| tools | ||
| vector_io | ||
| __init__.py | ||
| pagination.py | ||
| scheduler.py | ||