14 commits

Author	SHA1	Message	Date
Eric Huang	67cdeb1bef	fix shutdown ... # What does this PR do? Gets rid of this error message below (disclaimer: not sure why, but it does). ERROR 2025-10-06 12:04:22,837 asyncio:118 uncategorized: Task exception was never retrieved future: <Task finished name='Task-36' coro=<AsyncClient.aclose() done, defined at /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpx/_client.py:1978> exception=RuntimeError('unable to perform operation on <TCPTransport closed=True reading=False 0x122dc7ad0>; the handler is closed')> ╭─────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────╮ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpx/_client.py:1985 in aclose │ │ │ │ 1982 │ │ if self._state != ClientState.CLOSED: │ │ 1983 │ │ │ self._state = ClientState.CLOSED │ │ 1984 │ │ │ │ │ ❱ 1985 │ │ │ await self._transport.aclose() │ │ 1986 │ │ │ for proxy in self._mounts.values(): │ │ 1987 │ │ │ │ if proxy is not None: │ │ 1988 │ │ │ │ │ await proxy.aclose() │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpx/_transports/default.py:406 in aclose │ │ │ │ 403 │ │ ) │ │ 404 │ │ │ 405 │ async def aclose(self) -> None: │ │ ❱ 406 │ │ await self._pool.aclose() │ │ 407 │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py:353 in aclose │ │ │ │ 350 │ │ with self._optional_thread_lock: │ │ 351 │ │ │ closing_connections = list(self._connections) │ │ 352 │ │ │ self._connections = [] │ │ ❱ 353 │ │ await self._close_connections(closing_connections) │ │ 354 │ │ │ 355 │ async def __aenter__(self) -> AsyncConnectionPool: │ │ 356 │ │ return self │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py:345 in _close_connections │ │ │ │ 342 │ │ # Close connections which have been removed from the pool. │ │ 343 │ │ with AsyncShieldCancellation(): │ │ 344 │ │ │ for connection in closing: │ │ ❱ 345 │ │ │ │ await connection.aclose() │ │ 346 │ │ │ 347 │ async def aclose(self) -> None: │ │ 348 │ │ # Explicitly close the connection pool. │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py:173 in aclose │ │ │ │ 170 │ async def aclose(self) -> None: │ │ 171 │ │ if self._connection is not None: │ │ 172 │ │ │ async with Trace("close", logger, None, {}): │ │ ❱ 173 │ │ │ │ await self._connection.aclose() │ │ 174 │ │ │ 175 │ def is_available(self) -> bool: │ │ 176 │ │ if self._connection is None: │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py:258 in aclose │ │ │ │ 255 │ │ # Note that this method unilaterally closes the connection, and does │ │ 256 │ │ # not have any kind of locking in place around it. │ │ 257 │ │ self._state = HTTPConnectionState.CLOSED │ │ ❱ 258 │ │ await self._network_stream.aclose() │ │ 259 │ │ │ 260 │ # The AsyncConnectionInterface methods provide information about the state of │ │ 261 │ # the connection, allowing for a connection pooling implementation to │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/httpcore/_backends/anyio.py:53 in aclose │ │ │ │ 50 │ │ │ │ await self._stream.send(item=buffer) │ │ 51 │ │ │ 52 │ async def aclose(self) -> None: │ │ ❱ 53 │ │ await self._stream.aclose() │ │ 54 │ │ │ 55 │ async def start_tls( │ │ 56 │ │ self, │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/anyio/streams/tls.py:216 in aclose │ │ │ │ 213 │ │ │ │ await aclose_forcefully(self.transport_stream) │ │ 214 │ │ │ │ raise │ │ 215 │ │ │ │ ❱ 216 │ │ await self.transport_stream.aclose() │ │ 217 │ │ │ 218 │ async def receive(self, max_bytes: int = 65536) -> bytes: │ │ 219 │ │ data = await self._call_sslobject_method(self._ssl_object.read, max_bytes) │ │ │ │ /Users/erichuang/projects/llama-stack-git2/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py:1310 in aclose │ │ │ │ 1307 │ │ if not self._transport.is_closing(): │ │ 1308 │ │ │ self._closed = True │ │ 1309 │ │ │ try: │ │ ❱ 1310 │ │ │ │ self._transport.write_eof() │ │ 1311 │ │ │ except OSError: │ │ 1312 │ │ │ │ pass │ │ 1313 │ │ │ │ in uvloop.loop.UVStream.write_eof:703 │ │ │ │ in uvloop.loop.UVHandle._ensure_alive:159 │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x122dc7ad0>; the handler is closed ## Test Plan Run uv run --with llama-stack llama stack build --distro=starter --image-type=venv --run No more error	2025-10-06 13:50:26 -07:00
Matthew Farrellee	d23ed26238	chore: turn OpenAIMixin into a pydantic.BaseModel (#3671) ... # What does this PR do? - implement get_api_key instead of relying on LiteLLMOpenAIMixin.get_api_key - remove use of LiteLLMOpenAIMixin - add default initialize/shutdown methods to OpenAIMixin - remove __init__s to allow proper pydantic construction - remove dead code from vllm adapter and associated / duplicate unit tests - update vllm adapter to use openaimixin for model registration - remove ModelRegistryHelper from fireworks & together adapters - remove Inference from nvidia adapter - complete type hints on embedding_model_metadata - allow extra fields on OpenAIMixin, for model_store, __provider_id__, etc - new recordings for ollama - enhance the list models error handling - update cerebras (remove cerebras-cloud-sdk) and anthropic (custom model listing) inference adapters - parametrized test_inference_client_caching - remove cerebras, databricks, fireworks, together from blanket mypy exclude - removed unnecessary litellm deps ## Test Plan ci	2025-10-06 11:33:19 -04:00
Matthew Farrellee	724dac498c	chore: give OpenAIMixin subcalsses a change to list models without leaking _model_cache details (#3682) ... # What does this PR do? close the _model_cache abstraction leak ## Test Plan ci w/ new tests	2025-10-06 09:44:33 -04:00
Matthew Farrellee	0a41c4ead0	chore: OpenAIMixin implements ModelsProtocolPrivate (#3662) ... # What does this PR do? add ModelsProtocolPrivate methods to OpenAIMixin this will allow providers using OpenAIMixin to use a common interface ## Test Plan ci w/ new tests	2025-10-02 21:32:02 -07:00
Ashwin Bharambe	ef0736527d	feat(tools)!: substantial clean up of "Tool" related datatypes (#3627) ... This is a sweeping change to clean up some gunk around our "Tool" definitions. First, we had two types `Tool` and `ToolDef`. The first of these was a "Resource" type for the registry but we had stopped registering tools inside the Registry long back (and only registered ToolGroups.) The latter was for specifying tools for the Agents API. This PR removes the former and adds an optional `toolgroup_id` field to the latter. Secondly, as pointed out by @bbrowning in https://github.com/llamastack/llama-stack/pull/3003#issuecomment-3245270132, we were doing a lossy conversion from a full JSON schema from the MCP tool specification into our ToolDefinition to send it to the model. There is no necessity to do this -- we ourselves aren't doing any execution at all but merely passing it to the chat completions API which supports this. By doing this (and by doing it poorly), we encountered limitations like not supporting array items, or not resolving $refs, etc. To fix this, we replaced the `parameters` field by `{ input_schema, output_schema }` which can be full blown JSON schemas. Finally, there were some types in our llama-related chat format conversion which needed some cleanup. We are taking this opportunity to clean those up. This PR is a substantial breaking change to the API. However, given our window for introducing breaking changes, this suits us just fine. I will be landing a concurrent `llama-stack-client` change as well since API shapes are changing.	2025-10-02 15:12:03 -07:00
Matthew Farrellee	4dbe0593f9	chore: add provider-data-api-key support to openaimixin (#3639) ... # What does this PR do? the LiteLLMOpenAIMixin provides support for reading key from provider data (headers users send). this adds the same functionality to the OpenAIMixin. this is infrastructure for migrating providers. ## Test Plan ci w/ new tests	2025-10-01 13:44:59 -07:00
Matthew Farrellee	b48d5cfed7	feat(internal): add image_url download feature to OpenAIMixin (#3516) ... # What does this PR do? simplify Ollama inference adapter by - - moving image_url download code to OpenAIMixin - being a ModelRegistryHelper instead of having one (mypy blocks check_model_availability method assignment) ## Test Plan - add unit tests for new download feature - add integration tests for openai_chat_completion w/ image_url (close test gap)	2025-09-26 17:32:16 -04:00
Matthew Farrellee	b67aef2fc4	feat: add static embedding metadata to dynamic model listings for providers using OpenAIMixin (#3547) ... # What does this PR do? - remove auto-download of ollama embedding models - add embedding model metadata to dynamic listing w/ unit test - add support and tests for allowed_models - removed inference provider models.py files where dynamic listing is enabled - store embedding metadata in embedding_model_metadata field on inference providers - make model_entries optional on ModelRegistryHelper and LiteLLMOpenAIMixin - make OpenAIMixin a ModelRegistryHelper - skip base64 embedding test for remote::ollama, always returns floats - only use OpenAI client for ollama model listing - remove unused build_model_entry function - remove unused get_huggingface_repo function ## Test Plan ci w/ new tests	2025-09-25 17:17:00 -04:00
Matthew Farrellee	62e0aef7bc	fix: return llama stack model id from embeddings (#3525) ... # What does this PR do? the openai_embeddings method on OpenAIMixin was returning the provider's model id instead of the llama stack name ## Test Plan before - ``` $ ./scripts/integration-tests.sh --stack-config server:ci-tests --setup gpt --subdirs inference --inference-mode live --pattern test_openai_embeddings_single_string ... FAILED tests/integration/inference/test_openai_embeddings.py::test_openai_embeddings_single_string[openai_client-emb=openai/text-embedding-3-small] - AssertionError: assert 'text-embedding-3-small' == 'openai/text-...dding-3-small' FAILED tests/integration/inference/test_openai_embeddings.py::test_openai_embeddings_single_string[llama_stack_client-emb=openai/text-embedding-3-small] - AssertionError: assert 'text-embedding-3-small' == 'openai/text-...dding-3-small' ========================================== 2 failed, 95 deselected, 4 warnings in 3.87s =========================================== ``` after - ``` $ ./scripts/integration-tests.sh --stack-config server:ci-tests --setup gpt --subdirs inference --inference-mode live --pattern test_openai_embeddings_single_string ... ========================================== 2 passed, 95 deselected, 4 warnings in 2.12s =========================================== ```	2025-09-23 12:30:00 -04:00
Matthew Farrellee	521865c388	feat: include all models from provider's /v1/models (#3471) ... # What does this PR do? this replaces the static model listing for any provider using OpenAIMixin currently - - anthropic - azure openai - gemini - groq - llama-api - nvidia - openai - sambanova - tgi - vertexai - vllm - not changed: together has its own impl ## Test Plan - new unit tests - manual for llama-api, openai, groq, gemini ``` for provider in llama-openai-compat openai groq gemini; do uv run llama stack build --image-type venv --providers inference=remote::provider --run & uv run --with llama-stack-client llama-stack-client models list | grep Total ``` results (17 sep 2025): - llama-api: 4 - openai: 86 - groq: 21 - gemini: 66 closes #3467	2025-09-18 05:17:11 -04:00
Matthew Farrellee	f4ab154ade	feat: add dynamic model registration support to TGI inference (#3417) ... # What does this PR do? adds dynamic model support to TGI add new overwrite_completion_id feature to OpenAIMixin to deal with TGI always returning id="" ## Test Plan tgi: `docker run --gpus all --shm-size 1g -p 8080:80 -v /data:/data ghcr.io/huggingface/text-generation-inference --model-id Qwen/Qwen3-0.6B` stack: `TGI_URL=http://localhost:8080 uv run llama stack build --image-type venv --distro ci-tests --run` test: `./scripts/integration-tests.sh --stack-config http://localhost:8321 --setup tgi --subdirs inference --pattern openai`	2025-09-15 15:52:40 -04:00
Matthew Farrellee	8ef1189be7	chore: update the vLLM inference impl to use OpenAIMixin for openai-compat functions (#3404) ... # What does this PR do? update vLLM inference provider to use OpenAIMixin for openai-compat functions inference recordings from Qwen3-0.6B and vLLM 0.8.3 - ``` docker run --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host \ vllm/vllm-openai:latest \ --model Qwen/Qwen3-0.6B --enable-auto-tool-choice --tool-call-parser hermes ``` ## Test Plan ``` ./scripts/integration-tests.sh --stack-config server:ci-tests --setup vllm --subdirs inference ```	2025-09-11 09:04:38 -04:00
Mustafa Elbehery	c3b2b06974	refactor(logging): rename llama_stack logger categories (#3065) ... # What does this PR do? <!-- Provide a short summary of what this PR does and why. Link to relevant issues if applicable. --> This PR renames categories of llama_stack loggers. This PR aligns logging categories as per the package name, as well as reviews from initial https://github.com/meta-llama/llama-stack/pull/2868. This is a follow up to #3061. <!-- If resolving an issue, uncomment and update the line below --> <!-- Closes #[issue-number] --> Replaces https://github.com/meta-llama/llama-stack/pull/2868 Part of https://github.com/meta-llama/llama-stack/issues/2865 cc @leseb @rhuss Signed-off-by: Mustafa Elbehery <melbeher@redhat.com>	2025-08-21 17:31:04 -07:00
Matthew Farrellee	e1ed152779	chore: create OpenAIMixin for inference providers with an OpenAI-compat API that need to implement openai_* methods (#2835) ... # What does this PR do? add an `OpenAIMixin` for use by inference providers who remote endpoints support an OpenAI compatible API. use is demonstrated by refactoring - OpenAIInferenceAdapter - NVIDIAInferenceAdapter (adds embedding support) - LlamaCompatInferenceAdapter ## Test Plan existing unit and integration tests	2025-07-23 06:49:40 -04:00