16 commits

Author	SHA1	Message	Date
Derek Higgins	5abb7df41a	fix: ABAC bypass in vector store operations (#4394) ... Vector store operations were bypassing ABAC checks by calling providers directly instead of going through the routing table. This allowed unauthorized access to vector store data and operations. Changes: o Route all VectorIORouter methods through routing table instead of directly to providers o Update routing table to enforce ABAC checks on all vector store operations (read, update, delete) o Add test suite verifying ABAC enforcement for all vector store operations o Ensure providers are never called when authorization fails Fixes security issue where users could access vector stores they don't have permission for. Fixes: #4393 Signed-off-by: Derek Higgins <derekh@redhat.com>	2025-12-16 10:49:16 -08:00
Francisco Javier Arceo	62005dc1a9	feat: Making static prompt values in Rag/File Search configurable in Vector Store Config (#4368) ... # What does this PR do? - Enables users to configure prompts used throughout the File Search / Vector Retrieval - Configuration is defined in the Vector Stores Config so they can be modified at runtime - Backwards compatible, which means the fields are optional and default to the previously used values This is the summary of the new options in the `run.yaml` ```yaml vector_stores: file_search_params: header_template: 'knowledge_search tool found {num_chunks} chunks:\nBEGIN of knowledge_search tool results.\n' footer_template: 'END of knowledge_search tool results.\n' context_prompt_params: chunk_annotation_template: 'Result {index}\nContent: {chunk.content}\nMetadata: {metadata}\n' context_template: 'The above results were retrieved to help answer the user\'s query: "{query}". Use them as supporting information only in answering this query.{annotation_instruction}\n' annotation_prompt_params: enable_annotations: true annotation_instruction_template: 'Cite sources immediately at the end of sentences before punctuation, using `<|file-id|>` format like \'This is a fact <|file-Cn3MSNn72ENTiiq11Qda4A|>.\'. Do not add extra punctuation. Use only the file IDs provided, do not invent new ones.' chunk_annotation_template: '[{index}] {metadata_text} cite as <|{file_id}|>\n{chunk_text}\n' ``` <!-- If resolving an issue, uncomment and update the line below --> <!-- Closes #[issue-number] --> ## Test Plan Added tests. --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>	2025-12-15 11:39:01 -05:00
Francisco Javier Arceo	95b2948d11	feat: Add support for query rewrite in vector_store.search (#4171) ... # What does this PR do? Actualize query rewrite in search API, add `default_query_expansion_model` and `query_expansion_prompt` in `VectorStoresConfig`. Makes `rewrite_query` parameter functional in vector store search. - `rewrite_query=false` (default): Use original query - `rewrite_query=true`: Expand query via LLM, or fail gracefully if no LLM available Adds 4 parameters to`VectorStoresConfig`: - `default_query_expansion_model`: LLM model for query expansion (optional) - `query_expansion_prompt`: Custom prompt template (optional, uses built-in default) - `query_expansion_max_tokens`: Configurable token limit (default: 100) - `query_expansion_temperature`: Configurable temperature (default: 0.3) Enabled `run.yaml`: ```yaml vector_stores: rewrite_query_params: model: provider_id: "ollama" model_id: "llama3.2:3b-instruct-fp16" # prompt defaults to built-in # max_tokens defaults to 100 # temperature defaults to 0.3 ``` Fully customized `run.yaml`: ```yaml vector_stores: default_provider_id: faiss default_embedding_model: provider_id: sentence-transformers model_id: nomic-ai/nomic-embed-text-v1.5 rewrite_query_params: model: provider_id: ollama model_id: llama3.2:3b-instruct-fp16 prompt: "Rewrite this search query to improve retrieval results by expanding it with relevant synonyms and related terms: {query}" max_tokens: 100 temperature: 0.3 ``` ## Test Plan Added test and recording Example script as well: ```python import asyncio from llama_stack_client import LlamaStackClient from io import BytesIO def gen_file(client, text: str=""): file_buffer = BytesIO(text.encode('utf-8')) file_buffer.name = "my_file.txt" uploaded_file = client.files.create( file=file_buffer, purpose="assistants" ) return uploaded_file async def test_query_rewriting(): client = LlamaStackClient(base_url="http://0.0.0.0:8321/") uploaded_file = gen_file(client, "banana banana apple") uploaded_file2 = gen_file(client, "orange orange kiwi") vs = client.vector_stores.create() xf_vs = client.vector_stores.files.create(vector_store_id=vs.id, file_id=uploaded_file.id) xf_vs1 = client.vector_stores.files.create(vector_store_id=vs.id, file_id=uploaded_file2.id) response1 = client.vector_stores.search( vector_store_id=vs.id, query="apple", max_num_results=3, rewrite_query=False ) response2 = client.vector_stores.search( vector_store_id=vs.id, query="kiwi", max_num_results=3, rewrite_query=True, ) print(f"\n🔵 Response 1 (rewrite_query=False):\n\033[94m{response1}\033[0m") print(f"\n🟢 Response 2 (rewrite_query=True):\n\033[92m{response2}\033[0m") for f in [uploaded_file.id, uploaded_file2.id]: client.files.delete(file_id=f) client.vector_stores.delete(vector_store_id=vs.id) if __name__ == "__main__": asyncio.run(test_query_rewriting()) ``` And see the screen shot of the server logs showing it worked. <img width="1111" height="826" alt="Screenshot 2025-11-19 at 1 16 03 PM" src="https://github.com/user-attachments/assets/2d188b44-1fef-4df5-b465-2d6728ca49ce" /> Notice the log: ```bash Query rewritten: 'kiwi' → 'kiwi, a small brown or green fruit native to New Zealand, or a person having a fuzzy brown outer skin similar in appearance.' ``` So `kiwi` was expanded. --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com> Co-authored-by: Matthew Farrellee <matt@cs.wisc.edu>	2025-12-10 10:06:19 -05:00
Charlie Doern	661985e240	feat: remove usage of build yaml (#4192) ... # What does this PR do? the build.yaml is only used in the following ways: 1. list-deps 2. distribution code-gen since `llama stack build` no longer exists, I found myself asking "why do we need two different files for list-deps and run"? Removing the BuildConfig and altering the usage of the DistributionTemplate in llama stack list-deps is the first step in removing the build yaml entirely. Removing the BuildConfig and build.yaml cuts the files users need to maintain in half, and allows us to focus on the stability of _just_ the run.yaml This PR removes the build.yaml, BuildConfig datatype, and its usage throughout the codebase. Users are now expected to point to run.yaml files when running list-deps, and our codebase automatically uses these types now for things like `get_provider_registry`. **Additionally, two renames: `StackRunConfig` -> `StackConfig` and `run.yaml` -> `config.yaml`.** The build.yaml made sense for when we were managing the build process for the user and actually _producing_ a run.yaml _from_ the build.yaml, but now that we are simply just getting the provider registry and listing the deps, switching to config.yaml simplifies the scope here greatly. ## Test Plan existing list-deps usage should work in the tests. --------- Signed-off-by: Charlie Doern <cdoern@redhat.com>	2025-12-10 10:12:12 +01:00
Charlie Doern	91f1b352b4	chore: add storage sane defaults (#4182) ... # What does this PR do? since `StackRunConfig` requires certain parts of `StorageConfig`, it'd probably make sense to template in some defaults that will "just work" for most usecases specifically introduce`ServerStoresConfig` defaults for inference, metadata, conversations and prompts. We already actually funnel in defaults for these sections ad-hoc throughout the codebase additionally set some `backends` defaults for the `StorageConfig`. This will alleviate some weirdness for `--providers` for run/list-deps and also some work I have to better align our list-deps/run datatypes --------- Signed-off-by: Charlie Doern <cdoern@redhat.com>	2025-11-18 15:22:26 -08:00
Charlie Doern	a078f089d9	fix: rename llama_stack_api dir (#4155) ... # What does this PR do? the directory structure was src/llama-stack-api/llama_stack_api instead it should just be src/llama_stack_api to match the other packages. update the structure and pyproject/linting config --------- Signed-off-by: Charlie Doern <cdoern@redhat.com> Co-authored-by: Ashwin Bharambe <ashwin.bharambe@gmail.com>	2025-11-13 15:04:36 -08:00
Francisco Arceo	a82b79ce57	fix: Error out when creating vector store with unknown embedding model (#4154) ... # What does this PR do? Error out when creating vector store with unknown embedding model Closes https://github.com/llamastack/llama-stack/issues/4047 ## Test Plan Added tests Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>	2025-11-13 13:43:31 -08:00
Charlie Doern	840ad75fe9	feat: split API and provider specs into separate llama-stack-api pkg (#3895) ... # What does this PR do? Extract API definitions and provider specifications into a standalone llama-stack-api package that can be published to PyPI independently of the main llama-stack server. see: https://github.com/llamastack/llama-stack/pull/2978 and https://github.com/llamastack/llama-stack/pull/2978#issuecomment-3145115942 Motivation External providers currently import from llama-stack, which overrides the installed version and causes dependency conflicts. This separation allows external providers to: - Install only the type definitions they need without server dependencies - Avoid version conflicts with the installed llama-stack package - Be versioned and released independently This enables us to re-enable external provider module tests that were previously blocked by these import conflicts. Changes - Created llama-stack-api package with minimal dependencies (pydantic, jsonschema) - Moved APIs, providers datatypes, strong_typing, and schema_utils - Updated all imports from llama_stack.* to llama_stack_api.* - Configured local editable install for development workflow - Updated linting and type-checking configuration for both packages Next Steps - Publish llama-stack-api to PyPI - Update external provider dependencies - Re-enable external provider module tests Pre-cursor PRs to this one: - #4093 - #3954 - #4064 These PRs moved key pieces _out_ of the Api pkg, limiting the scope of change here. relates to #3237 ## Test Plan Package builds successfully and can be imported independently. All pre-commit hooks pass with expected exclusions maintained. --------- Signed-off-by: Charlie Doern <cdoern@redhat.com>	2025-11-13 11:51:17 -08:00
Francisco Arceo	eb3f9ac278	feat: allow returning embeddings and metadata from /vector_stores/ methods; disallow changing Provider ID (#4046) ... # What does this PR do? - Updates `/vector_stores/{vector_store_id}/files/{file_id}/content` to allow returning `embeddings` and `metadata` using the `extra_query` - Updates the UI accordingly to display them. - Update UI to support CRUD operations in the Vector Stores section and adds a new modal exposing the functionality. - Updates Vector Store update to fail if a user tries to update Provider ID (which doesn't make sense to allow) ```python In [1]: client.vector_stores.files.content( vector_store_id=vector_store.id, file_id=file.id, extra_query={"include_embeddings": True, "include_metadata": True} ) Out [1]: FileContentResponse(attributes={}, content=[Content(text='This is a test document to check if embeddings are generated properly.\n', type='text', embedding=[0.33760684728622437, ...,], chunk_metadata={'chunk_id': '62a63ae0-c202-f060-1b86-0a688995b8d3', 'document_id': 'file-27291dbc679642ac94ffac6d2810c339', 'source': None, 'created_timestamp': 1762053437, 'updated_timestamp': 1762053437, 'chunk_window': '0-13', 'chunk_tokenizer': 'DEFAULT_TIKTOKEN_TOKENIZER', 'chunk_embedding_model': 'sentence-transformers/nomic -ai/nomic-embed-text-v1.5', 'chunk_embedding_dimension': 768, 'content_token_count': 13, 'metadata_token_count': 9}, metadata={'filename': 'test-embedding.txt', 'chunk_id': '62a63ae0-c202-f060-1b86-0a688995b8d3', 'document_id': 'file-27291dbc679642ac94ffac6d2810c339', 'token_count': 13, 'metadata_token_count': 9})], file_id='file-27291dbc679642ac94ffac6d2810c339', filename='test-embedding.txt') ``` Screenshots of UI are displayed below: ### List Vector Store with Added "Create New Vector Store" <img width="1912" height="491" alt="Screenshot 2025-11-06 at 10 47 25 PM" src="https://github.com/user-attachments/assets/a3a3ddd9-758d-4005-ac9c-5047f03916f3" /> ### Create New Vector Store <img width="1918" height="1048" alt="Screenshot 2025-11-06 at 10 47 49 PM" src="https://github.com/user-attachments/assets/b4dc0d31-696f-4e68-b109-27915090f158" /> ### Edit Vector Store <img width="1916" height="1355" alt="Screenshot 2025-11-06 at 10 48 32 PM" src="https://github.com/user-attachments/assets/ec879c63-4cf7-489f-bb1e-57ccc7931414" /> ### Vector Store Files Contents page (with Embeddings) <img width="1914" height="849" alt="Screenshot 2025-11-06 at 11 54 32 PM" src="https://github.com/user-attachments/assets/3095520d-0e90-41f7-83bd-652f6c3fbf27" /> ### Vector Store Files Contents Details page (with Embeddings) <img width="1916" height="1221" alt="Screenshot 2025-11-06 at 11 55 00 PM" src="https://github.com/user-attachments/assets/e71dbdc5-5b49-472b-a43a-5785f58d196c" /> <!-- If resolving an issue, uncomment and update the line below --> <!-- Closes #[issue-number] --> ## Test Plan Tests added for Middleware extension and Provider failures. --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>	2025-11-12 09:59:48 -08:00
Ashwin Bharambe	4e6c769cc4	fix(context): prevent provider data leak between streaming requests (#3924) ... ## Summary - `preserve_contexts_async_generator` left `PROVIDER_DATA_VAR` (and other context vars) populated after a streaming generator completed on HEAD~1, so the asyncio context for request N+1 started with request N's provider payload. - FastAPI dependencies and middleware execute before `request_provider_data_context` rebinds the header data, meaning auth/logging hooks could observe a prior tenant's credentials or treat them as authenticated. Traces and any background work that inspects the context outside the `with` block leak as well—this is a real security regression, not just a CLI artifact. - The wrapper now restores each tracked `ContextVar` to the value it held before the iteration (falling back to clearing when necessary) after every yield and when the generator terminates, so provider data is wiped while callers that set their own defaults keep them. ## Test Plan - `uv run pytest tests/unit/core/test_provider_data_context.py -q` - `uv run pytest tests/unit/distribution/test_context.py -q` Both suites fail on HEAD~1 and pass with this change.	2025-10-27 23:01:12 -07:00
ehhuang	9916cb3b17	chore: support default model in moderations API (#3890) ... # What does this PR do? https://platform.openai.com/docs/api-reference/moderations supports optional model parameter. This PR adds support for using moderations API with model=None if a default shield id is provided via safety config. ## Test Plan added tests manual test: ``` > SAFETY_MODEL='together/meta-llama/Llama-Guard-4-12B' uv run llama stack run starter > curl http://localhost:8321/v1/moderations \ -H "Content-Type: application/json" \ -d '{ "input": [ "hello" ] }' ```	2025-10-23 16:03:53 -07:00
Ashwin Bharambe	122de785c4	chore(cleanup)!: kill vector_db references as far as possible (#3864) ... There should not be "vector db" anywhere.	2025-10-20 20:06:16 -07:00
Francisco Arceo	48581bf651	chore: Updating how default embedding model is set in stack (#3818) ... # What does this PR do? Refactor setting default vector store provider and embedding model to use an optional `vector_stores` config in the `StackRunConfig` and clean up code to do so (had to add back in some pieces of VectorDB). Also added remote Qdrant and Weaviate to starter distro (based on other PR where inference providers were added for UX). New config is simply (default for Starter distro): ```yaml vector_stores: default_provider_id: faiss default_embedding_model: provider_id: sentence-transformers model_id: nomic-ai/nomic-embed-text-v1.5 ``` ## Test Plan CI and Unit tests. --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com> Co-authored-by: Ashwin Bharambe <ashwin.bharambe@gmail.com>	2025-10-20 14:22:45 -07:00
Ashwin Bharambe	2c43285e22	feat(stores)!: use backend storage references instead of configs (#3697) ... **This PR changes configurations in a backward incompatible way.** Run configs today repeat full SQLite/Postgres snippets everywhere a store is needed, which means duplicated credentials, extra connection pools, and lots of drift between files. This PR introduces named storage backends so the stack and providers can share a single catalog and reference those backends by name. ## Key Changes - Add `storage.backends` to `StackRunConfig`, register each KV/SQL backend once at startup, and validate that references point to the right family. - Move server stores under `storage.stores` with lightweight references (backend + namespace/table) instead of full configs. - Update every provider/config/doc to use the new reference style; docs/codegen now surface the simplified YAML. ## Migration Before: ```yaml metadata_store: type: sqlite db_path: ~/.llama/distributions/foo/registry.db inference_store: type: postgres host: ${env.POSTGRES_HOST} port: ${env.POSTGRES_PORT} db: ${env.POSTGRES_DB} user: ${env.POSTGRES_USER} password: ${env.POSTGRES_PASSWORD} conversations_store: type: postgres host: ${env.POSTGRES_HOST} port: ${env.POSTGRES_PORT} db: ${env.POSTGRES_DB} user: ${env.POSTGRES_USER} password: ${env.POSTGRES_PASSWORD} ``` After: ```yaml storage: backends: kv_default: type: kv_sqlite db_path: ~/.llama/distributions/foo/kvstore.db sql_default: type: sql_postgres host: ${env.POSTGRES_HOST} port: ${env.POSTGRES_PORT} db: ${env.POSTGRES_DB} user: ${env.POSTGRES_USER} password: ${env.POSTGRES_PASSWORD} stores: metadata: backend: kv_default namespace: registry inference: backend: sql_default table_name: inference_store max_write_queue_size: 10000 num_writers: 4 conversations: backend: sql_default table_name: openai_conversations ``` Provider configs follow the same pattern—for example, a Chroma vector adapter switches from: ```yaml providers: vector_io: - provider_id: chromadb provider_type: remote::chromadb config: url: ${env.CHROMADB_URL} kvstore: type: sqlite db_path: ~/.llama/distributions/foo/chroma.db ``` to: ```yaml providers: vector_io: - provider_id: chromadb provider_type: remote::chromadb config: url: ${env.CHROMADB_URL} persistence: backend: kv_default namespace: vector_io::chroma_remote ``` Once the backends are declared, everything else just points at them, so rotating credentials or swapping to Postgres happens in one place and the stack reuses a single connection pool.	2025-10-20 13:20:09 -07:00
Francisco Arceo	ef4bc70bbe	feat: Enable setting a default embedding model in the stack (#3803) ... # What does this PR do? Enables automatic embedding model detection for vector stores and by using a `default_configured` boolean that can be defined in the `run.yaml`. <!-- If resolving an issue, uncomment and update the line below --> <!-- Closes #[issue-number] --> ## Test Plan - Unit tests - Integration tests - Simple example below: Spin up the stack: ```bash uv run llama stack build --distro starter --image-type venv --run ``` Then test with OpenAI's client: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8321/v1/", api_key="none") vs = client.vector_stores.create() ``` Previously you needed: ```python vs = client.vector_stores.create( extra_body={ "embedding_model": "sentence-transformers/all-MiniLM-L6-v2", "embedding_dimension": 384, } ) ``` The `extra_body` is now unnecessary. --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>	2025-10-14 18:25:13 -07:00
Francisco Arceo	968c364a3e	chore: Auto-detect Provider ID when only 1 Vector Store Provider avai… (#3802) ... # What does this PR do? 2 main changes: 1. Remove `provider_id` requirement in call to vector stores and 2. Removes "register first embedding model" logic - Now forces embedding model id as required on Vector Store creation Simplifies the UX for OpenAI to: ```python vs = client.vector_stores.create( name="my_citations_db", extra_body={ "embedding_model": "ollama/nomic-embed-text:latest", } ) ``` <!-- If resolving an issue, uncomment and update the line below --> <!-- Closes #[issue-number] --> ## Test Plan <!-- Describe the tests you ran to verify your changes with result summaries. *Provide clear instructions so the plan can be easily re-executed.* --> --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>	2025-10-13 10:25:36 -07:00