mirror of https://github.com/meta-llama/llama-stack.git synced 2025-12-12 12:06:04 +00:00

Composable building blocks to build Llama Apps https://llama-stack.readthedocs.io

Find a file

Sébastien Han dd513449de Some checks failed Unit Tests / unit-tests (3.13) (push) Failing after 2m9s Details Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 2s Details SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 1s Details Integration Tests (Replay) / generate-matrix (push) Successful in 5s Details Vector IO Integration Tests / test-matrix (push) Failing after 51s Details SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 53s Details Unit Tests / unit-tests (3.12) (push) Failing after 2m20s Details Pre-commit / pre-commit (push) Successful in 2m46s Details Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 2m51s Details fix: InferenceStore workers being cancelled on event loop change (#4373 ) # What does this PR do? The InferenceStore write queue workers were created during initialize() which runs in the app factory's event loop. When uvicorn starts, it creates its own event loop, causing the original worker tasks to be cancelled. This resulted in chat completions being queued but never written to the database. Root cause: asyncio tasks are bound to the event loop they were created in. When the event loop changes (app factory -> uvicorn server), tasks from the old loop are cancelled. However, the task references remained in self._worker_tasks, so _ensure_workers_started() thought workers were still running. The fix removes eager worker creation from initialize(). Workers must not be started during app factory execution because that event loop is temporary. Instead, _ensure_workers_started() creates workers lazily during store_chat_completion(), which runs during request handling and is guaranteed to be in uvicorn's event loop. Additionally, _ensure_workers_started() now checks for active (non-done) tasks rather than just checking if the list is non-empty. This makes the code resilient to any future event loop transitions. This follows the standard async Python pattern: defer background task creation until the server is actually running, not during app init. ## Test Plan Run the server with postgres or sqlite, I tested with postgres: ``` uv run llama stack run llama_stack/distributions/starter/run-with-postgres-store.yaml ``` Execute an inference call: ``` export INFERENCE_MODEL=gpt-4o-mini && curl -fsS http://127.0.0.1:8321/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"openai/$INFERENCE_MODEL\",\"messages\": [{\"role\": \"user\", \"content\": \"What color is grass?\"}], \"max_tokens\": 128, \"temperature\": 0.0}" ``` Check the DB being populated: ``` podman exec postgres psql -U llamastack -d llamastack -t -c "SELECT COUNT() FROM chat_completions;" 1 ``` AGAINST STABLE BRANCH 0.3.X SINCE MAIN DOES NOT HAVE THE ISSUE.* --------- Signed-off-by: Sébastien Han <seb@redhat.com>		2025-12-11 11:32:04 -05:00
.github	fix: harden storage semantics (backport #4118 ) (#4138 )	2025-11-12 13:01:21 -08:00
benchmarking/k8s-benchmark	feat(stores)!: use backend storage references instead of configs (#3697 )	2025-10-20 13:20:09 -07:00
client-sdks/stainless	revert: "chore(cleanup)!: remove tool_runtime.rag_tool" (#3877 )	2025-10-21 11:22:06 -07:00
containers	fix(ci): unset empty UV index env vars to prevent uv errors (#4013 )	2025-10-31 13:45:47 -07:00
docs	fix(docs): fix glob vulnerability (backport #4193 ) (#4227 )	2025-11-24 11:31:15 -08:00
llama_stack	fix: InferenceStore workers being cancelled on event loop change (#4373 )	2025-12-11 11:32:04 -05:00
scripts	chore(ci): remove unused recordings (backport #4074 ) (#4141 )	2025-11-12 12:36:28 -08:00
tests	fix(inference): respect table_name config in InferenceStore (backport #4371 ) (#4372 )	2025-12-11 15:07:35 +01:00
.coveragerc	test: Measure and track code coverage (#2636 )	2025-07-18 18:08:36 +02:00
.dockerignore	chore: use dockerfile for building containers (#3839 )	2025-10-20 10:23:01 -07:00
.gitattributes	chore: mark recordings as generated files (#3816 )	2025-10-15 11:06:42 -07:00
.gitignore	docs: docusaurus setup (#3541 )	2025-09-24 14:11:30 -07:00
.pre-commit-config.yaml	fix(ci): install client from release branch before uv sync (#4002 )	2025-10-31 11:44:05 -07:00
CHANGELOG.md	docs: Update changelog (#3343 )	2025-09-08 10:01:41 +02:00
CODE_OF_CONDUCT.md	Initial commit	2024-07-23 08:32:33 -07:00
CONTRIBUTING.md	refactor(build): rework CLI commands and build process (1/2) (#2974 )	2025-10-17 19:52:14 -07:00
coverage.svg	test: Measure and track code coverage (#2636 )	2025-07-18 18:08:36 +02:00
LICENSE	Update LICENSE (#47 )	2024-08-29 07:39:50 -07:00
MANIFEST.in	chore: MANIFEST maintenance (#3454 )	2025-09-27 11:28:11 -07:00
pyproject.toml	build: Bump version to 0.3.4	2025-12-03 19:00:01 +00:00
README.md	chore(docs): Remove Llama 4 support details from README (backport #4178 ) (#4323 )	2025-12-08 12:15:55 +01:00
SECURITY.md	Create SECURITY.md	2024-10-08 13:30:40 -04:00
uv.lock	fix: InferenceStore workers being cancelled on event loop change (#4373 )	2025-12-11 11:32:04 -05:00

README.md

Llama Stack

Quick Start | Documentation | Colab Notebook | Discord

🚀 One-Line Installer 🚀

To try Llama Stack locally, run:

curl -LsSf https://github.com/llamastack/llama-stack/raw/main/scripts/install.sh | bash

Overview

Llama Stack standardizes the core building blocks that simplify AI application development. It codifies best practices across the Llama ecosystem. More specifically, it provides

Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.
Plugin architecture to support the rich ecosystem of different API implementations in various environments, including local development, on-premises, cloud, and mobile.
Prepackaged verified distributions which offer a one-stop solution for developers to get started quickly and reliably in any environment.
Multiple developer interfaces like CLI and SDKs for Python, Typescript, iOS, and Android.
Standalone applications as examples for how to build production-grade AI applications with Llama Stack.

Llama Stack Benefits

Flexible Options: Developers can choose their preferred infrastructure without changing APIs and enjoy flexible deployment choices.
Consistent Experience: With its unified APIs, Llama Stack makes it easier to build, test, and deploy AI applications with consistent application behavior.
Robust Ecosystem: Llama Stack is already integrated with distribution partners (cloud providers, hardware vendors, and AI-focused companies) that offer tailored infrastructure, software, and services for deploying Llama models.

By reducing friction and complexity, Llama Stack empowers developers to focus on what they do best: building transformative generative AI applications.

API Providers

Here is a list of the various API providers and available distributions that can help developers get started easily with Llama Stack. Please checkout for full list

API Provider Builder	Environments	Agents	Inference	VectorIO	Safety	Telemetry	Post Training	Eval	DatasetIO
Meta Reference	Single Node	✅	✅	✅	✅	✅	✅	✅	✅
SambaNova	Hosted		✅		✅
Cerebras	Hosted		✅
Fireworks	Hosted	✅	✅	✅
AWS Bedrock	Hosted		✅		✅
Together	Hosted	✅	✅		✅
Groq	Hosted		✅
Ollama	Single Node		✅
TGI	Hosted/Single Node		✅
NVIDIA NIM	Hosted/Single Node		✅		✅
ChromaDB	Hosted/Single Node			✅
Milvus	Hosted/Single Node			✅
Qdrant	Hosted/Single Node			✅
Weaviate	Hosted/Single Node			✅
SQLite-vec	Single Node			✅
PG Vector	Single Node			✅
PyTorch ExecuTorch	On-device iOS	✅	✅
vLLM	Single Node		✅
OpenAI	Hosted		✅
Anthropic	Hosted		✅
Gemini	Hosted		✅
WatsonX	Hosted		✅
HuggingFace	Single Node						✅		✅
TorchTune	Single Node						✅
NVIDIA NEMO	Hosted		✅	✅			✅	✅	✅
NVIDIA	Hosted						✅	✅	✅

Note

: Additional providers are available through external packages. See External Providers documentation.

Distributions

A Llama Stack Distribution (or "distro") is a pre-configured bundle of provider implementations for each API component. Distributions make it easy to get started with a specific deployment scenario - you can begin with a local development setup (eg. ollama) and seamlessly transition to production (eg. Fireworks) without changing your application code. Here are some of the distributions we support:

Distribution	Llama Stack Docker	Start This Distribution
Starter Distribution	llamastack/distribution-starter	Guide
Meta Reference	llamastack/distribution-meta-reference-gpu	Guide
PostgreSQL	llamastack/distribution-postgres-demo

Documentation

Please checkout our Documentation page for more details.

CLI references
- llama (server-side) CLI Reference: Guide for using the llama CLI to work with Llama models (download, study prompts), and building/starting a Llama Stack distribution.
- llama (client-side) CLI Reference: Guide for using the llama-stack-client CLI, which allows you to query information about the distribution.
Getting Started
- Quick guide to start a Llama Stack server.
- Jupyter notebook to walk-through how to use simple text and vision inference llama_stack_client APIs
- The complete Llama Stack lesson Colab notebook of the new Llama 3.2 course on Deeplearning.ai.
- A Zero-to-Hero Guide that guide you through all the key components of llama stack with code samples.
Contributing
- Adding a new API Provider to walk-through how to add a new API provider.

Llama Stack Client SDKs

Language	Client SDK	Package
Python	llama-stack-client-python
Swift	llama-stack-client-swift
Typescript	llama-stack-client-typescript
Kotlin	llama-stack-client-kotlin

Check out our client SDKs for connecting to a Llama Stack server in your preferred language, you can choose from python, typescript, swift, and kotlin programming languages to quickly build your applications.

You can find more example scripts with client SDKs to talk with the Llama Stack server in our llama-stack-apps repo.

🌟 GitHub Star History

Star History

✨ Contributors

Thanks to all of our amazing contributors!