mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-10-21 08:03:09 +00:00
b11bcfde11
Some checks failed
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 0s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 0s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 3s
Test Llama Stack Build / generate-matrix (push) Successful in 22s
Test llama stack list-deps / show-single-provider (push) Failing after 53s
Test Llama Stack Build / build-single-provider (push) Failing after 3s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Python Package Build Test / build (3.12) (push) Failing after 18s
Python Package Build Test / build (3.13) (push) Failing after 24s
Test Llama Stack Build / build-ubi9-container-distribution (push) Failing after 26s
Test Llama Stack Build / build-custom-container-distribution (push) Failing after 27s
Unit Tests / unit-tests (3.12) (push) Failing after 26s
Vector IO Integration Tests / test-matrix (push) Failing after 44s
API Conformance Tests / check-schema-compatibility (push) Successful in 52s
Test llama stack list-deps / generate-matrix (push) Successful in 52s
Test Llama Stack Build / build (push) Failing after 29s
Test External API and Providers / test-external (venv) (push) Failing after 53s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1m2s
Unit Tests / unit-tests (3.13) (push) Failing after 1m30s
Test llama stack list-deps / list-deps-from-config (push) Failing after 1m59s
Test llama stack list-deps / list-deps (push) Failing after 1m10s
UI Tests / ui-tests (22) (push) Successful in 2m26s
Pre-commit / pre-commit (push) Successful in 3m8s
# What does this PR do? This PR does a few things outlined in #2878 namely: 1. adds `llama stack list-deps` a command which simply takes the build logic and instead of executing one of the `build_...` scripts, it displays all of the providers' dependencies using the `module` and `uv`. 2. deprecated `llama stack build` in favor of `llama stack list-deps` 3. updates all tests to use `list-deps` alongside `build`. PR 2/2 will migrate `llama stack run`'s default behavior to be `llama stack build --run` and use the new `list-deps` command under the hood before running the server. examples of `llama stack list-deps starter` ``` llama stack list-deps starter --format json { "name": "starter", "description": "Quick start template for running Llama Stack with several popular providers. This distribution is intended for CPU-only environments.", "apis": [ { "api": "inference", "provider": "remote::cerebras" }, { "api": "inference", "provider": "remote::ollama" }, { "api": "inference", "provider": "remote::vllm" }, { "api": "inference", "provider": "remote::tgi" }, { "api": "inference", "provider": "remote::fireworks" }, { "api": "inference", "provider": "remote::together" }, { "api": "inference", "provider": "remote::bedrock" }, { "api": "inference", "provider": "remote::nvidia" }, { "api": "inference", "provider": "remote::openai" }, { "api": "inference", "provider": "remote::anthropic" }, { "api": "inference", "provider": "remote::gemini" }, { "api": "inference", "provider": "remote::vertexai" }, { "api": "inference", "provider": "remote::groq" }, { "api": "inference", "provider": "remote::sambanova" }, { "api": "inference", "provider": "remote::azure" }, { "api": "inference", "provider": "inline::sentence-transformers" }, { "api": "vector_io", "provider": "inline::faiss" }, { "api": "vector_io", "provider": "inline::sqlite-vec" }, { "api": "vector_io", "provider": "inline::milvus" }, { "api": "vector_io", "provider": "remote::chromadb" }, { "api": "vector_io", "provider": "remote::pgvector" }, { "api": "files", "provider": "inline::localfs" }, { "api": "safety", "provider": "inline::llama-guard" }, { "api": "safety", "provider": "inline::code-scanner" }, { "api": "agents", "provider": "inline::meta-reference" }, { "api": "telemetry", "provider": "inline::meta-reference" }, { "api": "post_training", "provider": "inline::torchtune-cpu" }, { "api": "eval", "provider": "inline::meta-reference" }, { "api": "datasetio", "provider": "remote::huggingface" }, { "api": "datasetio", "provider": "inline::localfs" }, { "api": "scoring", "provider": "inline::basic" }, { "api": "scoring", "provider": "inline::llm-as-judge" }, { "api": "scoring", "provider": "inline::braintrust" }, { "api": "tool_runtime", "provider": "remote::brave-search" }, { "api": "tool_runtime", "provider": "remote::tavily-search" }, { "api": "tool_runtime", "provider": "inline::rag-runtime" }, { "api": "tool_runtime", "provider": "remote::model-context-protocol" }, { "api": "batches", "provider": "inline::reference" } ], "pip_dependencies": [ "pandas", "opentelemetry-exporter-otlp-proto-http", "matplotlib", "opentelemetry-sdk", "sentence-transformers", "datasets", "pymilvus[milvus-lite]>=2.4.10", "codeshield", "scipy", "torchvision", "tree_sitter", "h11>=0.16.0", "aiohttp", "pymongo", "tqdm", "pythainlp", "pillow", "torch", "emoji", "grpcio>=1.67.1,<1.71.0", "fireworks-ai", "langdetect", "psycopg2-binary", "asyncpg", "redis", "together", "torchao>=0.12.0", "openai", "sentencepiece", "aiosqlite", "google-cloud-aiplatform", "faiss-cpu", "numpy", "sqlite-vec", "nltk", "scikit-learn", "mcp>=1.8.1", "transformers", "boto3", "huggingface_hub", "ollama", "autoevals", "sqlalchemy[asyncio]", "torchtune>=0.5.0", "chromadb-client", "pypdf", "requests", "anthropic", "chardet", "aiosqlite", "fastapi", "fire", "httpx", "uvicorn", "opentelemetry-sdk", "opentelemetry-exporter-otlp-proto-http" ] } ``` <img width="1500" height="420" alt="Screenshot 2025-10-16 at 5 53 03 PM" src="https://github.com/user-attachments/assets/765929fb-93e2-44d7-9c3d-8918b70fc721" /> --------- Signed-off-by: Charlie Doern <cdoern@redhat.com>
210 lines
13 KiB
Markdown
210 lines
13 KiB
Markdown
# Llama Stack
|
|
|
|
[](https://pypi.org/project/llama_stack/)
|
|
[](https://pypi.org/project/llama-stack/)
|
|
[](https://github.com/meta-llama/llama-stack/blob/main/LICENSE)
|
|
[](https://discord.gg/llama-stack)
|
|
[](https://github.com/meta-llama/llama-stack/actions/workflows/unit-tests.yml?query=branch%3Amain)
|
|
[](https://github.com/meta-llama/llama-stack/actions/workflows/integration-tests.yml?query=branch%3Amain)
|
|
|
|
[**Quick Start**](https://llamastack.github.io/docs/getting_started/quickstart) | [**Documentation**](https://llamastack.github.io/docs) | [**Colab Notebook**](./docs/getting_started.ipynb) | [**Discord**](https://discord.gg/llama-stack)
|
|
|
|
|
|
### ✨🎉 Llama 4 Support 🎉✨
|
|
We released [Version 0.2.0](https://github.com/meta-llama/llama-stack/releases/tag/v0.2.0) with support for the Llama 4 herd of models released by Meta.
|
|
|
|
<details>
|
|
|
|
<summary>👋 Click here to see how to run Llama 4 models on Llama Stack </summary>
|
|
|
|
\
|
|
*Note you need 8xH100 GPU-host to run these models*
|
|
|
|
```bash
|
|
pip install -U llama_stack
|
|
|
|
MODEL="Llama-4-Scout-17B-16E-Instruct"
|
|
# get meta url from llama.com
|
|
huggingface-cli download meta-llama/$MODEL --local-dir ~/.llama/$MODEL
|
|
|
|
# install dependencies for the distribution
|
|
llama stack list-deps meta-reference-gpu | xargs -L1 uv pip install
|
|
|
|
# start a llama stack server
|
|
INFERENCE_MODEL=meta-llama/$MODEL llama stack run meta-reference-gpu
|
|
|
|
# install client to interact with the server
|
|
pip install llama-stack-client
|
|
```
|
|
### CLI
|
|
```bash
|
|
# Run a chat completion
|
|
MODEL="Llama-4-Scout-17B-16E-Instruct"
|
|
|
|
llama-stack-client --endpoint http://localhost:8321 \
|
|
inference chat-completion \
|
|
--model-id meta-llama/$MODEL \
|
|
--message "write a haiku for meta's llama 4 models"
|
|
|
|
OpenAIChatCompletion(
|
|
...
|
|
choices=[
|
|
OpenAIChatCompletionChoice(
|
|
finish_reason='stop',
|
|
index=0,
|
|
message=OpenAIChatCompletionChoiceMessageOpenAIAssistantMessageParam(
|
|
role='assistant',
|
|
content='...**Silent minds awaken,** \n**Whispers of billions of words,** \n**Reasoning breaks the night.** \n\n— \n*This haiku blends the essence of LLaMA 4\'s capabilities with nature-inspired metaphor, evoking its vast training data and transformative potential.*',
|
|
...
|
|
),
|
|
...
|
|
)
|
|
],
|
|
...
|
|
)
|
|
```
|
|
### Python SDK
|
|
```python
|
|
from llama_stack_client import LlamaStackClient
|
|
|
|
client = LlamaStackClient(base_url=f"http://localhost:8321")
|
|
|
|
model_id = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
|
|
prompt = "Write a haiku about coding"
|
|
|
|
print(f"User> {prompt}")
|
|
response = client.chat.completions.create(
|
|
model=model_id,
|
|
messages=[
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": prompt},
|
|
],
|
|
)
|
|
print(f"Assistant> {response.choices[0].message.content}")
|
|
```
|
|
As more providers start supporting Llama 4, you can use them in Llama Stack as well. We are adding to the list. Stay tuned!
|
|
|
|
|
|
</details>
|
|
|
|
### 🚀 One-Line Installer 🚀
|
|
|
|
To try Llama Stack locally, run:
|
|
|
|
```bash
|
|
curl -LsSf https://github.com/meta-llama/llama-stack/raw/main/scripts/install.sh | bash
|
|
```
|
|
|
|
### Overview
|
|
|
|
Llama Stack standardizes the core building blocks that simplify AI application development. It codifies best practices across the Llama ecosystem. More specifically, it provides
|
|
|
|
- **Unified API layer** for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.
|
|
- **Plugin architecture** to support the rich ecosystem of different API implementations in various environments, including local development, on-premises, cloud, and mobile.
|
|
- **Prepackaged verified distributions** which offer a one-stop solution for developers to get started quickly and reliably in any environment.
|
|
- **Multiple developer interfaces** like CLI and SDKs for Python, Typescript, iOS, and Android.
|
|
- **Standalone applications** as examples for how to build production-grade AI applications with Llama Stack.
|
|
|
|
<div style="text-align: center;">
|
|
<img
|
|
src="https://github.com/user-attachments/assets/33d9576d-95ea-468d-95e2-8fa233205a50"
|
|
width="480"
|
|
title="Llama Stack"
|
|
alt="Llama Stack"
|
|
/>
|
|
</div>
|
|
|
|
### Llama Stack Benefits
|
|
- **Flexible Options**: Developers can choose their preferred infrastructure without changing APIs and enjoy flexible deployment choices.
|
|
- **Consistent Experience**: With its unified APIs, Llama Stack makes it easier to build, test, and deploy AI applications with consistent application behavior.
|
|
- **Robust Ecosystem**: Llama Stack is already integrated with distribution partners (cloud providers, hardware vendors, and AI-focused companies) that offer tailored infrastructure, software, and services for deploying Llama models.
|
|
|
|
By reducing friction and complexity, Llama Stack empowers developers to focus on what they do best: building transformative generative AI applications.
|
|
|
|
### API Providers
|
|
Here is a list of the various API providers and available distributions that can help developers get started easily with Llama Stack.
|
|
Please checkout for [full list](https://llamastack.github.io/docs/providers)
|
|
|
|
| API Provider Builder | Environments | Agents | Inference | VectorIO | Safety | Telemetry | Post Training | Eval | DatasetIO |
|
|
|:--------------------:|:------------:|:------:|:---------:|:--------:|:------:|:---------:|:-------------:|:----:|:--------:|
|
|
| Meta Reference | Single Node | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
|
| SambaNova | Hosted | | ✅ | | ✅ | | | | |
|
|
| Cerebras | Hosted | | ✅ | | | | | | |
|
|
| Fireworks | Hosted | ✅ | ✅ | ✅ | | | | | |
|
|
| AWS Bedrock | Hosted | | ✅ | | ✅ | | | | |
|
|
| Together | Hosted | ✅ | ✅ | | ✅ | | | | |
|
|
| Groq | Hosted | | ✅ | | | | | | |
|
|
| Ollama | Single Node | | ✅ | | | | | | |
|
|
| TGI | Hosted/Single Node | | ✅ | | | | | | |
|
|
| NVIDIA NIM | Hosted/Single Node | | ✅ | | ✅ | | | | |
|
|
| ChromaDB | Hosted/Single Node | | | ✅ | | | | | |
|
|
| Milvus | Hosted/Single Node | | | ✅ | | | | | |
|
|
| Qdrant | Hosted/Single Node | | | ✅ | | | | | |
|
|
| Weaviate | Hosted/Single Node | | | ✅ | | | | | |
|
|
| SQLite-vec | Single Node | | | ✅ | | | | | |
|
|
| PG Vector | Single Node | | | ✅ | | | | | |
|
|
| PyTorch ExecuTorch | On-device iOS | ✅ | ✅ | | | | | | |
|
|
| vLLM | Single Node | | ✅ | | | | | | |
|
|
| OpenAI | Hosted | | ✅ | | | | | | |
|
|
| Anthropic | Hosted | | ✅ | | | | | | |
|
|
| Gemini | Hosted | | ✅ | | | | | | |
|
|
| WatsonX | Hosted | | ✅ | | | | | | |
|
|
| HuggingFace | Single Node | | | | | | ✅ | | ✅ |
|
|
| TorchTune | Single Node | | | | | | ✅ | | |
|
|
| NVIDIA NEMO | Hosted | | ✅ | ✅ | | | ✅ | ✅ | ✅ |
|
|
| NVIDIA | Hosted | | | | | | ✅ | ✅ | ✅ |
|
|
|
|
> **Note**: Additional providers are available through external packages. See [External Providers](https://llamastack.github.io/docs/providers/external) documentation.
|
|
|
|
### Distributions
|
|
|
|
A Llama Stack Distribution (or "distro") is a pre-configured bundle of provider implementations for each API component. Distributions make it easy to get started with a specific deployment scenario - you can begin with a local development setup (eg. ollama) and seamlessly transition to production (eg. Fireworks) without changing your application code.
|
|
Here are some of the distributions we support:
|
|
|
|
| **Distribution** | **Llama Stack Docker** | Start This Distribution |
|
|
|:---------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------:|
|
|
| Starter Distribution | [llamastack/distribution-starter](https://hub.docker.com/repository/docker/llamastack/distribution-starter/general) | [Guide](https://llamastack.github.io/latest/distributions/self_hosted_distro/starter.html) |
|
|
| Meta Reference | [llamastack/distribution-meta-reference-gpu](https://hub.docker.com/repository/docker/llamastack/distribution-meta-reference-gpu/general) | [Guide](https://llamastack.github.io/latest/distributions/self_hosted_distro/meta-reference-gpu.html) |
|
|
| PostgreSQL | [llamastack/distribution-postgres-demo](https://hub.docker.com/repository/docker/llamastack/distribution-postgres-demo/general) | |
|
|
|
|
### Documentation
|
|
|
|
Please checkout our [Documentation](https://llamastack.github.io/latest/index.html) page for more details.
|
|
|
|
* CLI references
|
|
* [llama (server-side) CLI Reference](https://llamastack.github.io/latest/references/llama_cli_reference/index.html): Guide for using the `llama` CLI to work with Llama models (download, study prompts), and building/starting a Llama Stack distribution.
|
|
* [llama (client-side) CLI Reference](https://llamastack.github.io/latest/references/llama_stack_client_cli_reference.html): Guide for using the `llama-stack-client` CLI, which allows you to query information about the distribution.
|
|
* Getting Started
|
|
* [Quick guide to start a Llama Stack server](https://llamastack.github.io/latest/getting_started/index.html).
|
|
* [Jupyter notebook](./docs/getting_started.ipynb) to walk-through how to use simple text and vision inference llama_stack_client APIs
|
|
* The complete Llama Stack lesson [Colab notebook](https://colab.research.google.com/drive/1dtVmxotBsI4cGZQNsJRYPrLiDeT0Wnwt) of the new [Llama 3.2 course on Deeplearning.ai](https://learn.deeplearning.ai/courses/introducing-multimodal-llama-3-2/lesson/8/llama-stack).
|
|
* A [Zero-to-Hero Guide](https://github.com/meta-llama/llama-stack/tree/main/docs/zero_to_hero_guide) that guide you through all the key components of llama stack with code samples.
|
|
* [Contributing](CONTRIBUTING.md)
|
|
* [Adding a new API Provider](https://llamastack.github.io/latest/contributing/new_api_provider.html) to walk-through how to add a new API provider.
|
|
|
|
### Llama Stack Client SDKs
|
|
|
|
| **Language** | **Client SDK** | **Package** |
|
|
| :----: | :----: | :----: |
|
|
| Python | [llama-stack-client-python](https://github.com/meta-llama/llama-stack-client-python) | [](https://pypi.org/project/llama_stack_client/)
|
|
| Swift | [llama-stack-client-swift](https://github.com/meta-llama/llama-stack-client-swift) | [](https://swiftpackageindex.com/meta-llama/llama-stack-client-swift)
|
|
| Typescript | [llama-stack-client-typescript](https://github.com/meta-llama/llama-stack-client-typescript) | [](https://npmjs.org/package/llama-stack-client)
|
|
| Kotlin | [llama-stack-client-kotlin](https://github.com/meta-llama/llama-stack-client-kotlin) | [](https://central.sonatype.com/artifact/com.llama.llamastack/llama-stack-client-kotlin)
|
|
|
|
Check out our client SDKs for connecting to a Llama Stack server in your preferred language, you can choose from [python](https://github.com/meta-llama/llama-stack-client-python), [typescript](https://github.com/meta-llama/llama-stack-client-typescript), [swift](https://github.com/meta-llama/llama-stack-client-swift), and [kotlin](https://github.com/meta-llama/llama-stack-client-kotlin) programming languages to quickly build your applications.
|
|
|
|
You can find more example scripts with client SDKs to talk with the Llama Stack server in our [llama-stack-apps](https://github.com/meta-llama/llama-stack-apps/tree/main/examples) repo.
|
|
|
|
|
|
## 🌟 GitHub Star History
|
|
## Star History
|
|
|
|
[](https://www.star-history.com/#meta-llama/llama-stack&Date)
|
|
|
|
## ✨ Contributors
|
|
|
|
Thanks to all of our amazing contributors!
|
|
|
|
<a href="https://github.com/meta-llama/llama-stack/graphs/contributors">
|
|
<img src="https://contrib.rocks/image?repo=meta-llama/llama-stack" />
|
|
</a>
|