forked from phoenix-oss/llama-stack-mirror

Composable building blocks to build Llama Apps

Find a file

Sébastien Han c91548fe07 build(container): misc improvements (#1291 ) # What does this PR do? See individual commit messages. [//]: # (If resolving an issue, uncomment and update the line below) [//]: # (Closes #[issue-number]) ## Test Plan Apply this diff: ``` diff --git a/llama_stack/templates/ollama/build.yaml b/llama_stack/templates/ollama/build.yaml index da33b8d5..4a702f6f 100644 --- a/llama_stack/templates/ollama/build.yaml +++ b/llama_stack/templates/ollama/build.yaml @@ -28,5 +28,5 @@ distribution_spec: - remote::tavily-search - inline::code-interpreter - inline::rag-runtime - - remote::model-context-protocol + container_image: "registry.access.redhat.com/ubi9" image_type: conda ``` Then run: ``` CONTAINER_BINARY=podman llama stack build --template ollama --image-type container --image-name registry.access.redhat.com/ubi9 Containerfile created successfully in /var/folders/mq/rnm5w_7s2d3fxmtkx02knvhm0000gn/T/tmp.I7E5V6zbVI/Containerfile FROM registry.access.redhat.com/ubi9 WORKDIR /app RUN dnf -y update && dnf install -y iputils net-tools wget vim-minimal python3.11 python3.11-pip python3.11-wheel python3.11-setuptools && ln -s /bin/pip3.11 /bin/pip && ln -s /bin/python3.11 /bin/python && dnf clean all ENV UV_SYSTEM_PYTHON=1 RUN pip install uv RUN uv pip install --no-cache ollama nltk opentelemetry-sdk aiosqlite matplotlib datasets sqlite-vec scipy chromadb-client psycopg2-binary numpy scikit-learn openai redis pandas tqdm blobfile sentencepiece aiohttp requests pillow pymongo transformers autoevals opentelemetry-exporter-otlp-proto-http pypdf chardet aiosqlite fastapi fire httpx uvicorn RUN uv pip install --no-cache llama-stack RUN pip uninstall -y uv ENTRYPOINT ["python", "-m", "llama_stack.distribution.server.server", "--template", "ollama"] # Allows running as non-root user RUN mkdir -p /.llama /.cache RUN chmod -R g+rw /app /.llama /.cache PWD: /Users/leseb/Documents/AI/llama-stack Containerfile: /var/folders/mq/rnm5w_7s2d3fxmtkx02knvhm0000gn/T/tmp.I7E5V6zbVI/Containerfile + podman build --platform linux/arm64 -t distribution-ollama:0.1.4 -f /var/folders/mq/rnm5w_7s2d3fxmtkx02knvhm0000gn/T/tmp.I7E5V6zbVI/Containerfile . --progress=plain STEP 1/11: FROM registry.access.redhat.com/ubi9 STEP 2/11: WORKDIR /app --> Using cache d73dafd4caddd75bc29242a9031258fea759dc571c5bb53a64b5e6d86b3b1335 --> d73dafd4cadd STEP 3/11: RUN dnf -y update && dnf install -y iputils net-tools wget vim-minimal python3.11 python3.11-pip python3.11-wheel python3.11-setuptools && ln -s /bin/pip3.11 /bin/pip && ln -s /bin/python3.11 /bin/python && dnf clean all --> Using cache b74ad682db149771612a3ea1e4796e0760ab8a4e07c26ad672b46a86d38178c2 --> b74ad682db14 STEP 4/11: ENV UV_SYSTEM_PYTHON=1 --> Using cache 0812a05e6576506aa2fe646cbf239d0cb504cac30a50cb5cf4dc88e49039466d --> 0812a05e6576 STEP 5/11: RUN pip install uv --> Using cache a0ce1705f87e52f70f6eb34e66f67b68ebc7c1a073f4d2a664b189cfa89a4e88 --> a0ce1705f87e STEP 6/11: RUN uv pip install --no-cache ollama nltk opentelemetry-sdk aiosqlite matplotlib datasets sqlite-vec scipy chromadb-client psycopg2-binary numpy scikit-learn openai redis pandas tqdm blobfile sentencepiece aiohttp requests pillow pymongo transformers autoevals opentelemetry-exporter-otlp-proto-http pypdf chardet aiosqlite fastapi fire httpx uvicorn Using Python 3.11.9 environment at: /usr Resolved 107 packages in 1.78s Downloading kiwisolver (1.4MiB) Downloading aiohttp (1.6MiB) Downloading grpcio (5.4MiB) Downloading nltk (1.4MiB) Downloading transformers (9.5MiB) Downloading pydantic-core (1.7MiB) Downloading lxml (4.6MiB) Downloading psycopg2-binary (2.7MiB) Downloading scipy (33.8MiB) Downloading scikit-learn (12.0MiB) Downloading tokenizers (2.8MiB) Downloading fonttools (4.6MiB) Downloading pymongo (1.3MiB) Downloading rapidfuzz (1.4MiB) Downloading sentencepiece (1.2MiB) Downloading pyarrow (38.7MiB) Downloading matplotlib (8.1MiB) Downloading pycryptodomex (2.1MiB) Downloading pillow (4.2MiB) Downloading pandas (14.9MiB) Downloading numpy (13.6MiB) Building fire==0.7.0 Downloaded sentencepiece Downloaded kiwisolver Downloaded pymongo Downloaded rapidfuzz Downloaded nltk Downloaded aiohttp Built fire==0.7.0 Downloaded pydantic-core Downloaded pycryptodomex Downloaded psycopg2-binary Downloaded tokenizers Downloaded pillow Downloaded lxml Downloaded fonttools Downloaded grpcio Downloaded matplotlib Downloaded transformers Downloaded scikit-learn Downloaded numpy Downloaded pandas Downloaded scipy Downloaded pyarrow Prepared 107 packages in 3.03s Installed 107 packages in 62ms + aiohappyeyeballs==2.4.6 + aiohttp==3.11.13 + aiosignal==1.3.2 + aiosqlite==0.21.0 + annotated-types==0.7.0 + anyio==4.8.0 + attrs==25.1.0 + autoevals==0.0.120 + backoff==2.2.1 + blobfile==3.0.0 + braintrust-core==0.0.58 + certifi==2025.1.31 + chardet==5.2.0 + charset-normalizer==3.4.1 + chevron==0.14.0 + chromadb-client==0.6.3 + click==8.1.8 + contourpy==1.3.1 + cycler==0.12.1 + datasets==3.3.2 + deprecated==1.2.18 + dill==0.3.8 + distro==1.9.0 + dnspython==2.7.0 + fastapi==0.115.8 + filelock==3.17.0 + fire==0.7.0 + fonttools==4.56.0 + frozenlist==1.5.0 + fsspec==2024.12.0 + googleapis-common-protos==1.68.0 + grpcio==1.70.0 + h11==0.14.0 + httpcore==1.0.7 + httpx==0.28.1 + huggingface-hub==0.29.1 + idna==3.10 + importlib-metadata==8.5.0 + jiter==0.8.2 + joblib==1.4.2 + jsonschema==4.23.0 + jsonschema-specifications==2024.10.1 + kiwisolver==1.4.8 + levenshtein==0.26.1 + lxml==5.3.1 + matplotlib==3.10.0 + monotonic==1.6 + multidict==6.1.0 + multiprocess==0.70.16 + nltk==3.9.1 + numpy==1.26.4 + ollama==0.4.7 + openai==1.64.0 + opentelemetry-api==1.30.0 + opentelemetry-exporter-otlp-proto-common==1.30.0 + opentelemetry-exporter-otlp-proto-grpc==1.30.0 + opentelemetry-exporter-otlp-proto-http==1.30.0 + opentelemetry-proto==1.30.0 + opentelemetry-sdk==1.30.0 + opentelemetry-semantic-conventions==0.51b0 + orjson==3.10.15 + overrides==7.7.0 + packaging==24.2 + pandas==2.2.3 + pillow==11.1.0 + posthog==3.16.0 + propcache==0.3.0 + protobuf==5.29.3 + psycopg2-binary==2.9.10 + pyarrow==19.0.1 + pycryptodomex==3.21.0 + pydantic==2.10.6 + pydantic-core==2.27.2 + pymongo==4.11.1 + pyparsing==3.2.1 + pypdf==5.3.0 + python-dateutil==2.9.0.post0 + pytz==2025.1 + pyyaml==6.0.2 + rapidfuzz==3.12.1 + redis==5.2.1 + referencing==0.36.2 + regex==2024.11.6 + requests==2.32.3 + rpds-py==0.23.1 + safetensors==0.5.3 + scikit-learn==1.6.1 + scipy==1.15.2 + sentencepiece==0.2.0 + six==1.17.0 + sniffio==1.3.1 + sqlite-vec==0.1.6 + starlette==0.45.3 + tenacity==9.0.0 + termcolor==2.5.0 + threadpoolctl==3.5.0 + tokenizers==0.21.0 + tqdm==4.67.1 + transformers==4.49.0 + typing-extensions==4.12.2 + tzdata==2025.1 + urllib3==2.3.0 + uvicorn==0.34.0 + wrapt==1.17.2 + xxhash==3.5.0 + yarl==1.18.3 + zipp==3.21.0 --> 5b5b823605a1 STEP 7/11: RUN uv pip install --no-cache llama-stack Using Python 3.11.9 environment at: /usr Resolved 55 packages in 1.08s Downloading setuptools (1.2MiB) Downloading pygments (1.2MiB) Downloading llama-models (1.5MiB) Downloading tiktoken (1.1MiB) Downloaded tiktoken Downloaded llama-models Downloaded pygments Downloaded setuptools Prepared 15 packages in 402ms Installed 15 packages in 15ms + jinja2==3.1.5 + llama-models==0.1.4 + llama-stack==0.1.4 + llama-stack-client==0.1.4 + markdown-it-py==3.0.0 + markupsafe==3.0.2 + mdurl==0.1.2 + prompt-toolkit==3.0.50 + pyaml==25.1.0 + pygments==2.19.1 + python-dotenv==1.0.1 + rich==13.9.4 + setuptools==75.8.2 + tiktoken==0.9.0 + wcwidth==0.2.13 --> 38a037443807 STEP 8/11: RUN pip uninstall -y uv Found existing installation: uv 0.6.3 Uninstalling uv-0.6.3: Successfully uninstalled uv-0.6.3 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv --> 54f749dc5ece STEP 9/11: ENTRYPOINT ["python", "-m", "llama_stack.distribution.server.server", "--template", "ollama"] --> 481c138b1982 STEP 10/11: RUN mkdir -p /.llama /.cache --> 0fc174f014a8 STEP 11/11: RUN chmod -R g+rw /app /.llama /.cache COMMIT distribution-ollama:0.1.4 --> d41b4ab4b136 Successfully tagged localhost/distribution-ollama:0.1.4 d41b4ab4b1363bfbaf6239e6f313bcb37873ef4b5f2fd816a4ee55acf2ac54d3 + set +x Success! Build Successful! ``` UBI9 container successfully builds. Run the container: ``` podman run d41b4ab4b1363bfbaf6239e6f313bcb37873ef4b5f2fd816a4ee55acf2ac54d3 --env INFERENCE_MODEL=meta-llama/Llama-3.2-3B-Instruct INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:213: Resolved 30 providers INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: inner-inference => ollama INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: models => __routing_table__ INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: inference => __autorouted__ INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: inner-vector_io => sqlite-vec INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: inner-safety => llama-guard INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: shields => __routing_table__ INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: safety => __autorouted__ INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: vector_dbs => __routing_table__ INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: vector_io => __autorouted__ INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: inner-tool_runtime => brave-search INFO 2025-02-27 13:08:03,666 llama_stack.distribution.resolver:215: inner-tool_runtime => tavily-search ``` [//]: # (## Documentation) --------- Signed-off-by: Sébastien Han <seb@redhat.com>		2025-02-28 10:01:52 -08:00
.github	ci: improve GitHub Actions workflow for website builds (#1151 )	2025-02-20 21:37:37 -08:00
distributions	fix: precommits ugh why wont they run correctly because they dont have the right dependencies	2025-02-27 15:02:04 -08:00
docs	docs: Add link to distributions guide in quick start guide (#1326 )	2025-02-28 09:18:02 -08:00
llama_stack	build(container): misc improvements (#1291 )	2025-02-28 10:01:52 -08:00
rfcs	docs: Fix url to the llama-stack-spec yaml/html files (#1081 )	2025-02-13 12:39:26 -08:00
tests/client-sdk	chore(lint): update Ruff ignores for project conventions and maintainability (#1184 )	2025-02-28 09:36:49 -08:00
.gitignore	github: ignore non-hidden python virtual environments (#939 )	2025-02-03 11:53:05 -08:00
.gitmodules	chore: removed executorch submodule (#1265 )	2025-02-25 21:57:21 -08:00
.pre-commit-config.yaml	chore: upgrade uv pre-commit version, uv-sync -> uv-lock (#1284 )	2025-02-26 14:57:48 -08:00
.python-version	build: hint on Python version for uv venv (#1172 )	2025-02-25 10:37:45 -05:00
.readthedocs.yaml	first version of readthedocs (#278 )	2024-10-22 10:15:58 +05:30
CODE_OF_CONDUCT.md	Initial commit	2024-07-23 08:32:33 -07:00
CONTRIBUTING.md	build: Add dotenv file for running tests with uv (#1251 )	2025-02-27 16:42:55 -08:00
LICENSE	Update LICENSE (#47 )	2024-08-29 07:39:50 -07:00
MANIFEST.in	feat: completing text /chat-completion and /completion tests (#1223 )	2025-02-25 11:37:04 -08:00
pyproject.toml	chore(lint): update Ruff ignores for project conventions and maintainability (#1184 )	2025-02-28 09:36:49 -08:00
README.md	docs: Simplify installation guide with `uv` (#1196 )	2025-02-20 21:05:47 -08:00
requirements.txt	fix: pre-commit updates (#1243 )	2025-02-24 17:20:29 -08:00
SECURITY.md	Create SECURITY.md	2024-10-08 13:30:40 -04:00
uv.lock	fix: run uv-sync manually. locally pre-commit is not triggering	2025-02-26 13:54:08 -08:00

README.md

Llama Stack

Quick Start | Documentation | Colab Notebook

Llama Stack standardizes the core building blocks that simplify AI application development. It codifies best practices across the Llama ecosystem. More specifically, it provides

Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.
Plugin architecture to support the rich ecosystem of different API implementations in various environments, including local development, on-premises, cloud, and mobile.
Prepackaged verified distributions which offer a one-stop solution for developers to get started quickly and reliably in any environment.
Multiple developer interfaces like CLI and SDKs for Python, Typescript, iOS, and Android.
Standalone applications as examples for how to build production-grade AI applications with Llama Stack.

Llama Stack Benefits

Flexible Options: Developers can choose their preferred infrastructure without changing APIs and enjoy flexible deployment choices.
Consistent Experience: With its unified APIs, Llama Stack makes it easier to build, test, and deploy AI applications with consistent application behavior.
Robust Ecosystem: Llama Stack is already integrated with distribution partners (cloud providers, hardware vendors, and AI-focused companies) that offer tailored infrastructure, software, and services for deploying Llama models.

By reducing friction and complexity, Llama Stack empowers developers to focus on what they do best: building transformative generative AI applications.

API Providers

Here is a list of the various API providers and available distributions that can help developers get started easily with Llama Stack.

API Provider Builder	Environments	Agents	Inference	Memory	Safety	Telemetry
Meta Reference	Single Node	✅	✅	✅	✅	✅
SambaNova	Hosted		✅
Cerebras	Hosted		✅
Fireworks	Hosted	✅	✅	✅
AWS Bedrock	Hosted		✅		✅
Together	Hosted	✅	✅		✅
Groq	Hosted		✅
Ollama	Single Node		✅
TGI	Hosted and Single Node		✅
NVIDIA NIM	Hosted and Single Node		✅
Chroma	Single Node			✅
PG Vector	Single Node			✅
PyTorch ExecuTorch	On-device iOS	✅	✅
vLLM	Hosted and Single Node		✅

Distributions

A Llama Stack Distribution (or "distro") is a pre-configured bundle of provider implementations for each API component. Distributions make it easy to get started with a specific deployment scenario - you can begin with a local development setup (eg. ollama) and seamlessly transition to production (eg. Fireworks) without changing your application code. Here are some of the distributions we support:

Distribution	Llama Stack Docker	Start This Distribution
Meta Reference	llamastack/distribution-meta-reference-gpu	Guide
Meta Reference Quantized	llamastack/distribution-meta-reference-quantized-gpu	Guide
SambaNova	llamastack/distribution-sambanova	Guide
Cerebras	llamastack/distribution-cerebras	Guide
Ollama	llamastack/distribution-ollama	Guide
TGI	llamastack/distribution-tgi	Guide
Together	llamastack/distribution-together	Guide
Fireworks	llamastack/distribution-fireworks	Guide
vLLM	llamastack/distribution-remote-vllm	Guide

Installation

You have two ways to install this repository:

Install as a package: You can install the repository directly from PyPI by running the following command:
```
pip install llama-stack
```
Install from source: If you prefer to install from the source code, we recommend using uv. Then, run the following commands:
```
 git clone git@github.com:meta-llama/llama-stack.git
 cd llama-stack

 uv sync
 uv pip install -e .
```

Documentation

Please checkout our Documentation page for more details.

CLI references
- llama (server-side) CLI Reference: Guide for using the llama CLI to work with Llama models (download, study prompts), and building/starting a Llama Stack distribution.
- llama (client-side) CLI Reference: Guide for using the llama-stack-client CLI, which allows you to query information about the distribution.
Getting Started
- Quick guide to start a Llama Stack server.
- Jupyter notebook to walk-through how to use simple text and vision inference llama_stack_client APIs
- The complete Llama Stack lesson Colab notebook of the new Llama 3.2 course on Deeplearning.ai.
- A Zero-to-Hero Guide that guide you through all the key components of llama stack with code samples.
Contributing
- Adding a new API Provider to walk-through how to add a new API provider.

Llama Stack Client SDKs

Language	Client SDK	Package
Python	llama-stack-client-python
Swift	llama-stack-client-swift
Typescript	llama-stack-client-typescript
Kotlin	llama-stack-client-kotlin

Check out our client SDKs for connecting to a Llama Stack server in your preferred language, you can choose from python, typescript, swift, and kotlin programming languages to quickly build your applications.

You can find more example scripts with client SDKs to talk with the Llama Stack server in our llama-stack-apps repo.