llama-stack-mirror/docs/source/providers/index.md

Providers Overview

The goal of Llama Stack is to build an ecosystem where users can easily swap out different implementations for the same API. Examples for these include:

• LLM inference providers (e.g., Meta Reference, Ollama, Fireworks, Together, AWS Bedrock, Groq, Cerebras, SambaNova, vLLM, OpenAI, Anthropic, Gemini, WatsonX, etc.),
• Vector databases (e.g., FAISS, SQLite-Vec, ChromaDB, Weaviate, Qdrant, Milvus, PGVector, etc.),
• Safety providers (e.g., Meta's Llama Guard, Prompt Guard, Code Scanner, AWS Bedrock Guardrails, etc.),
• Tool Runtime providers (e.g., RAG Runtime, Brave Search, etc.)

Providers come in two flavors:

• Remote: the provider runs as a separate service external to the Llama Stack codebase. Llama Stack contains a small amount of adapter code.
• Inline: the provider is fully specified and implemented within the Llama Stack codebase. It may be a simple wrapper around an existing library, or a full fledged implementation within Llama Stack.

Importantly, Llama Stack always strives to provide at least one fully inline provider for each API so you can iterate on a fully featured environment locally.

Available Providers

Here is a comprehensive list of all available API providers in Llama Stack:

API Provider Builder	Environments	Agents	Inference	VectorIO	Safety	Telemetry	Post Training	Eval	DatasetIO	Tool Runtime
Meta Reference	Single Node	✅	✅	✅	✅	✅	✅	✅	✅	✅
SambaNova	Hosted		✅		✅
Cerebras	Hosted		✅
Fireworks	Hosted	✅	✅	✅
AWS Bedrock	Hosted		✅		✅
Together	Hosted	✅	✅		✅
Groq	Hosted		✅
Ollama	Single Node		✅
TGI	Hosted/Single Node		✅
NVIDIA NIM	Hosted/Single Node		✅
Chroma	Single Node			✅
PG Vector	Single Node			✅
PyTorch ExecuTorch	On-device iOS	✅	✅
vLLM	Single Node		✅
OpenAI	Hosted		✅
Anthropic	Hosted		✅
Gemini	Hosted		✅
WatsonX	Hosted		✅
HuggingFace	Single Node						✅		✅
TorchTune	Single Node						✅
NVIDIA NEMO	Hosted		✅	✅			✅	✅	✅
FAISS	Single Node			✅
SQLite-Vec	Single Node			✅
Qdrant	Hosted/Single Node			✅
Weaviate	Hosted			✅
Milvus	Hosted/Single Node			✅
Prompt Guard	Single Node				✅
Llama Guard	Single Node				✅
Code Scanner	Single Node				✅
Brave Search	Hosted									✅
RAG Runtime	Single Node									✅

External Providers

Llama Stack supports external providers that live outside of the main codebase. This allows you to create and maintain your own providers independently. See the External Providers Guide for details.

Agents

Run multi-step agentic workflows with LLMs with tool usage, memory (RAG), etc.

DatasetIO

Interfaces with datasets and data loaders.

Eval

Generates outputs (via Inference or Agents) and perform scoring.

Inference

Runs inference with an LLM.

Post Training

Fine-tunes a model.

Post Training Providers

The following providers are available for Post Training:

:maxdepth: 1

external
post_training/huggingface
post_training/torchtune
post_training/nvidia_nemo

Safety

Applies safety policies to the output at a Systems (not only model) level.

Scoring

Evaluates the outputs of the system.

Telemetry

Collects telemetry data from the system.

Tool Runtime

Is associated with the ToolGroup resouces.

Vector IO

Vector IO refers to operations on vector databases, such as adding documents, searching, and deleting documents. Vector IO plays a crucial role in Retreival Augmented Generation (RAG), where the vector io and database are used to store and retrieve documents for retrieval.

Vector IO Providers

The following providers (i.e., databases) are available for Vector IO:

:maxdepth: 1

external
vector_io/faiss
vector_io/sqlite-vec
vector_io/chromadb
vector_io/pgvector
vector_io/qdrant
vector_io/milvus
vector_io/weaviate