mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-07-03 12:52:41 +00:00
faaeccc6fd
Some checks failed
Integration Tests / test-matrix (http, 3.12, agents) (push) Failing after 4s
Integration Tests / test-matrix (http, 3.12, post_training) (push) Failing after 14s
Integration Tests / test-matrix (library, 3.12, post_training) (push) Failing after 10s
Integration Tests / test-matrix (http, 3.12, scoring) (push) Failing after 15s
Integration Tests / test-matrix (http, 3.12, datasets) (push) Failing after 21s
Integration Tests / test-matrix (http, 3.12, tool_runtime) (push) Failing after 19s
Integration Tests / test-matrix (http, 3.13, datasets) (push) Failing after 13s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 23s
Integration Tests / test-matrix (http, 3.12, inference) (push) Failing after 23s
Integration Tests / test-matrix (http, 3.13, post_training) (push) Failing after 18s
Integration Tests / test-matrix (library, 3.12, datasets) (push) Failing after 16s
Integration Tests / test-matrix (http, 3.13, tool_runtime) (push) Failing after 11s
Integration Tests / test-matrix (http, 3.12, providers) (push) Failing after 26s
Integration Tests / test-matrix (library, 3.12, agents) (push) Failing after 17s
Integration Tests / test-matrix (http, 3.13, inference) (push) Failing after 26s
Integration Tests / test-matrix (http, 3.12, vector_io) (push) Failing after 20s
Integration Tests / test-matrix (http, 3.13, providers) (push) Failing after 19s
Integration Tests / test-matrix (library, 3.12, vector_io) (push) Failing after 15s
Integration Tests / test-matrix (library, 3.13, datasets) (push) Failing after 12s
Integration Tests / test-matrix (http, 3.13, inspect) (push) Failing after 18s
Integration Tests / test-matrix (library, 3.12, providers) (push) Failing after 22s
Integration Tests / test-matrix (library, 3.12, scoring) (push) Failing after 15s
Integration Tests / test-matrix (library, 3.12, inference) (push) Failing after 17s
Integration Tests / test-matrix (library, 3.13, inspect) (push) Failing after 7s
Integration Tests / test-matrix (http, 3.13, vector_io) (push) Failing after 25s
Integration Tests / test-matrix (http, 3.13, agents) (push) Failing after 33s
Vector IO Integration Tests / test-matrix (3.12, inline::faiss) (push) Failing after 8s
Integration Tests / test-matrix (http, 3.12, inspect) (push) Failing after 36s
Vector IO Integration Tests / test-matrix (3.12, inline::milvus) (push) Failing after 9s
Integration Tests / test-matrix (http, 3.13, scoring) (push) Failing after 31s
Integration Tests / test-matrix (library, 3.13, agents) (push) Failing after 28s
Integration Tests / test-matrix (library, 3.13, post_training) (push) Failing after 9s
Vector IO Integration Tests / test-matrix (3.12, inline::sqlite-vec) (push) Failing after 8s
Integration Tests / test-matrix (library, 3.13, inference) (push) Failing after 8s
Integration Tests / test-matrix (library, 3.13, tool_runtime) (push) Failing after 7s
Integration Tests / test-matrix (library, 3.12, inspect) (push) Failing after 29s
Vector IO Integration Tests / test-matrix (3.12, remote::pgvector) (push) Failing after 7s
Vector IO Integration Tests / test-matrix (3.12, remote::chromadb) (push) Failing after 10s
Integration Tests / test-matrix (library, 3.13, providers) (push) Failing after 10s
Vector IO Integration Tests / test-matrix (3.13, inline::faiss) (push) Failing after 8s
Integration Tests / test-matrix (library, 3.13, vector_io) (push) Failing after 11s
Vector IO Integration Tests / test-matrix (3.13, inline::sqlite-vec) (push) Failing after 7s
Integration Tests / test-matrix (library, 3.13, scoring) (push) Failing after 14s
Python Package Build Test / build (3.12) (push) Failing after 9s
Python Package Build Test / build (3.13) (push) Failing after 7s
Vector IO Integration Tests / test-matrix (3.13, remote::chromadb) (push) Failing after 14s
Test External Providers / test-external-providers (venv) (push) Failing after 7s
Vector IO Integration Tests / test-matrix (3.13, inline::milvus) (push) Failing after 16s
Integration Tests / test-matrix (library, 3.12, tool_runtime) (push) Failing after 15s
Vector IO Integration Tests / test-matrix (3.13, remote::pgvector) (push) Failing after 14s
Unit Tests / unit-tests (3.12) (push) Failing after 10s
Unit Tests / unit-tests (3.13) (push) Failing after 8s
Update ReadTheDocs / update-readthedocs (push) Failing after 6s
Pre-commit / pre-commit (push) Successful in 1m23s
# What does this PR do? The external providers guide can now be accessed directly from the sidebar ## Test Plan Build locally to test the changes Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
2.5 KiB
2.5 KiB
Providers Overview
The goal of Llama Stack is to build an ecosystem where users can easily swap out different implementations for the same API. Examples for these include:
- LLM inference providers (e.g., Ollama, Fireworks, Together, AWS Bedrock, Groq, Cerebras, SambaNova, vLLM, etc.),
- Vector databases (e.g., ChromaDB, Weaviate, Qdrant, Milvus, FAISS, PGVector, SQLite-Vec, etc.),
- Safety providers (e.g., Meta's Llama Guard, AWS Bedrock Guardrails, etc.)
Providers come in two flavors:
- Remote: the provider runs as a separate service external to the Llama Stack codebase. Llama Stack contains a small amount of adapter code.
- Inline: the provider is fully specified and implemented within the Llama Stack codebase. It may be a simple wrapper around an existing library, or a full fledged implementation within Llama Stack.
Importantly, Llama Stack always strives to provide at least one fully inline provider for each API so you can iterate on a fully featured environment locally.
External Providers
Llama Stack supports external providers that live outside of the main codebase. This allows you to create and maintain your own providers independently.
:maxdepth: 1
external
Agents
Run multi-step agentic workflows with LLMs with tool usage, memory (RAG), etc.
:maxdepth: 1
agents/index
DatasetIO
Interfaces with datasets and data loaders.
:maxdepth: 1
datasetio/index
Eval
Generates outputs (via Inference or Agents) and perform scoring.
:maxdepth: 1
eval/index
Inference
Runs inference with an LLM.
:maxdepth: 1
inference/index
Post Training
Fine-tunes a model.
:maxdepth: 1
post_training/index
Safety
Applies safety policies to the output at a Systems (not only model) level.
:maxdepth: 1
safety/index
Scoring
Evaluates the outputs of the system.
:maxdepth: 1
scoring/index
Telemetry
Collects telemetry data from the system.
:maxdepth: 1
telemetry/index
Tool Runtime
Is associated with the ToolGroup resouces.
:maxdepth: 1
tool_runtime/index
Vector IO
Vector IO refers to operations on vector databases, such as adding documents, searching, and deleting documents. Vector IO plays a crucial role in Retreival Augmented Generation (RAG), where the vector io and database are used to store and retrieve documents for retrieval.
:maxdepth: 1
vector_io/index