+
+## Benefits of Llama stack
+
+### Current challenges in custom AI applications
+
+Building production AI applications today requires solving multiple challenges:
+
+**Infrastructure Complexity**
+
+- Running large language models efficiently requires specialized infrastructure.
+- Different deployment scenarios (local development, cloud, edge) need different solutions.
+- Moving from development to production often requires significant rework.
+
+**Essential Capabilities**
+
+- Safety guardrails and content filtering are necessary in an enterprise setting.
+- Just model inference is not enough - Knowledge retrieval and RAG capabilities are required.
+- Nearly any application needs composable multi-step workflows.
+- Without monitoring, observability and evaluation, you end up operating in the dark.
+
+**Lack of Flexibility and Choice**
+
+- Directly integrating with multiple providers creates tight coupling.
+- Different providers have different APIs and abstractions.
+- Changing providers requires significant code changes.
+
+### Our Solution: A Universal Stack
+
+Llama Stack addresses these challenges through a service-oriented, API-first approach:
+
+**Develop Anywhere, Deploy Everywhere**
+- Start locally with CPU-only setups
+- Move to GPU acceleration when needed
+- Deploy to cloud or edge without code changes
+- Same APIs and developer experience everywhere
+
+**Production-Ready Building Blocks**
+- Pre-built safety guardrails and content filtering
+- Built-in RAG and agent capabilities
+- Comprehensive evaluation toolkit
+- Full observability and monitoring
+
+**True Provider Independence**
+- Swap providers without application changes
+- Mix and match best-in-class implementations
+- Federation and fallback support
+- No vendor lock-in
+
+**Robust Ecosystem**
+- Llama Stack is already integrated with distribution partners (cloud providers, hardware vendors, and AI-focused companies).
+- Ecosystem offers tailored infrastructure, software, and services for deploying a variety of models.
+
+
+## Our Philosophy
+
+- **Service-Oriented**: REST APIs enforce clean interfaces and enable seamless transitions across different environments.
+- **Composability**: Every component is independent but works together seamlessly
+- **Production Ready**: Built for real-world applications, not just demos
+- **Turnkey Solutions**: Easy to deploy built in solutions for popular deployment scenarios
+
+
+With Llama Stack, you can focus on building your application while we handle the infrastructure complexity, essential capabilities, and provider integrations.
diff --git a/docs/docs/concepts/distributions.mdx b/docs/docs/concepts/distributions.mdx
new file mode 100644
index 000000000..3b911ed7c
--- /dev/null
+++ b/docs/docs/concepts/distributions.mdx
@@ -0,0 +1,16 @@
+---
+title: Distributions
+description: Pre-packaged provider configurations for different deployment scenarios
+sidebar_label: Distributions
+sidebar_position: 5
+---
+
+# Distributions
+
+While there is a lot of flexibility to mix-and-match providers, often users will work with a specific set of providers (hardware support, contractual obligations, etc.) We therefore need to provide a _convenient shorthand_ for such collections. We call this shorthand a **Llama Stack Distribution** or a **Distro**. One can think of it as specific pre-packaged versions of the Llama Stack. Here are some examples:
+
+**Remotely Hosted Distro**: These are the simplest to consume from a user perspective. You can simply obtain the API key for these providers, point to a URL and have _all_ Llama Stack APIs working out of the box. Currently, [Fireworks](https://fireworks.ai/) and [Together](https://together.xyz/) provide such easy-to-consume Llama Stack distributions.
+
+**Locally Hosted Distro**: You may want to run Llama Stack on your own hardware. Typically though, you still need to use Inference via an external service. You can use providers like HuggingFace TGI, Fireworks, Together, etc. for this purpose. Or you may have access to GPUs and can run a [vLLM](https://github.com/vllm-project/vllm) or [NVIDIA NIM](https://build.nvidia.com/nim?filters=nimType%3Anim_type_run_anywhere&q=llama) instance. If you "just" have a regular desktop machine, you can use [Ollama](https://ollama.com/) for inference. To provide convenient quick access to these options, we provide a number of such pre-configured locally-hosted Distros.
+
+**On-device Distro**: To run Llama Stack directly on an edge device (mobile phone or a tablet), we provide Distros for [iOS](/docs/distributions/ondevice_distro/ios_sdk) and [Android](/docs/distributions/ondevice_distro/android_sdk)
diff --git a/docs/docs/concepts/evaluation-concepts.mdx b/docs/docs/concepts/evaluation-concepts.mdx
new file mode 100644
index 000000000..1cf2e2361
--- /dev/null
+++ b/docs/docs/concepts/evaluation-concepts.mdx
@@ -0,0 +1,71 @@
+# Evaluation Concepts
+
+The Llama Stack Evaluation flow allows you to run evaluations on your GenAI application datasets or pre-registered benchmarks.
+
+We introduce a set of APIs in Llama Stack for supporting running evaluations of LLM applications:
+- `/datasetio` + `/datasets` API
+- `/scoring` + `/scoring_functions` API
+- `/eval` + `/benchmarks` API
+
+This guide goes over the sets of APIs and developer experience flow of using Llama Stack to run evaluations for different use cases. Checkout our Colab notebook on working examples with evaluations [here](https://colab.research.google.com/drive/10CHyykee9j2OigaIcRv47BKG9mrNm0tJ?usp=sharing).
+
+The Evaluation APIs are associated with a set of Resources. Please visit the Resources section in our [Core Concepts](./index.mdx) guide for better high-level understanding.
+
+- **DatasetIO**: defines interface with datasets and data loaders.
+ - Associated with `Dataset` resource.
+- **Scoring**: evaluate outputs of the system.
+ - Associated with `ScoringFunction` resource. We provide a suite of out-of-the box scoring functions and also the ability for you to add custom evaluators. These scoring functions are the core part of defining an evaluation task to output evaluation metrics.
+- **Eval**: generate outputs (via Inference or Agents) and perform scoring.
+ - Associated with `Benchmark` resource.
+
+## Open-benchmark Eval
+
+### List of open-benchmarks Llama Stack support
+
+Llama stack pre-registers several popular open-benchmarks to easily evaluate model perfomance via CLI.
+
+The list of open-benchmarks we currently support:
+- [MMLU-COT](https://arxiv.org/abs/2009.03300) (Measuring Massive Multitask Language Understanding): Benchmark designed to comprehensively evaluate the breadth and depth of a model's academic and professional understanding
+- [GPQA-COT](https://arxiv.org/abs/2311.12022) (A Graduate-Level Google-Proof Q&A Benchmark): A challenging benchmark of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.
+- [SimpleQA](https://openai.com/index/introducing-simpleqa/): Benchmark designed to access models to answer short, fact-seeking questions.
+- [MMMU](https://arxiv.org/abs/2311.16502) (A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI)]: Benchmark designed to evaluate multimodal models.
+
+You can follow this [contributing guide](../references/evals-reference.mdx#open-benchmark-contributing-guide) to add more open-benchmarks to Llama Stack
+
+### Run evaluation on open-benchmarks via CLI
+
+We have built-in functionality to run the supported open-benckmarks using llama-stack-client CLI
+
+#### Spin up Llama Stack server
+
+Spin up llama stack server with 'open-benchmark' template
+```bash
+llama stack run llama_stack/distributions/open-benchmark/run.yaml
+```
+
+#### Run eval CLI
+There are 3 necessary inputs to run a benchmark eval
+- `list of benchmark_ids`: The list of benchmark ids to run evaluation on
+- `model-id`: The model id to evaluate on
+- `output_dir`: Path to store the evaluate results
+
+```bash
+llama-stack-client eval run-benchmark
+
+
+Let's break this down into the different sections. The first section specifies the set of APIs that the stack server will serve:
+
+```yaml
+apis:
+- agents
+- inference
+- vector_io
+- safety
+- telemetry
+```
+
+## Providers
+
+Next up is the most critical part: the set of providers that the stack will use to serve the above APIs. Consider the `inference` API:
+
+```yaml
+providers:
+ inference:
+ # provider_id is a string you can choose freely
+ - provider_id: ollama
+ # provider_type is a string that specifies the type of provider.
+ # in this case, the provider for inference is ollama and it runs remotely (outside of the distribution)
+ provider_type: remote::ollama
+ # config is a dictionary that contains the configuration for the provider.
+ # in this case, the configuration is the url of the ollama server
+ config:
+ url: ${env.OLLAMA_URL:=http://localhost:11434}
+```
+
+A few things to note:
+- A _provider instance_ is identified with an (id, type, config) triplet.
+- The id is a string you can choose freely.
+- You can instantiate any number of provider instances of the same type.
+- The configuration dictionary is provider-specific.
+- Notice that configuration can reference environment variables (with default values), which are expanded at runtime. When you run a stack server (via docker or via `llama stack run`), you can specify `--env OLLAMA_URL=http://my-server:11434` to override the default value.
+
+### Environment Variable Substitution
+
+Llama Stack supports environment variable substitution in configuration values using the
+`${env.VARIABLE_NAME}` syntax. This allows you to externalize configuration values and provide
+different settings for different environments. The syntax is inspired by [bash parameter expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html)
+and follows similar patterns.
+
+#### Basic Syntax
+
+The basic syntax for environment variable substitution is:
+
+```yaml
+config:
+ api_key: ${env.API_KEY}
+ url: ${env.SERVICE_URL}
+```
+
+If the environment variable is not set, the server will raise an error during startup.
+
+#### Default Values
+
+You can provide default values using the `:=` operator:
+
+```yaml
+config:
+ url: ${env.OLLAMA_URL:=http://localhost:11434}
+ port: ${env.PORT:=8321}
+ timeout: ${env.TIMEOUT:=60}
+```
+
+If the environment variable is not set, the default value `http://localhost:11434` will be used.
+Empty defaults are allowed so `url: ${env.OLLAMA_URL:=}` will be set to `None` if the environment variable is not set.
+
+#### Conditional Values
+
+You can use the `:+` operator to provide a value only when the environment variable is set:
+
+```yaml
+config:
+ # Only include this field if ENVIRONMENT is set
+ environment: ${env.ENVIRONMENT:+production}
+```
+
+If the environment variable is set, the value after `:+` will be used. If it's not set, the field
+will be omitted with a `None` value.
+
+Do not use conditional values (`${env.OLLAMA_URL:+}`) for empty defaults (`${env.OLLAMA_URL:=}`).
+This will be set to `None` if the environment variable is not set.
+Conditional must only be used when the environment variable is set.
+
+#### Examples
+
+Here are some common patterns:
+
+```yaml
+# Required environment variable (will error if not set)
+api_key: ${env.OPENAI_API_KEY}
+
+# Optional with default
+base_url: ${env.API_BASE_URL:=https://api.openai.com/v1}
+
+# Conditional field
+debug_mode: ${env.DEBUG:+true}
+
+# Optional field that becomes None if not set
+optional_token: ${env.OPTIONAL_TOKEN:+}
+```
+
+#### Runtime Override
+
+You can override environment variables at runtime when starting the server:
+
+```bash
+# Override specific environment variables
+llama stack run --config run.yaml --env API_KEY=sk-123 --env BASE_URL=https://custom-api.com
+
+# Or set them in your shell
+export API_KEY=sk-123
+export BASE_URL=https://custom-api.com
+llama stack run --config run.yaml
+```
+
+#### Type Safety
+
+The environment variable substitution system is type-safe:
+
+- String values remain strings
+- Empty defaults (`${env.VAR:+}`) are converted to `None` for fields that accept `str | None`
+- Numeric defaults are properly typed (e.g., `${env.PORT:=8321}` becomes an integer)
+- Boolean defaults work correctly (e.g., `${env.DEBUG:=false}` becomes a boolean)
+
+## Resources
+
+Let's look at the `models` section:
+
+```yaml
+models:
+- metadata: {}
+ model_id: ${env.INFERENCE_MODEL}
+ provider_id: ollama
+ provider_model_id: null
+ model_type: llm
+```
+
+A Model is an instance of a "Resource" (see [Concepts](/docs/concepts/)) and is associated with a specific inference provider (in this case, the provider with identifier `ollama`). This is an instance of a "pre-registered" model. While we always encourage the clients to register models before using them, some Stack servers may come up a list of "already known and available" models.
+
+What's with the `provider_model_id` field? This is an identifier for the model inside the provider's model catalog. Contrast it with `model_id` which is the identifier for the same model for Llama Stack's purposes. For example, you may want to name "llama3.2:vision-11b" as "image_captioning_model" when you use it in your Stack interactions. When omitted, the server will set `provider_model_id` to be the same as `model_id`.
+
+If you need to conditionally register a model in the configuration, such as only when specific environment variable(s) are set, this can be accomplished by utilizing a special `__disabled__` string as the default value of an environment variable substitution, as shown below:
+
+```yaml
+models:
+- metadata: {}
+ model_id: ${env.INFERENCE_MODEL:__disabled__}
+ provider_id: ollama
+ provider_model_id: ${env.INFERENCE_MODEL:__disabled__}
+```
+
+The snippet above will only register this model if the environment variable `INFERENCE_MODEL` is set and non-empty. If the environment variable is not set, the model will not get registered at all.
+
+## Server Configuration
+
+The `server` section configures the HTTP server that serves the Llama Stack APIs:
+
+```yaml
+server:
+ port: 8321 # Port to listen on (default: 8321)
+ tls_certfile: "/path/to/cert.pem" # Optional: Path to TLS certificate for HTTPS
+ tls_keyfile: "/path/to/key.pem" # Optional: Path to TLS key for HTTPS
+ cors: true # Optional: Enable CORS (dev mode) or full config object
+```
+
+### CORS Configuration
+
+CORS (Cross-Origin Resource Sharing) can be configured in two ways:
+
+**Local development** (allows localhost origins only):
+```yaml
+server:
+ cors: true
+```
+
+**Explicit configuration** (custom origins and settings):
+```yaml
+server:
+ cors:
+ allow_origins: ["https://myapp.com", "https://app.example.com"]
+ allow_methods: ["GET", "POST", "PUT", "DELETE"]
+ allow_headers: ["Content-Type", "Authorization"]
+ allow_credentials: true
+ max_age: 3600
+```
+
+When `cors: true`, the server enables secure localhost-only access for local development. For production, specify exact origins to maintain security.
+
+### Authentication Configuration
+
+:::info[Breaking Change (v0.2.14)]
+The authentication configuration structure has changed. The previous format with `provider_type` and `config` fields has been replaced with a unified `provider_config` field that includes the `type` field. Update your configuration files accordingly.
+:::
+
+The `auth` section configures authentication for the server. When configured, all API requests must include a valid Bearer token in the Authorization header:
+
+```
+Authorization: Bearer ๐ Click here for a Sample Configuration File
+ +```yaml +version: 2 +apis: +- agents +- inference +- vector_io +- safety +- telemetry +providers: + inference: + - provider_id: ollama + provider_type: remote::ollama + config: + url: ${env.OLLAMA_URL:=http://localhost:11434} + vector_io: + - provider_id: faiss + provider_type: inline::faiss + config: + kvstore: + type: sqlite + namespace: null + db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/ollama}/faiss_store.db + safety: + - provider_id: llama-guard + provider_type: inline::llama-guard + config: {} + agents: + - provider_id: meta-reference + provider_type: inline::meta-reference + config: + persistence_store: + type: sqlite + namespace: null + db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/ollama}/agents_store.db + telemetry: + - provider_id: meta-reference + provider_type: inline::meta-reference + config: {} +metadata_store: + namespace: null + type: sqlite + db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/ollama}/registry.db +models: +- metadata: {} + model_id: ${env.INFERENCE_MODEL} + provider_id: ollama + provider_model_id: null +shields: [] +server: + port: 8321 + auth: + provider_config: + type: "oauth2_token" + jwks: + uri: "https://my-token-issuing-svc.com/jwks" +``` + +| Method | +Best For | +Complexity | +Use Cases | +
|---|---|---|---|
| Library | +Development & External Services | +Low | +Prototyping, using remote inference providers | +
| Container | +Local Development & Production | +Medium | +Consistent environments, local inference | +
| Kubernetes | +Production & Scale | +High | +Production deployments, high availability | +
+
+ ๐ Click here to see the sample output
+ Non-streaming ... + agent> I'm an artificial intelligence designed to assist and communicate with users like you. I don't have a personal identity, but I can provide information, answer questions, and help with tasks to the best of my abilities. + + I'm a large language model, which means I've been trained on a massive dataset of text from various sources, allowing me to understand and respond to a wide range of topics and questions. My purpose is to provide helpful and accurate information, and I'm constantly learning and improving my responses based on the interactions I have with users like you. + + I can help with: + + * Answering questions on various subjects + * Providing definitions and explanations + * Offering suggestions and ideas + * Assisting with language-related tasks, such as proofreading and editing + * Generating text and content + * And more! + + Feel free to ask me anything, and I'll do my best to help! + Streaming ... + AgentTurnResponseStreamChunk( + โ event=TurnResponseEvent( + โ โ payload=AgentTurnResponseStepStartPayload( + โ โ โ event_type='step_start', + โ โ โ step_id='69831607-fa75-424a-949b-e2049e3129d1', + โ โ โ step_type='inference', + โ โ โ metadata={} + โ โ ) + โ ) + ) + AgentTurnResponseStreamChunk( + โ event=TurnResponseEvent( + โ โ payload=AgentTurnResponseStepProgressPayload( + โ โ โ delta=TextDelta(text='As', type='text'), + โ โ โ event_type='step_progress', + โ โ โ step_id='69831607-fa75-424a-949b-e2049e3129d1', + โ โ โ step_type='inference' + โ โ ) + โ ) + ) + AgentTurnResponseStreamChunk( + โ event=TurnResponseEvent( + โ โ payload=AgentTurnResponseStepProgressPayload( + โ โ โ delta=TextDelta(text=' a', type='text'), + โ โ โ event_type='step_progress', + โ โ โ step_id='69831607-fa75-424a-949b-e2049e3129d1', + โ โ โ step_type='inference' + โ โ ) + โ ) + ) + ... + AgentTurnResponseStreamChunk( + โ event=TurnResponseEvent( + โ โ payload=AgentTurnResponseStepCompletePayload( + โ โ โ event_type='step_complete', + โ โ โ step_details=InferenceStep( + โ โ โ โ api_model_response=CompletionMessage( + โ โ โ โ โ content='As a conversational AI, I don\'t have a personal identity in the classical sense. I exist as a program running on computer servers, designed to process and respond to text-based inputs.\n\nI\'m an instance of a type of artificial intelligence called a "language model," which is trained on vast amounts of text data to generate human-like responses. My primary function is to understand and respond to natural language inputs, like our conversation right now.\n\nThink of me as a virtual assistant, a chatbot, or a conversational interface โ I\'m here to provide information, answer questions, and engage in conversation to the best of my abilities. I don\'t have feelings, emotions, or consciousness like humans do, but I\'m designed to simulate human-like interactions to make our conversations feel more natural and helpful.\n\nSo, that\'s me in a nutshell! What can I help you with today?', + โ โ โ โ โ role='assistant', + โ โ โ โ โ stop_reason='end_of_turn', + โ โ โ โ โ tool_calls=[] + โ โ โ โ ), + โ โ โ โ step_id='69831607-fa75-424a-949b-e2049e3129d1', + โ โ โ โ step_type='inference', + โ โ โ โ turn_id='8b360202-f7cb-4786-baa9-166a1b46e2ca', + โ โ โ โ completed_at=datetime.datetime(2025, 4, 3, 1, 15, 21, 716174, tzinfo=TzInfo(UTC)), + โ โ โ โ started_at=datetime.datetime(2025, 4, 3, 1, 15, 14, 28823, tzinfo=TzInfo(UTC)) + โ โ โ ), + โ โ โ step_id='69831607-fa75-424a-949b-e2049e3129d1', + โ โ โ step_type='inference' + โ โ ) + โ ) + ) + AgentTurnResponseStreamChunk( + โ event=TurnResponseEvent( + โ โ payload=AgentTurnResponseTurnCompletePayload( + โ โ โ event_type='turn_complete', + โ โ โ turn=Turn( + โ โ โ โ input_messages=[UserMessage(content='Who are you?', role='user', context=None)], + โ โ โ โ output_message=CompletionMessage( + โ โ โ โ โ content='As a conversational AI, I don\'t have a personal identity in the classical sense. I exist as a program running on computer servers, designed to process and respond to text-based inputs.\n\nI\'m an instance of a type of artificial intelligence called a "language model," which is trained on vast amounts of text data to generate human-like responses. My primary function is to understand and respond to natural language inputs, like our conversation right now.\n\nThink of me as a virtual assistant, a chatbot, or a conversational interface โ I\'m here to provide information, answer questions, and engage in conversation to the best of my abilities. I don\'t have feelings, emotions, or consciousness like humans do, but I\'m designed to simulate human-like interactions to make our conversations feel more natural and helpful.\n\nSo, that\'s me in a nutshell! What can I help you with today?', + โ โ โ โ โ role='assistant', + โ โ โ โ โ stop_reason='end_of_turn', + โ โ โ โ โ tool_calls=[] + โ โ โ โ ), + โ โ โ โ session_id='abd4afea-4324-43f4-9513-cfe3970d92e8', + โ โ โ โ started_at=datetime.datetime(2025, 4, 3, 1, 15, 14, 28722, tzinfo=TzInfo(UTC)), + โ โ โ โ steps=[ + โ โ โ โ โ InferenceStep( + โ โ โ โ โ โ api_model_response=CompletionMessage( + โ โ โ โ โ โ โ content='As a conversational AI, I don\'t have a personal identity in the classical sense. I exist as a program running on computer servers, designed to process and respond to text-based inputs.\n\nI\'m an instance of a type of artificial intelligence called a "language model," which is trained on vast amounts of text data to generate human-like responses. My primary function is to understand and respond to natural language inputs, like our conversation right now.\n\nThink of me as a virtual assistant, a chatbot, or a conversational interface โ I\'m here to provide information, answer questions, and engage in conversation to the best of my abilities. I don\'t have feelings, emotions, or consciousness like humans do, but I\'m designed to simulate human-like interactions to make our conversations feel more natural and helpful.\n\nSo, that\'s me in a nutshell! What can I help you with today?', + โ โ โ โ โ โ โ role='assistant', + โ โ โ โ โ โ โ stop_reason='end_of_turn', + โ โ โ โ โ โ โ tool_calls=[] + โ โ โ โ โ โ ), + โ โ โ โ โ โ step_id='69831607-fa75-424a-949b-e2049e3129d1', + โ โ โ โ โ โ step_type='inference', + โ โ โ โ โ โ turn_id='8b360202-f7cb-4786-baa9-166a1b46e2ca', + โ โ โ โ โ โ completed_at=datetime.datetime(2025, 4, 3, 1, 15, 21, 716174, tzinfo=TzInfo(UTC)), + โ โ โ โ โ โ started_at=datetime.datetime(2025, 4, 3, 1, 15, 14, 28823, tzinfo=TzInfo(UTC)) + โ โ โ โ โ ) + โ โ โ โ ], + โ โ โ โ turn_id='8b360202-f7cb-4786-baa9-166a1b46e2ca', + โ โ โ โ completed_at=datetime.datetime(2025, 4, 3, 1, 15, 21, 727364, tzinfo=TzInfo(UTC)), + โ โ โ โ output_attachments=[] + โ โ โ ) + โ โ ) + โ ) + ) + + + Streaming with print helper... + inference> Dรฉjร vu! You're asking me again! + + As I mentioned earlier, I'm a computer program designed to simulate conversation and answer questions. I don't have a personal identity or consciousness like a human would. I exist solely as a digital entity, running on computer servers and responding to inputs from users like you. + + I'm a type of artificial intelligence (AI) called a large language model, which means I've been trained on a massive dataset of text from various sources. This training allows me to understand and respond to a wide range of questions and topics. + + My purpose is to provide helpful and accurate information, answer questions, and assist users like you with tasks and conversations. I don't have personal preferences, emotions, or opinions like humans do. My goal is to be informative, neutral, and respectful in my responses. + + So, that's me in a nutshell! + +
+
+ ๐ Click here to see the sample output
+ + ``` + user> what is torchtune + inference> [knowledge_search(query='TorchTune')] + tool_execution> Tool:knowledge_search Args:{'query': 'TorchTune'} + tool_execution> Tool:knowledge_search Response:[TextContentItem(text='knowledge_search tool found 5 chunks:\nBEGIN of knowledge_search tool results.\n', type='text'), TextContentItem(text='Result 1:\nDocument_id:num-1\nContent: conversational data, :func:`~torchtune.datasets.chat_dataset` seems to be a good fit. ..., type='text'), TextContentItem(text='END of knowledge_search tool results.\n', type='text')] + inference> Here is a high-level overview of the text: + + **LoRA Finetuning with PyTorch Tune** + + PyTorch Tune provides a recipe for LoRA (Low-Rank Adaptation) finetuning, which is a technique to adapt pre-trained models to new tasks. The recipe uses the `lora_finetune_distributed` command. + ... + Overall, DORA is a powerful reinforcement learning algorithm that can learn complex tasks from human demonstrations. However, it requires careful consideration of the challenges and limitations to achieve optimal results. + ``` + +
+
+Our goal is to provide pre-packaged implementations (aka "distributions") which can be run in a variety of deployment environments. LlamaStack can assist you in your entire app development lifecycle - start iterating on local, mobile or desktop and seamlessly transition to on-prem or public cloud deployments. At every point in this transition, the same set of APIs and the same developer experience is available.
+
+## How does Llama Stack work?
+
+Llama Stack consists of a server (with multiple pluggable API providers) and Client SDKs meant to be used in your applications. The server can be run in a variety of environments, including local (inline) development, on-premises, and cloud. The client SDKs are available for Python, Swift, Node, and Kotlin.
+
+## Quick Links
+
+- Ready to build? Check out the [Getting Started Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/index.html) to get started.
+- Want to contribute? See the [Contributing Guide](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md).
+- Explore [Example Applications](https://github.com/meta-llama/llama-stack-apps) built with Llama Stack.
+
+## Rich Ecosystem Support
+
+Llama Stack provides adapters for popular providers across all API categories:
+
+- **Inference**: Meta Reference, Ollama, Fireworks, Together, NVIDIA, vLLM, AWS Bedrock, OpenAI, Anthropic, and more
+- **Vector Databases**: FAISS, Chroma, Milvus, Postgres, Weaviate, Qdrant, and others
+- **Safety**: Llama Guard, Prompt Guard, Code Scanner, AWS Bedrock
+- **Training & Evaluation**: HuggingFace, TorchTune, NVIDIA NEMO
+
+:::info Provider Details
+For complete provider compatibility and setup instructions, see our [Providers Documentation](https://llama-stack.readthedocs.io/en/latest/providers/index.html).
+:::
+
+## Get Started Today
+
+
diff --git a/docs/docs/providers/agents/index.mdx b/docs/docs/providers/agents/index.mdx
new file mode 100644
index 000000000..b4718ca77
--- /dev/null
+++ b/docs/docs/providers/agents/index.mdx
@@ -0,0 +1,25 @@
+---
+description: Available providers for the agents API
+sidebar_label: Overview
+sidebar_position: 1
+title: Agents
+---
+
+# Agents
+
+## Overview
+
+Agents API for creating and interacting with agentic systems.
+
+ Main functionalities provided by this API:
+ - Create agents with specific instructions and ability to use tools.
+ - Interactions with agents are grouped into sessions ("threads"), and each interaction is called a "turn".
+ - Agents can be provided with various tools (see the ToolGroups and ToolRuntime APIs for more details).
+ - Agents can be provided with various shields (see the Safety API for more details).
+ - Agents can also use Memory to retrieve information from knowledge bases. See the RAG Tool and Vector IO APIs for more details.
+
+This section contains documentation for all available providers for the **agents** API.
+
+## Providers
+
+- **[Meta Reference](./inline_meta-reference)** - Inline provider
diff --git a/docs/docs/providers/agents/inline_meta-reference.mdx b/docs/docs/providers/agents/inline_meta-reference.mdx
new file mode 100644
index 000000000..0dae53c4c
--- /dev/null
+++ b/docs/docs/providers/agents/inline_meta-reference.mdx
@@ -0,0 +1,31 @@
+---
+description: Meta's reference implementation of an agent system that can use tools,
+ access vector databases, and perform complex reasoning tasks
+sidebar_label: Meta Reference
+sidebar_position: 2
+title: inline::meta-reference
+---
+
+# inline::meta-reference
+
+## Description
+
+Meta's reference implementation of an agent system that can use tools, access vector databases, and perform complex reasoning tasks.
+
+## Configuration
+
+| Field | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `persistence_store` | `utils.kvstore.config.RedisKVStoreConfig \| utils.kvstore.config.SqliteKVStoreConfig \| utils.kvstore.config.PostgresKVStoreConfig \| utils.kvstore.config.MongoDBKVStoreConfig` | No | sqlite | |
+| `responses_store` | `utils.sqlstore.sqlstore.SqliteSqlStoreConfig \| utils.sqlstore.sqlstore.PostgresSqlStoreConfig` | No | sqlite | |
+
+## Sample Configuration
+
+```yaml
+persistence_store:
+ type: sqlite
+ db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/agents_store.db
+responses_store:
+ type: sqlite
+ db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/responses_store.db
+```
diff --git a/docs/docs/providers/batches/index.mdx b/docs/docs/providers/batches/index.mdx
new file mode 100644
index 000000000..08893bd06
--- /dev/null
+++ b/docs/docs/providers/batches/index.mdx
@@ -0,0 +1,27 @@
+---
+description: Available providers for the batches API
+sidebar_label: Overview
+sidebar_position: 1
+title: Batches
+---
+
+# Batches
+
+## Overview
+
+The Batches API enables efficient processing of multiple requests in a single operation,
+ particularly useful for processing large datasets, batch evaluation workflows, and
+ cost-effective inference at scale.
+
+ The API is designed to allow use of openai client libraries for seamless integration.
+
+ This API provides the following extensions:
+ - idempotent batch creation
+
+ Note: This API is currently under active development and may undergo changes.
+
+This section contains documentation for all available providers for the **batches** API.
+
+## Providers
+
+- **[Reference](./inline_reference)** - Inline provider
diff --git a/docs/docs/providers/batches/inline_reference.mdx b/docs/docs/providers/batches/inline_reference.mdx
new file mode 100644
index 000000000..d85c412eb
--- /dev/null
+++ b/docs/docs/providers/batches/inline_reference.mdx
@@ -0,0 +1,28 @@
+---
+description: Reference implementation of batches API with KVStore persistence
+sidebar_label: Reference
+sidebar_position: 2
+title: inline::reference
+---
+
+# inline::reference
+
+## Description
+
+Reference implementation of batches API with KVStore persistence.
+
+## Configuration
+
+| Field | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `kvstore` | `utils.kvstore.config.RedisKVStoreConfig \| utils.kvstore.config.SqliteKVStoreConfig \| utils.kvstore.config.PostgresKVStoreConfig \| utils.kvstore.config.MongoDBKVStoreConfig` | No | sqlite | Configuration for the key-value store backend. |
+| `max_concurrent_batches` | `