diff --git a/docs/source/building_applications/agent_execution_loop.md b/docs/source/building_applications/agent_execution_loop.md index 62fb314bc..eec8fee95 100644 --- a/docs/source/building_applications/agent_execution_loop.md +++ b/docs/source/building_applications/agent_execution_loop.md @@ -1,4 +1,4 @@ -# Agent Execution Loop +## Agent Execution Loop Agents are the heart of complex AI applications. They combine inference, memory, safety, and tool usage into coherent workflows. At its core, an agent follows a sophisticated execution loop that enables multi-step reasoning, tool usage, and safety checks. diff --git a/docs/source/benchmark_evaluations/index.md b/docs/source/building_applications/evals.md similarity index 95% rename from docs/source/benchmark_evaluations/index.md rename to docs/source/building_applications/evals.md index 56852c89c..511a3d31d 100644 --- a/docs/source/benchmark_evaluations/index.md +++ b/docs/source/building_applications/evals.md @@ -1,8 +1,8 @@ -# Benchmark Evaluations +# Evals [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/10CHyykee9j2OigaIcRv47BKG9mrNm0tJ?usp=sharing) -Llama Stack provides the building blocks needed to run benchmark and application evaluations. This guide will walk you through how to use these components to run open benchmark evaluations. Visit our [Evaluation Concepts](../concepts/evaluation_concepts.md) guide for more details on how evaluations work in Llama Stack, and our [Evaluation Reference](../references/evals_reference/index.md) guide for a comprehensive reference on the APIs. Check out our [Colab notebook](https://colab.research.google.com/drive/10CHyykee9j2OigaIcRv47BKG9mrNm0tJ?usp=sharing) on working examples on how you can use Llama Stack for running benchmark evaluations. +Llama Stack provides the building blocks needed to run benchmark and application evaluations. This guide will walk you through how to use these components to run open benchmark evaluations. Visit our [Evaluation Concepts](../concepts/evaluation_concepts.md) guide for more details on how evaluations work in Llama Stack, and our [Evaluation Reference](../references/evals_reference/index.md) guide for a comprehensive reference on the APIs. ### 1. Open Benchmark Model Evaluation diff --git a/docs/source/building_applications/index.md b/docs/source/building_applications/index.md index 6e1e9454f..55485ddbc 100644 --- a/docs/source/building_applications/index.md +++ b/docs/source/building_applications/index.md @@ -6,12 +6,14 @@ The best way to get started is to look at this notebook which walks through the **Notebook**: [Building AI Applications](docs/notebooks/Llama_Stack_Building_AI_Applications.ipynb) -## Agentic Concepts +Here are some key topics that will help you build effective agents: + - **[Agent Execution Loop](agent_execution_loop)** - **[RAG](rag)** - **[Safety](safety)** - **[Tools](tools)** - **[Telemetry](telemetry)** +- **[Evals](evals)** ```{toctree} @@ -23,4 +25,5 @@ rag safety tools telemetry +evals ``` diff --git a/docs/source/building_applications/telemetry.md b/docs/source/building_applications/telemetry.md index 45bc7a1c2..25b637821 100644 --- a/docs/source/building_applications/telemetry.md +++ b/docs/source/building_applications/telemetry.md @@ -1,11 +1,10 @@ -# Telemetry - +## Telemetry The Llama Stack telemetry system provides comprehensive tracing, metrics, and logging capabilities. It supports multiple sink types including OpenTelemetry, SQLite, and Console output. -## Key Concepts +#### Key Concepts -### Events +#### Events The telemetry system supports three main types of events: - **Unstructured Log Events**: Free-form log messages with severity levels @@ -31,24 +30,24 @@ structured_log_event = SpanStartPayload( ) ``` -### Spans and Traces +#### Spans and Traces - **Spans**: Represent operations with timing and hierarchical relationships - **Traces**: Collection of related spans forming a complete request flow -### Sinks +#### Sinks - **OpenTelemetry**: Send events to an OpenTelemetry Collector. This is useful for visualizing traces in a tool like Jaeger. - **SQLite**: Store events in a local SQLite database. This is needed if you want to query the events later through the Llama Stack API. - **Console**: Print events to the console. -## Providers +#### Providers -### Meta-Reference Provider +#### Meta-Reference Provider Currently, only the meta-reference provider is implemented. It can be configured to send events to three sink types: 1) OpenTelemetry Collector 2) SQLite 3) Console -## Configuration +#### Configuration Here's an example that sends telemetry signals to all three sink types. Your configuration might use only one. ```yaml @@ -61,7 +60,7 @@ Here's an example that sends telemetry signals to all three sink types. Your con sqlite_db_path: "/path/to/telemetry.db" ``` -## Jaeger to visualize traces +#### Jaeger to visualize traces The `otel` sink works with any service compatible with the OpenTelemetry collector. Let's use Jaeger to visualize this data. @@ -75,6 +74,6 @@ $ docker run --rm --name jaeger \ Once the Jaeger instance is running, you can visualize traces by navigating to http://localhost:16686/. -## Querying Traces Stored in SQLIte +#### Querying Traces Stored in SQLIte The `sqlite` sink allows you to query traces without an external system. Here are some example queries. Refer to the notebook at [Llama Stack Building AI Applications](https://github.com/meta-llama/llama-stack/blob/main/docs/getting_started.ipynb) for more examples on how to query traces and spaces. diff --git a/docs/source/contributing/memory_api.md b/docs/source/contributing/memory_api.md deleted file mode 100644 index be486ae8f..000000000 --- a/docs/source/contributing/memory_api.md +++ /dev/null @@ -1,53 +0,0 @@ -# Memory API Providers - -This guide gives you references to switch between different memory API providers. - -##### pgvector -1. Start running the pgvector server: - -``` -$ docker run --network host --name mypostgres -it -p 5432:5432 -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres pgvector/pgvector:pg16 -``` - -2. Edit the `run.yaml` file to point to the pgvector server. -``` -memory: - - provider_id: pgvector - provider_type: remote::pgvector - config: - host: 127.0.0.1 - port: 5432 - db: postgres - user: postgres - password: mysecretpassword -``` - -> [!NOTE] -> If you get a `RuntimeError: Vector extension is not installed.`. You will need to run `CREATE EXTENSION IF NOT EXISTS vector;` to include the vector extension. E.g. - -``` -docker exec -it mypostgres ./bin/psql -U postgres -postgres=# CREATE EXTENSION IF NOT EXISTS vector; -postgres=# SELECT extname from pg_extension; - extname -``` - -3. Run `docker compose up` with the updated `run.yaml` file. - -##### chromadb -1. Start running chromadb server -``` -docker run -it --network host --name chromadb -p 6000:6000 -v ./chroma_vdb:/chroma/chroma -e IS_PERSISTENT=TRUE chromadb/chroma:latest -``` - -2. Edit the `run.yaml` file to point to the chromadb server. -``` -memory: - - provider_id: remote::chromadb - provider_type: remote::chromadb - config: - host: localhost - port: 6000 -``` - -3. Run `docker compose up` with the updated `run.yaml` file. diff --git a/docs/source/index.md b/docs/source/index.md index 77afd9d22..532e0fa20 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -15,7 +15,7 @@ Our goal is to provide pre-packaged implementations (aka "distributions") which - New to Llama Stack? Start with the [Introduction](introduction/index) to understand our motivation and vision. - Ready to build? Check out the [Quick Start](getting_started/index) to get started. -- Need specific providers? Browse [Distributions](distributions/index) to see all the options available. +- Need specific providers? Browse [Distributions](distributions/selection) to see all the options available. - Want to contribute? See the [Contributing](contributing/index) guide. ## Available SDKs @@ -60,7 +60,6 @@ concepts/index distributions/index distributions/selection building_applications/index -benchmark_evaluations/index playground/index contributing/index references/index