diff --git a/docs/source/benchmark_evaluations/index.md b/docs/source/building_applications/evals.md similarity index 95% rename from docs/source/benchmark_evaluations/index.md rename to docs/source/building_applications/evals.md index 56852c89c..511a3d31d 100644 --- a/docs/source/benchmark_evaluations/index.md +++ b/docs/source/building_applications/evals.md @@ -1,8 +1,8 @@ -# Benchmark Evaluations +# Evals [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/10CHyykee9j2OigaIcRv47BKG9mrNm0tJ?usp=sharing) -Llama Stack provides the building blocks needed to run benchmark and application evaluations. This guide will walk you through how to use these components to run open benchmark evaluations. Visit our [Evaluation Concepts](../concepts/evaluation_concepts.md) guide for more details on how evaluations work in Llama Stack, and our [Evaluation Reference](../references/evals_reference/index.md) guide for a comprehensive reference on the APIs. Check out our [Colab notebook](https://colab.research.google.com/drive/10CHyykee9j2OigaIcRv47BKG9mrNm0tJ?usp=sharing) on working examples on how you can use Llama Stack for running benchmark evaluations. +Llama Stack provides the building blocks needed to run benchmark and application evaluations. This guide will walk you through how to use these components to run open benchmark evaluations. Visit our [Evaluation Concepts](../concepts/evaluation_concepts.md) guide for more details on how evaluations work in Llama Stack, and our [Evaluation Reference](../references/evals_reference/index.md) guide for a comprehensive reference on the APIs. ### 1. Open Benchmark Model Evaluation diff --git a/docs/source/building_applications/index.md b/docs/source/building_applications/index.md index 6e1e9454f..55485ddbc 100644 --- a/docs/source/building_applications/index.md +++ b/docs/source/building_applications/index.md @@ -6,12 +6,14 @@ The best way to get started is to look at this notebook which walks through the **Notebook**: [Building AI Applications](docs/notebooks/Llama_Stack_Building_AI_Applications.ipynb) -## Agentic Concepts +Here are some key topics that will help you build effective agents: + - **[Agent Execution Loop](agent_execution_loop)** - **[RAG](rag)** - **[Safety](safety)** - **[Tools](tools)** - **[Telemetry](telemetry)** +- **[Evals](evals)** ```{toctree} @@ -23,4 +25,5 @@ rag safety tools telemetry +evals ``` diff --git a/docs/source/contributing/memory_api.md b/docs/source/contributing/memory_api.md deleted file mode 100644 index be486ae8f..000000000 --- a/docs/source/contributing/memory_api.md +++ /dev/null @@ -1,53 +0,0 @@ -# Memory API Providers - -This guide gives you references to switch between different memory API providers. - -##### pgvector -1. Start running the pgvector server: - -``` -$ docker run --network host --name mypostgres -it -p 5432:5432 -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres pgvector/pgvector:pg16 -``` - -2. Edit the `run.yaml` file to point to the pgvector server. -``` -memory: - - provider_id: pgvector - provider_type: remote::pgvector - config: - host: 127.0.0.1 - port: 5432 - db: postgres - user: postgres - password: mysecretpassword -``` - -> [!NOTE] -> If you get a `RuntimeError: Vector extension is not installed.`. You will need to run `CREATE EXTENSION IF NOT EXISTS vector;` to include the vector extension. E.g. - -``` -docker exec -it mypostgres ./bin/psql -U postgres -postgres=# CREATE EXTENSION IF NOT EXISTS vector; -postgres=# SELECT extname from pg_extension; - extname -``` - -3. Run `docker compose up` with the updated `run.yaml` file. - -##### chromadb -1. Start running chromadb server -``` -docker run -it --network host --name chromadb -p 6000:6000 -v ./chroma_vdb:/chroma/chroma -e IS_PERSISTENT=TRUE chromadb/chroma:latest -``` - -2. Edit the `run.yaml` file to point to the chromadb server. -``` -memory: - - provider_id: remote::chromadb - provider_type: remote::chromadb - config: - host: localhost - port: 6000 -``` - -3. Run `docker compose up` with the updated `run.yaml` file. diff --git a/docs/source/index.md b/docs/source/index.md index bc4666be3..41910a973 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -59,7 +59,6 @@ concepts/index distributions/index distributions/selection building_applications/index -benchmark_evaluations/index playground/index contributing/index references/index