Merge 407c3e3bad into 632cf9eb72

2025-07-27 06:28:50 +00:00 · 2025-07-24 23:57:19 +02:00 · 2025-07-24 23:57:19 +02:00 · 172d578b20
commit 172d578b20
parent 632cf9eb72 407c3e3bad
50 changed files with 5611 additions and 508 deletions
--- a/docs/source/distributions/self_hosted_distro/ollama.md
+++ b/docs/source/distributions/self_hosted_distro/ollama.md
@ -0,0 +1,172 @@
+<!-- This file was auto-generated by distro_codegen.py, please edit source -->
+# Ollama Distribution
+
+The `llamastack/distribution-ollama` distribution consists of the following provider configurations.
+
+| API | Provider(s) |
+|-----|-------------|
+| agents | `inline::meta-reference` |
+| datasetio | `remote::huggingface`, `inline::localfs` |
+| eval | `inline::meta-reference` |
+| files | `inline::localfs` |
+| inference | `remote::ollama` |
+| post_training | `inline::huggingface` |
+| safety | `inline::llama-guard` |
+| scoring | `inline::basic`, `inline::llm-as-judge`, `inline::braintrust` |
+| telemetry | `inline::meta-reference` |
+| tool_runtime | `remote::brave-search`, `remote::tavily-search`, `inline::rag-runtime`, `remote::model-context-protocol`, `remote::wolfram-alpha` |
+| vector_io | `inline::faiss`, `remote::chromadb`, `remote::pgvector` |
+
+
+### Environment Variables
+
+The following environment variables can be configured:
+
+- `LLAMA_STACK_PORT`: Port for the Llama Stack distribution server (default: `8321`)
+- `OLLAMA_URL`: URL of the Ollama server (default: `http://127.0.0.1:11434`)
+- `INFERENCE_MODEL`: Inference model loaded into the Ollama server (default: `meta-llama/Llama-3.2-3B-Instruct`)
+- `SAFETY_MODEL`: Safety model loaded into the Ollama server (default: `meta-llama/Llama-Guard-3-1B`)
+
+
+## Prerequisites
+
+### Ollama Server
+
+This distribution requires an external Ollama server to be running. You can install and run Ollama by following these steps:
+
+1. **Install Ollama**: Download and install Ollama from [https://ollama.ai/](https://ollama.ai/)
+
+2. **Start the Ollama server**:
+   ```bash
+   ollama serve
+   ```
+   By default, Ollama serves on `http://127.0.0.1:11434`
+
+3. **Pull the required models**:
+   ```bash
+   # Pull the inference model
+   ollama pull meta-llama/Llama-3.2-3B-Instruct
+
+   # Pull the embedding model
+   ollama pull all-minilm:latest
+
+   # (Optional) Pull the safety model for run-with-safety.yaml
+   ollama pull meta-llama/Llama-Guard-3-1B
+   ```
+
+## Supported Services
+
+### Inference: Ollama
+Uses an external Ollama server for running LLM inference. The server should be accessible at the URL specified in the `OLLAMA_URL` environment variable.
+
+### Vector IO: FAISS
+Provides vector storage capabilities using FAISS for embeddings and similarity search operations.
+
+### Safety: Llama Guard (Optional)
+When using the `run-with-safety.yaml` configuration, provides safety checks using Llama Guard models running on the Ollama server.
+
+### Agents: Meta Reference
+Provides agent execution capabilities using the meta-reference implementation.
+
+### Post-Training: Hugging Face
+Supports model fine-tuning using Hugging Face integration.
+
+### Tool Runtime
+Supports various external tools including:
+- Brave Search
+- Tavily Search
+- RAG Runtime
+- Model Context Protocol
+- Wolfram Alpha
+
+## Running Llama Stack with Ollama
+
+You can do this via Conda or venv (build code), or Docker which has a pre-built image.
+
+### Via Docker
+
+This method allows you to get started quickly without having to build the distribution code.
+
+```bash
+LLAMA_STACK_PORT=8321
+docker run \
+  -it \
+  --pull always \
+  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
+  -v ./run.yaml:/root/my-run.yaml \
+  llamastack/distribution-ollama \
+  --config /root/my-run.yaml \
+  --port $LLAMA_STACK_PORT \
+  --env OLLAMA_URL=$OLLAMA_URL \
+  --env INFERENCE_MODEL=$INFERENCE_MODEL
+```
+
+### Via Conda
+
+```bash
+llama stack build --template ollama --image-type conda
+llama stack run ./run.yaml \
+  --port 8321 \
+  --env OLLAMA_URL=$OLLAMA_URL \
+  --env INFERENCE_MODEL=$INFERENCE_MODEL
+```
+
+### Via venv
+
+If you've set up your local development environment, you can also build the image using your local virtual environment.
+
+```bash
+llama stack build --template ollama --image-type venv
+llama stack run ./run.yaml \
+  --port 8321 \
+  --env OLLAMA_URL=$OLLAMA_URL \
+  --env INFERENCE_MODEL=$INFERENCE_MODEL
+```
+
+### Running with Safety
+
+To enable safety checks, use the `run-with-safety.yaml` configuration:
+
+```bash
+llama stack run ./run-with-safety.yaml \
+  --port 8321 \
+  --env OLLAMA_URL=$OLLAMA_URL \
+  --env INFERENCE_MODEL=$INFERENCE_MODEL \
+  --env SAFETY_MODEL=$SAFETY_MODEL
+```
+
+## Example Usage
+
+Once your Llama Stack server is running with Ollama, you can interact with it using the Llama Stack client:
+
+```python
+from llama_stack_client import LlamaStackClient
+
+client = LlamaStackClient(base_url="http://localhost:8321")
+
+# Run inference
+response = client.inference.chat_completion(
+    model_id="meta-llama/Llama-3.2-3B-Instruct",
+    messages=[{"role": "user", "content": "Hello, how are you?"}],
+)
+print(response.completion_message.content)
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Connection refused errors**: Ensure your Ollama server is running and accessible at the configured URL.
+
+2. **Model not found errors**: Make sure you've pulled the required models using `ollama pull <model-name>`.
+
+3. **Performance issues**: Consider using more powerful models or adjusting the Ollama server configuration for better performance.
+
+### Logs
+
+Check the Ollama server logs for any issues:
+```bash
+# Ollama logs are typically available in:
+# - macOS: ~/Library/Logs/Ollama/
+# - Linux: ~/.ollama/logs/
+```
--- a/docs/source/getting_started/xdg_compliance.md
+++ b/docs/source/getting_started/xdg_compliance.md
@ -0,0 +1,191 @@
+# XDG Base Directory Specification Compliance
+
+Starting with version 0.2.14, Llama Stack follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) for organizing configuration and data files. This provides better integration with modern desktop environments and allows for more flexible customization of where files are stored.
+
+## Overview
+
+The XDG Base Directory Specification defines standard locations for different types of application data:
+
+- **Configuration files** (`XDG_CONFIG_HOME`): User-specific configuration files
+- **Data files** (`XDG_DATA_HOME`): User-specific data files that should persist
+- **Cache files** (`XDG_CACHE_HOME`): User-specific cache files
+- **State files** (`XDG_STATE_HOME`): User-specific state files
+
+## Directory Mapping
+
+Llama Stack now uses the following XDG-compliant directory structure:
+
+| Data Type | XDG Directory | Default Location | Description |
+|-----------|---------------|------------------|-------------|
+| Configuration | `XDG_CONFIG_HOME` | `~/.config/llama-stack` | Distribution configs, provider configs |
+| Data | `XDG_DATA_HOME` | `~/.local/share/llama-stack` | Model checkpoints, persistent files |
+| Cache | `XDG_CACHE_HOME` | `~/.cache/llama-stack` | Temporary cache files |
+| State | `XDG_STATE_HOME` | `~/.local/state/llama-stack` | Runtime state, databases |
+
+## Environment Variables
+
+You can customize the locations by setting these environment variables:
+
+```bash
+# Override the base directories
+export XDG_CONFIG_HOME="/custom/config/path"
+export XDG_DATA_HOME="/custom/data/path"
+export XDG_CACHE_HOME="/custom/cache/path"
+export XDG_STATE_HOME="/custom/state/path"
+
+# Or override specific Llama Stack directories
+export SQLITE_STORE_DIR="/custom/database/path"
+export FILES_STORAGE_DIR="/custom/files/path"
+```
+
+## Backwards Compatibility
+
+Llama Stack maintains full backwards compatibility with existing installations:
+
+1. **Legacy Environment Variable**: If `LLAMA_STACK_CONFIG_DIR` is set, it will be used for all directories
+2. **Legacy Directory Detection**: If `~/.llama` exists and contains data, it will continue to be used
+3. **Gradual Migration**: New installations use XDG paths, existing installations continue to work
+
+## Migration Guide
+
+### Automatic Migration
+
+Use the built-in migration command to move from legacy `~/.llama` to XDG-compliant directories:
+
+```bash
+# Preview what would be migrated
+llama migrate-xdg --dry-run
+
+# Perform the migration
+llama migrate-xdg
+```
+
+### Manual Migration
+
+If you prefer to migrate manually, here's the mapping:
+
+```bash
+# Create XDG directories
+mkdir -p ~/.config/llama-stack
+mkdir -p ~/.local/share/llama-stack
+mkdir -p ~/.local/state/llama-stack
+
+# Move configuration files
+mv ~/.llama/distributions ~/.config/llama-stack/
+mv ~/.llama/providers.d ~/.config/llama-stack/
+
+# Move data files
+mv ~/.llama/checkpoints ~/.local/share/llama-stack/
+
+# Move state files
+mv ~/.llama/runtime ~/.local/state/llama-stack/
+
+# Clean up empty legacy directory
+rmdir ~/.llama
+```
+
+### Environment Variables in Configurations
+
+Template configurations now use XDG-compliant environment variables:
+
+```yaml
+# Old format
+db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/ollama}/registry.db
+
+# New format
+db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_STATE_HOME:-~/.local/state}/llama-stack/distributions/ollama}/registry.db
+```
+
+## Configuration Examples
+
+### Using Custom XDG Directories
+
+```bash
+# Set custom XDG directories
+export XDG_CONFIG_HOME="/opt/llama-stack/config"
+export XDG_DATA_HOME="/opt/llama-stack/data"
+export XDG_STATE_HOME="/opt/llama-stack/state"
+
+# Start Llama Stack
+llama stack run my-distribution.yaml
+```
+
+### Using Legacy Directory
+
+```bash
+# Continue using legacy directory
+export LLAMA_STACK_CONFIG_DIR="/home/user/.llama"
+
+# Start Llama Stack
+llama stack run my-distribution.yaml
+```
+
+### Custom Database and File Locations
+
+```bash
+# Override specific directories
+export SQLITE_STORE_DIR="/fast/ssd/llama-stack/databases"
+export FILES_STORAGE_DIR="/large/disk/llama-stack/files"
+
+# Start Llama Stack
+llama stack run my-distribution.yaml
+```
+
+## Benefits of XDG Compliance
+
+1. **Standards Compliance**: Follows established Linux/Unix conventions
+2. **Better Organization**: Separates configuration, data, cache, and state files
+3. **Flexibility**: Easy to customize storage locations
+4. **Backup-Friendly**: Easier to backup just data files or just configuration
+5. **Multi-User Support**: Better support for shared systems
+6. **Cross-Platform**: Works consistently across different environments
+
+## Template Updates
+
+All distribution templates have been updated to use XDG-compliant paths:
+
+- Database files use `XDG_STATE_HOME`
+- Model checkpoints use `XDG_DATA_HOME`
+- Configuration files use `XDG_CONFIG_HOME`
+- Cache files use `XDG_CACHE_HOME`
+
+## Troubleshooting
+
+### Migration Issues
+
+If you encounter issues during migration:
+
+1. **Check Permissions**: Ensure you have write permissions to target directories
+2. **Disk Space**: Verify sufficient disk space in target locations
+3. **Existing Files**: Handle conflicts with existing files in target locations
+
+### Environment Variable Conflicts
+
+If you have multiple environment variables set:
+
+1. `LLAMA_STACK_CONFIG_DIR` takes highest precedence
+2. Individual `XDG_*` variables override defaults
+3. Fallback to legacy `~/.llama` if it exists
+4. Default to XDG standard paths for new installations
+
+### Debugging Path Resolution
+
+To see which paths Llama Stack is using:
+
+```python
+from llama_stack.distribution.utils.xdg_utils import (
+    get_llama_stack_config_dir,
+    get_llama_stack_data_dir,
+    get_llama_stack_state_dir,
+)
+
+print(f"Config: {get_llama_stack_config_dir()}")
+print(f"Data: {get_llama_stack_data_dir()}")
+print(f"State: {get_llama_stack_state_dir()}")
+```
+
+## Future Considerations
+
+- Container deployments will continue to use `/app` or similar paths
+- Cloud deployments may use provider-specific storage systems
+- The XDG specification primarily applies to local development and single-user systems
--- a/docs/source/providers/agents/inline_meta-reference.md
+++ b/docs/source/providers/agents/inline_meta-reference.md
@ -16,10 +16,10 @@ Meta's reference implementation of an agent system that can use tools, access ve
 ```yaml
 persistence_store:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/agents_store.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/agents_store.db
 responses_store:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/responses_store.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/responses_store.db

 ```

--- a/docs/source/providers/datasetio/inline_localfs.md
+++ b/docs/source/providers/datasetio/inline_localfs.md
@ -15,7 +15,7 @@ Local filesystem-based dataset I/O provider for reading and writing datasets to
 ```yaml
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/localfs_datasetio.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/localfs_datasetio.db

 ```

--- a/docs/source/providers/datasetio/remote_huggingface.md
+++ b/docs/source/providers/datasetio/remote_huggingface.md
@ -15,7 +15,7 @@ HuggingFace datasets provider for accessing and managing datasets from the Huggi
 ```yaml
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/huggingface_datasetio.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/huggingface_datasetio.db

 ```

--- a/docs/source/providers/eval/inline_meta-reference.md
+++ b/docs/source/providers/eval/inline_meta-reference.md
@ -15,7 +15,7 @@ Meta's reference implementation of evaluation tasks with support for multiple la
 ```yaml
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/meta_reference_eval.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/meta_reference_eval.db

 ```

--- a/docs/source/providers/files/inline_localfs.md
+++ b/docs/source/providers/files/inline_localfs.md
@ -15,10 +15,10 @@ Local filesystem-based file storage provider for managing files and documents lo
 ## Sample Configuration

 ```yaml
-storage_dir: ${env.FILES_STORAGE_DIR:=~/.llama/dummy/files}
+storage_dir: ${env.FILES_STORAGE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy/files}
 metadata_store:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/files_metadata.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/files_metadata.db

 ```

--- a/docs/source/providers/telemetry/inline_meta-reference.md
+++ b/docs/source/providers/telemetry/inline_meta-reference.md
@ -11,14 +11,14 @@ Meta's reference implementation of telemetry and observability using OpenTelemet
 | `otel_exporter_otlp_endpoint` | `str \| None` | No |  | The OpenTelemetry collector endpoint URL (base URL for traces, metrics, and logs). If not set, the SDK will use OTEL_EXPORTER_OTLP_ENDPOINT environment variable. |
 | `service_name` | `<class 'str'>` | No |  | The service name to use for telemetry |
 | `sinks` | `list[inline.telemetry.meta_reference.config.TelemetrySink` | No | [<TelemetrySink.CONSOLE: 'console'>, <TelemetrySink.SQLITE: 'sqlite'>] | List of telemetry sinks to enable (possible values: otel_trace, otel_metric, sqlite, console) |
-| `sqlite_db_path` | `<class 'str'>` | No | ~/.llama/runtime/trace_store.db | The path to the SQLite database to use for storing traces |
+| `sqlite_db_path` | `<class 'str'>` | No | ${env.XDG_STATE_HOME:-~/.local/state}/llama-stack/runtime/trace_store.db | The path to the SQLite database to use for storing traces |

 ## Sample Configuration

 ```yaml
 service_name: "${env.OTEL_SERVICE_NAME:=\u200B}"
 sinks: ${env.TELEMETRY_SINKS:=console,sqlite}
-sqlite_db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/trace_store.db
+sqlite_db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/trace_store.db
 otel_exporter_otlp_endpoint: ${env.OTEL_EXPORTER_OTLP_ENDPOINT:=}

 ```
--- a/docs/source/providers/vector_io/inline_faiss.md
+++ b/docs/source/providers/vector_io/inline_faiss.md
@ -44,7 +44,7 @@ more details about Faiss in general.
 ```yaml
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/faiss_store.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/faiss_store.db

 ```

--- a/docs/source/providers/vector_io/inline_meta-reference.md
+++ b/docs/source/providers/vector_io/inline_meta-reference.md
@ -15,7 +15,7 @@ Meta's reference implementation of a vector database.
 ```yaml
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/faiss_store.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/faiss_store.db

 ```

--- a/docs/source/providers/vector_io/inline_milvus.md
+++ b/docs/source/providers/vector_io/inline_milvus.md
@ -17,10 +17,10 @@ Please refer to the remote provider documentation.
 ## Sample Configuration

 ```yaml
-db_path: ${env.MILVUS_DB_PATH:=~/.llama/dummy}/milvus.db
+db_path: ${env.MILVUS_DB_PATH:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/milvus.db
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/milvus_registry.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/milvus_registry.db

 ```

--- a/docs/source/providers/vector_io/inline_qdrant.md
+++ b/docs/source/providers/vector_io/inline_qdrant.md
@ -55,7 +55,7 @@ See the [Qdrant documentation](https://qdrant.tech/documentation/) for more deta
 ## Sample Configuration

 ```yaml
-path: ${env.QDRANT_PATH:=~/.llama/~/.llama/dummy}/qdrant.db
+path: ${env.QDRANT_PATH:=~/.llama/${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/qdrant.db

 ```

--- a/docs/source/providers/vector_io/inline_sqlite-vec.md
+++ b/docs/source/providers/vector_io/inline_sqlite-vec.md
@ -211,10 +211,10 @@ See [sqlite-vec's GitHub repo](https://github.com/asg017/sqlite-vec/tree/main) f
 ## Sample Configuration

 ```yaml
-db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/sqlite_vec.db
+db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/sqlite_vec.db
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/sqlite_vec_registry.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/sqlite_vec_registry.db

 ```

--- a/docs/source/providers/vector_io/inline_sqlite_vec.md
+++ b/docs/source/providers/vector_io/inline_sqlite_vec.md
@ -16,10 +16,10 @@ Please refer to the sqlite-vec provider documentation.
 ## Sample Configuration

 ```yaml
-db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/sqlite_vec.db
+db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/sqlite_vec.db
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/sqlite_vec_registry.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/sqlite_vec_registry.db

 ```

--- a/docs/source/providers/vector_io/remote_milvus.md
+++ b/docs/source/providers/vector_io/remote_milvus.md
@ -126,7 +126,7 @@ uri: ${env.MILVUS_ENDPOINT}
 token: ${env.MILVUS_TOKEN}
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/milvus_remote_registry.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/milvus_remote_registry.db

 ```

--- a/docs/source/providers/vector_io/remote_pgvector.md
+++ b/docs/source/providers/vector_io/remote_pgvector.md
@ -52,7 +52,7 @@ user: ${env.PGVECTOR_USER}
 password: ${env.PGVECTOR_PASSWORD}
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/pgvector_registry.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/pgvector_registry.db

 ```

--- a/docs/source/providers/vector_io/remote_weaviate.md
+++ b/docs/source/providers/vector_io/remote_weaviate.md
@ -38,7 +38,7 @@ See [Weaviate's documentation](https://weaviate.io/developers/weaviate) for more
 ```yaml
 kvstore:
  type: sqlite
-  db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/weaviate_registry.db
+  db_path: ${env.SQLITE_STORE_DIR:=${env.XDG_DATA_HOME:-~/.local/share}/llama-stack/dummy}/weaviate_registry.db

 ```