mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-06-27 18:50:41 +00:00
0883944bc3
Some checks failed
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 5s
Integration Tests / test-matrix (http, 3.12, agents) (push) Failing after 5s
Integration Tests / test-matrix (http, 3.12, post_training) (push) Failing after 6s
Integration Tests / test-matrix (http, 3.12, inspect) (push) Failing after 7s
Integration Tests / test-matrix (http, 3.12, inference) (push) Failing after 9s
Integration Tests / test-matrix (http, 3.12, providers) (push) Failing after 9s
Integration Tests / test-matrix (http, 3.12, vector_io) (push) Failing after 7s
Integration Tests / test-matrix (http, 3.13, inspect) (push) Failing after 6s
Integration Tests / test-matrix (http, 3.13, tool_runtime) (push) Failing after 6s
Integration Tests / test-matrix (http, 3.13, inference) (push) Failing after 9s
Integration Tests / test-matrix (http, 3.12, tool_runtime) (push) Failing after 11s
Integration Tests / test-matrix (library, 3.12, datasets) (push) Failing after 6s
Integration Tests / test-matrix (http, 3.13, scoring) (push) Failing after 10s
Integration Tests / test-matrix (http, 3.13, post_training) (push) Failing after 12s
Integration Tests / test-matrix (http, 3.13, agents) (push) Failing after 14s
Integration Tests / test-matrix (library, 3.12, inference) (push) Failing after 9s
Integration Tests / test-matrix (http, 3.13, vector_io) (push) Failing after 11s
Integration Tests / test-matrix (library, 3.12, post_training) (push) Failing after 7s
Integration Tests / test-matrix (library, 3.12, providers) (push) Failing after 7s
Integration Tests / test-matrix (library, 3.12, inspect) (push) Failing after 9s
Integration Tests / test-matrix (library, 3.12, tool_runtime) (push) Failing after 6s
Integration Tests / test-matrix (library, 3.13, agents) (push) Failing after 6s
Integration Tests / test-matrix (library, 3.13, inference) (push) Failing after 6s
Integration Tests / test-matrix (http, 3.12, datasets) (push) Failing after 25s
Integration Tests / test-matrix (library, 3.13, inspect) (push) Failing after 8s
Integration Tests / test-matrix (http, 3.12, scoring) (push) Failing after 25s
Integration Tests / test-matrix (http, 3.13, datasets) (push) Failing after 25s
Integration Tests / test-matrix (http, 3.13, providers) (push) Failing after 23s
Integration Tests / test-matrix (library, 3.12, scoring) (push) Failing after 17s
Integration Tests / test-matrix (library, 3.12, vector_io) (push) Failing after 15s
Integration Tests / test-matrix (library, 3.13, datasets) (push) Failing after 13s
Integration Tests / test-matrix (library, 3.13, providers) (push) Failing after 9s
Integration Tests / test-matrix (library, 3.13, scoring) (push) Failing after 8s
Integration Tests / test-matrix (library, 3.13, tool_runtime) (push) Failing after 7s
Integration Tests / test-matrix (library, 3.13, post_training) (push) Failing after 12s
Vector IO Integration Tests / test-matrix (3.12, remote::pgvector) (push) Failing after 4s
Vector IO Integration Tests / test-matrix (3.12, inline::faiss) (push) Failing after 9s
Integration Tests / test-matrix (library, 3.12, agents) (push) Failing after 28s
Python Package Build Test / build (3.13) (push) Failing after 2s
Vector IO Integration Tests / test-matrix (3.13, remote::chromadb) (push) Failing after 8s
Test Llama Stack Build / generate-matrix (push) Successful in 6s
Test Llama Stack Build / build-custom-container-distribution (push) Failing after 5s
Test External Providers / test-external-providers (venv) (push) Failing after 3s
Unit Tests / unit-tests (3.12) (push) Failing after 5s
Python Package Build Test / build (3.12) (push) Failing after 9s
Test Llama Stack Build / build-single-provider (push) Failing after 11s
Vector IO Integration Tests / test-matrix (3.12, remote::chromadb) (push) Failing after 18s
Vector IO Integration Tests / test-matrix (3.13, inline::sqlite-vec) (push) Failing after 16s
Test Llama Stack Build / build (push) Failing after 6s
Unit Tests / unit-tests (3.13) (push) Failing after 8s
Integration Tests / test-matrix (library, 3.13, vector_io) (push) Failing after 34s
Vector IO Integration Tests / test-matrix (3.13, inline::faiss) (push) Failing after 30s
Vector IO Integration Tests / test-matrix (3.12, inline::sqlite-vec) (push) Failing after 32s
Test Llama Stack Build / build-ubi9-container-distribution (push) Failing after 24s
Vector IO Integration Tests / test-matrix (3.13, remote::pgvector) (push) Failing after 29s
Pre-commit / pre-commit (push) Successful in 1m1s
# What does this PR do?
Some templates were still using the old environment variable substition
syntax instead of the new one and were not getting substituted properly.
Also, some places didn't handle the new None vs old empty string ("")
values that come from the conditional environment variable substitution.
This gets the starter and remote-vllm distributions starting again, and
I tested various permutations of the starter as chroma and pgvector
needed some adjustments to their config classes to handle the new
possible `None` values. And, I had to tweak our `Provider` class to also
handle `None` values, for cases where we disable providers in the
starter config via environment variables.
This may not have caught everything that was missed, but I did grep
around quite a bit to try and find anything lingering.
## Test Plan
The following permutations now all run (or attempt to run to the point
of complaining that they can't connect to chroma, vllm, etc) when before
they failed immediately on startup because of bad environment variable
substitions:
```
uv run llama stack run llama_stack/templates/starter/run.yaml
ENABLE_SQLITE_VEC=true uv run llama stack run llama_stack/templates/starter/run.yaml
ENABLE_PGVECTOR=true uv run llama stack run llama_stack/templates/starter/run.yaml
ENABLE_CHROMADB=true uv run llama stack run llama_stack/templates/starter/run.yaml
uv run llama stack run llama_stack/templates/remote-vllm/run.yaml
```
<!-- Describe the tests you ran to verify your changes with result
summaries. *Provide clear instructions so the plan can be easily
re-executed.* -->
Signed-off-by: Ben Browning <bbrownin@redhat.com>
Co-authored-by: raghotham <rsm@meta.com>
826 lines
26 KiB
YAML
826 lines
26 KiB
YAML
version: 2
|
|
image_name: starter
|
|
apis:
|
|
- agents
|
|
- datasetio
|
|
- eval
|
|
- files
|
|
- inference
|
|
- safety
|
|
- scoring
|
|
- telemetry
|
|
- tool_runtime
|
|
- vector_io
|
|
providers:
|
|
inference:
|
|
- provider_id: openai
|
|
provider_type: remote::openai
|
|
config:
|
|
api_key: ${env.OPENAI_API_KEY:+}
|
|
- provider_id: fireworks
|
|
provider_type: remote::fireworks
|
|
config:
|
|
url: https://api.fireworks.ai/inference/v1
|
|
api_key: ${env.FIREWORKS_API_KEY:+}
|
|
- provider_id: together
|
|
provider_type: remote::together
|
|
config:
|
|
url: https://api.together.xyz/v1
|
|
api_key: ${env.TOGETHER_API_KEY:+}
|
|
- provider_id: ollama
|
|
provider_type: remote::ollama
|
|
config:
|
|
url: ${env.OLLAMA_URL:=http://localhost:11434}
|
|
raise_on_connect_error: false
|
|
- provider_id: anthropic
|
|
provider_type: remote::anthropic
|
|
config:
|
|
api_key: ${env.ANTHROPIC_API_KEY:+}
|
|
- provider_id: gemini
|
|
provider_type: remote::gemini
|
|
config:
|
|
api_key: ${env.GEMINI_API_KEY:+}
|
|
- provider_id: groq
|
|
provider_type: remote::groq
|
|
config:
|
|
url: https://api.groq.com
|
|
api_key: ${env.GROQ_API_KEY:+}
|
|
- provider_id: sambanova
|
|
provider_type: remote::sambanova
|
|
config:
|
|
url: https://api.sambanova.ai/v1
|
|
api_key: ${env.SAMBANOVA_API_KEY:+}
|
|
- provider_id: vllm
|
|
provider_type: remote::vllm
|
|
config:
|
|
url: ${env.VLLM_URL:=http://localhost:8000/v1}
|
|
max_tokens: ${env.VLLM_MAX_TOKENS:=4096}
|
|
api_token: ${env.VLLM_API_TOKEN:=fake}
|
|
tls_verify: ${env.VLLM_TLS_VERIFY:=true}
|
|
- provider_id: sentence-transformers
|
|
provider_type: inline::sentence-transformers
|
|
config: {}
|
|
vector_io:
|
|
- provider_id: faiss
|
|
provider_type: inline::faiss
|
|
config:
|
|
kvstore:
|
|
type: sqlite
|
|
namespace: null
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/faiss_store.db
|
|
- provider_id: ${env.ENABLE_SQLITE_VEC:+sqlite-vec}
|
|
provider_type: inline::sqlite-vec
|
|
config:
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/sqlite_vec.db
|
|
- provider_id: ${env.ENABLE_CHROMADB:+chromadb}
|
|
provider_type: remote::chromadb
|
|
config:
|
|
url: ${env.CHROMADB_URL:+}
|
|
- provider_id: ${env.ENABLE_PGVECTOR:+pgvector}
|
|
provider_type: remote::pgvector
|
|
config:
|
|
host: ${env.PGVECTOR_HOST:=localhost}
|
|
port: ${env.PGVECTOR_PORT:=5432}
|
|
db: ${env.PGVECTOR_DB:+}
|
|
user: ${env.PGVECTOR_USER:+}
|
|
password: ${env.PGVECTOR_PASSWORD:+}
|
|
files:
|
|
- provider_id: meta-reference-files
|
|
provider_type: inline::localfs
|
|
config:
|
|
storage_dir: ${env.FILES_STORAGE_DIR:=~/.llama/distributions/starter/files}
|
|
metadata_store:
|
|
type: sqlite
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/files_metadata.db
|
|
safety:
|
|
- provider_id: llama-guard
|
|
provider_type: inline::llama-guard
|
|
config:
|
|
excluded_categories: []
|
|
agents:
|
|
- provider_id: meta-reference
|
|
provider_type: inline::meta-reference
|
|
config:
|
|
persistence_store:
|
|
type: sqlite
|
|
namespace: null
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/agents_store.db
|
|
responses_store:
|
|
type: sqlite
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/responses_store.db
|
|
telemetry:
|
|
- provider_id: meta-reference
|
|
provider_type: inline::meta-reference
|
|
config:
|
|
service_name: "${env.OTEL_SERVICE_NAME:=\u200B}"
|
|
sinks: ${env.TELEMETRY_SINKS:=console,sqlite}
|
|
sqlite_db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/trace_store.db
|
|
eval:
|
|
- provider_id: meta-reference
|
|
provider_type: inline::meta-reference
|
|
config:
|
|
kvstore:
|
|
type: sqlite
|
|
namespace: null
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/meta_reference_eval.db
|
|
datasetio:
|
|
- provider_id: huggingface
|
|
provider_type: remote::huggingface
|
|
config:
|
|
kvstore:
|
|
type: sqlite
|
|
namespace: null
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/huggingface_datasetio.db
|
|
- provider_id: localfs
|
|
provider_type: inline::localfs
|
|
config:
|
|
kvstore:
|
|
type: sqlite
|
|
namespace: null
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/localfs_datasetio.db
|
|
scoring:
|
|
- provider_id: basic
|
|
provider_type: inline::basic
|
|
config: {}
|
|
- provider_id: llm-as-judge
|
|
provider_type: inline::llm-as-judge
|
|
config: {}
|
|
- provider_id: braintrust
|
|
provider_type: inline::braintrust
|
|
config:
|
|
openai_api_key: ${env.OPENAI_API_KEY:+}
|
|
tool_runtime:
|
|
- provider_id: brave-search
|
|
provider_type: remote::brave-search
|
|
config:
|
|
api_key: ${env.BRAVE_SEARCH_API_KEY:+}
|
|
max_results: 3
|
|
- provider_id: tavily-search
|
|
provider_type: remote::tavily-search
|
|
config:
|
|
api_key: ${env.TAVILY_SEARCH_API_KEY:+}
|
|
max_results: 3
|
|
- provider_id: rag-runtime
|
|
provider_type: inline::rag-runtime
|
|
config: {}
|
|
- provider_id: model-context-protocol
|
|
provider_type: remote::model-context-protocol
|
|
config: {}
|
|
metadata_store:
|
|
type: sqlite
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/registry.db
|
|
inference_store:
|
|
type: sqlite
|
|
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/distributions/starter}/inference_store.db
|
|
models:
|
|
- metadata: {}
|
|
model_id: openai/gpt-4o
|
|
provider_id: openai
|
|
provider_model_id: openai/gpt-4o
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-4o-mini
|
|
provider_id: openai
|
|
provider_model_id: openai/gpt-4o-mini
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/chatgpt-4o-latest
|
|
provider_id: openai
|
|
provider_model_id: openai/chatgpt-4o-latest
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-3.5-turbo-0125
|
|
provider_id: openai
|
|
provider_model_id: gpt-3.5-turbo-0125
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-3.5-turbo
|
|
provider_id: openai
|
|
provider_model_id: gpt-3.5-turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-3.5-turbo-instruct
|
|
provider_id: openai
|
|
provider_model_id: gpt-3.5-turbo-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-4
|
|
provider_id: openai
|
|
provider_model_id: gpt-4
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-4-turbo
|
|
provider_id: openai
|
|
provider_model_id: gpt-4-turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-4o
|
|
provider_id: openai
|
|
provider_model_id: gpt-4o
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-4o-2024-08-06
|
|
provider_id: openai
|
|
provider_model_id: gpt-4o-2024-08-06
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-4o-mini
|
|
provider_id: openai
|
|
provider_model_id: gpt-4o-mini
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/gpt-4o-audio-preview
|
|
provider_id: openai
|
|
provider_model_id: gpt-4o-audio-preview
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/chatgpt-4o-latest
|
|
provider_id: openai
|
|
provider_model_id: chatgpt-4o-latest
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/o1
|
|
provider_id: openai
|
|
provider_model_id: o1
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/o1-mini
|
|
provider_id: openai
|
|
provider_model_id: o1-mini
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/o3-mini
|
|
provider_id: openai
|
|
provider_model_id: o3-mini
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: openai/o4-mini
|
|
provider_id: openai
|
|
provider_model_id: o4-mini
|
|
model_type: llm
|
|
- metadata:
|
|
embedding_dimension: 1536
|
|
context_length: 8192
|
|
model_id: openai/text-embedding-3-small
|
|
provider_id: openai
|
|
provider_model_id: openai/text-embedding-3-small
|
|
model_type: embedding
|
|
- metadata:
|
|
embedding_dimension: 3072
|
|
context_length: 8192
|
|
model_id: openai/text-embedding-3-large
|
|
provider_id: openai
|
|
provider_model_id: openai/text-embedding-3-large
|
|
model_type: embedding
|
|
- metadata:
|
|
embedding_dimension: 1536
|
|
context_length: 8192
|
|
model_id: openai/text-embedding-3-small
|
|
provider_id: openai
|
|
provider_model_id: text-embedding-3-small
|
|
model_type: embedding
|
|
- metadata:
|
|
embedding_dimension: 3072
|
|
context_length: 8192
|
|
model_id: openai/text-embedding-3-large
|
|
provider_id: openai
|
|
provider_model_id: text-embedding-3-large
|
|
model_type: embedding
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-v3p1-8b-instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p1-8b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-3.1-8B-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p1-8b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-v3p1-70b-instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p1-70b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-3.1-70B-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p1-70b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-v3p1-405b-instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p1-405b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-3.1-405B-Instruct-FP8
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p1-405b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-v3p2-3b-instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p2-3b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-3.2-3B-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p2-3b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-v3p2-11b-vision-instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p2-11b-vision-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-3.2-11B-Vision-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p2-11b-vision-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-v3p2-90b-vision-instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p2-90b-vision-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-3.2-90B-Vision-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p2-90b-vision-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-v3p3-70b-instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p3-70b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-3.3-70B-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-v3p3-70b-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-guard-3-8b
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-guard-3-8b
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-Guard-3-8B
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-guard-3-8b
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama-guard-3-11b-vision
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-guard-3-11b-vision
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-Guard-3-11B-Vision
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama-guard-3-11b-vision
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama4-scout-instruct-basic
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama4-scout-instruct-basic
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama4-scout-instruct-basic
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: accounts/fireworks/models/llama4-maverick-instruct-basic
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama4-maverick-instruct-basic
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: fireworks/meta-llama/Llama-4-Maverick-17B-128E-Instruct
|
|
provider_id: fireworks
|
|
provider_model_id: accounts/fireworks/models/llama4-maverick-instruct-basic
|
|
model_type: llm
|
|
- metadata:
|
|
embedding_dimension: 768
|
|
context_length: 8192
|
|
model_id: fireworks/nomic-ai/nomic-embed-text-v1.5
|
|
provider_id: fireworks
|
|
provider_model_id: nomic-ai/nomic-embed-text-v1.5
|
|
model_type: embedding
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.1-8B-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.1-70B-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.1-405B-Instruct-FP8
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.2-3B-Instruct-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.2-3B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.2-3B-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.2-3B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.2-11B-Vision-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.2-90B-Vision-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.3-70B-Instruct-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.3-70B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-3.3-70B-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-3.3-70B-Instruct-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Meta-Llama-Guard-3-8B
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-Guard-3-8B
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-Guard-3-8B
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Meta-Llama-Guard-3-8B
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-Guard-3-11B-Vision-Turbo
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-Guard-3-11B-Vision-Turbo
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-Guard-3-11B-Vision
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-Guard-3-11B-Vision-Turbo
|
|
model_type: llm
|
|
- metadata:
|
|
embedding_dimension: 768
|
|
context_length: 8192
|
|
model_id: togethercomputer/m2-bert-80M-8k-retrieval
|
|
provider_id: together
|
|
provider_model_id: togethercomputer/m2-bert-80M-8k-retrieval
|
|
model_type: embedding
|
|
- metadata:
|
|
embedding_dimension: 768
|
|
context_length: 32768
|
|
model_id: togethercomputer/m2-bert-80M-32k-retrieval
|
|
provider_id: together
|
|
provider_model_id: togethercomputer/m2-bert-80M-32k-retrieval
|
|
model_type: embedding
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-4-Maverick-17B-128E-Instruct
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: together/meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
|
provider_id: together
|
|
provider_model_id: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: ollama/${env.OLLAMA_INFERENCE_MODEL:=__disabled__}
|
|
provider_id: ollama
|
|
provider_model_id: ${env.OLLAMA_INFERENCE_MODEL:=__disabled__}
|
|
model_type: llm
|
|
- metadata:
|
|
embedding_dimension: ${env.OLLAMA_EMBEDDING_DIMENSION:=384}
|
|
model_id: ollama/${env.OLLAMA_EMBEDDING_MODEL:=__disabled__}
|
|
provider_id: ollama
|
|
provider_model_id: ${env.OLLAMA_EMBEDDING_MODEL:=__disabled__}
|
|
model_type: embedding
|
|
- metadata: {}
|
|
model_id: anthropic/claude-3-5-sonnet-latest
|
|
provider_id: anthropic
|
|
provider_model_id: anthropic/claude-3-5-sonnet-latest
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: anthropic/claude-3-7-sonnet-latest
|
|
provider_id: anthropic
|
|
provider_model_id: anthropic/claude-3-7-sonnet-latest
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: anthropic/claude-3-5-haiku-latest
|
|
provider_id: anthropic
|
|
provider_model_id: anthropic/claude-3-5-haiku-latest
|
|
model_type: llm
|
|
- metadata:
|
|
embedding_dimension: 1024
|
|
context_length: 32000
|
|
model_id: anthropic/voyage-3
|
|
provider_id: anthropic
|
|
provider_model_id: anthropic/voyage-3
|
|
model_type: embedding
|
|
- metadata:
|
|
embedding_dimension: 512
|
|
context_length: 32000
|
|
model_id: anthropic/voyage-3-lite
|
|
provider_id: anthropic
|
|
provider_model_id: anthropic/voyage-3-lite
|
|
model_type: embedding
|
|
- metadata:
|
|
embedding_dimension: 1024
|
|
context_length: 32000
|
|
model_id: anthropic/voyage-code-3
|
|
provider_id: anthropic
|
|
provider_model_id: anthropic/voyage-code-3
|
|
model_type: embedding
|
|
- metadata: {}
|
|
model_id: gemini/gemini-1.5-flash
|
|
provider_id: gemini
|
|
provider_model_id: gemini/gemini-1.5-flash
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: gemini/gemini-1.5-pro
|
|
provider_id: gemini
|
|
provider_model_id: gemini/gemini-1.5-pro
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: gemini/gemini-2.0-flash
|
|
provider_id: gemini
|
|
provider_model_id: gemini/gemini-2.0-flash
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: gemini/gemini-2.5-flash
|
|
provider_id: gemini
|
|
provider_model_id: gemini/gemini-2.5-flash
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: gemini/gemini-2.5-pro
|
|
provider_id: gemini
|
|
provider_model_id: gemini/gemini-2.5-pro
|
|
model_type: llm
|
|
- metadata:
|
|
embedding_dimension: 768
|
|
context_length: 2048
|
|
model_id: gemini/text-embedding-004
|
|
provider_id: gemini
|
|
provider_model_id: gemini/text-embedding-004
|
|
model_type: embedding
|
|
- metadata: {}
|
|
model_id: groq/llama3-8b-8192
|
|
provider_id: groq
|
|
provider_model_id: groq/llama3-8b-8192
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-3.1-8B-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama3-8b-8192
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/llama-3.1-8b-instant
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-3.1-8b-instant
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/llama3-70b-8192
|
|
provider_id: groq
|
|
provider_model_id: groq/llama3-70b-8192
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-3-70B-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama3-70b-8192
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/llama-3.3-70b-versatile
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-3.3-70b-versatile
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-3.3-70B-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-3.3-70b-versatile
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/llama-3.2-3b-preview
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-3.2-3b-preview
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-3.2-3B-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-3.2-3b-preview
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/llama-4-scout-17b-16e-instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-4-scout-17b-16e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-4-scout-17b-16e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/llama-4-scout-17b-16e-instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/meta-llama/llama-4-scout-17b-16e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/meta-llama/llama-4-scout-17b-16e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/llama-4-maverick-17b-128e-instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-4-maverick-17b-128e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-4-Maverick-17B-128E-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/llama-4-maverick-17b-128e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/llama-4-maverick-17b-128e-instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/meta-llama/llama-4-maverick-17b-128e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: groq/meta-llama/Llama-4-Maverick-17B-128E-Instruct
|
|
provider_id: groq
|
|
provider_model_id: groq/meta-llama/llama-4-maverick-17b-128e-instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Meta-Llama-3.1-8B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.1-8B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-3.1-8B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.1-8B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Meta-Llama-3.1-405B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.1-405B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-3.1-405B-Instruct-FP8
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.1-405B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Meta-Llama-3.2-1B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.2-1B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-3.2-1B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.2-1B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Meta-Llama-3.2-3B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.2-3B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-3.2-3B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.2-3B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Meta-Llama-3.3-70B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.3-70B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-3.3-70B-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-3.3-70B-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Llama-3.2-11B-Vision-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-3.2-11B-Vision-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-3.2-11B-Vision-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-3.2-11B-Vision-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Llama-3.2-90B-Vision-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-3.2-90B-Vision-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-3.2-90B-Vision-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-3.2-90B-Vision-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-4-Scout-17B-16E-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-4-Scout-17B-16E-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-4-Scout-17B-16E-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Llama-4-Maverick-17B-128E-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-4-Maverick-17B-128E-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-4-Maverick-17B-128E-Instruct
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Llama-4-Maverick-17B-128E-Instruct
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/Meta-Llama-Guard-3-8B
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-Guard-3-8B
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: sambanova/meta-llama/Llama-Guard-3-8B
|
|
provider_id: sambanova
|
|
provider_model_id: sambanova/Meta-Llama-Guard-3-8B
|
|
model_type: llm
|
|
- metadata: {}
|
|
model_id: vllm/${env.VLLM_INFERENCE_MODEL:=__disabled__}
|
|
provider_id: vllm
|
|
provider_model_id: ${env.VLLM_INFERENCE_MODEL:=__disabled__}
|
|
model_type: llm
|
|
- metadata:
|
|
embedding_dimension: 384
|
|
model_id: all-MiniLM-L6-v2
|
|
provider_id: sentence-transformers
|
|
model_type: embedding
|
|
shields:
|
|
- shield_id: meta-llama/Llama-Guard-3-8B
|
|
vector_dbs: []
|
|
datasets: []
|
|
scoring_fns: []
|
|
benchmarks: []
|
|
tool_groups:
|
|
- toolgroup_id: builtin::websearch
|
|
provider_id: tavily-search
|
|
- toolgroup_id: builtin::rag
|
|
provider_id: rag-runtime
|
|
server:
|
|
port: 8321
|