phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-12-23 03:12:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix: LLAMACPP_URL and LLAMACPP_API_KEY format

Browse source
This commit is contained in:
Young Han 2025-07-14 17:25:18 -07:00
parent b884bc490b
commit d3d50121cf
5 changed files with 7 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/source/providers/inference/remote_llamacpp.md
View file

@ -5,12 +5,12 @@
| Field | Type | Required | Default | Description | | Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------| |-------|------|----------|---------|-------------|
| `api_key` | `str \| None` | No | | The llama.cpp server API key (optional for local servers) | | `api_key` | `str \| None` | No | | The llama.cpp server API key (optional for local servers) |
| `openai_compat_api_base` | `<class 'str'>` | No | http://localhost:8080/v1 | The URL for the llama.cpp server with OpenAI-compatible API | | `openai_compat_api_base` | `<class 'str'>` | No | http://localhost:8080 | The URL for the llama.cpp server with OpenAI-compatible API |
## Sample Configuration ## Sample Configuration
```yaml ```yaml
openai_compat_api_base: ${env.LLAMACPP_URL:http://localhost:8080}/v1 openai_compat_api_base: ${env.LLAMACPP_URL:=http://localhost:8080}
api_key: ${env.LLAMACPP_API_KEY:=} api_key: ${env.LLAMACPP_API_KEY:=}
``` ```

4
llama_stack/providers/remote/inference/llamacpp/config.py
View file

@ -26,13 +26,13 @@ class LlamaCppImplConfig(BaseModel):
) )
openai_compat_api_base: str = Field( openai_compat_api_base: str = Field(
default="http://localhost:8080/v1", default="http://localhost:8080",
description="The URL for the llama.cpp server with OpenAI-compatible API", description="The URL for the llama.cpp server with OpenAI-compatible API",
) )
@classmethod @classmethod
def sample_run_config(cls, api_key: str = "${env.LLAMACPP_API_KEY:=}") -> dict[str, Any]: def sample_run_config(cls, api_key: str = "${env.LLAMACPP_API_KEY:=}") -> dict[str, Any]:
return { return {
"openai_compat_api_base": "${env.LLAMACPP_URL:http://localhost:8080}/v1", "openai_compat_api_base": "${env.LLAMACPP_URL:=http://localhost:8080}",
"api_key": api_key, "api_key": api_key,
} }

2
llama_stack/templates/llamacpp/run-with-safety.yaml
View file

@ -16,7 +16,7 @@ providers:
- provider_id: llamacpp - provider_id: llamacpp
provider_type: remote::llamacpp provider_type: remote::llamacpp
config: config:
openai_compat_api_base: ${env.LLAMACPP_URL:http://localhost:8080}/v1 openai_compat_api_base: ${env.LLAMACPP_URL:=http://localhost:8080}
api_key: ${env.LLAMACPP_API_KEY:=} api_key: ${env.LLAMACPP_API_KEY:=}
- provider_id: sentence-transformers - provider_id: sentence-transformers
provider_type: inline::sentence-transformers provider_type: inline::sentence-transformers

2
llama_stack/templates/llamacpp/run.yaml
View file

@ -16,7 +16,7 @@ providers:
- provider_id: llamacpp - provider_id: llamacpp
provider_type: remote::llamacpp provider_type: remote::llamacpp
config: config:
openai_compat_api_base: ${env.LLAMACPP_URL:http://localhost:8080}/v1 openai_compat_api_base: ${env.LLAMACPP_URL:=http://localhost:8080}
api_key: ${env.LLAMACPP_API_KEY:=} api_key: ${env.LLAMACPP_API_KEY:=}
- provider_id: sentence-transformers - provider_id: sentence-transformers
provider_type: inline::sentence-transformers provider_type: inline::sentence-transformers

2
llama_stack/templates/starter/run.yaml
View file

@ -133,7 +133,7 @@ providers:
- provider_id: ${env.ENABLE_LLAMACPP:=__disabled__} - provider_id: ${env.ENABLE_LLAMACPP:=__disabled__}
provider_type: remote::llamacpp provider_type: remote::llamacpp
config: config:
openai_compat_api_base: ${env.LLAMACPP_URL:http://localhost:8080}/v1 openai_compat_api_base: ${env.LLAMACPP_URL:=http://localhost:8080}
api_key: ${env.LLAMACPP_API_KEY:=} api_key: ${env.LLAMACPP_API_KEY:=}
- provider_id: ${env.ENABLE_SENTENCE_TRANSFORMERS:=sentence-transformers} - provider_id: ${env.ENABLE_SENTENCE_TRANSFORMERS:=sentence-transformers}
provider_type: inline::sentence-transformers provider_type: inline::sentence-transformers