mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-06-28 02:53:30 +00:00
add NVIDIA_BASE_URL and NVIDIA_API_KEY to control hosted vs local endpoints (#897)
# What does this PR do? allows template distribution connect to hosted or local NIM: use --env NVIDIA_BASE_URL=http://localhost:8000 to connect to a local NIM running at localhost:8000 use --env NVIDIA_API_KEY=blah when connecting to hosted NIM, e.g. NVIDIA_BASE_URL=https://integrate.api.nvidia.com ## Test Plan - `llama stack run ./llama_stack/templates/nvidia/run.yaml` -> error, e.g. API key is required for hosted NVIDIA NIM - `llama stack run ./llama_stack/templates/nvidia/run.yaml --env NVIDIA_BASE_URL=https://integrate.api.nvidia.com` -> error, e.g. API key is required for hosted NVIDIA NIM - `llama stack run ./llama_stack/templates/nvidia/run.yaml --env NVIDIA_API_KEY=REDACTED` -> successful connection to NIM on https://integrate.api.nvidia.com - `llama stack run ./llama_stack/templates/nvidia/run.yaml --env NVIDIA_BASE_URL=https://integrate.api.nvidia.com --env NVIDIA_API_KEY=REDACTED` -> successful connection to NIM running on integrate.api.nvidia.com - `llama stack run ./llama_stack/templates/nvidia/run.yaml --env NVIDIA_BASE_URL=http://localhost:8000` -> successful connection to NIM running on localhost:8000 - `llama stack run ./llama_stack/templates/nvidia/run.yaml --env NVIDIA_BASE_URL=http://localhost:8000 --env NVIDIA_API_KEY=REDACTED` -> successful connection to NIM running on http://localhost:8000 - `llama stack run ./llama_stack/templates/nvidia/run.yaml --env NVIDIA_BASE_URL=http://bogus` -> runtime error, e.g. ConnectionError (TODO: this should be a startup error) ## Before submitting - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). - [x] Ran pre-commit to handle lint / formatting issues. - [x] Read the [contributor guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md), Pull Request section? - [ ] Updated relevant documentation. - [ ] Wrote necessary unit or integration tests.
This commit is contained in:
parent
1a5c17a92f
commit
11b1cdf31d
1 changed files with 2 additions and 2 deletions
|
@ -15,8 +15,8 @@ providers:
|
|||
- provider_id: nvidia
|
||||
provider_type: remote::nvidia
|
||||
config:
|
||||
url: https://integrate.api.nvidia.com
|
||||
api_key: ${env.NVIDIA_API_KEY}
|
||||
url: ${env.NVIDIA_BASE_URL:https://integrate.api.nvidia.com}
|
||||
api_key: ${env.NVIDIA_API_KEY:}
|
||||
vector_io:
|
||||
- provider_id: faiss
|
||||
provider_type: inline::faiss
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue