phoenix-oss/llama-stack
1
0
Fork 0
forked from phoenix-oss/llama-stack-mirror
Code Pull requests Releases Packages 2 Activity Actions

Add a pre-commit for distro_codegen but it does not work yet

Browse source
This commit is contained in:
Ashwin Bharambe 2024-11-18 15:20:49 -08:00
parent 57a9b4d57f
commit 3aedde2ab4
8 changed files with 66 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

10
docs/source/getting_started/distributions/self_hosted_distro/remote-vllm.md
View file

@ -13,6 +13,16 @@ The `llamastack/distribution-remote-vllm` distribution consists of the following
You can use this distribution if you have GPUs and want to run an independent vLLM server container for running inference.
### Environment Variables
The following environment variables can be configured:
- `LLAMASTACK_PORT`: Port for the Llama Stack distribution server (default: `5001`)
- `INFERENCE_MODEL`: Inference model loaded into the vLLM server (default: `meta-llama/Llama-3.2-3B-Instruct`)
- `VLLM_URL`: URL of the vLLM server with the main inference model (default: `http://host.docker.internal:5100}/v1`)
- `MAX_TOKENS`: Maximum number of tokens for generation (default: `4096`)
- `SAFETY_VLLM_URL`: URL of the vLLM server with the safety model (default: `http://host.docker.internal:5101/v1`)
- `SAFETY_MODEL`: Name of the safety (Llama-Guard) model to use (default: `meta-llama/Llama-Guard-3-1B`)
## Setting up vLLM server