phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-02 16:54:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: add INFERENCE_MODEL env

Browse source
This commit is contained in:
seyeong-han 2024-11-24 01:41:55 -06:00
parent 1d3cc0b138
commit 839f4a4779
1 changed files with 12 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

14
llama_stack/templates/ssambanova/doc_template.md
View file

@ -43,6 +43,14 @@ Make sure you have access to a Ssambanova API Key. You can get one by visiting [
You can do this via Conda (build code) or Docker which has a pre-built image. You can do this via Conda (build code) or Docker which has a pre-built image.
### Available INFERENCE_MODEL
- Meta-Llama-3.1-8B-Instruct
- Meta-Llama-3.1-70B-Instruct
- Meta-Llama-3.1-405B-Instruct
- Meta-Llama-3.2-1B-Instruct
- Meta-Llama-3.2-3B-Instruct
### Via Docker ### Via Docker
This method allows you to get started quickly without having to build the distribution code. This method allows you to get started quickly without having to build the distribution code.
@ -54,7 +62,8 @@ docker run \
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \ -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
llamastack/distribution-{{ name }} \ llamastack/distribution-{{ name }} \
--port $LLAMA_STACK_PORT \ --port $LLAMA_STACK_PORT \
--env SSAMBANOVA_API_KEY=$SSAMBANOVA_API_KEY --env SSAMBANOVA_API_KEY=$SSAMBANOVA_API_KEY \
--env INFERENCE_MODEL=$INFERENCE_MODEL
``` ```
### Via Conda ### Via Conda
@ -63,5 +72,6 @@ docker run \
llama stack build --template ssambanova --image-type conda llama stack build --template ssambanova --image-type conda
llama stack run ./run.yaml \ llama stack run ./run.yaml \
--port $LLAMA_STACK_PORT \ --port $LLAMA_STACK_PORT \
--env SSAMBANOVA_API_KEY=$SSAMBANOVA_API_KEY --env SSAMBANOVA_API_KEY=$SSAMBANOVA_API_KEY \
--env INFERENCE_MODEL=$INFERENCE_MODEL
``` ```