phoenix-oss/llama-stack
1
0
Fork 0
forked from phoenix-oss/llama-stack-mirror
Code Pull requests Releases Packages 2 Activity Actions

Update getting_started.md

Browse source
This commit is contained in:
Xi Yan 2024-10-25 13:30:33 -07:00 • committed by GitHub
parent f168752bba
commit 8a74e400d6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 19 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

38
docs/getting_started.md
View file

@ -35,31 +35,31 @@ You have two ways to start up Llama stack server:
1. **Starting up server via docker**: 1. **Starting up server via docker**:
We provide pre-built Docker image of Llama Stack distribution, which can be found in the following links in the [distributions](../distributions/) folder. We provide pre-built Docker image of Llama Stack distribution, which can be found in the following links in the [distributions](../distributions/) folder.
> [!NOTE] > [!NOTE]
> For GPU inference, you need to set these environment variables for specifying local directory containing your model checkpoints, and enable GPU inference to start running docker container. > For GPU inference, you need to set these environment variables for specifying local directory containing your model checkpoints, and enable GPU inference to start running docker container.
``` ```
export LLAMA_CHECKPOINT_DIR=~/.llama export LLAMA_CHECKPOINT_DIR=~/.llama
``` ```
> [!NOTE] > [!NOTE]
> `~/.llama` should be the path containing downloaded weights of Llama models. > `~/.llama` should be the path containing downloaded weights of Llama models.
To download llama models, use To download llama models, use
``` ```
llama download --model-id Llama3.1-8B-Instruct llama download --model-id Llama3.1-8B-Instruct
``` ```
To download and start running a pre-built docker container, you may use the following commands: To download and start running a pre-built docker container, you may use the following commands:
``` ```
$ cd llama-stack/distributions/meta-reference-gpu cd llama-stack/distributions/meta-reference-gpu
$ docker run -it -p 5000:5000 -v ~/.llama:/root/.llama -v ./run.yaml:/root/my-run.yaml --gpus=all distribution-meta-reference-gpu --yaml_config /root/my-run.yaml docker run -it -p 5000:5000 -v ~/.llama:/root/.llama -v ./run.yaml:/root/my-run.yaml --gpus=all distribution-meta-reference-gpu --yaml_config /root/my-run.yaml
``` ```
> [!TIP] > [!TIP]
> Pro Tip: We may use `docker compose up` for starting up a distribution with remote providers (e.g. TGI) using [llamastack-local-cpu](https://hub.docker.com/repository/docker/llamastack/llamastack-local-cpu/general). You can checkout [these scripts](../distributions/) to help you get started. > Pro Tip: We may use `docker compose up` for starting up a distribution with remote providers (e.g. TGI) using [llamastack-local-cpu](https://hub.docker.com/repository/docker/llamastack/llamastack-local-cpu/general). You can checkout [these scripts](../distributions/) to help you get started.
2. **Build->Configure->Run Llama Stack server via conda**: 2. **Build->Configure->Run Llama Stack server via conda**: