phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-07-21 20:18:52 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

add more distro templates (#279)

Browse source 
* verify dockers

* together distro verified

* readme

* fireworks distro

* fireworks compose up

* fireworks verified
This commit is contained in:
Xi Yan 2024-10-21 18:15:08 -07:00 • committed by GitHub
parent cf27d19dd5
commit 4d2bd2d39e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 265 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

11
distributions/meta-reference-gpu/README.md
View file

@ -11,13 +11,8 @@ The `llamastack/distribution-meta-reference-gpu` distribution consists of the fo
### Start the Distribution (Single Node GPU)
> [!NOTE]
> This assumes you have access to GPU to start a TGI server with access to your GPU.
> This assumes you have access to GPU to start a local server with access to your GPU.
> [!NOTE]
> For GPU inference, you need to set these environment variables for specifying local directory containing your model checkpoints, and enable GPU inference to start running docker container.
```
export LLAMA_CHECKPOINT_DIR=~/.llama
```
> [!NOTE]
> `~/.llama` should be the path containing downloaded weights of Llama models.
@ -26,8 +21,8 @@ export LLAMA_CHECKPOINT_DIR=~/.llama
To download and start running a pre-built docker container, you may use the following commands:
```
docker run -it -p 5000:5000 -v ~/.llama:/root/.llama --gpus=all llamastack/llamastack-local-gpu
docker run -it -p 5000:5000 -v ~/.llama:/root/.llama -v ./run.yaml:/root/my-run.yaml --gpus=all distribution-meta-reference-gpu --yaml_config /root/my-run.yaml
```
### Alternative (Build and start distribution locally via conda)
- You may checkout the [Getting Started](../../docs/getting_started.md) for more details on starting up a meta-reference distribution.
- You may checkout the [Getting Started](../../docs/getting_started.md) for more details on building locally via conda and starting up a meta-reference distribution.