phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-16 06:53:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

distributions

Browse source
This commit is contained in:
Xi Yan 2024-10-29 15:00:58 -07:00
parent 3fb9a8e82e
commit 39872ca4b4
7 changed files with 336 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

35
distributions/tgi/README.md
View file

@ -8,17 +8,14 @@ The `llamastack/distribution-tgi` distribution consists of the following provide
| **Provider(s)** | remote::tgi | meta-reference | meta-reference, remote::pgvector, remote::chroma | meta-reference | meta-reference |
### Start the Distribution (Single Node GPU)
### Docker: Start the Distribution (Single Node GPU)
> [!NOTE]
> This assumes you have access to GPU to start a TGI server with access to your GPU.
```
$ cd distributions/tgi/gpu
$ ls
compose.yaml tgi-run.yaml
$ docker compose up
$ cd distributions/tgi/gpu && docker compose up
```
The script will first start up TGI server, then start up Llama Stack distribution server hooking up to the remote TGI provider for inference. You should be able to see the following outputs --
@ -37,16 +34,13 @@ To kill the server
docker compose down
```
### Start the Distribution (Single Node CPU)
### Docker: Start the Distribution (Single Node CPU)
> [!NOTE]
> This assumes you have an hosted endpoint compatible with TGI server.
```
$ cd distributions/tgi/cpu
$ ls
compose.yaml run.yaml
$ docker compose up
$ cd distributions/tgi/cpu && docker compose up
```
Replace <ENTER_YOUR_TGI_HOSTED_ENDPOINT> in `run.yaml` file with your TGI endpoint.
@ -58,20 +52,28 @@ inference:
url: <ENTER_YOUR_TGI_HOSTED_ENDPOINT>
```
### (Alternative) TGI server + llama stack run (Single Node GPU)
### Conda: TGI server + llama stack run
If you wish to separately spin up a TGI server, and connect with Llama Stack, you may use the following commands.
#### (optional) Start TGI server locally
#### Start TGI server locally
- Please check the [TGI Getting Started Guide](https://github.com/huggingface/text-generation-inference?tab=readme-ov-file#get-started) to get a TGI endpoint.
```
docker run --rm -it -v $HOME/.cache/huggingface:/data -p 5009:5009 --gpus all ghcr.io/huggingface/text-generation-inference:latest --dtype bfloat16 --usage-stats on --sharded false --model-id meta-llama/Llama-3.1-8B-Instruct --port 5009
```
#### Start Llama Stack server pointing to TGI server
**Via Conda**
```bash
llama stack build --template tgi --image-type conda
# -- start a TGI server endpoint
llama stack run ./gpu/run.yaml
```
**Via Docker**
```
docker run --network host -it -p 5000:5000 -v ./run.yaml:/root/my-run.yaml --gpus=all llamastack/distribution-tgi --yaml_config /root/my-run.yaml
```
@ -85,13 +87,6 @@ inference:
url: http://127.0.0.1:5009
```
**Via Conda**
```bash
llama stack build --template tgi --image-type conda
# -- start a TGI server endpoint
llama stack run ./gpu/run.yaml
```
### Model Serving
To serve a new model with `tgi`, change the docker command flag `--model-id <model-to-serve>`.