phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-12-13 04:22:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

move distribution/templates to distributions/

Browse source
This commit is contained in:
Xi Yan 2024-10-18 17:21:50 -07:00
parent b4aca0aeb6
commit cbb423a32f
10 changed files with 143 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

11
distributions/tgi/README.md
View file

@ -40,7 +40,7 @@ docker compose down
### Start the Distribution (Single Node CPU)
> [!NOTE]
> This assumes you have an hosted endpoint
> This assumes you have an hosted endpoint compatible with TGI server.
```
$ cd llama-stack/distribution/tgi/cpu
@ -49,6 +49,15 @@ compose.yaml run.yaml
$ docker compose up
```
Replace <ENTER_YOUR_TGI_HOSTED_ENDPOINT> in `run.yaml` file with your TGI endpoint.
```
inference:
- provider_id: tgi0
provider_type: remote::tgi
config:
url: <ENTER_YOUR_TGI_HOSTED_ENDPOINT>
```
### (Alternative) TGI server + llama stack run (Single Node GPU)
If you wish to separately spin up a TGI server, and connect with Llama Stack, you may use the following commands.