phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-06-28 02:53:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

kill tgi/cpu

Browse source
This commit is contained in:
Xi Yan 2024-11-08 10:33:45 -08:00
parent 6192bf43a4
commit b1d7376730
5 changed files with 1 additions and 97 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/source/getting_started/distributions/self_hosted_distro/tgi.md
View file

@ -15,7 +15,7 @@ The `llamastack/distribution-tgi` distribution consists of the following provide
```
$ cd distributions/tgi/gpu && docker compose up
$ cd distributions/tgi && docker compose up
```
The script will first start up TGI server, then start up Llama Stack distribution server hooking up to the remote TGI provider for inference. You should be able to see the following outputs --
@ -34,23 +34,6 @@ To kill the server
docker compose down
```
### Docker: Start the Distribution (Single Node CPU)
> [!NOTE]
> This assumes you have an hosted endpoint compatible with TGI server.
```
$ cd distributions/tgi/cpu && docker compose up
```
Replace <ENTER_YOUR_TGI_HOSTED_ENDPOINT> in `run.yaml` file with your TGI endpoint.
```
inference:
- provider_id: tgi0
provider_type: remote::tgi
config:
url: <ENTER_YOUR_TGI_HOSTED_ENDPOINT>
```
### Conda: TGI server + llama stack run