update getting started guide

This commit is contained in:
Xi Yan 2024-09-18 15:35:54 -07:00
parent 29ce73ff7a
commit 2f9e952813
2 changed files with 64 additions and 53 deletions

View file

@ -28,6 +28,7 @@ Build spec configuration saved at ~/.conda/envs/llamastack-my-local-llama-stack/
``` ```
**`llama stack configure`** **`llama stack configure`**
- Run `llama stack configure <name>` with the name you have previously defined in `build` step.
``` ```
llama stack configure my-local-llama-stack llama stack configure my-local-llama-stack
@ -61,6 +62,7 @@ You can now run `llama stack run my-local-llama-stack --port PORT` or `llama sta
``` ```
**`llama stack run`** **`llama stack run`**
- Run `llama stack run <name>` with the name you have previously defined.
``` ```
llama stack run my-local-llama-stack llama stack run my-local-llama-stack
@ -110,74 +112,83 @@ In the following steps, imagine we'll be working with a `Meta-Llama3.1-8B-Instru
- `providers`: specifies the underlying implementation for serving each API endpoint - `providers`: specifies the underlying implementation for serving each API endpoint
- `image_type`: `conda` | `docker` to specify whether to build the distribution in the form of Docker image or Conda environment. - `image_type`: `conda` | `docker` to specify whether to build the distribution in the form of Docker image or Conda environment.
#### Build a local distribution with conda
The following command and specifications allows you to get started with building.
```
llama stack build <path/to/config>
```
- You will be required to pass in a file path to the build.config file (e.g. `./llama_stack/distribution/example_configs/conda/local-conda-example-build.yaml`). We provide some example build config files for configuring different types of distributions in the `./llama_stack/distribution/example_configs/` folder.
The file will be of the contents At the end of build command, we will generate `<name>-build.yaml` file storing the build configurations.
```
$ cat ./llama_stack/distribution/example_configs/conda/local-conda-example-build.yaml
name: 8b-instruct #### Building from scratch
- For a new user, we could start off with running `llama stack build` which will allow you to a interactively enter wizard where you will be prompted to enter build configurations.
```
llama stack build
```
Running the command above will allow you to fill in the configuration to build your Llama Stack distribution, you will see the following outputs.
```
> Enter an unique name for identifying your Llama Stack build distribution (e.g. my-local-stack): my-local-llama-stack
> Enter the image type you want your distribution to be built with (docker or conda): conda
Llama Stack is composed of several APIs working together. Let's configure the providers (implementations) you want to use for these APIs.
> Enter the API provider for the inference API: (default=meta-reference): meta-reference
> Enter the API provider for the safety API: (default=meta-reference): meta-reference
> Enter the API provider for the agents API: (default=meta-reference): meta-reference
> Enter the API provider for the memory API: (default=meta-reference): meta-reference
> Enter the API provider for the telemetry API: (default=meta-reference): meta-reference
> (Optional) Enter a short description for your Llama Stack distribution:
Build spec configuration saved at ~/.conda/envs/llamastack-my-local-llama-stack/my-local-llama-stack-build.yaml
```
#### Building from templates
- To build from alternative API providers, we provide distribution templates for users to get started building a distribution backed by different providers.
The following command will allow you to see the available templates and their corresponding providers.
```
llama stack build --list-templates
```
![alt text](list-templates.png)
You may then pick a template to build your distribution with providers fitted to your liking.
```
llama stack build --template local-tgi --name my-tgi-stack
```
```
```
#### Building from config file
- In addition to templates, you may customize the build to your liking through editing config files and build from config files with the following command.
- The config file will be of contents like the ones in `llama_stack/distributions/templates/`.
```
$ cat llama_stack/distribution/templates/local-ollama-build.yaml
name: local-ollama
distribution_spec: distribution_spec:
distribution_type: local description: Like local, but use ollama for running LLM inference
description: Use code from `llama_stack` itself to serve all llama stack APIs
docker_image: null
providers: providers:
inference: meta-reference inference: remote::ollama
memory: meta-reference-faiss memory: meta-reference
safety: meta-reference safety: meta-reference
agentic_system: meta-reference agents: meta-reference
telemetry: console telemetry: meta-reference
image_type: conda image_type: conda
``` ```
You may run the `llama stack build` command to generate your distribution with `--name` to override the name for your distribution.
``` ```
$ llama stack build ~/.llama/distributions/conda/8b-instruct-build.yaml --name 8b-instruct llama stack build --config llama_stack/distribution/templates/local-ollama-build.yaml
...
...
Build spec configuration saved at ~/.llama/distributions/conda/8b-instruct-build.yaml
``` ```
After this step is complete, a file named `8b-instruct-build.yaml` will be generated and saved at `~/.llama/distributions/conda/8b-instruct-build.yaml`. After this step is complete, a file named `8b-instruct-build.yaml` will be generated and saved at `~/.llama/distributions/conda/8b-instruct-build.yaml`.
#### How to build distribution with different API providers using configs
To specify a different API provider, we can change the `distribution_spec` in our `<name>-build.yaml` config. For example, the following build spec allows you to build a distribution using TGI as the inference API provider.
```
$ cat ./llama_stack/distribution/example_configs/conda/local-tgi-conda-example-build.yaml
name: local-tgi-conda-example
distribution_spec:
description: Use TGI (local or with Hugging Face Inference Endpoints for running LLM inference. When using HF Inference Endpoints, you must provide the name of the endpoint).
docker_image: null
providers:
inference: remote::tgi
memory: meta-reference-faiss
safety: meta-reference
agentic_system: meta-reference
telemetry: console
image_type: conda
```
The following command allows you to build a distribution with TGI as the inference API provider, with the name `tgi`.
```
llama stack build ./llama_stack/distribution/example_configs/conda/local-tgi-conda-example-build.yaml --name tgi
```
We provide some example build configs to help you get started with building with different API providers.
#### How to build distribution with Docker image #### How to build distribution with Docker image
To build a docker image, simply change the `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. To build a docker image, simply change the `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`.
``` ```
$ cat ./llama_stack/distribution/example_configs/docker/local-docker-example-build.yaml
name: local-docker-example name: local-docker-example
distribution_spec: distribution_spec:
description: Use code from `llama_stack` itself to serve all llama stack APIs description: Use code from `llama_stack` itself to serve all llama stack APIs
@ -206,7 +217,7 @@ Build spec configuration saved at /home/xiyan/.llama/distributions/docker/docker
## Step 2. Configure ## Step 2. Configure
After our distribution is built (either in form of docker or conda environment), we will run the following command to After our distribution is built (either in form of docker or conda environment), we will run the following command to
``` ```
llama stack configure [<path/to/name.build.yaml> | <docker-image-name>] llama stack configure [ <name> | <docker-image-name> | <path/to/name.build.yaml>]
``` ```
- For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1. - For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1.
- For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument. - For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument.
@ -298,7 +309,7 @@ INFO: Uvicorn running on http://[::]:5000 (Press CTRL+C to quit)
``` ```
> [!NOTE] > [!NOTE]
> Configuration is in `~/.llama/builds/local/conda/8b-instruct.yaml`. Feel free to increase `max_seq_len`. > Configuration is in `~/.llama/builds/local/conda/8b-instruct-run.yaml`. Feel free to increase `max_seq_len`.
> [!IMPORTANT] > [!IMPORTANT]
> The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines. > The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines.

BIN
docs/list-templates.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 220 KiB