phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-06-27 18:50:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Kill --name from llama stack build (#340)

Browse source
This commit is contained in:
Ashwin Bharambe 2024-10-28 23:07:32 -07:00 • committed by GitHub
parent 26d1668f7d
commit 4aa1bf6a60
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 71 additions and 130 deletions
Download patch file Download diff file Expand all files Collapse all files

26
docs/building_distro.md
View file

@ -58,15 +58,15 @@ llama stack build --list-templates
You may then pick a template to build your distribution with providers fitted to your liking. You may then pick a template to build your distribution with providers fitted to your liking.
``` ```
llama stack build --template local-tgi --name my-tgi-stack llama stack build --template tgi
``` ```
``` ```
$ llama stack build --template local-tgi --name my-tgi-stack $ llama stack build --template tgi
... ...
... ...
Build spec configuration saved at ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml Build spec configuration saved at ~/.conda/envs/llamastack-tgi/tgi-build.yaml
You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml` You may now run `llama stack configure tgi` or `llama stack configure ~/.conda/envs/llamastack-tgi/tgi-build.yaml`
``` ```
#### Building from config file #### Building from config file
@ -75,9 +75,9 @@ You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~
- The config file will be of contents like the ones in `llama_stack/distributions/templates/`. - The config file will be of contents like the ones in `llama_stack/distributions/templates/`.
``` ```
$ cat llama_stack/distribution/templates/local-ollama-build.yaml $ cat llama_stack/templates/ollama/build.yaml
name: local-ollama name: ollama
distribution_spec: distribution_spec:
description: Like local, but use ollama for running LLM inference description: Like local, but use ollama for running LLM inference
providers: providers:
@ -90,7 +90,7 @@ image_type: conda
``` ```
``` ```
llama stack build --config llama_stack/distribution/templates/local-ollama-build.yaml llama stack build --config llama_stack/templates/ollama/build.yaml
``` ```
#### How to build distribution with Docker image #### How to build distribution with Docker image
@ -101,7 +101,7 @@ llama stack build --config llama_stack/distribution/templates/local-ollama-build
To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type. To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type.
``` ```
llama stack build --template local --image-type docker --name docker-0 llama stack build --template local --image-type docker
``` ```
Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like: Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like:
@ -136,14 +136,14 @@ Build spec configuration saved at ~/.llama/distributions/docker/docker-local-bui
## Step 2. Configure ## Step 2. Configure
After our distribution is built (either in form of docker or conda environment), we will run the following command to After our distribution is built (either in form of docker or conda environment), we will run the following command to
``` ```
llama stack configure [ <name> | <docker-image-name> | <path/to/name.build.yaml>] llama stack configure [ <docker-image-name> | <path/to/name.build.yaml>]
``` ```
- For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1. - For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1.
- For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument. - For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument.
- Run `docker images` to check list of available images on your machine. - Run `docker images` to check list of available images on your machine.
``` ```
$ llama stack configure 8b-instruct $ llama stack configure tgi
Configuring API: inference (meta-reference) Configuring API: inference (meta-reference)
Enter value for model (existing: Meta-Llama3.1-8B-Instruct) (required): Enter value for model (existing: Meta-Llama3.1-8B-Instruct) (required):
@ -172,10 +172,10 @@ Enter value for wolfram_api_key (optional):
Configuring API: telemetry (console) Configuring API: telemetry (console)
YAML configuration has been written to ~/.llama/builds/conda/8b-instruct-run.yaml YAML configuration has been written to ~/.llama/builds/conda/tgi-run.yaml
``` ```
After this step is successful, you should be able to find a run configuration spec in `~/.llama/builds/conda/8b-instruct-run.yaml` with the following contents. You may edit this file to change the settings. After this step is successful, you should be able to find a run configuration spec in `~/.llama/builds/conda/tgi-run.yaml` with the following contents. You may edit this file to change the settings.
As you can see, we did basic configuration above and configured: As you can see, we did basic configuration above and configured:
- inference to run on model `Meta-Llama3.1-8B-Instruct` (obtained from `llama model list`) - inference to run on model `Meta-Llama3.1-8B-Instruct` (obtained from `llama model list`)
@ -228,7 +228,7 @@ INFO: Uvicorn running on http://[::]:5000 (Press CTRL+C to quit)
``` ```
> [!NOTE] > [!NOTE]
> Configuration is in `~/.llama/builds/local/conda/8b-instruct-run.yaml`. Feel free to increase `max_seq_len`. > Configuration is in `~/.llama/builds/local/conda/tgi-run.yaml`. Feel free to increase `max_seq_len`.
> [!IMPORTANT] > [!IMPORTANT]
> The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines. > The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines.

26
docs/cli_reference.md
View file

@ -229,8 +229,8 @@ You will be shown a Markdown formatted description of the model interface and ho
- Please see our [Getting Started](getting_started.md) guide for more details on how to build and start a Llama Stack distribution. - Please see our [Getting Started](getting_started.md) guide for more details on how to build and start a Llama Stack distribution.
### Step 3.1 Build ### Step 3.1 Build
In the following steps, imagine we'll be working with a `Llama3.1-8B-Instruct` model. We will name our build `8b-instruct` to help us remember the config. We will start build our distribution (in the form of a Conda environment, or Docker image). In this step, we will specify: In the following steps, imagine we'll be working with a `Llama3.1-8B-Instruct` model. We will name our build `tgi` to help us remember the config. We will start build our distribution (in the form of a Conda environment, or Docker image). In this step, we will specify:
- `name`: the name for our distribution (e.g. `8b-instruct`) - `name`: the name for our distribution (e.g. `tgi`)
- `image_type`: our build image type (`conda | docker`) - `image_type`: our build image type (`conda | docker`)
- `distribution_spec`: our distribution specs for specifying API providers - `distribution_spec`: our distribution specs for specifying API providers
- `description`: a short description of the configurations for the distribution - `description`: a short description of the configurations for the distribution
@ -279,15 +279,15 @@ llama stack build --list-templates
You may then pick a template to build your distribution with providers fitted to your liking. You may then pick a template to build your distribution with providers fitted to your liking.
``` ```
llama stack build --template local-tgi --name my-tgi-stack --image-type conda llama stack build --template tgi --image-type conda
``` ```
``` ```
$ llama stack build --template local-tgi --name my-tgi-stack --image-type conda $ llama stack build --template tgi --image-type conda
... ...
... ...
Build spec configuration saved at ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml Build spec configuration saved at ~/.conda/envs/llamastack-tgi/tgi-build.yaml
You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml` You may now run `llama stack configure tgi` or `llama stack configure ~/.conda/envs/llamastack-tgi/tgi-build.yaml`
``` ```
#### Building from config file #### Building from config file
@ -298,7 +298,7 @@ You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~
``` ```
$ cat build.yaml $ cat build.yaml
name: local-ollama name: ollama
distribution_spec: distribution_spec:
description: Like local, but use ollama for running LLM inference description: Like local, but use ollama for running LLM inference
providers: providers:
@ -319,7 +319,7 @@ llama stack build --config build.yaml
To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type. To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type.
``` ```
llama stack build --template local --image-type docker --name docker-0 llama stack build --template tgi --image-type docker
``` ```
Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like: Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like:
@ -354,14 +354,14 @@ Build spec configuration saved at ~/.llama/distributions/docker/docker-local-bui
### Step 3.2 Configure ### Step 3.2 Configure
After our distribution is built (either in form of docker or conda environment), we will run the following command to After our distribution is built (either in form of docker or conda environment), we will run the following command to
``` ```
llama stack configure [ <name> | <docker-image-name> | <path/to/name.build.yaml>] llama stack configure [ <docker-image-name> | <path/to/name-build.yaml>]
``` ```
- For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1. - For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1.
- For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument. - For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument.
- Run `docker images` to check list of available images on your machine. - Run `docker images` to check list of available images on your machine.
``` ```
$ llama stack configure ~/.llama/distributions/conda/8b-instruct-build.yaml $ llama stack configure ~/.llama/distributions/conda/tgi-build.yaml
Configuring API: inference (meta-reference) Configuring API: inference (meta-reference)
Enter value for model (existing: Llama3.1-8B-Instruct) (required): Enter value for model (existing: Llama3.1-8B-Instruct) (required):
@ -409,13 +409,13 @@ Note that all configurations as well as models are stored in `~/.llama`
Now, let's start the Llama Stack Distribution Server. You will need the YAML configuration file which was written out at the end by the `llama stack configure` step. Now, let's start the Llama Stack Distribution Server. You will need the YAML configuration file which was written out at the end by the `llama stack configure` step.
``` ```
llama stack run ~/.llama/builds/conda/8b-instruct-run.yaml llama stack run ~/.llama/builds/conda/tgi-run.yaml
``` ```
You should see the Llama Stack server start and print the APIs that it is supporting You should see the Llama Stack server start and print the APIs that it is supporting
``` ```
$ llama stack run ~/.llama/builds/local/conda/8b-instruct.yaml $ llama stack run ~/.llama/builds/local/conda/tgi-run.yaml
> initializing model parallel with size 1 > initializing model parallel with size 1
> initializing ddp with size 1 > initializing ddp with size 1
@ -446,7 +446,7 @@ INFO: Uvicorn running on http://[::]:5000 (Press CTRL+C to quit)
``` ```
> [!NOTE] > [!NOTE]
> Configuration is in `~/.llama/builds/local/conda/8b-instruct-run.yaml`. Feel free to increase `max_seq_len`. > Configuration is in `~/.llama/builds/local/conda/tgi-run.yaml`. Feel free to increase `max_seq_len`.
> [!IMPORTANT] > [!IMPORTANT]
> The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines. > The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines.

1
docs/getting_started.ipynb
View file

@ -68,7 +68,6 @@
"```\n", "```\n",
"$ llama stack configure llamastack-local-gpu\n", "$ llama stack configure llamastack-local-gpu\n",
"\n", "\n",
"Could not find llamastack-local-gpu. Trying conda build name instead...\n",
"Could not find /home/hjshah/.conda/envs/llamastack-llamastack-local-gpu/llamastack-local-gpu-build.yaml. Trying docker image name instead...\n", "Could not find /home/hjshah/.conda/envs/llamastack-llamastack-local-gpu/llamastack-local-gpu-build.yaml. Trying docker image name instead...\n",
"+ podman run --network host -it -v /home/hjshah/.llama/builds/docker:/app/builds llamastack-local-gpu llama stack configure ./llamastack-build.yaml --output-dir /app/builds\n", "+ podman run --network host -it -v /home/hjshah/.llama/builds/docker:/app/builds llamastack-local-gpu llama stack configure ./llamastack-build.yaml --output-dir /app/builds\n",
"\n", "\n",

1
docs/getting_started.md
View file

@ -97,7 +97,6 @@ docker run -it -p 5000:5000 -v ~/.llama:/root/.llama -v ./run.yaml:/root/my-run.
``` ```
$ llama stack configure my-local-stack $ llama stack configure my-local-stack
Could not find my-local-stack. Trying conda build name instead...
Configuring API `inference`... Configuring API `inference`...
=== Configuring provider `meta-reference` for API inference... === Configuring provider `meta-reference` for API inference...
Enter value for model (default: Llama3.1-8B-Instruct) (required): Enter value for model (default: Llama3.1-8B-Instruct) (required):

30
docs/source/cli_reference.md
View file

@ -279,15 +279,15 @@ llama stack build --list-templates
You may then pick a template to build your distribution with providers fitted to your liking. You may then pick a template to build your distribution with providers fitted to your liking.
``` ```
llama stack build --template local-tgi --name my-tgi-stack llama stack build --template tgi
``` ```
``` ```
$ llama stack build --template local-tgi --name my-tgi-stack $ llama stack build --template tgi
... ...
... ...
Build spec configuration saved at ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml Build spec configuration saved at ~/.conda/envs/llamastack-tgi/tgi-build.yaml
You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml` You may now run `llama stack configure tgi` or `llama stack configure ~/.conda/envs/llamastack-tgi/tgi-build.yaml`
``` ```
#### Building from config file #### Building from config file
@ -296,9 +296,9 @@ You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~
- The config file will be of contents like the ones in `llama_stack/distributions/templates/`. - The config file will be of contents like the ones in `llama_stack/distributions/templates/`.
``` ```
$ cat llama_stack/distribution/templates/local-ollama-build.yaml $ cat llama_stack/templates/ollama/build.yaml
name: local-ollama name: ollama
distribution_spec: distribution_spec:
description: Like local, but use ollama for running LLM inference description: Like local, but use ollama for running LLM inference
providers: providers:
@ -311,7 +311,7 @@ image_type: conda
``` ```
``` ```
llama stack build --config llama_stack/distribution/templates/local-ollama-build.yaml llama stack build --config llama_stack/templates/ollama/build.yaml
``` ```
#### How to build distribution with Docker image #### How to build distribution with Docker image
@ -319,7 +319,7 @@ llama stack build --config llama_stack/distribution/templates/local-ollama-build
To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type. To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type.
``` ```
llama stack build --template local --image-type docker --name docker-0 llama stack build --template local --image-type docker
``` ```
Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like: Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like:
@ -354,14 +354,14 @@ Build spec configuration saved at ~/.llama/distributions/docker/docker-local-bui
### Step 3.2 Configure ### Step 3.2 Configure
After our distribution is built (either in form of docker or conda environment), we will run the following command to After our distribution is built (either in form of docker or conda environment), we will run the following command to
``` ```
llama stack configure [ <name> | <docker-image-name> | <path/to/name.build.yaml>] llama stack configure [ <docker-image-name> | <path/to/name.build.yaml>]
``` ```
- For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1. - For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1.
- For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument. - For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument.
- Run `docker images` to check list of available images on your machine. - Run `docker images` to check list of available images on your machine.
``` ```
$ llama stack configure ~/.llama/distributions/conda/8b-instruct-build.yaml $ llama stack configure ~/.llama/distributions/conda/tgi-build.yaml
Configuring API: inference (meta-reference) Configuring API: inference (meta-reference)
Enter value for model (existing: Llama3.1-8B-Instruct) (required): Enter value for model (existing: Llama3.1-8B-Instruct) (required):
@ -390,10 +390,10 @@ Enter value for wolfram_api_key (optional):
Configuring API: telemetry (console) Configuring API: telemetry (console)
YAML configuration has been written to ~/.llama/builds/conda/8b-instruct-run.yaml YAML configuration has been written to ~/.llama/builds/conda/tgi-run.yaml
``` ```
After this step is successful, you should be able to find a run configuration spec in `~/.llama/builds/conda/8b-instruct-run.yaml` with the following contents. You may edit this file to change the settings. After this step is successful, you should be able to find a run configuration spec in `~/.llama/builds/conda/tgi-run.yaml` with the following contents. You may edit this file to change the settings.
As you can see, we did basic configuration above and configured: As you can see, we did basic configuration above and configured:
- inference to run on model `Llama3.1-8B-Instruct` (obtained from `llama model list`) - inference to run on model `Llama3.1-8B-Instruct` (obtained from `llama model list`)
@ -409,13 +409,13 @@ Note that all configurations as well as models are stored in `~/.llama`
Now, let's start the Llama Stack Distribution Server. You will need the YAML configuration file which was written out at the end by the `llama stack configure` step. Now, let's start the Llama Stack Distribution Server. You will need the YAML configuration file which was written out at the end by the `llama stack configure` step.
``` ```
llama stack run ~/.llama/builds/conda/8b-instruct-run.yaml llama stack run ~/.llama/builds/conda/tgi-run.yaml
``` ```
You should see the Llama Stack server start and print the APIs that it is supporting You should see the Llama Stack server start and print the APIs that it is supporting
``` ```
$ llama stack run ~/.llama/builds/local/conda/8b-instruct.yaml $ llama stack run ~/.llama/builds/conda/tgi-run.yaml
> initializing model parallel with size 1 > initializing model parallel with size 1
> initializing ddp with size 1 > initializing ddp with size 1
@ -446,7 +446,7 @@ INFO: Uvicorn running on http://[::]:5000 (Press CTRL+C to quit)
``` ```
> [!NOTE] > [!NOTE]
> Configuration is in `~/.llama/builds/local/conda/8b-instruct-run.yaml`. Feel free to increase `max_seq_len`. > Configuration is in `~/.llama/builds/local/conda/tgi-run.yaml`. Feel free to increase `max_seq_len`.
> [!IMPORTANT] > [!IMPORTANT]
> The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines. > The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines.

29
docs/source/getting_started.md
View file

@ -86,7 +86,6 @@ llama stack configure <name>
``` ```
$ llama stack configure my-local-stack $ llama stack configure my-local-stack
Could not find my-local-stack. Trying conda build name instead...
Configuring API `inference`... Configuring API `inference`...
=== Configuring provider `meta-reference` for API inference... === Configuring provider `meta-reference` for API inference...
Enter value for model (default: Llama3.1-8B-Instruct) (required): Enter value for model (default: Llama3.1-8B-Instruct) (required):
@ -218,15 +217,15 @@ llama stack build --list-templates
You may then pick a template to build your distribution with providers fitted to your liking. You may then pick a template to build your distribution with providers fitted to your liking.
``` ```
llama stack build --template local-tgi --name my-tgi-stack llama stack build --template tgi
``` ```
``` ```
$ llama stack build --template local-tgi --name my-tgi-stack $ llama stack build --template tgi
... ...
... ...
Build spec configuration saved at ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml Build spec configuration saved at ~/.conda/envs/llamastack-tgi/tgi-build.yaml
You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~/.conda/envs/llamastack-my-tgi-stack/my-tgi-stack-build.yaml` You may now run `llama stack configure tgi` or `llama stack configure ~/.conda/envs/llamastack-tgi/tgi-build.yaml`
``` ```
#### Building from config file #### Building from config file
@ -235,9 +234,9 @@ You may now run `llama stack configure my-tgi-stack` or `llama stack configure ~
- The config file will be of contents like the ones in `llama_stack/distributions/templates/`. - The config file will be of contents like the ones in `llama_stack/distributions/templates/`.
``` ```
$ cat llama_stack/distribution/templates/local-ollama-build.yaml $ cat llama_stack/templates/ollama/build.yaml
name: local-ollama name: ollama
distribution_spec: distribution_spec:
description: Like local, but use ollama for running LLM inference description: Like local, but use ollama for running LLM inference
providers: providers:
@ -250,7 +249,7 @@ image_type: conda
``` ```
``` ```
llama stack build --config llama_stack/distribution/templates/local-ollama-build.yaml llama stack build --config llama_stack/templates/ollama/build.yaml
``` ```
#### How to build distribution with Docker image #### How to build distribution with Docker image
@ -261,7 +260,7 @@ llama stack build --config llama_stack/distribution/templates/local-ollama-build
To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type. To build a docker image, you may start off from a template and use the `--image-type docker` flag to specify `docker` as the build image type.
``` ```
llama stack build --template local --image-type docker --name docker-0 llama stack build --template tgi --image-type docker
``` ```
Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like: Alternatively, you may use a config file and set `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`. The `<name>-build.yaml` will be of contents like:
@ -296,14 +295,14 @@ Build spec configuration saved at ~/.llama/distributions/docker/docker-local-bui
### Step 2. Configure ### Step 2. Configure
After our distribution is built (either in form of docker or conda environment), we will run the following command to After our distribution is built (either in form of docker or conda environment), we will run the following command to
``` ```
llama stack configure [ <name> | <docker-image-name> | <path/to/name.build.yaml>] llama stack configure [ <docker-image-name> | <path/to/name.build.yaml>]
``` ```
- For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1. - For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1.
- For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument. - For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument.
- Run `docker images` to check list of available images on your machine. - Run `docker images` to check list of available images on your machine.
``` ```
$ llama stack configure 8b-instruct $ llama stack configure tgi
Configuring API: inference (meta-reference) Configuring API: inference (meta-reference)
Enter value for model (existing: Meta-Llama3.1-8B-Instruct) (required): Enter value for model (existing: Meta-Llama3.1-8B-Instruct) (required):
@ -332,10 +331,10 @@ Enter value for wolfram_api_key (optional):
Configuring API: telemetry (console) Configuring API: telemetry (console)
YAML configuration has been written to ~/.llama/builds/conda/8b-instruct-run.yaml YAML configuration has been written to ~/.llama/builds/conda/tgi-run.yaml
``` ```
After this step is successful, you should be able to find a run configuration spec in `~/.llama/builds/conda/8b-instruct-run.yaml` with the following contents. You may edit this file to change the settings. After this step is successful, you should be able to find a run configuration spec in `~/.llama/builds/conda/tgi-run.yaml` with the following contents. You may edit this file to change the settings.
As you can see, we did basic configuration above and configured: As you can see, we did basic configuration above and configured:
- inference to run on model `Meta-Llama3.1-8B-Instruct` (obtained from `llama model list`) - inference to run on model `Meta-Llama3.1-8B-Instruct` (obtained from `llama model list`)
@ -351,13 +350,13 @@ Note that all configurations as well as models are stored in `~/.llama`
Now, let's start the Llama Stack Distribution Server. You will need the YAML configuration file which was written out at the end by the `llama stack configure` step. Now, let's start the Llama Stack Distribution Server. You will need the YAML configuration file which was written out at the end by the `llama stack configure` step.
``` ```
llama stack run 8b-instruct llama stack run tgi
``` ```
You should see the Llama Stack server start and print the APIs that it is supporting You should see the Llama Stack server start and print the APIs that it is supporting
``` ```
$ llama stack run 8b-instruct $ llama stack run tgi
> initializing model parallel with size 1 > initializing model parallel with size 1
> initializing ddp with size 1 > initializing ddp with size 1

62
llama_stack/cli/stack/build.py
View file

@ -62,17 +62,12 @@ class StackBuild(Subcommand):
help="Show the available templates for building a Llama Stack distribution", help="Show the available templates for building a Llama Stack distribution",
) )
self.parser.add_argument(
"--name",
type=str,
help="Name of the Llama Stack build to override from template config. This name will be used as paths to store configuration files, build conda environments/docker images. If not specified, will use the name from the template config. ",
)
self.parser.add_argument( self.parser.add_argument(
"--image-type", "--image-type",
type=str, type=str,
help="Image Type to use for the build. This can be either conda or docker. If not specified, will use the image type from the template config.", help="Image Type to use for the build. This can be either conda or docker. If not specified, will use the image type from the template config.",
choices=["conda", "docker"], choices=["conda", "docker"],
default="conda",
) )
def _run_stack_build_command(self, args: argparse.Namespace) -> None: def _run_stack_build_command(self, args: argparse.Namespace) -> None:
@ -91,15 +86,9 @@ class StackBuild(Subcommand):
return return
if args.template: if args.template:
if not args.name:
self.parser.error(
"You must specify a name for the build using --name when using a template"
)
return
available_templates = available_templates_specs() available_templates = available_templates_specs()
for build_config in available_templates: for build_config in available_templates:
if build_config.name == args.template: if build_config.name == args.template:
build_config.name = args.name
if args.image_type: if args.image_type:
build_config.image_type = args.image_type build_config.image_type = args.image_type
else: else:
@ -114,30 +103,14 @@ class StackBuild(Subcommand):
) )
return return
# try to see if we can find a pre-existing build config file through name
if args.name:
maybe_build_config = self._get_build_config_from_name(args)
if maybe_build_config:
cprint(
f"Building from existing build config for {args.name} in {str(maybe_build_config)}...",
"green",
)
with open(maybe_build_config, "r") as f:
build_config = BuildConfig(**yaml.safe_load(f))
self._run_stack_build_command_from_build_config(build_config)
return
if not args.config and not args.template: if not args.config and not args.template:
if not args.name: name = prompt(
name = prompt( "> Enter a name for your Llama Stack (e.g. my-local-stack): ",
"> Enter a name for your Llama Stack (e.g. my-local-stack): ", validator=Validator.from_callable(
validator=Validator.from_callable( lambda x: len(x) > 0,
lambda x: len(x) > 0, error_message="Name cannot be empty, please enter a name",
error_message="Name cannot be empty, please enter a name", ),
), )
)
else:
name = args.name
image_type = prompt( image_type = prompt(
"> Enter the image type you want your Llama Stack to be built as (docker or conda): ", "> Enter the image type you want your Llama Stack to be built as (docker or conda): ",
@ -203,25 +176,6 @@ class StackBuild(Subcommand):
return return
self._run_stack_build_command_from_build_config(build_config) self._run_stack_build_command_from_build_config(build_config)
def _get_build_config_from_name(self, args: argparse.Namespace) -> Optional[Path]:
if os.getenv("CONDA_PREFIX", ""):
conda_dir = (
Path(os.getenv("CONDA_PREFIX")).parent / f"llamastack-{args.name}"
)
else:
cprint(
"Cannot find CONDA_PREFIX. Trying default conda path ~/.conda/envs...",
color="green",
)
conda_dir = (
Path(os.path.expanduser("~/.conda/envs")) / f"llamastack-{args.name}"
)
build_config_file = Path(conda_dir) / f"{args.name}-build.yaml"
if build_config_file.exists():
return build_config_file
return None
def _run_stack_build_command_from_build_config( def _run_stack_build_command_from_build_config(
self, build_config: BuildConfig self, build_config: BuildConfig
) -> None: ) -> None:

19
llama_stack/cli/stack/configure.py
View file

@ -55,19 +55,12 @@ class StackConfigure(Subcommand):
docker_image = None docker_image = None
build_config_file = Path(args.config) build_config_file = Path(args.config)
if build_config_file.exists(): if build_config_file.exists():
with open(build_config_file, "r") as f: with open(build_config_file, "r") as f:
build_config = BuildConfig(**yaml.safe_load(f)) build_config = BuildConfig(**yaml.safe_load(f))
self._configure_llama_distribution(build_config, args.output_dir) self._configure_llama_distribution(build_config, args.output_dir)
return return
# if we get here, we need to try to find the conda build config file
cprint(
f"Could not find {build_config_file}. Trying conda build name instead...",
color="green",
)
conda_dir = ( conda_dir = (
Path(os.path.expanduser("~/.conda/envs")) / f"llamastack-{args.config}" Path(os.path.expanduser("~/.conda/envs")) / f"llamastack-{args.config}"
) )
@ -80,19 +73,14 @@ class StackConfigure(Subcommand):
break break
build_config_file = Path(conda_dir) / f"{args.config}-build.yaml" build_config_file = Path(conda_dir) / f"{args.config}-build.yaml"
if build_config_file.exists(): if build_config_file.exists():
with open(build_config_file, "r") as f: with open(build_config_file, "r") as f:
build_config = BuildConfig(**yaml.safe_load(f)) build_config = BuildConfig(**yaml.safe_load(f))
cprint(f"Using {build_config_file}...", "green")
self._configure_llama_distribution(build_config, args.output_dir) self._configure_llama_distribution(build_config, args.output_dir)
return return
# if we get here, we need to try to find the docker image
cprint(
f"Could not find {build_config_file}. Trying docker image name instead...",
color="green",
)
docker_image = args.config docker_image = args.config
builds_dir = BUILDS_BASE_DIR / ImageType.docker.value builds_dir = BUILDS_BASE_DIR / ImageType.docker.value
if args.output_dir: if args.output_dir:
@ -105,15 +93,10 @@ class StackConfigure(Subcommand):
script_args = [script, docker_image, str(builds_dir)] script_args = [script, docker_image, str(builds_dir)]
return_code = run_with_pty(script_args) return_code = run_with_pty(script_args)
# we have regenerated the build config file with script, now check if it exists
if return_code != 0: if return_code != 0:
self.parser.error( self.parser.error(
f"Failed to configure container {docker_image} with return code {return_code}. Please run `llama stack build` first. " f"Failed to configure container {docker_image} with return code {return_code}. Please run `llama stack build` first. "
) )
return
return
def _configure_llama_distribution( def _configure_llama_distribution(
self, self,

7
llama_stack/distribution/configure.py
View file

@ -99,6 +99,13 @@ def configure_api_providers(
cprint(f"Configuring API `{api_str}`...", "green", attrs=["bold"]) cprint(f"Configuring API `{api_str}`...", "green", attrs=["bold"])
updated_providers = [] updated_providers = []
for i, provider_type in enumerate(plist): for i, provider_type in enumerate(plist):
if i >= 1:
others = ", ".join(plist[i:])
print(
f"Not configuring other providers ({others}) interactively. Please edit the resulting YAML directly.\n"
)
break
print(f"> Configuring provider `({provider_type})`") print(f"> Configuring provider `({provider_type})`")
updated_providers.append( updated_providers.append(
configure_single_provider( configure_single_provider(