mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-07-22 12:37:53 +00:00
2a31163178
# What does this PR do? Automatically generates - build.yaml - run.yaml - run-with-safety.yaml - parts of markdown docs for the distributions. ## Test Plan At this point, this only updates the YAMLs and the docs. Some testing (especially with ollama and vllm) has been performed but needs to be much more tested.
134 lines
4.4 KiB
Markdown
134 lines
4.4 KiB
Markdown
# Ollama Distribution
|
|
|
|
The `llamastack/distribution-{{ name }}` distribution consists of the following provider configurations.
|
|
|
|
{{ providers_table }}
|
|
|
|
You should use this distribution if you have a regular desktop machine without very powerful GPUs. Of course, if you have powerful GPUs, you can still continue using this distribution since Ollama supports GPU acceleration.
|
|
|
|
{%- if run_config_env_vars %}
|
|
### Environment Variables
|
|
|
|
The following environment variables can be configured:
|
|
|
|
{% for var, (default_value, description) in run_config_env_vars.items() %}
|
|
- `{{ var }}`: {{ description }} (default: `{{ default_value }}`)
|
|
{% endfor %}
|
|
{% endif %}
|
|
|
|
|
|
## Setting up Ollama server
|
|
|
|
Please check the [Ollama Documentation](https://github.com/ollama/ollama) on how to install and run Ollama. After installing Ollama, you need to run `ollama serve` to start the server.
|
|
|
|
In order to load models, you can run:
|
|
|
|
```bash
|
|
export INFERENCE_MODEL="meta-llama/Llama-3.2-3B-Instruct"
|
|
|
|
# ollama names this model differently, and we must use the ollama name when loading the model
|
|
export OLLAMA_INFERENCE_MODEL="llama3.2:3b-instruct-fp16"
|
|
ollama run $OLLAMA_INFERENCE_MODEL --keepalive 60m
|
|
```
|
|
|
|
If you are using Llama Stack Safety / Shield APIs, you will also need to pull and run the safety model.
|
|
|
|
```bash
|
|
export SAFETY_MODEL="meta-llama/Llama-Guard-3-1B"
|
|
|
|
# ollama names this model differently, and we must use the ollama name when loading the model
|
|
export OLLAMA_SAFETY_MODEL="llama-guard3:1b"
|
|
ollama run $OLLAMA_SAFETY_MODEL --keepalive 60m
|
|
```
|
|
|
|
## Running Llama Stack
|
|
|
|
Now you are ready to run Llama Stack with Ollama as the inference provider. You can do this via Conda (build code) or Docker which has a pre-built image.
|
|
|
|
### Via Docker
|
|
|
|
This method allows you to get started quickly without having to build the distribution code.
|
|
|
|
```bash
|
|
LLAMA_STACK_PORT=5001
|
|
docker run \
|
|
-it \
|
|
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
|
|
-v ~/.llama:/root/.llama \
|
|
-v ./run.yaml:/root/my-run.yaml \
|
|
--gpus=all \
|
|
llamastack/distribution-{{ name }} \
|
|
/root/my-run.yaml \
|
|
--port $LLAMA_STACK_PORT \
|
|
--env INFERENCE_MODEL=$INFERENCE_MODEL \
|
|
--env OLLAMA_URL=http://host.docker.internal:11434
|
|
```
|
|
|
|
If you are using Llama Stack Safety / Shield APIs, use:
|
|
|
|
```bash
|
|
docker run \
|
|
-it \
|
|
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
|
|
-v ~/.llama:/root/.llama \
|
|
-v ./run-with-safety.yaml:/root/my-run.yaml \
|
|
--gpus=all \
|
|
llamastack/distribution-{{ name }} \
|
|
/root/my-run.yaml \
|
|
--port $LLAMA_STACK_PORT \
|
|
--env INFERENCE_MODEL=$INFERENCE_MODEL \
|
|
--env SAFETY_MODEL=$SAFETY_MODEL \
|
|
--env OLLAMA_URL=http://host.docker.internal:11434
|
|
```
|
|
|
|
### Via Conda
|
|
|
|
Make sure you have done `pip install llama-stack` and have the Llama Stack CLI available.
|
|
|
|
```bash
|
|
llama stack build --template ollama --image-type conda
|
|
llama stack run ./run.yaml \
|
|
--port 5001 \
|
|
--env INFERENCE_MODEL=$INFERENCE_MODEL \
|
|
--env OLLAMA_URL=http://127.0.0.1:11434
|
|
```
|
|
|
|
If you are using Llama Stack Safety / Shield APIs, use:
|
|
|
|
```bash
|
|
llama stack run ./run-with-safety.yaml \
|
|
--port 5001 \
|
|
--env INFERENCE_MODEL=$INFERENCE_MODEL \
|
|
--env SAFETY_MODEL=$SAFETY_MODEL \
|
|
--env OLLAMA_URL=http://127.0.0.1:11434
|
|
```
|
|
|
|
|
|
### (Optional) Update Model Serving Configuration
|
|
|
|
> [!NOTE]
|
|
> Please check the [OLLAMA_SUPPORTED_MODELS](https://github.com/meta-llama/llama-stack/blob/main/llama_stack/providers.remote/inference/ollama/ollama.py) for the supported Ollama models.
|
|
|
|
|
|
To serve a new model with `ollama`
|
|
```bash
|
|
ollama run <model_name>
|
|
```
|
|
|
|
To make sure that the model is being served correctly, run `ollama ps` to get a list of models being served by ollama.
|
|
```
|
|
$ ollama ps
|
|
|
|
NAME ID SIZE PROCESSOR UNTIL
|
|
llama3.1:8b-instruct-fp16 4aacac419454 17 GB 100% GPU 4 minutes from now
|
|
```
|
|
|
|
To verify that the model served by ollama is correctly connected to Llama Stack server
|
|
```bash
|
|
$ llama-stack-client models list
|
|
+----------------------+----------------------+---------------+-----------------------------------------------+
|
|
| identifier | llama_model | provider_id | metadata |
|
|
+======================+======================+===============+===============================================+
|
|
| Llama3.1-8B-Instruct | Llama3.1-8B-Instruct | ollama0 | {'ollama_model': 'llama3.1:8b-instruct-fp16'} |
|
|
+----------------------+----------------------+---------------+-----------------------------------------------+
|
|
```
|