mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-08-03 17:29:01 +00:00
fix: rewording
Signed-off-by: Jeff MAURY <jmaury@redhat.com>
This commit is contained in:
parent
dd86427ce3
commit
aa68e98b7a
5 changed files with 11 additions and 13 deletions
|
@ -26,7 +26,7 @@ The `llamastack/distribution-podman-ai-lab` distribution consists of the followi
|
||||||
| vector_io | `inline::faiss`, `remote::chromadb`, `remote::pgvector` |
|
| vector_io | `inline::faiss`, `remote::chromadb`, `remote::pgvector` |
|
||||||
|
|
||||||
|
|
||||||
You should use this distribution if you have a regular desktop machine without very powerful GPUs. Of course, if you have powerful GPUs, you can still continue using this distribution since Ollama supports GPU acceleration.
|
You should use this distribution if you have a regular desktop machine without very powerful GPUs. Of course, if you have powerful GPUs, you can still continue using this distribution since Podman AI Lab supports GPU acceleration.
|
||||||
|
|
||||||
### Environment Variables
|
### Environment Variables
|
||||||
|
|
||||||
|
@ -34,12 +34,12 @@ The following environment variables can be configured:
|
||||||
|
|
||||||
- `LLAMA_STACK_PORT`: Port for the Llama Stack distribution server (default: `5001`)
|
- `LLAMA_STACK_PORT`: Port for the Llama Stack distribution server (default: `5001`)
|
||||||
- `PODMAN_AI_LAB_URL`: URL of the Podman AI Lab server (default: `http://127.0.0.1:10434`)
|
- `PODMAN_AI_LAB_URL`: URL of the Podman AI Lab server (default: `http://127.0.0.1:10434`)
|
||||||
- `SAFETY_MODEL`: Safety model loaded into the Ollama server (default: `meta-llama/Llama-Guard-3-1B`)
|
- `SAFETY_MODEL`: Safety model loaded into the Podman AI Lab server (default: `meta-llama/Llama-Guard-3-1B`)
|
||||||
|
|
||||||
|
|
||||||
## Setting up Podman AI Lab server
|
## Setting up Podman AI Lab server
|
||||||
|
|
||||||
Please check the [Podman AI Lab Documentation](https://github.com/containers/podman-desktop-extension-ai-lab) on how to install and run Ollama. After installing Ollama, you need to run `ollama serve` to start the server.
|
Please check the [Podman AI Lab Documentation](https://github.com/containers/podman-desktop-extension-ai-lab) on how to install and run Podman AI Lab.
|
||||||
|
|
||||||
|
|
||||||
If you are using Llama Stack Safety / Shield APIs, you will also need to pull and run the safety model.
|
If you are using Llama Stack Safety / Shield APIs, you will also need to pull and run the safety model.
|
||||||
|
@ -47,7 +47,6 @@ If you are using Llama Stack Safety / Shield APIs, you will also need to pull an
|
||||||
```bash
|
```bash
|
||||||
export SAFETY_MODEL="meta-llama/Llama-Guard-3-1B"
|
export SAFETY_MODEL="meta-llama/Llama-Guard-3-1B"
|
||||||
|
|
||||||
# ollama names this model differently, and we must use the ollama name when loading the model
|
|
||||||
export PODMAN_AI_LAB_SAFETY_MODEL="llama-guard3:1b"
|
export PODMAN_AI_LAB_SAFETY_MODEL="llama-guard3:1b"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -81,7 +80,7 @@ docker run \
|
||||||
-it \
|
-it \
|
||||||
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
|
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
|
||||||
-v ~/.llama:/root/.llama \
|
-v ~/.llama:/root/.llama \
|
||||||
-v ./llama_stack/templates/ollama/run-with-safety.yaml:/root/my-run.yaml \
|
-v ./llama_stack/templates/podman-ai-lab/run-with-safety.yaml:/root/my-run.yaml \
|
||||||
llamastack/distribution-podman-ai-lab \
|
llamastack/distribution-podman-ai-lab \
|
||||||
--yaml-config /root/my-run.yaml \
|
--yaml-config /root/my-run.yaml \
|
||||||
--port $LLAMA_STACK_PORT \
|
--port $LLAMA_STACK_PORT \
|
||||||
|
|
|
@ -14,7 +14,7 @@ The `llamastack/distribution-{{ name }}` distribution consists of the following
|
||||||
|
|
||||||
{{ providers_table }}
|
{{ providers_table }}
|
||||||
|
|
||||||
You should use this distribution if you have a regular desktop machine without very powerful GPUs. Of course, if you have powerful GPUs, you can still continue using this distribution since Ollama supports GPU acceleration.
|
You should use this distribution if you have a regular desktop machine without very powerful GPUs. Of course, if you have powerful GPUs, you can still continue using this distribution since Podman AI Lab supports GPU acceleration.
|
||||||
|
|
||||||
{% if run_config_env_vars %}
|
{% if run_config_env_vars %}
|
||||||
### Environment Variables
|
### Environment Variables
|
||||||
|
@ -29,7 +29,7 @@ The following environment variables can be configured:
|
||||||
|
|
||||||
## Setting up Podman AI Lab server
|
## Setting up Podman AI Lab server
|
||||||
|
|
||||||
Please check the [Podman AI Lab Documentation](https://github.com/containers/podman-desktop-extension-ai-lab) on how to install and run Ollama. After installing Ollama, you need to run `ollama serve` to start the server.
|
Please check the [Podman AI Lab Documentation](https://github.com/containers/podman-desktop-extension-ai-lab) on how to install and run Podman AI Lab.
|
||||||
|
|
||||||
|
|
||||||
If you are using Llama Stack Safety / Shield APIs, you will also need to pull and run the safety model.
|
If you are using Llama Stack Safety / Shield APIs, you will also need to pull and run the safety model.
|
||||||
|
@ -37,7 +37,6 @@ If you are using Llama Stack Safety / Shield APIs, you will also need to pull an
|
||||||
```bash
|
```bash
|
||||||
export SAFETY_MODEL="meta-llama/Llama-Guard-3-1B"
|
export SAFETY_MODEL="meta-llama/Llama-Guard-3-1B"
|
||||||
|
|
||||||
# ollama names this model differently, and we must use the ollama name when loading the model
|
|
||||||
export PODMAN_AI_LAB_SAFETY_MODEL="llama-guard3:1b"
|
export PODMAN_AI_LAB_SAFETY_MODEL="llama-guard3:1b"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -71,7 +70,7 @@ docker run \
|
||||||
-it \
|
-it \
|
||||||
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
|
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
|
||||||
-v ~/.llama:/root/.llama \
|
-v ~/.llama:/root/.llama \
|
||||||
-v ./llama_stack/templates/ollama/run-with-safety.yaml:/root/my-run.yaml \
|
-v ./llama_stack/templates/podman-ai-lab/run-with-safety.yaml:/root/my-run.yaml \
|
||||||
llamastack/distribution-{{ name }} \
|
llamastack/distribution-{{ name }} \
|
||||||
--yaml-config /root/my-run.yaml \
|
--yaml-config /root/my-run.yaml \
|
||||||
--port $LLAMA_STACK_PORT \
|
--port $LLAMA_STACK_PORT \
|
||||||
|
|
|
@ -50,7 +50,7 @@ def get_distribution_template() -> DistributionTemplate:
|
||||||
|
|
||||||
safety_model = ModelInput(
|
safety_model = ModelInput(
|
||||||
model_id="${env.SAFETY_MODEL}",
|
model_id="${env.SAFETY_MODEL}",
|
||||||
provider_id="ollama",
|
provider_id="podman-ai-lab",
|
||||||
)
|
)
|
||||||
default_tool_groups = [
|
default_tool_groups = [
|
||||||
ToolGroupInput(
|
ToolGroupInput(
|
||||||
|
@ -131,7 +131,7 @@ def get_distribution_template() -> DistributionTemplate:
|
||||||
),
|
),
|
||||||
"SAFETY_MODEL": (
|
"SAFETY_MODEL": (
|
||||||
"meta-llama/Llama-Guard-3-1B",
|
"meta-llama/Llama-Guard-3-1B",
|
||||||
"Safety model loaded into the Ollama server",
|
"Safety model loaded into the Podman AI Lab server",
|
||||||
),
|
),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
# Report for Podman AI Lab distribution
|
# Report for Podman AI Lab distribution
|
||||||
|
|
||||||
## Supported Models
|
## Supported Models
|
||||||
| Model Descriptor | ollama |
|
| Model Descriptor | Podman AI Lab |
|
||||||
|:---|:---|
|
|:---|:---|
|
||||||
| Llama-3-8B-Instruct | ❌ |
|
| Llama-3-8B-Instruct | ❌ |
|
||||||
| Llama-3-70B-Instruct | ❌ |
|
| Llama-3-70B-Instruct | ❌ |
|
||||||
|
|
|
@ -109,7 +109,7 @@ metadata_store:
|
||||||
models:
|
models:
|
||||||
- metadata: {}
|
- metadata: {}
|
||||||
model_id: ${env.SAFETY_MODEL}
|
model_id: ${env.SAFETY_MODEL}
|
||||||
provider_id: ollama
|
provider_id: podman-ai-lab
|
||||||
model_type: llm
|
model_type: llm
|
||||||
shields:
|
shields:
|
||||||
- shield_id: ${env.SAFETY_MODEL}
|
- shield_id: ${env.SAFETY_MODEL}
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue