llama-stack-mirror/docs/source/distributions/self_hosted_distro/ramalama.md

orphan
true

RamaLama Distribution

:maxdepth: 2
:hidden:

self

The llamastack/distribution-ramalama distribution consists of the following provider configurations.

API	Provider(s)
agents	inline::meta-reference
datasetio	remote::huggingface, inline::localfs
eval	inline::meta-reference
inference	remote::ramalama
safety	inline::llama-guard
scoring	inline::basic, inline::llm-as-judge, inline::braintrust
telemetry	inline::meta-reference
tool_runtime	remote::brave-search, remote::tavily-search, inline::code-interpreter, inline::rag-runtime
vector_io	inline::faiss, remote::chromadb, remote::pgvector

You should use this distribution if you have a regular desktop machine without very powerful GPUs. Of course, if you have powerful GPUs, you can still continue using this distribution since RamaLama supports GPU acceleration.

Environment Variables

The following environment variables can be configured:

• LLAMA_STACK_PORT: Port for the Llama Stack distribution server (default: 8321)
• RAMALAMA_URL: URL of the RamaLama server (default: http://0.0.0.0:8080/v1)
• INFERENCE_MODEL: Inference model loaded into the RamaLama server (default: meta-llama/Llama-3.2-3B-Instruct)
• SAFETY_MODEL: Safety model loaded into the RamaLama server (default: meta-llama/Llama-Guard-3-1B)

Setting up RamaLama server

Please check the RamaLama Documentation on how to install and run RamaLama. After installing RamaLama, you need to run ramalama serve to start the server.

In order to load models, you can run:

export RAMALAMA_INFERENCE_MODEL="llama3.2:3b-instruct-fp16"

export INFERENCE_MODEL="~/path_to_model/meta-llama/Llama-3.2-3B-Instruct"

ramalama serve $RAMALAMA_INFERENCE_MODEL

RamaLama requires the inference model to be the fully qualified path to the model on disk when running on MacOS, on Linux it can just be the model name.

If you are using Llama Stack Safety / Shield APIs, you will also need to pull and run the safety model.

export SAFETY_MODEL="meta-llama/Llama-Guard-3-1B"

# ramalama names this model differently, and we must use the ramalama name when loading the model
export RAMALAMA_SAFETY_MODEL="llama-guard3:1b"
ramalama run $RAMALAMA_SAFETY_MODEL --keepalive 60m

Running Llama Stack

Now you are ready to run Llama Stack with RamaLama as the inference provider. You can do this via Conda, Venv, or Podman which has a pre-built image.

Via Podman

This method allows you to get started quickly without having to build the distribution code.

export LLAMA_STACK_PORT=5001
podman run \
  -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama:z \
  llamastack/distribution-ramalama \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env RAMALAMA_URL=http://0.0.0.0:8080/v1

If you are using Llama Stack Safety / Shield APIs, use:

# You need a local checkout of llama-stack to run this, get it using
# git clone https://github.com/meta-llama/llama-stack.git
cd /path/to/llama-stack

podman run \
  -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama:z \
  -v ./llama_stack/templates/ramalama/run-with-safety.yaml:/root/my-run.yaml:z \
  llamastack/distribution-ramalama \
  --yaml-config /root/my-run.yaml \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env SAFETY_MODEL=$SAFETY_MODEL \
  --env RAMALAMA_URL=http://host.containers.internal:8080/v1

Via Docker

This method allows you to get started quickly without having to build the distribution code.

export LLAMA_STACK_PORT=5001
docker run \
  -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama \
  llamastack/distribution-ramalama \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env RAMALAMA_URL=http://host.docker.internal:8080/v1

If you are using Llama Stack Safety / Shield APIs, use:

# You need a local checkout of llama-stack to run this, get it using
# git clone https://github.com/meta-llama/llama-stack.git
cd /path/to/llama-stack

docker run \
  -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama \
  -v ./llama_stack/templates/ramalama/run-with-safety.yaml:/root/my-run.yaml \
  llamastack/distribution-ramalama \
  --yaml-config /root/my-run.yaml \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env SAFETY_MODEL=$SAFETY_MODEL \
  --env RAMALAMA_URL=http://host.docker.internal:8080/v1

Via Conda

Make sure you have done uv pip install llama-stack and have the Llama Stack CLI available.

export LLAMA_STACK_PORT=5001

llama stack build --template ramalama --image-type conda
llama stack run ./run.yaml \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env RAMALAMA_URL=http://host.docker.internal:8080/v1

If you are using Llama Stack Safety / Shield APIs, use:

llama stack run ./run-with-safety.yaml \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env SAFETY_MODEL=$SAFETY_MODEL \
  --env RAMALAMA_URL=http://host.docker.internal:8080/v1

(Optional) Update Model Serving Configuration

Please check the [model_aliases](https://github.com/meta-llama/llama-stack/blob/main/llama_stack/providers/remote/inference/ramalama/ramalama.py#L45) for the supported RamaLama models.

To serve a new model with ramalama

ramalama run <model_name>

To make sure that the model is being served correctly, run ramalama ps to get a list of models being served by ramalama.

$ ramalama ps

NAME                         ID              SIZE     PROCESSOR    UNTIL
llama3.1:8b-instruct-fp16    4aacac419454    17 GB    100% GPU     4 minutes from now

To verify that the model served by ramalama is correctly connected to Llama Stack server

$ llama-stack-client models list
+----------------------+----------------------+---------------+-----------------------------------------------+
| identifier           | llama_model          | provider_id   | metadata                                      |
+======================+======================+===============+===============================================+
| Llama3.1-8B-Instruct | Llama3.1-8B-Instruct | ramalama0       | {'ramalama_model': 'llama3.1:8b-instruct-fp16'} |
+----------------------+----------------------+---------------+-----------------------------------------------+