llama-stack-mirror/docs/docs/providers/external/external-providers-guide.mdx

# Creating External Providers

## Configuration

To enable external providers, you need to add `module` into your build yaml, allowing Llama Stack to install the required package corresponding to the external provider.

an example entry in your build.yaml should look like:

```
- provider_type: remote::ramalama
  module: ramalama_stack
```

## Provider Types

Llama Stack supports two types of external providers:

1. **Remote Providers**: Providers that communicate with external services (e.g., cloud APIs)
2. **Inline Providers**: Providers that run locally within the Llama Stack process


### Provider Specification (Common between inline and remote providers)

- `provider_type`: The type of the provider to be installed (remote or inline). eg. `remote::ollama`
- `api`: The API for this provider, eg. `inference`
- `config_class`: The full path to the configuration class
- `module`: The Python module containing the provider implementation
- `optional_api_dependencies`: List of optional Llama Stack APIs that this provider can use
- `api_dependencies`:  List of Llama Stack APIs that this provider depends on
- `provider_data_validator`: Optional validator for provider data.
- `pip_packages`: List of Python packages required by the provider

### Remote Provider Specification

Remote providers are used when you need to communicate with external services. Here's an example for a custom Ollama provider:

```yaml
adapter_type: custom_ollama
provider_type: "remote::ollama"
pip_packages:
- ollama
- aiohttp
config_class: llama_stack_ollama_provider.config.OllamaImplConfig
module: llama_stack_ollama_provider
api_dependencies: []
optional_api_dependencies: []
```

#### Remote Provider Configuration

- `adapter_type`: A unique identifier for this adapter, eg. `ollama`

### Inline Provider Specification

Inline providers run locally within the Llama Stack process. Here's an example for a custom vector store provider:

```yaml
module: llama_stack_vector_provider
provider_type: inline::llama_stack_vector_provider
config_class: llama_stack_vector_provider.config.VectorStoreConfig
pip_packages:
  - faiss-cpu
  - numpy
api_dependencies:
  - inference
optional_api_dependencies:
  - vector_io
provider_data_validator: llama_stack_vector_provider.validator.VectorStoreValidator
container_image: custom-vector-store:latest  # optional
```

#### Inline Provider Fields

- `container_image`: Optional container image to use instead of pip packages

## Required Fields

### All Providers

All providers must contain a `get_provider_spec` function in their `provider` module. This is a standardized structure that Llama Stack expects and is necessary for getting things such as the config class. The `get_provider_spec` method returns a structure identical to the `adapter`. An example function may look like:

```python
from llama_stack.providers.datatypes import (
    ProviderSpec,
    Api,
    RemoteProviderSpec,
)


def get_provider_spec() -> ProviderSpec:
    return RemoteProviderSpec(
        api=Api.inference,
        adapter_type="ramalama",
        pip_packages=["ramalama>=0.8.5", "pymilvus"],
        config_class="ramalama_stack.config.RamalamaImplConfig",
        module="ramalama_stack",
    )
```

#### Remote Providers

Remote providers must expose a `get_adapter_impl()` function in their module that takes two arguments:
1. `config`: An instance of the provider's config class
2. `deps`: A dictionary of API dependencies

This function must return an instance of the provider's adapter class that implements the required protocol for the API.

Example:
```python
async def get_adapter_impl(
    config: OllamaImplConfig, deps: Dict[Api, Any]
) -> OllamaInferenceAdapter:
    return OllamaInferenceAdapter(config)
```

#### Inline Providers

Inline providers must expose a `get_provider_impl()` function in their module that takes two arguments:
1. `config`: An instance of the provider's config class
2. `deps`: A dictionary of API dependencies

Example:
```python
async def get_provider_impl(
    config: VectorStoreConfig, deps: Dict[Api, Any]
) -> VectorStoreImpl:
    impl = VectorStoreImpl(config, deps[Api.inference])
    await impl.initialize()
    return impl
```

## Dependencies

The provider package must be installed on the system. For example:

```bash
$ uv pip show llama-stack-ollama-provider
Name: llama-stack-ollama-provider
Version: 0.1.0
Location: /path/to/venv/lib/python3.10/site-packages
```

## Best Practices

1. **Package Naming**: Use the prefix `llama-stack-provider-` for your provider packages to make them easily identifiable.

2. **Version Management**: Keep your provider package versioned and compatible with the Llama Stack version you're using.

3. **Dependencies**: Only include the minimum required dependencies in your provider package.

4. **Documentation**: Include clear documentation in your provider package about:
   - Installation requirements
   - Configuration options
   - Usage examples
   - Any limitations or known issues

5. **Testing**: Include tests in your provider package to ensure it works correctly with Llama Stack.
You can refer to the [integration tests
guide](https://github.com/meta-llama/llama-stack/blob/main/tests/integration/README.md) for more
information. Execute the test for the Provider type you are developing.

## Troubleshooting

If your external provider isn't being loaded:

1. Check that `module` points to a published pip package with a top level `provider` module including `get_provider_spec`.
2. Verify that the YAML files are properly formatted.
3. Ensure all required Python packages are installed.
4. Check the Llama Stack server logs for any error messages - turn on debug logging to get more
   information using `LLAMA_STACK_LOGGING=all=debug`.

## Examples

### How to create an external provider module

If you are creating a new external provider called `llama-stack-provider-ollama` here is how you would set up the package properly:

1. First, create the provider package:

```bash
mkdir -p llama-stack-provider-ollama
cd llama-stack-provider-ollama
git init
uv init
```

2. Edit `pyproject.toml`:

```toml
[project]
name = "llama-stack-provider-ollama"
version = "0.1.0"
description = "Ollama provider for Llama Stack"
requires-python = ">=3.12"
dependencies = ["llama-stack", "pydantic", "ollama", "aiohttp"]
```

3. Install the provider:

```bash
uv pip install -e .
```

4. Edit `provider.py`

provider.py must be updated to contain `get_provider_spec`. This is used by llama stack to install the provider.

```python
def get_provider_spec() -> ProviderSpec:
    return RemoteProviderSpec(
        api=Api.inference,
        adapter_type="llama-stack-provider-ollama",
        pip_packages=["ollama", "aiohttp"],
        config_class="llama_stack_provider_ollama.config.OllamaImplConfig",
        module="llama_stack_provider_ollama",
    )
```

5. Implement the provider as outlined above with `get_provider_impl` or `get_adapter_impl`, etc.

### Example using `module`: ramalama-stack

[ramalama-stack](https://github.com/containers/ramalama-stack) is a recognized external provider that supports installation via module.

To install Llama Stack with this external provider a user can provider the following build.yaml:

```yaml
version: 2
distribution_spec:
  description: Use (an external) Ramalama server for running LLM inference
  container_image: null
  providers:
    inference:
    - provider_type: remote::ramalama
      module: ramalama_stack==0.3.0a0
image_type: venv
image_name: null
additional_pip_packages:
- aiosqlite
- sqlalchemy[asyncio]
```

No other steps are required beyond installing dependencies with `llama stack list-deps <distro> | xargs -L1 uv pip install` and then running `llama stack run`. The CLI will use `module` to install the provider dependencies, retrieve the spec, etc.

The provider will now be available in Llama Stack with the type `remote::ramalama`.