llama-stack-mirror/docs/source/contributing/new_api_provider.md

Adding a New API Provider

This guide will walk you through the process of adding a new API provider to Llama Stack.

• Begin by reviewing the core concepts of Llama Stack and choose the API your provider belongs to (Inference, Safety, VectorIO, etc.)
• Determine the provider type ({repopath}Remote::llama_stack/providers/remote or {repopath}Inline::llama_stack/providers/inline). Remote providers make requests to external services, while inline providers execute implementation locally.
• Add your provider to the appropriate {repopath}Registry::llama_stack/providers/registry/. Specify pip dependencies necessary.
• Update any distribution {repopath}Templates::llama_stack/templates/ build.yaml and run.yaml files if they should include your provider by default. Run {repopath}./scripts/distro_codegen.py if necessary. Note that distro_codegen.py will fail if the new provider causes any distribution template to attempt to import provider-specific dependencies. This usually means the distribution's get_distribution_template() code path should only import any necessary Config or model alias definitions from each provider and not the provider's actual implementation.

Here are some example PRs to help you get started:

• Grok Inference Implementation
• Nvidia Inference Implementation
• Model context protocol Tool Runtime

Testing the Provider

Before running tests, you must have required dependencies installed. This depends on the providers or distributions you are testing. For example, if you are testing the together distribution, you should install dependencies via llama stack build --template together.

1. Integration Testing

Integration tests are located in {repopath}tests/integration. These tests use the python client-SDK APIs (from the llama_stack_client package) to test functionality. Since these tests use client APIs, they can be run either by pointing to an instance of the Llama Stack server or "inline" by using LlamaStackAsLibraryClient.

Consult {repopath}tests/integration/README.md for more details on how to run the tests.

Note that each provider's sample_run_config() method (in the configuration class for that provider) typically references some environment variables for specifying API keys and the like. You can set these in the environment or pass these via the --env flag to the test command.

2. Unit Testing

Unit tests are located in {repopath}tests/unit. Provider-specific unit tests are located in {repopath}tests/unit/providers. These tests are all run automatically as part of the CI process.

3. Additional end-to-end testing

1. Start a Llama Stack server with your new provider
2. Verify compatibility with existing client scripts in the llama-stack-apps repository
3. Document which scripts are compatible with your provider

Submitting Your PR

1. Ensure all tests pass
2. Include a comprehensive test plan in your PR summary
3. Document any known limitations or considerations