phoenix-oss/llama-stack
1
0
Fork 0
forked from phoenix-oss/llama-stack-mirror
Code Pull requests Releases Packages 2 Activity Actions

update docs for adding new API providers (#855)

Browse source 
# What does this PR do?

update docs for adding new API providers
![Screenshot 2025-01-23 at 11 21
42 AM](https://github.com/user-attachments/assets/0d4621d4-ef7e-43cd-9c4a-3e8e0b49242f)
This commit is contained in:
Dinesh Yeduguru 2025-01-23 12:05:57 -08:00 • committed by GitHub
parent d0be9288a3
commit 86466b71a9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 44 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

58
docs/source/contributing/new_api_provider.md
View file

@ -1,27 +1,57 @@
# Adding a New API Provider # Adding a New API Provider
This guide contains references to walk you through adding a new API provider. This guide will walk you through the process of adding a new API provider to Llama Stack.
1. First, decide which API your provider falls into (e.g. Inference, Safety, Agents, Memory). ## Getting Started
2. Decide whether your provider is a remote provider, or inline implementation. A remote provider is a provider that makes a remote request to a service. An inline provider is a provider where implementation is executed locally. Checkout the examples, and follow the structure to add your own API provider. Please find the following code pointers:
- {repopath}`Remote Providers::llama_stack/providers/remote` 1. **Choose Your API Category**
- {repopath}`Inline Providers::llama_stack/providers/inline` - Determine which API category your provider belongs to (Inference, Safety, Agents, VectorIO)
- Review the core concepts of Llama Stack in the [concepts guide](../concepts/index.md)
3. [Build a Llama Stack distribution](https://llama-stack.readthedocs.io/en/latest/distributions/building_distro.html) with your API provider. 2. **Determine Provider Type**
4. Test your code! - **Remote Provider**: Makes requests to external services
- **Inline Provider**: Executes implementation locally
## Testing your newly added API providers Reference existing implementations:
- {repopath}`Remote Providers::llama_stack/providers/remote`
- {repopath}`Inline Providers::llama_stack/providers/inline`
1. Start with an _integration test_ for your provider. That means we will instantiate the real provider, pass it real configuration and if it is a remote service, we will actually hit the remote service. We **strongly** discourage mocking for these tests at the provider level. Llama Stack is first and foremost about integration so we need to make sure stuff works end-to-end. See {repopath}`tests/client-sdk` for an example. Example PRs:
- [Grok Inference Implementation](https://github.com/meta-llama/llama-stack/pull/609)
- [Nvidia Inference Implementation](https://github.com/meta-llama/llama-stack/pull/355)
- [Model context protocol Tool Runtime](https://github.com/meta-llama/llama-stack/pull/665)
3. **Register Your Provider**
- Add your provider to the appropriate {repopath}`Registry::llama_stack/providers/registry/`
- Specify any required pip dependencies
2. In addition, if you want to unit test functionality within your provider, feel free to do so. You can find some tests in {repopath}`llama_stack/providers/tests/inference/test_text_inference.py`. 4. **Integration**
- Update the run.yaml file to include your provider
- To make your provider a default option or create a new distribution, look at the teamplates in {repopath}`llama_stack/templates/` and run {repopath}`llama_stack/scripts/distro_codegen.py`
- Example PRs:
- [Adding Model Context Protocol Tool Runtime](https://github.com/meta-llama/llama-stack/pull/816)
3. Test with a client-server Llama Stack setup. (a) Start a Llama Stack server with your own distribution which includes the new provider. (b) Send a client request to the server. These client scripts can serve as lightweight tests. ## Testing Guidelines
You can find more complex client scripts [llama-stack-apps](https://github.com/meta-llama/llama-stack-apps/tree/main) repo. Note down which scripts works and do not work with your distribution. ### 1. Integration Testing
- Create integration tests that use real provider instances and configurations
- For remote services, test actual API interactions
- Avoid mocking at the provider level
- Reference examples in {repopath}`tests/client-sdk`
## Submit your PR ### 2. Unit Testing (Optional)
- Add unit tests for provider-specific functionality
- See examples in {repopath}`llama_stack/providers/tests/inference/test_text_inference.py`
After you have fully tested your newly added API provider, submit a PR with the attached test plan. You must have a Test Plan in the summary section of your PR. ### 3. End-to-End Testing
1. Start a Llama Stack server with your new provider
2. Test using client requests
3. Verify compatibility with existing client scripts in the [llama-stack-apps](https://github.com/meta-llama/llama-stack-apps/tree/main) repository
4. Document which scripts are compatible with your provider
## Submitting Your PR
1. Ensure all tests pass
2. Include a comprehensive test plan in your PR summary
3. Document any known limitations or considerations
4. Submit your pull request for review