From d9433d9f9432afdc75f3362ab42227a85a7ad25c Mon Sep 17 00:00:00 2001 From: Ishaan Jaff Date: Thu, 29 Aug 2024 14:14:37 -0700 Subject: [PATCH] doc Tag Based Routing --- docs/my-website/docs/proxy/tag_routing.md | 159 +++++++++++++++++++++- 1 file changed, 157 insertions(+), 2 deletions(-) diff --git a/docs/my-website/docs/proxy/tag_routing.md b/docs/my-website/docs/proxy/tag_routing.md index c3d533564..81012c900 100644 --- a/docs/my-website/docs/proxy/tag_routing.md +++ b/docs/my-website/docs/proxy/tag_routing.md @@ -1,7 +1,11 @@ # Tag Based Routing Route requests based on tags. -This is useful for implementing free / paid tiers for users +This is useful for +- implementing free / paid tiers for users +- controlling model access per team, example Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B + +## Quick Start ### 1. Define tags on config.yaml @@ -130,4 +134,155 @@ Response "total_tokens": 22 } } -``` \ No newline at end of file +``` + +## ✨ Team based tag routing (Enterprise) + +LiteLLM Proxy supports team-based tag routing, allowing you to associate specific tags with teams and route requests accordingly. Example **Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B** + + +:::info + +This is an enterprise feature, [Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) + +::: + +Here's how to set up and use team-based tag routing using curl commands: + +1. **Enable tag filtering in your proxy configuration:** + + In your `proxy_config.yaml`, ensure you have the following setting: + + ```yaml + model_list: + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + tags: ["teamA"] # 👈 Key Change + model_info: + id: "team-a-model" # used for identifying model in response headers + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + tags: ["teamB"] # 👈 Key Change + model_info: + id: "team-b-model" # used for identifying model in response headers + + router_settings: + enable_tag_filtering: True # 👈 Key Change + + general_settings: + master_key: sk-1234 + ``` + +2. **Create teams with tags:** + + Use the `/team/new` endpoint to create teams with specific tags: + + ```shell + # Create Team A + curl -X POST http://0.0.0.0:4000/team/new \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"tags": ["teamA"]}' + ``` + + ```shell + # Create Team B + curl -X POST http://0.0.0.0:4000/team/new \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"tags": ["teamB"]}' + ``` + + These commands will return JSON responses containing the `team_id` for each team. + +3. **Generate keys for team members:** + + Use the `/key/generate` endpoint to create keys associated with specific teams: + + ```shell + # Generate key for Team A + curl -X POST http://0.0.0.0:4000/key/generate \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"team_id": "team_a_id_here"}' + ``` + + ```shell + # Generate key for Team B + curl -X POST http://0.0.0.0:4000/key/generate \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"team_id": "team_b_id_here"}' + ``` + + Replace `team_a_id_here` and `team_b_id_here` with the actual team IDs received from step 2. + +4. **Make requests with team-specific keys:** + + When making requests to the `/chat/completions` endpoint, use the team-specific keys. The proxy will automatically route the request to the appropriate model based on the team's tags: + + ```shell + # Request using Team A's key + curl -X POST http://0.0.0.0:4000/chat/completions \ + -H "Authorization: Bearer team_a_key_here" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "fake-openai-endpoint", + "messages": [ + {"role": "user", "content": "Hello!"} + ] + }' + ``` + + ```shell + # Request using Team B's key + curl -X POST http://0.0.0.0:4000/chat/completions \ + -H "Authorization: Bearer team_b_key_here" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "fake-openai-endpoint", + "messages": [ + {"role": "user", "content": "Hello!"} + ] + }' + ``` + + Replace `team_a_key_here` and `team_b_key_here` with the actual keys generated in step 3. + +5. **Verify routing:** + + Check the `x-litellm-model-id` header in the response to confirm that the request was routed to the correct model based on the team's tags. You can use the `-i` flag with curl to include the response headers: + + Request with Team A's key (including headers) + ```shell + curl -i -X POST http://0.0.0.0:4000/chat/completions \ + -H "Authorization: Bearer team_a_key_here" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "fake-openai-endpoint", + "messages": [ + {"role": "user", "content": "Hello!"} + ] + }' + ``` + + In the response headers, you should see: + ``` + x-litellm-model-id: teama + ``` + + Similarly, when using Team B's key, you should see: + ``` + x-litellm-model-id: teamb + ``` + +By following these steps and using these curl commands, you can implement and test team-based tag routing in your LiteLLM Proxy setup, ensuring that different teams are routed to the appropriate models or deployments based on their assigned tags. + + +