From 1132253b4cf1a263514be133cb0044419ab00f73 Mon Sep 17 00:00:00 2001 From: Ishaan Jaff Date: Sat, 17 Aug 2024 17:37:23 -0700 Subject: [PATCH] docs virtual key access groups --- docs/my-website/docs/proxy/virtual_keys.md | 242 ++++++++++++++++++--- 1 file changed, 213 insertions(+), 29 deletions(-) diff --git a/docs/my-website/docs/proxy/virtual_keys.md b/docs/my-website/docs/proxy/virtual_keys.md index ea4eefb302..65582b0f73 100644 --- a/docs/my-website/docs/proxy/virtual_keys.md +++ b/docs/my-website/docs/proxy/virtual_keys.md @@ -226,7 +226,68 @@ Expected Response ## **Model Access** -### Restrict models by `team_id` +### **Restrict models by Virtual Key** + +Set allowed models for a key using the `models` param + +:::info + +This key can only make requests to `models` that are `gpt-3.5-turbo` or `gpt-4` + +::: + +```shell +curl 'http://0.0.0.0:4000/key/generate' \ +--header 'Authorization: Bearer ' \ +--header 'Content-Type: application/json' \ +--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4"]}' +``` + + +Verify this is set correctly by + + + + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "gpt-4", + "messages": [ + {"role": "user", "content": "Hello"} + ] + }' +``` + + + + + +:::info + +Expect this to fail since gpt-4o is not in the `models` for the key generated + +::: + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "gpt-4o", + "messages": [ + {"role": "user", "content": "Hello"} + ] + }' +``` + + + + + +### **Restrict models by `team_id`** `litellm-dev` can only access `azure-gpt-3.5` **1. Create a team via `/team/new`** @@ -270,6 +331,157 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \ {"error":{"message":"Invalid model for team litellm-dev: BEDROCK_GROUP. Valid models for team are: ['azure-gpt-3.5']\n\n\nTraceback (most recent call last):\n File \"/Users/ishaanjaffer/Github/litellm/litellm/proxy/proxy_server.py\", line 2298, in chat_completion\n _is_valid_team_configs(\n File \"/Users/ishaanjaffer/Github/litellm/litellm/proxy/utils.py\", line 1296, in _is_valid_team_configs\n raise Exception(\nException: Invalid model for team litellm-dev: BEDROCK_GROUP. Valid models for team are: ['azure-gpt-3.5']\n\n","type":"None","param":"None","code":500}}% ``` +### **Grant Access to new model (Access Groups)** + +Use model access groups to give users access to select models, and add new ones to it over time (e.g. mistral, llama-2, etc.) + +**Step 1. Assign model, access group in config.yaml** + +```yaml +model_list: + - model_name: gpt-4 + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + model_info: + access_groups: ["beta-models"] # 👈 Model Access Group + - model_name: fireworks-llama-v3-70b-instruct + litellm_params: + model: fireworks_ai/accounts/fireworks/models/llama-v3-70b-instruct + api_key: "os.environ/FIREWORKS" + model_info: + access_groups: ["beta-models"] # 👈 Model Access Group +``` + + + + + +**Create key with access group** + +```bash +curl --location 'http://localhost:4000/key/generate' \ +-H 'Authorization: Bearer ' \ +-H 'Content-Type: application/json' \ +-d '{"models": ["beta-models"], # 👈 Model Access Group + "max_budget": 0,}' +``` + +Test Key + + + + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-" \ + -d '{ + "model": "gpt-4", + "messages": [ + {"role": "user", "content": "Hello"} + ] + }' +``` + + + + + +:::info + +Expect this to fail since gpt-4o is not in the `beta-models` access group + +::: + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-" \ + -d '{ + "model": "gpt-4o", + "messages": [ + {"role": "user", "content": "Hello"} + ] + }' +``` + + + + + + + + + +Create Team + +```shell +curl --location 'http://localhost:4000/team/new' \ +-H 'Authorization: Bearer sk-' \ +-H 'Content-Type: application/json' \ +-d '{"models": ["beta-models"]}' +``` + +Create Key for Team + +```shell +curl --location 'http://0.0.0.0:4000/key/generate' \ +--header 'Authorization: Bearer sk-' \ +--header 'Content-Type: application/json' \ +--data '{"team_id": "0ac97648-c194-4c90-8cd6-40af7b0d2d2a"} +``` + + +Test Key + + + + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-" \ + -d '{ + "model": "gpt-4", + "messages": [ + {"role": "user", "content": "Hello"} + ] + }' +``` + + + + + +:::info + +Expect this to fail since gpt-4o is not in the `beta-models` access group + +::: + +```shell +curl -i http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-" \ + -d '{ + "model": "gpt-4o", + "messages": [ + {"role": "user", "content": "Hello"} + ] + }' +``` + + + + + + + + + + ### Model Aliases If a user is expected to use a given model (i.e. gpt3-5), and you want to: @@ -320,34 +532,6 @@ curl -X POST "https://0.0.0.0:4000/key/generate" \ - **How are routing between diff keys/api bases done?** litellm handles this by shuffling between different models in the model list with the same model_name. [**See Code**](https://github.com/BerriAI/litellm/blob/main/litellm/router.py) -### Grant Access to new model - -Use model access groups to give users access to select models, and add new ones to it over time (e.g. mistral, llama-2, etc.) - -**Step 1. Assign model, access group in config.yaml** - -```yaml -model_list: - - model_name: text-embedding-ada-002 - litellm_params: - model: azure/azure-embedding-model - api_base: "os.environ/AZURE_API_BASE" - api_key: "os.environ/AZURE_API_KEY" - api_version: "2023-07-01-preview" - model_info: - access_groups: ["beta-models"] # 👈 Model Access Group -``` - -**Step 2. Create key with access group** - -```bash -curl --location 'http://localhost:4000/key/generate' \ --H 'Authorization: Bearer ' \ --H 'Content-Type: application/json' \ --d '{"models": ["beta-models"], # 👈 Model Access Group - "max_budget": 0,}' -``` - ## Advanced ### Pass LiteLLM Key in custom header