mirror of
https://github.com/BerriAI/litellm.git
synced 2025-04-27 03:34:10 +00:00
docs virtual key access groups
This commit is contained in:
Ishaan Jaff
parent
888afa2d08
commit
1132253b4c
1 changed files with 213 additions and 29 deletions
|
|
@ -226,7 +226,68 @@ Expected Response
|
|||
|
||||
## **Model Access**
|
||||
|
||||
### Restrict models by `team_id`
|
||||
### **Restrict models by Virtual Key**
|
||||
|
||||
Set allowed models for a key using the `models` param
|
||||
|
||||
:::info
|
||||
|
||||
This key can only make requests to `models` that are `gpt-3.5-turbo` or `gpt-4`
|
||||
|
||||
:::
|
||||
|
||||
```shell
|
||||
curl 'http://0.0.0.0:4000/key/generate' \
|
||||
--header 'Authorization: Bearer <your-master-key>' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4"]}'
|
||||
```
|
||||
|
||||
|
||||
Verify this is set correctly by
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Allowed Access" value = "allowed">
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "gpt-4",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Disallowed Access" value = "not-allowed">
|
||||
|
||||
:::info
|
||||
|
||||
Expect this to fail since gpt-4o is not in the `models` for the key generated
|
||||
|
||||
:::
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "gpt-4o",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
### **Restrict models by `team_id`**
|
||||
`litellm-dev` can only access `azure-gpt-3.5`
|
||||
|
||||
**1. Create a team via `/team/new`**
|
||||
|
|
@ -270,6 +331,157 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
|||
{"error":{"message":"Invalid model for team litellm-dev: BEDROCK_GROUP. Valid models for team are: ['azure-gpt-3.5']\n\n\nTraceback (most recent call last):\n File \"/Users/ishaanjaffer/Github/litellm/litellm/proxy/proxy_server.py\", line 2298, in chat_completion\n _is_valid_team_configs(\n File \"/Users/ishaanjaffer/Github/litellm/litellm/proxy/utils.py\", line 1296, in _is_valid_team_configs\n raise Exception(\nException: Invalid model for team litellm-dev: BEDROCK_GROUP. Valid models for team are: ['azure-gpt-3.5']\n\n","type":"None","param":"None","code":500}}%
|
||||
```
|
||||
|
||||
### **Grant Access to new model (Access Groups)**
|
||||
|
||||
Use model access groups to give users access to select models, and add new ones to it over time (e.g. mistral, llama-2, etc.)
|
||||
|
||||
**Step 1. Assign model, access group in config.yaml**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-4
|
||||
litellm_params:
|
||||
model: openai/fake
|
||||
api_key: fake-key
|
||||
api_base: https://exampleopenaiendpoint-production.up.railway.app/
|
||||
model_info:
|
||||
access_groups: ["beta-models"] # 👈 Model Access Group
|
||||
- model_name: fireworks-llama-v3-70b-instruct
|
||||
litellm_params:
|
||||
model: fireworks_ai/accounts/fireworks/models/llama-v3-70b-instruct
|
||||
api_key: "os.environ/FIREWORKS"
|
||||
model_info:
|
||||
access_groups: ["beta-models"] # 👈 Model Access Group
|
||||
```
|
||||
|
||||
<Tabs>
|
||||
|
||||
<TabItem value="key" label="Key Access Groups">
|
||||
|
||||
**Create key with access group**
|
||||
|
||||
```bash
|
||||
curl --location 'http://localhost:4000/key/generate' \
|
||||
-H 'Authorization: Bearer <your-master-key>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"models": ["beta-models"], # 👈 Model Access Group
|
||||
"max_budget": 0,}'
|
||||
```
|
||||
|
||||
Test Key
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Allowed Access" value = "allowed">
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-<key-from-previous-step>" \
|
||||
-d '{
|
||||
"model": "gpt-4",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Disallowed Access" value = "not-allowed">
|
||||
|
||||
:::info
|
||||
|
||||
Expect this to fail since gpt-4o is not in the `beta-models` access group
|
||||
|
||||
:::
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-<key-from-previous-step>" \
|
||||
-d '{
|
||||
"model": "gpt-4o",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="team" label="Team Access Groups">
|
||||
|
||||
Create Team
|
||||
|
||||
```shell
|
||||
curl --location 'http://localhost:4000/team/new' \
|
||||
-H 'Authorization: Bearer sk-<key-from-previous-step>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"models": ["beta-models"]}'
|
||||
```
|
||||
|
||||
Create Key for Team
|
||||
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/key/generate' \
|
||||
--header 'Authorization: Bearer sk-<key-from-previous-step>' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{"team_id": "0ac97648-c194-4c90-8cd6-40af7b0d2d2a"}
|
||||
```
|
||||
|
||||
|
||||
Test Key
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Allowed Access" value = "allowed">
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-<key-from-previous-step>" \
|
||||
-d '{
|
||||
"model": "gpt-4",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Disallowed Access" value = "not-allowed">
|
||||
|
||||
:::info
|
||||
|
||||
Expect this to fail since gpt-4o is not in the `beta-models` access group
|
||||
|
||||
:::
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-<key-from-previous-step>" \
|
||||
-d '{
|
||||
"model": "gpt-4o",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
|
||||
### Model Aliases
|
||||
|
||||
If a user is expected to use a given model (i.e. gpt3-5), and you want to:
|
||||
|
|
@ -320,34 +532,6 @@ curl -X POST "https://0.0.0.0:4000/key/generate" \
|
|||
- **How are routing between diff keys/api bases done?** litellm handles this by shuffling between different models in the model list with the same model_name. [**See Code**](https://github.com/BerriAI/litellm/blob/main/litellm/router.py)
|
||||
|
||||
|
||||
### Grant Access to new model
|
||||
|
||||
Use model access groups to give users access to select models, and add new ones to it over time (e.g. mistral, llama-2, etc.)
|
||||
|
||||
**Step 1. Assign model, access group in config.yaml**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: text-embedding-ada-002
|
||||
litellm_params:
|
||||
model: azure/azure-embedding-model
|
||||
api_base: "os.environ/AZURE_API_BASE"
|
||||
api_key: "os.environ/AZURE_API_KEY"
|
||||
api_version: "2023-07-01-preview"
|
||||
model_info:
|
||||
access_groups: ["beta-models"] # 👈 Model Access Group
|
||||
```
|
||||
|
||||
**Step 2. Create key with access group**
|
||||
|
||||
```bash
|
||||
curl --location 'http://localhost:4000/key/generate' \
|
||||
-H 'Authorization: Bearer <your-master-key>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"models": ["beta-models"], # 👈 Model Access Group
|
||||
"max_budget": 0,}'
|
||||
```
|
||||
|
||||
## Advanced
|
||||
|
||||
### Pass LiteLLM Key in custom header
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue