forked from phoenix/litellm-mirror
docs(virtual_keys.md): add model access groups to docs
This commit is contained in:
parent
0414e40d4a
commit
a6cd068c49
2 changed files with 71 additions and 0 deletions
|
@ -39,4 +39,46 @@ curl --location 'http://0.0.0.0:8000/key/generate' \
|
|||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{"duration": "20m", "max_parallel_requests": 1}' # 👈 max parallel requests = 1
|
||||
```
|
||||
|
||||
## Grant Access to new model
|
||||
|
||||
Use model access groups to give users access to select models, and add new ones to it over time (e.g. mistral, llama-2, etc.).
|
||||
|
||||
Difference between doing this with `/key/generate` vs. `/user/new`? If you do it on `/user/new` it'll persist across multiple keys generated for that user.
|
||||
|
||||
**Step 1. Assign model, access group in config.yaml**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: text-embedding-ada-002
|
||||
litellm_params:
|
||||
model: azure/azure-embedding-model
|
||||
api_base: "os.environ/AZURE_API_BASE"
|
||||
api_key: "os.environ/AZURE_API_KEY"
|
||||
api_version: "2023-07-01-preview"
|
||||
model_info:
|
||||
access_groups: ["beta-models"] # 👈 Model Access Group
|
||||
```
|
||||
|
||||
**Step 2. Create key with access group**
|
||||
|
||||
```bash
|
||||
curl --location 'http://localhost:8000/user/new' \
|
||||
-H 'Authorization: Bearer <your-master-key>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"models": ["beta-models"], # 👈 Model Access Group
|
||||
"max_budget": 0}'
|
||||
```
|
||||
|
||||
|
||||
## Create new keys for existing user
|
||||
|
||||
Just include user_id in the `/key/generate` request.
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:8000/key/generate' \
|
||||
--header 'Authorization: Bearer <your-master-key>' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{"models": ["azure-models"], "user_id": "krrish@berri.ai"}'
|
||||
```
|
|
@ -136,6 +136,35 @@ curl -X POST "https://0.0.0.0:8000/key/generate" \
|
|||
- **How to upgrade / downgrade request?** Change the alias mapping
|
||||
- **How are routing between diff keys/api bases done?** litellm handles this by shuffling between different models in the model list with the same model_name. [**See Code**](https://github.com/BerriAI/litellm/blob/main/litellm/router.py)
|
||||
|
||||
|
||||
## Grant Access to new model
|
||||
|
||||
Use model access groups to give users access to select models, and add new ones to it over time (e.g. mistral, llama-2, etc.)
|
||||
|
||||
**Step 1. Assign model, access group in config.yaml**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: text-embedding-ada-002
|
||||
litellm_params:
|
||||
model: azure/azure-embedding-model
|
||||
api_base: "os.environ/AZURE_API_BASE"
|
||||
api_key: "os.environ/AZURE_API_KEY"
|
||||
api_version: "2023-07-01-preview"
|
||||
model_info:
|
||||
access_groups: ["beta-models"] # 👈 Model Access Group
|
||||
```
|
||||
|
||||
**Step 2. Create key with access group**
|
||||
|
||||
```bash
|
||||
curl --location 'http://localhost:8000/key/generate' \
|
||||
-H 'Authorization: Bearer <your-master-key>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"models": ["beta-models"], # 👈 Model Access Group
|
||||
"max_budget": 0,}'
|
||||
```
|
||||
|
||||
## Tracking Spend
|
||||
|
||||
You can get spend for a key by using the `/key/info` endpoint.
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue