forked from phoenix/litellm-mirror
docs - end user rate limiting
This commit is contained in:
parent
42078ac285
commit
a848a676af
1 changed files with 56 additions and 0 deletions
|
@ -374,6 +374,62 @@ curl --location 'http://0.0.0.0:4000/key/generate' \
|
|||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="per-end-user" label="For End User">
|
||||
|
||||
Use this to set rate limits for `user` passed to `/chat/completions`, without needing to create a key for every user
|
||||
|
||||
#### Step 1. Create Budget
|
||||
|
||||
Set a `tpm_limit` on the budget (You can also pass `rpm_limit` if needed)
|
||||
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/budget/new' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"budget_id" : "free-tier",
|
||||
"tpm_limit": 5
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
#### Step 2. Create `End-User` with Budget
|
||||
|
||||
We use `budget_id="free-tier"` from Step 1 when creating this new end user
|
||||
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/end_user/new' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"user_id" : "palantir",
|
||||
"budget_id": "free-tier"
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
#### Step 3. Pass end user id in `/chat/completions` requests
|
||||
|
||||
Pass the `user_id` from Step 2 as `user="palantir"`
|
||||
|
||||
```shell
|
||||
curl --location 'http://localhost:4000/chat/completions' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"model": "llama3",
|
||||
"user": "palantir",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "gm"
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue