mirror of
https://github.com/BerriAI/litellm.git
synced 2025-04-25 02:34:29 +00:00
update docs
This commit is contained in:
parent
7bf6769aa7
commit
04eb718f7a
2 changed files with 56 additions and 15 deletions
|
@ -6,17 +6,13 @@ import TabItem from '@theme/TabItem';
|
|||
|
||||
Get alerts for:
|
||||
|
||||
- Hanging LLM api calls
|
||||
- Slow LLM api calls
|
||||
- Failed LLM api calls
|
||||
- Budget Tracking per key/user
|
||||
- Spend Reports - Weekly & Monthly spend per Team, Tag
|
||||
- Failed db read/writes
|
||||
- Model outage alerting
|
||||
- Daily Reports:
|
||||
- **LLM** Top 5 slowest deployments
|
||||
- **LLM** Top 5 deployments with most failed requests
|
||||
- **Spend** Weekly & Monthly spend per Team, Tag
|
||||
| Category | Alert Type |
|
||||
|----------|------------|
|
||||
| **LLM Performance** | Hanging API calls, Slow API calls, Failed API calls, Model outage alerting |
|
||||
| **Budget & Spend** | Budget tracking per key/user, Soft budget alerts, Weekly & Monthly spend reports per Team/Tag |
|
||||
| **System Health** | Failed database read/writes |
|
||||
| **Daily Reports** | Top 5 slowest LLM deployments, Top 5 LLM deployments with most failed requests, Weekly & Monthly spend per Team/Tag |
|
||||
|
||||
|
||||
|
||||
Works across:
|
||||
|
@ -93,6 +89,51 @@ litellm_settings:
|
|||
redact_messages_in_exceptions: True
|
||||
```
|
||||
|
||||
### Soft Budget Alerts for Virtual Keys
|
||||
|
||||
Use this to send an alert when a key/team is close to it's budget running out
|
||||
|
||||
Step 1. Create a virtual key with a soft budget
|
||||
|
||||
Set the `soft_budget` to 0.001
|
||||
|
||||
```shell
|
||||
curl -X 'POST' \
|
||||
'http://localhost:4000/key/generate' \
|
||||
-H 'accept: application/json' \
|
||||
-H 'x-goog-api-key: sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"key_alias": "prod-app1",
|
||||
"team_id": "113c1a22-e347-4506-bfb2-b320230ea414",
|
||||
"soft_budget": 0.001
|
||||
}'
|
||||
```
|
||||
|
||||
Step 2. Send a request to the proxy with the virtual key
|
||||
|
||||
```shell
|
||||
curl http://0.0.0.0:4000/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-Nb5eCf427iewOlbxXIH4Ow" \
|
||||
-d '{
|
||||
"model": "openai/gpt-4",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "this is a test request, write a short poem"
|
||||
}
|
||||
]
|
||||
}'
|
||||
|
||||
```
|
||||
|
||||
Step 3. Check slack for Expected Alert
|
||||
|
||||
<Image img={require('../../img/soft_budget_alert.png')}/>
|
||||
|
||||
|
||||
|
||||
|
||||
### Add Metadata to alerts
|
||||
|
||||
|
@ -123,7 +164,7 @@ response = client.chat.completions.create(
|
|||
|
||||
<Image img={require('../../img/alerting_metadata.png')}/>
|
||||
|
||||
### Opting into specific alert types
|
||||
### Select specific alert types
|
||||
|
||||
Set `alert_types` if you want to Opt into only specific alert types. When alert_types is not set, all Default Alert Types are enabled.
|
||||
|
||||
|
@ -145,7 +186,7 @@ general_settings:
|
|||
]
|
||||
```
|
||||
|
||||
### Set specific slack channels per alert type
|
||||
### Map slack channels to alert type
|
||||
|
||||
Use this if you want to set specific channels per alert type
|
||||
|
||||
|
@ -243,7 +284,7 @@ curl -i http://localhost:4000/v1/chat/completions \
|
|||
```
|
||||
|
||||
|
||||
### Using MS Teams Webhooks
|
||||
### MS Teams Webhooks
|
||||
|
||||
MS Teams provides a slack compatible webhook url that you can use for alerting
|
||||
|
||||
|
@ -285,7 +326,7 @@ curl --location 'http://0.0.0.0:4000/health/services?service=slack' \
|
|||
|
||||
<Image img={require('../../img/ms_teams_alerting.png')}/>
|
||||
|
||||
### Using Discord Webhooks
|
||||
### Discord Webhooks
|
||||
|
||||
Discord provides a slack compatible webhook url that you can use for alerting
|
||||
|
||||
|
|
BIN
docs/my-website/img/soft_budget_alert.png
Normal file
BIN
docs/my-website/img/soft_budget_alert.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 47 KiB |
Loading…
Add table
Add a link
Reference in a new issue