phoenix/litellm
1
0
Fork 0
forked from phoenix/litellm-mirror
Code Pull requests Activity

(QOL improvement) Provider budget routing - allow using 1s, 1d, 1mo, 2mo etc (#6885)

Browse source 
* use 1 file for duration_in_seconds

* add to readme.md

* re use duration_in_seconds

* fix importing _extract_from_regex, get_last_day_of_month

* fix import

* update provider budget routing

* fix - remove dup test
This commit is contained in:
Ishaan Jaff 2024-11-23 16:59:46 -08:00 • committed by GitHub
parent e69678a9b3
commit 34bfebe470
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 149 additions and 142 deletions
Download patch file Download diff file Expand all files Collapse all files

24
docs/my-website/docs/proxy/provider_budget_routing.md
View file

@ -22,14 +22,14 @@ router_settings:
provider_budget_config:
openai:
budget_limit: 0.000000000001 # float of $ value budget for time period
time_period: 1d # can be 1d, 2d, 30d
time_period: 1d # can be 1d, 2d, 30d, 1mo, 2mo
azure:
budget_limit: 100
time_period: 1d
anthropic:
budget_limit: 100
time_period: 10d
vertexai:
vertex_ai:
budget_limit: 100
time_period: 12d
gemini:
@ -112,8 +112,11 @@ Expected response on failure
- If all providers exceed budget, raises an error
3. **Supported Time Periods**:
- Format: "Xd" where X is number of days
- Examples: "1d" (1 day), "30d" (30 days)
- Seconds: "Xs" (e.g., "30s")
- Minutes: "Xm" (e.g., "10m")
- Hours: "Xh" (e.g., "24h")
- Days: "Xd" (e.g., "1d", "30d")
- Months: "Xmo" (e.g., "1mo", "2mo")
4. **Requirements**:
- Redis required for tracking spend across instances
@ -136,7 +139,12 @@ The `provider_budget_config` is a dictionary where:
- **Key**: Provider name (string) - Must be a valid [LiteLLM provider name](https://docs.litellm.ai/docs/providers)
- **Value**: Budget configuration object with the following parameters:
- `budget_limit`: Float value representing the budget in USD
- `time_period`: String in the format "Xd" where X is the number of days (e.g., "1d", "30d")
- `time_period`: Duration string in one of the following formats:
- Seconds: `"Xs"` (e.g., "30s")
- Minutes: `"Xm"` (e.g., "10m")
- Hours: `"Xh"` (e.g., "24h")
- Days: `"Xd"` (e.g., "1d", "30d")
- Months: `"Xmo"` (e.g., "1mo", "2mo")
Example structure:
```yaml
@ -147,4 +155,10 @@ provider_budget_config:
azure:
budget_limit: 500.0 # $500 USD
time_period: "30d" # 30 day period
anthropic:
budget_limit: 200.0 # $200 USD
time_period: "1mo" # 1 month period
gemini:
budget_limit: 50.0 # $50 USD
time_period: "24h" # 24 hour period
```