docs(reliability.md): add tutorial on setting wildcard models as fallbacks

This commit is contained in:
Krrish Dholakia 2024-09-28 19:08:20 -07:00
parent 81d6c5e5a5
commit c9d6925a42

View file

@ -699,3 +699,53 @@ print(response)
```
</TabItem>
</Tabs>
### Setting Fallbacks for Wildcard Models
You can set fallbacks for wildcard models (e.g. `azure/*`) in your config file.
1. Setup config
```yaml
model_list:
- model_name: "gpt-4o"
litellm_params:
model: "openai/gpt-4o"
api_key: os.environ/OPENAI_API_KEY
- model_name: "azure/*"
litellm_params:
model: "azure/*"
api_key: os.environ/AZURE_API_KEY
api_base: os.environ/AZURE_API_BASE
litellm_settings:
fallbacks: [{"gpt-4o": ["azure/gpt-4o"]}]
```
2. Start Proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "what color is red"
}
]
}
],
"max_tokens": 300,
"mock_testing_fallbacks": true
}'
```