docs(routing.md): adding comparison to docs

This commit is contained in:
Krrish Dholakia 2023-11-06 18:57:50 -08:00
parent ba66294a3d
commit 50564ab38e
3 changed files with 9 additions and 15 deletions

View file

@ -2,22 +2,9 @@ import Image from '@theme/IdealImage';
# Reliability - Fallbacks, Azure Deployments, etc.
Prevent failed calls and slow response times with multiple deployments for API calls (E.g. multiple azure-openai deployments).
<Image img={require('../img/multiple_deployment.png')} alt="HF_Dashboard" style={{ maxWidth: '100%', height: 'auto' }}/>
# Reliability
LiteLLM helps prevent failed requests in 3 ways:
- Retries
- Fallbacks: Context Window + General
- RateLimitManager
## Helper utils
LiteLLM supports the following functions for reliability:
* `litellm.longer_context_model_fallback_dict`: Dictionary which has a mapping for those models which have larger equivalents
* `num_retries`: use tenacity retries
* `completion()` with fallbacks: switch between models/keys/api bases in case of errors.
* `router()`: An abstraction on top of completion + embeddings to route the request to a deployment with capacity (available tpm/rpm).
<Image img={require('../img/multiple_deployments.png')} alt="HF_Dashboard" style={{ maxWidth: '100%', height: 'auto' }}/>
## Manage Multiple Deployments
@ -135,6 +122,13 @@ response = completion(
## Fallbacks
## Helper utils
LiteLLM supports the following functions for reliability:
* `litellm.longer_context_model_fallback_dict`: Dictionary which has a mapping for those models which have larger equivalents
* `num_retries`: use tenacity retries
* `completion()` with fallbacks: switch between models/keys/api bases in case of errors.
### Context Window Fallbacks
```python
from litellm import completion

Binary file not shown.

Before

Width:  |  Height:  |  Size: 135 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 268 KiB