forked from phoenix/litellm-mirror
docs(routing.md): adding comparison to docs
This commit is contained in:
parent
ba66294a3d
commit
50564ab38e
3 changed files with 9 additions and 15 deletions
|
@ -2,22 +2,9 @@ import Image from '@theme/IdealImage';
|
|||
|
||||
# Reliability - Fallbacks, Azure Deployments, etc.
|
||||
|
||||
Prevent failed calls and slow response times with multiple deployments for API calls (E.g. multiple azure-openai deployments).
|
||||
|
||||
<Image img={require('../img/multiple_deployment.png')} alt="HF_Dashboard" style={{ maxWidth: '100%', height: 'auto' }}/>
|
||||
|
||||
# Reliability
|
||||
|
||||
LiteLLM helps prevent failed requests in 3 ways:
|
||||
- Retries
|
||||
- Fallbacks: Context Window + General
|
||||
- RateLimitManager
|
||||
|
||||
## Helper utils
|
||||
LiteLLM supports the following functions for reliability:
|
||||
* `litellm.longer_context_model_fallback_dict`: Dictionary which has a mapping for those models which have larger equivalents
|
||||
* `num_retries`: use tenacity retries
|
||||
* `completion()` with fallbacks: switch between models/keys/api bases in case of errors.
|
||||
* `router()`: An abstraction on top of completion + embeddings to route the request to a deployment with capacity (available tpm/rpm).
|
||||
<Image img={require('../img/multiple_deployments.png')} alt="HF_Dashboard" style={{ maxWidth: '100%', height: 'auto' }}/>
|
||||
|
||||
## Manage Multiple Deployments
|
||||
|
||||
|
@ -135,6 +122,13 @@ response = completion(
|
|||
|
||||
## Fallbacks
|
||||
|
||||
## Helper utils
|
||||
LiteLLM supports the following functions for reliability:
|
||||
* `litellm.longer_context_model_fallback_dict`: Dictionary which has a mapping for those models which have larger equivalents
|
||||
* `num_retries`: use tenacity retries
|
||||
* `completion()` with fallbacks: switch between models/keys/api bases in case of errors.
|
||||
|
||||
|
||||
### Context Window Fallbacks
|
||||
```python
|
||||
from litellm import completion
|
||||
|
|
Binary file not shown.
Before Width: | Height: | Size: 135 KiB |
BIN
docs/my-website/img/multiple_deployments.png
Normal file
BIN
docs/my-website/img/multiple_deployments.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 268 KiB |
Loading…
Add table
Add a link
Reference in a new issue