forked from phoenix/litellm-mirror
docs(caching.md): add batch redis requests to docs
This commit is contained in:
parent
f3cf1ec71f
commit
2d2731c3b5
1 changed files with 26 additions and 0 deletions
|
@ -225,6 +225,32 @@ litellm_settings:
|
||||||
supported_call_types: ["acompletion", "completion", "embedding", "aembedding"] # defaults to all litellm call types
|
supported_call_types: ["acompletion", "completion", "embedding", "aembedding"] # defaults to all litellm call types
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
|
### Turn on `batch_redis_requests`
|
||||||
|
|
||||||
|
**What it does?**
|
||||||
|
When a request is made:
|
||||||
|
|
||||||
|
- Check if a key starting with `litellm:<hashed_api_key>:<call_type>:` exists in-memory, if no - get the last 100 cached requests for this key and store it
|
||||||
|
|
||||||
|
- New requests are stored with this `litellm:..` as the namespace
|
||||||
|
|
||||||
|
**Why?**
|
||||||
|
Reduce number of redis GET requests. This improved latency by 46% in prod load tests.
|
||||||
|
|
||||||
|
**Usage**
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
litellm_settings:
|
||||||
|
cache: true
|
||||||
|
cache_params:
|
||||||
|
type: redis
|
||||||
|
... # remaining redis args (host, port, etc.)
|
||||||
|
callbacks: ["batch_redis_requests"] # 👈 KEY CHANGE!
|
||||||
|
```
|
||||||
|
|
||||||
|
[**SEE CODE**](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/hooks/batch_redis_get.py)
|
||||||
|
|
||||||
### Turn on / off caching per request.
|
### Turn on / off caching per request.
|
||||||
|
|
||||||
The proxy support 3 cache-controls:
|
The proxy support 3 cache-controls:
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue