forked from phoenix/litellm-mirror
docs: adds redis url to router + proxy docs
This commit is contained in:
parent
4d7ff1b33b
commit
68ca2a28d4
4 changed files with 32 additions and 4 deletions
|
@ -1,11 +1,14 @@
|
|||
# Redis Cache
|
||||
|
||||
[**See Code**](https://github.com/BerriAI/litellm/blob/4d7ff1b33b9991dcf38d821266290631d9bcd2dd/litellm/caching.py#L71)
|
||||
|
||||
### Pre-requisites
|
||||
Install redis
|
||||
```
|
||||
pip install redis
|
||||
```
|
||||
For the hosted version you can setup your own Redis DB here: https://app.redislabs.com/
|
||||
### Usage
|
||||
### Quick Start
|
||||
```python
|
||||
import litellm
|
||||
from litellm import completion
|
||||
|
@ -55,6 +58,11 @@ litellm.cache = cache # set litellm.cache to your cache
|
|||
### Detecting Cached Responses
|
||||
For resposes that were returned as cache hit, the response includes a param `cache` = True
|
||||
|
||||
:::info
|
||||
|
||||
Only valid for OpenAI <= 0.28.1 [Let us know if you still need this](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=bug&projects=&template=bug_report.yml&title=%5BBug%5D%3A+)
|
||||
:::
|
||||
|
||||
Example response with cache hit
|
||||
```python
|
||||
{
|
||||
|
|
|
@ -12,17 +12,27 @@ model_list:
|
|||
litellm_settings:
|
||||
set_verbose: True
|
||||
cache: # init cache
|
||||
type: redis # tell litellm to use redis caching
|
||||
type: redis # tell litellm to use redis caching (Also: `pip install redis`)
|
||||
```
|
||||
|
||||
#### Step 2: Add Redis Credentials to .env
|
||||
LiteLLM requires the following REDIS credentials in your env to enable caching
|
||||
Set either `REDIS_URL` or the `REDIS_HOST` in your os environment, to enable caching.
|
||||
|
||||
```shell
|
||||
REDIS_URL = "" # REDIS_URL='redis://username:password@hostname:port/database'
|
||||
## OR ##
|
||||
REDIS_HOST = "" # REDIS_HOST='redis-18841.c274.us-east-1-3.ec2.cloud.redislabs.com'
|
||||
REDIS_PORT = "" # REDIS_PORT='18841'
|
||||
REDIS_PASSWORD = "" # REDIS_PASSWORD='liteLlmIsAmazing'
|
||||
```
|
||||
|
||||
**Additional kwargs**
|
||||
You can pass in any additional redis.Redis arg, by storing the variable + value in your os environment, like this:
|
||||
```shell
|
||||
REDIS_<redis-kwarg-name> = ""
|
||||
```
|
||||
|
||||
[**See how it's read from the environment**](https://github.com/BerriAI/litellm/blob/4d7ff1b33b9991dcf38d821266290631d9bcd2dd/litellm/_redis.py#L40)
|
||||
#### Step 3: Run proxy with config
|
||||
```shell
|
||||
$ litellm --config /path/to/config.yaml
|
||||
|
|
|
@ -356,6 +356,16 @@ router = Router(model_list=model_list,
|
|||
|
||||
print(response)
|
||||
```
|
||||
|
||||
**Pass in Redis URL, additional kwargs**
|
||||
```python
|
||||
router = Router(model_list: Optional[list] = None,
|
||||
## CACHING ##
|
||||
redis_url=os.getenv("REDIS_URL")",
|
||||
cache_kwargs= {}, # additional kwargs to pass to RedisCache (see caching.py)
|
||||
cache_responses=True)
|
||||
```
|
||||
|
||||
#### Default litellm.completion/embedding params
|
||||
|
||||
You can also set default params for litellm completion/embedding calls. Here's how to do that:
|
||||
|
|
|
@ -36,7 +36,7 @@ def test_caching_v2(): # test in memory cache
|
|||
print(f"error occurred: {traceback.format_exc()}")
|
||||
pytest.fail(f"Error occurred: {e}")
|
||||
|
||||
# test_caching_v2()
|
||||
test_caching_v2()
|
||||
|
||||
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue