(docs) s3 cache

This commit is contained in:
ishaan-jaff 2024-01-03 16:13:15 +05:30
parent d45101b652
commit 344e8d8508

View file

@ -1,18 +1,29 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Redis Cache, s3 Cache
[**See Code**](https://github.com/BerriAI/litellm/blob/4d7ff1b33b9991dcf38d821266290631d9bcd2dd/litellm/caching.py#L71)
[**See Code**](https://github.com/BerriAI/litellm/blob/main/litellm/caching.py)
## Initialize Cache - In Memory, Redis, s3 Bucket
<Tabs>
<TabItem value="redis" label="redis-cache">
### Pre-requisites
Install redis
```shell
pip install redis
```
For the hosted version you can setup your own Redis DB here: https://app.redislabs.com/
### Quick Start
```python
import litellm
from litellm import completion
from litellm.caching import Cache
litellm.cache = Cache(type="redis", host=<host>, port=<port>, password=<password>)
# Make completion calls
@ -28,6 +39,48 @@ response2 = completion(
# response1 == response2, response 1 is cached
```
</TabItem>
<TabItem value="s3" label="s3-cache">
Install boto3
```shell
pip install boto3
```
Set AWS environment variables
```shell
AWS_ACCESS_KEY_ID = "AKI*******"
AWS_SECRET_ACCESS_KEY = "WOl*****"
```
### Quick Start
```python
import litellm
from litellm import completion
from litellm.caching import Cache
# pass s3-bucket name
litellm.cache = Cache(type="s3", s3_bucket_name="cache-bucket-litellm", s3_region_name="us-west-2")
# Make completion calls
response1 = completion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Tell me a joke."}]
)
response2 = completion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Tell me a joke."}]
)
# response1 == response2, response 1 is cached
```
</TabItem>
</Tabs>
## Cache Context Manager - Enable, Disable, Update Cache
Use the context manager for easily enabling, disabling & updating the litellm cache
@ -103,35 +156,34 @@ litellm.cache = cache # set litellm.cache to your cache
## Cache Initialization Parameters
#### `type` (str, optional)
```python
def __init__(
self,
type: Optional[Literal["local", "redis", "s3"]] = "local",
supported_call_types: Optional[
List[Literal["completion", "acompletion", "embedding", "aembedding"]]
] = ["completion", "acompletion", "embedding", "aembedding"], # A list of litellm call types to cache for. Defaults to caching for all litellm call types.
The type of cache to initialize. It can be either "local" or "redis". Defaults to "local".
# redis cache params
host: Optional[str] = None,
port: Optional[str] = None,
password: Optional[str] = None,
#### `host` (str, optional)
The host address for the Redis cache. This parameter is required if the `type` is set to "redis".
#### `port` (int, optional)
The port number for the Redis cache. This parameter is required if the `type` is set to "redis".
#### `password` (str, optional)
The password for the Redis cache. This parameter is required if the `type` is set to "redis".
#### `supported_call_types` (list, optional)
A list of call types to cache for. Defaults to caching for all call types. The available call types are:
- "completion"
- "acompletion"
- "embedding"
- "aembedding"
#### `**kwargs` (additional keyword arguments)
Additional keyword arguments are accepted for the initialization of the Redis cache using the `redis.Redis()` constructor. These arguments allow you to fine-tune the Redis cache configuration based on your specific needs.
# s3 Bucket, boto3 configuration
s3_bucket_name: Optional[str] = None,
s3_region_name: Optional[str] = None,
s3_api_version: Optional[str] = None,
s3_use_ssl: Optional[bool] = True,
s3_verify: Optional[Union[bool, str]] = None,
s3_endpoint_url: Optional[str] = None,
s3_aws_access_key_id: Optional[str] = None,
s3_aws_secret_access_key: Optional[str] = None,
s3_aws_session_token: Optional[str] = None,
s3_config: Optional[Any] = None,
**kwargs,
):
```
## Logging