diff --git a/docs/my-website/docs/caching/redis_cache.md b/docs/my-website/docs/caching/redis_cache.md
index f0fcc6952..521c4d00f 100644
--- a/docs/my-website/docs/caching/redis_cache.md
+++ b/docs/my-website/docs/caching/redis_cache.md
@@ -1,11 +1,14 @@
# Redis Cache
+
+[**See Code**](https://github.com/BerriAI/litellm/blob/4d7ff1b33b9991dcf38d821266290631d9bcd2dd/litellm/caching.py#L71)
+
### Pre-requisites
Install redis
```
pip install redis
```
For the hosted version you can setup your own Redis DB here: https://app.redislabs.com/
-### Usage
+### Quick Start
```python
import litellm
from litellm import completion
@@ -55,6 +58,11 @@ litellm.cache = cache # set litellm.cache to your cache
### Detecting Cached Responses
For resposes that were returned as cache hit, the response includes a param `cache` = True
+:::info
+
+Only valid for OpenAI <= 0.28.1 [Let us know if you still need this](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=bug&projects=&template=bug_report.yml&title=%5BBug%5D%3A+)
+:::
+
Example response with cache hit
```python
{
diff --git a/docs/my-website/docs/completion/input.md b/docs/my-website/docs/completion/input.md
index 30bd20f84..047be5395 100644
--- a/docs/my-website/docs/completion/input.md
+++ b/docs/my-website/docs/completion/input.md
@@ -6,7 +6,7 @@ import TabItem from '@theme/TabItem';
## Common Params
LiteLLM accepts and translates the [OpenAI Chat Completion params](https://platform.openai.com/docs/api-reference/chat/create) across all providers.
-### usage
+### Usage
```python
import litellm
@@ -23,7 +23,7 @@ response = litellm.completion(
print(response)
```
-### translated OpenAI params
+### Translated OpenAI params
This is a list of openai params we translate across providers.
This list is constantly being updated.
@@ -40,7 +40,7 @@ This list is constantly being updated.
|AI21| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | |
|VertexAI| ✅ | ✅ | | ✅ | | | | | | |
|Bedrock| ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
-|Sagemaker| ✅ | ✅ | | ✅ | | | | | | |
+|Sagemaker| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | | |
|TogetherAI| ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
|AlephAlpha| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | | |
|Palm| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | | |
@@ -185,6 +185,25 @@ def completion(
- `metadata`: *dict (optional)* - Any additional data you want to be logged when the call is made (sent to logging integrations, eg. promptlayer and accessible via custom callback function)
+**CUSTOM MODEL COST**
+- `input_cost_per_token`: *float (optional)* - The cost per input token for the completion call
+
+- `output_cost_per_token`: *float (optional)* - The cost per output token for the completion call
+
+**CUSTOM PROMPT TEMPLATE** (See [prompt formatting for more info](./prompt_formatting.md#format-prompt-yourself))
+- `initial_prompt_value`: *string (optional)* - Initial string applied at the start of the input messages
+
+- `roles`: *dict (optional)* - Dictionary specifying how to format the prompt based on the role + message passed in via `messages`.
+
+- `final_prompt_value`: *string (optional)* - Final string applied at the end of the input messages
+
+- `bos_token`: *string (optional)* - Initial string applied at the start of a sequence
+
+- `eos_token`: *string (optional)* - Initial string applied at the end of a sequence
+
+- `hf_model_name`: *string (optional)* - [Sagemaker Only] The corresponding huggingface name of the model, used to pull the right chat template for the model.
+
+
## Provider-specific Params
Providers might offer params not supported by OpenAI (e.g. top_k). You can pass those in 2 ways:
- via completion(): We'll pass the non-openai param, straight to the provider as part of the request body.
diff --git a/docs/my-website/docs/embedding/supported_embedding.md b/docs/my-website/docs/embedding/supported_embedding.md
index 009e5337b..3c13c5732 100644
--- a/docs/my-website/docs/embedding/supported_embedding.md
+++ b/docs/my-website/docs/embedding/supported_embedding.md
@@ -182,6 +182,17 @@ response = embedding(
input=["good morning from litellm"]
)
```
+### Usage - Custom API Base
+```python
+from litellm import embedding
+import os
+os.environ['HUGGINGFACE_API_KEY'] = ""
+response = embedding(
+ model='huggingface/microsoft/codebert-base',
+ input=["good morning from litellm"],
+ api_base = "https://p69xlsj6rpno5drq.us-east-1.aws.endpoints.huggingface.cloud"
+)
+```
| Model Name | Function Call | Required OS Variables |
|-----------------------|--------------------------------------------------------------|-------------------------------------------------|
diff --git a/docs/my-website/docs/observability/custom_callback.md b/docs/my-website/docs/observability/custom_callback.md
index 580bd819a..78b7499a8 100644
--- a/docs/my-website/docs/observability/custom_callback.md
+++ b/docs/my-website/docs/observability/custom_callback.md
@@ -85,6 +85,43 @@ print(response)
```
+## Async Callback Functions
+
+LiteLLM currently supports just async success callback functions for async completion/embedding calls.
+
+```python
+import asyncio, litellm
+
+async def async_test_logging_fn(kwargs, completion_obj, start_time, end_time):
+ print(f"On Async Success!")
+
+async def test_chat_openai():
+ try:
+ # litellm.set_verbose = True
+ litellm.success_callback = [async_test_logging_fn]
+ response = await litellm.acompletion(model="gpt-3.5-turbo",
+ messages=[{
+ "role": "user",
+ "content": "Hi 👋 - i'm openai"
+ }],
+ stream=True)
+ async for chunk in response:
+ continue
+ except Exception as e:
+ print(e)
+ pytest.fail(f"An error occurred - {str(e)}")
+
+asyncio.run(test_chat_openai())
+```
+
+:::info
+
+We're actively trying to expand this to other event types. [Tell us if you need this!](https://github.com/BerriAI/litellm/issues/1007)
+
+
+
+:::
+
## What's in kwargs?
Notice we pass in a kwargs argument to custom callback.
diff --git a/docs/my-website/docs/providers/aws_sagemaker.md b/docs/my-website/docs/providers/aws_sagemaker.md
index 264d6d9f2..606268ad1 100644
--- a/docs/my-website/docs/providers/aws_sagemaker.md
+++ b/docs/my-website/docs/providers/aws_sagemaker.md
@@ -1,5 +1,5 @@
# AWS Sagemaker
-LiteLLM supports Llama2 on Sagemaker
+LiteLLM supports All Sagemaker Huggingface Jumpstart Models
### API KEYS
```python
@@ -42,6 +42,27 @@ response = completion(
)
```
+### Specifying HF Model Name
+To apply the correct prompt template for your sagemaker deployment, pass in it's hf model name as well.
+
+```python
+import os
+from litellm import completion
+
+os.environ["AWS_ACCESS_KEY_ID"] = ""
+os.environ["AWS_SECRET_ACCESS_KEY"] = ""
+os.environ["AWS_REGION_NAME"] = ""
+
+response = completion(
+ model="sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b",
+ messages=messages,
+ temperature=0.2,
+ max_tokens=80,
+ hf_model_name="meta-llama/Llama-2-7b",
+ )
+```
+
+
### Usage - Streaming
Sagemaker currently does not support streaming - LiteLLM fakes streaming by returning chunks of the response string
diff --git a/docs/my-website/docs/providers/replicate.md b/docs/my-website/docs/providers/replicate.md
index d8ab035e1..3384ba35c 100644
--- a/docs/my-website/docs/providers/replicate.md
+++ b/docs/my-website/docs/providers/replicate.md
@@ -49,8 +49,8 @@ Below are examples on how to call replicate LLMs using liteLLM
Model Name | Function Call | Required OS Variables |
-----------------------------|----------------------------------------------------------------|--------------------------------------|
- replicate/llama-2-70b-chat | `completion(model='replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf', messages)` | `os.environ['REPLICATE_API_KEY']` |
- a16z-infra/llama-2-13b-chat| `completion(model='replicate/a16z-infra/llama-2-13b-chat:2a7f981751ec7fdf87b5b91ad4db53683a98082e9ff7bfd12c8cd5ea85980a52', messages)`| `os.environ['REPLICATE_API_KEY']` |
+ replicate/llama-2-70b-chat | `completion(model='replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf', messages, supports_system_prompt=True)` | `os.environ['REPLICATE_API_KEY']` |
+ a16z-infra/llama-2-13b-chat| `completion(model='replicate/a16z-infra/llama-2-13b-chat:2a7f981751ec7fdf87b5b91ad4db53683a98082e9ff7bfd12c8cd5ea85980a52', messages, supports_system_prompt=True)`| `os.environ['REPLICATE_API_KEY']` |
replicate/vicuna-13b | `completion(model='replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b', messages)` | `os.environ['REPLICATE_API_KEY']` |
daanelson/flan-t5-large | `completion(model='replicate/daanelson/flan-t5-large:ce962b3f6792a57074a601d3979db5839697add2e4e02696b3ced4c022d4767f', messages)` | `os.environ['REPLICATE_API_KEY']` |
custom-llm | `completion(model='replicate/custom-llm-version-id', messages)` | `os.environ['REPLICATE_API_KEY']` |
diff --git a/docs/my-website/docs/proxy/caching.md b/docs/my-website/docs/proxy/caching.md
index d052102db..56e4b4c1c 100644
--- a/docs/my-website/docs/proxy/caching.md
+++ b/docs/my-website/docs/proxy/caching.md
@@ -12,17 +12,27 @@ model_list:
litellm_settings:
set_verbose: True
cache: # init cache
- type: redis # tell litellm to use redis caching
+ type: redis # tell litellm to use redis caching (Also: `pip install redis`)
```
#### Step 2: Add Redis Credentials to .env
-LiteLLM requires the following REDIS credentials in your env to enable caching
+Set either `REDIS_URL` or the `REDIS_HOST` in your os environment, to enable caching.
```shell
+ REDIS_URL = "" # REDIS_URL='redis://username:password@hostname:port/database'
+ ## OR ##
REDIS_HOST = "" # REDIS_HOST='redis-18841.c274.us-east-1-3.ec2.cloud.redislabs.com'
REDIS_PORT = "" # REDIS_PORT='18841'
REDIS_PASSWORD = "" # REDIS_PASSWORD='liteLlmIsAmazing'
```
+
+**Additional kwargs**
+You can pass in any additional redis.Redis arg, by storing the variable + value in your os environment, like this:
+```shell
+REDIS_