@@ -62,6 +62,22 @@ response = completion(model="command-nightly", messages=messages)
print(response)
```
+## Async ([Docs](https://docs.litellm.ai/docs/completion/stream#async-completion))
+
+```python
+from litellm import acompletion
+import asyncio
+
+async def test_get_response():
+ user_message = "Hello, how are you?"
+ messages = [{"content": user_message, "role": "user"}]
+ response = await acompletion(model="gpt-3.5-turbo", messages=messages)
+ return response
+
+response = asyncio.run(test_get_response())
+print(response)
+```
+
## Streaming ([Docs](https://docs.litellm.ai/docs/completion/stream))
liteLLM supports streaming the model response back, pass `stream=True` to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
diff --git a/dist/litellm-1.12.5.dev1-py3-none-any.whl b/dist/litellm-1.12.5.dev1-py3-none-any.whl
new file mode 100644
index 000000000..395d5c567
Binary files /dev/null and b/dist/litellm-1.12.5.dev1-py3-none-any.whl differ
diff --git a/dist/litellm-1.12.5.dev1.tar.gz b/dist/litellm-1.12.5.dev1.tar.gz
new file mode 100644
index 000000000..8fdfd9d5e
Binary files /dev/null and b/dist/litellm-1.12.5.dev1.tar.gz differ
diff --git a/docker-compose.example.yml b/docker-compose.example.yml
new file mode 100644
index 000000000..6fdaf1506
--- /dev/null
+++ b/docker-compose.example.yml
@@ -0,0 +1,12 @@
+version: "3.9"
+services:
+ litellm:
+ image: ghcr.io/berriai/litellm:main
+ ports:
+ - "8000:8000" # Map the container port to the host, change the host port if necessary
+ volumes:
+ - ./litellm-config.yaml:/app/config.yaml # Mount the local configuration file
+ # You can change the port or number of workers as per your requirements or pass any new supported CLI augument. Make sure the port passed here matches with the container port defined above in `ports` value
+ command: [ "--config", "/app/config.yaml", "--port", "8000", "--num_workers", "8" ]
+
+# ...rest of your docker-compose config if any
diff --git a/docs/my-website/docs/caching/redis_cache.md b/docs/my-website/docs/caching/redis_cache.md
index f0fcc6952..521c4d00f 100644
--- a/docs/my-website/docs/caching/redis_cache.md
+++ b/docs/my-website/docs/caching/redis_cache.md
@@ -1,11 +1,14 @@
# Redis Cache
+
+[**See Code**](https://github.com/BerriAI/litellm/blob/4d7ff1b33b9991dcf38d821266290631d9bcd2dd/litellm/caching.py#L71)
+
### Pre-requisites
Install redis
```
pip install redis
```
For the hosted version you can setup your own Redis DB here: https://app.redislabs.com/
-### Usage
+### Quick Start
```python
import litellm
from litellm import completion
@@ -55,6 +58,11 @@ litellm.cache = cache # set litellm.cache to your cache
### Detecting Cached Responses
For resposes that were returned as cache hit, the response includes a param `cache` = True
+:::info
+
+Only valid for OpenAI <= 0.28.1 [Let us know if you still need this](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=bug&projects=&template=bug_report.yml&title=%5BBug%5D%3A+)
+:::
+
Example response with cache hit
```python
{
diff --git a/docs/my-website/docs/completion/input.md b/docs/my-website/docs/completion/input.md
index 30bd20f84..047be5395 100644
--- a/docs/my-website/docs/completion/input.md
+++ b/docs/my-website/docs/completion/input.md
@@ -6,7 +6,7 @@ import TabItem from '@theme/TabItem';
## Common Params
LiteLLM accepts and translates the [OpenAI Chat Completion params](https://platform.openai.com/docs/api-reference/chat/create) across all providers.
-### usage
+### Usage
```python
import litellm
@@ -23,7 +23,7 @@ response = litellm.completion(
print(response)
```
-### translated OpenAI params
+### Translated OpenAI params
This is a list of openai params we translate across providers.
This list is constantly being updated.
@@ -40,7 +40,7 @@ This list is constantly being updated.
|AI21| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | |
|VertexAI| ✅ | ✅ | | ✅ | | | | | | |
|Bedrock| ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
-|Sagemaker| ✅ | ✅ | | ✅ | | | | | | |
+|Sagemaker| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | | |
|TogetherAI| ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
|AlephAlpha| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | | |
|Palm| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | | |
@@ -185,6 +185,25 @@ def completion(
- `metadata`: *dict (optional)* - Any additional data you want to be logged when the call is made (sent to logging integrations, eg. promptlayer and accessible via custom callback function)
+**CUSTOM MODEL COST**
+- `input_cost_per_token`: *float (optional)* - The cost per input token for the completion call
+
+- `output_cost_per_token`: *float (optional)* - The cost per output token for the completion call
+
+**CUSTOM PROMPT TEMPLATE** (See [prompt formatting for more info](./prompt_formatting.md#format-prompt-yourself))
+- `initial_prompt_value`: *string (optional)* - Initial string applied at the start of the input messages
+
+- `roles`: *dict (optional)* - Dictionary specifying how to format the prompt based on the role + message passed in via `messages`.
+
+- `final_prompt_value`: *string (optional)* - Final string applied at the end of the input messages
+
+- `bos_token`: *string (optional)* - Initial string applied at the start of a sequence
+
+- `eos_token`: *string (optional)* - Initial string applied at the end of a sequence
+
+- `hf_model_name`: *string (optional)* - [Sagemaker Only] The corresponding huggingface name of the model, used to pull the right chat template for the model.
+
+
## Provider-specific Params
Providers might offer params not supported by OpenAI (e.g. top_k). You can pass those in 2 ways:
- via completion(): We'll pass the non-openai param, straight to the provider as part of the request body.
diff --git a/docs/my-website/docs/embedding/supported_embedding.md b/docs/my-website/docs/embedding/supported_embedding.md
index 009e5337b..d860e0074 100644
--- a/docs/my-website/docs/embedding/supported_embedding.md
+++ b/docs/my-website/docs/embedding/supported_embedding.md
@@ -142,6 +142,8 @@ print(response)
| Model Name | Function Call |
|----------------------|---------------------------------------------|
| Titan Embeddings - G1 | `embedding(model="amazon.titan-embed-text-v1", input=input)` |
+| Cohere Embeddings - English | `embedding(model="cohere.embed-english-v3", input=input)` |
+| Cohere Embeddings - Multilingual | `embedding(model="cohere.embed-multilingual-v3", input=input)` |
## Cohere Embedding Models
@@ -182,6 +184,17 @@ response = embedding(
input=["good morning from litellm"]
)
```
+### Usage - Custom API Base
+```python
+from litellm import embedding
+import os
+os.environ['HUGGINGFACE_API_KEY'] = ""
+response = embedding(
+ model='huggingface/microsoft/codebert-base',
+ input=["good morning from litellm"],
+ api_base = "https://p69xlsj6rpno5drq.us-east-1.aws.endpoints.huggingface.cloud"
+)
+```
| Model Name | Function Call | Required OS Variables |
|-----------------------|--------------------------------------------------------------|-------------------------------------------------|
diff --git a/docs/my-website/docs/observability/custom_callback.md b/docs/my-website/docs/observability/custom_callback.md
index 580bd819a..78b7499a8 100644
--- a/docs/my-website/docs/observability/custom_callback.md
+++ b/docs/my-website/docs/observability/custom_callback.md
@@ -85,6 +85,43 @@ print(response)
```
+## Async Callback Functions
+
+LiteLLM currently supports just async success callback functions for async completion/embedding calls.
+
+```python
+import asyncio, litellm
+
+async def async_test_logging_fn(kwargs, completion_obj, start_time, end_time):
+ print(f"On Async Success!")
+
+async def test_chat_openai():
+ try:
+ # litellm.set_verbose = True
+ litellm.success_callback = [async_test_logging_fn]
+ response = await litellm.acompletion(model="gpt-3.5-turbo",
+ messages=[{
+ "role": "user",
+ "content": "Hi 👋 - i'm openai"
+ }],
+ stream=True)
+ async for chunk in response:
+ continue
+ except Exception as e:
+ print(e)
+ pytest.fail(f"An error occurred - {str(e)}")
+
+asyncio.run(test_chat_openai())
+```
+
+:::info
+
+We're actively trying to expand this to other event types. [Tell us if you need this!](https://github.com/BerriAI/litellm/issues/1007)
+
+
+
+:::
+
## What's in kwargs?
Notice we pass in a kwargs argument to custom callback.
diff --git a/docs/my-website/docs/providers/aws_sagemaker.md b/docs/my-website/docs/providers/aws_sagemaker.md
index 264d6d9f2..328981c70 100644
--- a/docs/my-website/docs/providers/aws_sagemaker.md
+++ b/docs/my-website/docs/providers/aws_sagemaker.md
@@ -1,5 +1,5 @@
# AWS Sagemaker
-LiteLLM supports Llama2 on Sagemaker
+LiteLLM supports All Sagemaker Huggingface Jumpstart Models
### API KEYS
```python
@@ -42,6 +42,28 @@ response = completion(
)
```
+### Applying Prompt Templates
+To apply the correct prompt template for your sagemaker deployment, pass in it's hf model name as well.
+
+```python
+import os
+from litellm import completion
+
+os.environ["AWS_ACCESS_KEY_ID"] = ""
+os.environ["AWS_SECRET_ACCESS_KEY"] = ""
+os.environ["AWS_REGION_NAME"] = ""
+
+response = completion(
+ model="sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b",
+ messages=messages,
+ temperature=0.2,
+ max_tokens=80,
+ hf_model_name="meta-llama/Llama-2-7b",
+ )
+```
+
+You can also pass in your own [custom prompt template](../completion/prompt_formatting.md#format-prompt-yourself)
+
### Usage - Streaming
Sagemaker currently does not support streaming - LiteLLM fakes streaming by returning chunks of the response string
@@ -64,14 +86,32 @@ for chunk in response:
print(chunk)
```
-### AWS Sagemaker Models
+### Completion Models
Here's an example of using a sagemaker model with LiteLLM
| Model Name | Function Call |
|-------------------------------|-------------------------------------------------------------------------------------------|
+| Your Custom Huggingface Model | `completion(model='sagemaker/