-```python
+```python showLineNumbers
litellm._turn_on_debug()
litellm.modify_params = True
model = "anthropic/claude-3-7-sonnet-20250219" # works across Anthropic, Bedrock, Vertex AI
@@ -150,7 +153,7 @@ response = litellm.completion(
messages=messages,
tools=tools,
tool_choice="auto", # auto is default, but we'll be explicit
- thinking={"type": "enabled", "budget_tokens": 1024},
+ reasoning_effort="low",
)
print("Response\n", response)
response_message = response.choices[0].message
@@ -198,9 +201,9 @@ if tool_calls:
model=model,
messages=messages,
seed=22,
+ reasoning_effort="low",
# tools=tools,
drop_params=True,
- thinking={"type": "enabled", "budget_tokens": 1024},
) # get a new response from the model where it can see the function response
print("second response\n", second_response)
```
@@ -210,7 +213,7 @@ if tool_calls:
1. Setup config.yaml
-```yaml
+```yaml showLineNumbers
model_list:
- model_name: claude-3-7-sonnet-thinking
litellm_params:
@@ -224,7 +227,7 @@ model_list:
2. Run proxy
-```bash
+```bash showLineNumbers
litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000
@@ -332,7 +335,7 @@ curl http://0.0.0.0:4000/v1/chat/completions \
Set `drop_params=True` to drop the 'thinking' blocks when swapping from Anthropic to Deepseek models. Suggest improvements to this approach [here](https://github.com/BerriAI/litellm/discussions/8927).
-```python
+```python showLineNumbers
litellm.drop_params = True # 👈 EITHER GLOBALLY or per request
# or per request
@@ -340,7 +343,7 @@ litellm.drop_params = True # 👈 EITHER GLOBALLY or per request
response = litellm.completion(
model="anthropic/claude-3-7-sonnet-20250219",
messages=[{"role": "user", "content": "What is the capital of France?"}],
- thinking={"type": "enabled", "budget_tokens": 1024},
+ reasoning_effort="low",
drop_params=True,
)
@@ -348,7 +351,7 @@ response = litellm.completion(
response = litellm.completion(
model="deepseek/deepseek-chat",
messages=[{"role": "user", "content": "What is the capital of France?"}],
- thinking={"type": "enabled", "budget_tokens": 1024},
+ reasoning_effort="low",
drop_params=True,
)
```
@@ -364,3 +367,123 @@ These fields can be accessed via `response.choices[0].message.reasoning_content`
- `thinking` - str: The thinking from the model.
- `signature` - str: The signature delta from the model.
+
+
+## Pass `thinking` to Anthropic models
+
+You can also pass the `thinking` parameter to Anthropic models.
+
+
+
+
+```python showLineNumbers
+response = litellm.completion(
+ model="anthropic/claude-3-7-sonnet-20250219",
+ messages=[{"role": "user", "content": "What is the capital of France?"}],
+ thinking={"type": "enabled", "budget_tokens": 1024},
+)
+```
+
+
+
+
+```bash
+curl http://0.0.0.0:4000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -H "Authorization: Bearer $LITELLM_KEY" \
+ -d '{
+ "model": "anthropic/claude-3-7-sonnet-20250219",
+ "messages": [{"role": "user", "content": "What is the capital of France?"}],
+ "thinking": {"type": "enabled", "budget_tokens": 1024}
+ }'
+```
+
+
+
+
+## Checking if a model supports reasoning
+
+
+
+
+Use `litellm.supports_reasoning(model="")` -> returns `True` if model supports reasoning and `False` if not.
+
+```python showLineNumbers title="litellm.supports_reasoning() usage"
+import litellm
+
+# Example models that support reasoning
+assert litellm.supports_reasoning(model="anthropic/claude-3-7-sonnet-20250219") == True
+assert litellm.supports_reasoning(model="deepseek/deepseek-chat") == True
+
+# Example models that do not support reasoning
+assert litellm.supports_reasoning(model="openai/gpt-3.5-turbo") == False
+```
+
+
+
+
+1. Define models that support reasoning in your `config.yaml`. You can optionally add `supports_reasoning: True` to the `model_info` if LiteLLM does not automatically detect it for your custom model.
+
+```yaml showLineNumbers title="litellm proxy config.yaml"
+model_list:
+ - model_name: claude-3-sonnet-reasoning
+ litellm_params:
+ model: anthropic/claude-3-7-sonnet-20250219
+ api_key: os.environ/ANTHROPIC_API_KEY
+ - model_name: deepseek-reasoning
+ litellm_params:
+ model: deepseek/deepseek-chat
+ api_key: os.environ/DEEPSEEK_API_KEY
+ # Example for a custom model where detection might be needed
+ - model_name: my-custom-reasoning-model
+ litellm_params:
+ model: openai/my-custom-model # Assuming it's OpenAI compatible
+ api_base: http://localhost:8000
+ api_key: fake-key
+ model_info:
+ supports_reasoning: True # Explicitly mark as supporting reasoning
+```
+
+2. Run the proxy server:
+
+```bash showLineNumbers title="litellm --config config.yaml"
+litellm --config config.yaml
+```
+
+3. Call `/model_group/info` to check if your model supports `reasoning`
+
+```shell showLineNumbers title="curl /model_group/info"
+curl -X 'GET' \
+ 'http://localhost:4000/model_group/info' \
+ -H 'accept: application/json' \
+ -H 'x-api-key: sk-1234'
+```
+
+Expected Response
+
+```json showLineNumbers title="response from /model_group/info"
+{
+ "data": [
+ {
+ "model_group": "claude-3-sonnet-reasoning",
+ "providers": ["anthropic"],
+ "mode": "chat",
+ "supports_reasoning": true,
+ },
+ {
+ "model_group": "deepseek-reasoning",
+ "providers": ["deepseek"],
+ "supports_reasoning": true,
+ },
+ {
+ "model_group": "my-custom-reasoning-model",
+ "providers": ["openai"],
+ "supports_reasoning": true,
+ }
+ ]
+}
+````
+
+
+
+
diff --git a/docs/my-website/docs/response_api.md b/docs/my-website/docs/response_api.md
index 0604a42586..532f20bc05 100644
--- a/docs/my-website/docs/response_api.md
+++ b/docs/my-website/docs/response_api.md
@@ -14,22 +14,22 @@ LiteLLM provides a BETA endpoint in the spec of [OpenAI's `/responses` API](http
| Fallbacks | ✅ | Works between supported models |
| Loadbalancing | ✅ | Works between supported models |
| Supported LiteLLM Versions | 1.63.8+ | |
-| Supported LLM providers | `openai` | |
+| Supported LLM providers | **All LiteLLM supported providers** | `openai`, `anthropic`, `bedrock`, `vertex_ai`, `gemini`, `azure`, `azure_ai` etc. |
## Usage
-## Create a model response
+### LiteLLM Python SDK
-
+
#### Non-streaming
-```python
+```python showLineNumbers title="OpenAI Non-streaming Response"
import litellm
# Non-streaming response
response = litellm.responses(
- model="gpt-4o",
+ model="openai/o1-pro",
input="Tell me a three sentence bedtime story about a unicorn.",
max_output_tokens=100
)
@@ -38,12 +38,12 @@ print(response)
```
#### Streaming
-```python
+```python showLineNumbers title="OpenAI Streaming Response"
import litellm
# Streaming response
response = litellm.responses(
- model="gpt-4o",
+ model="openai/o1-pro",
input="Tell me a three sentence bedtime story about a unicorn.",
stream=True
)
@@ -53,58 +53,169 @@ for event in response:
```
-
-First, add this to your litellm proxy config.yaml:
-```yaml
-model_list:
- - model_name: gpt-4o
- litellm_params:
- model: openai/gpt-4o
- api_key: os.environ/OPENAI_API_KEY
-```
-
-Start your LiteLLM proxy:
-```bash
-litellm --config /path/to/config.yaml
-
-# RUNNING on http://0.0.0.0:4000
-```
-
-Then use the OpenAI SDK pointed to your proxy:
+
#### Non-streaming
-```python
-from openai import OpenAI
+```python showLineNumbers title="Anthropic Non-streaming Response"
+import litellm
+import os
-# Initialize client with your proxy URL
-client = OpenAI(
- base_url="http://localhost:4000", # Your proxy URL
- api_key="your-api-key" # Your proxy API key
-)
+# Set API key
+os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"
# Non-streaming response
-response = client.responses.create(
- model="gpt-4o",
- input="Tell me a three sentence bedtime story about a unicorn."
+response = litellm.responses(
+ model="anthropic/claude-3-5-sonnet-20240620",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ max_output_tokens=100
)
print(response)
```
#### Streaming
-```python
-from openai import OpenAI
+```python showLineNumbers title="Anthropic Streaming Response"
+import litellm
+import os
-# Initialize client with your proxy URL
-client = OpenAI(
- base_url="http://localhost:4000", # Your proxy URL
- api_key="your-api-key" # Your proxy API key
-)
+# Set API key
+os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"
# Streaming response
-response = client.responses.create(
- model="gpt-4o",
+response = litellm.responses(
+ model="anthropic/claude-3-5-sonnet-20240620",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+
+#### Non-streaming
+```python showLineNumbers title="Vertex AI Non-streaming Response"
+import litellm
+import os
+
+# Set credentials - Vertex AI uses application default credentials
+# Run 'gcloud auth application-default login' to authenticate
+os.environ["VERTEXAI_PROJECT"] = "your-gcp-project-id"
+os.environ["VERTEXAI_LOCATION"] = "us-central1"
+
+# Non-streaming response
+response = litellm.responses(
+ model="vertex_ai/gemini-1.5-pro",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ max_output_tokens=100
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="Vertex AI Streaming Response"
+import litellm
+import os
+
+# Set credentials - Vertex AI uses application default credentials
+# Run 'gcloud auth application-default login' to authenticate
+os.environ["VERTEXAI_PROJECT"] = "your-gcp-project-id"
+os.environ["VERTEXAI_LOCATION"] = "us-central1"
+
+# Streaming response
+response = litellm.responses(
+ model="vertex_ai/gemini-1.5-pro",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+
+#### Non-streaming
+```python showLineNumbers title="AWS Bedrock Non-streaming Response"
+import litellm
+import os
+
+# Set AWS credentials
+os.environ["AWS_ACCESS_KEY_ID"] = "your-access-key-id"
+os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret-access-key"
+os.environ["AWS_REGION_NAME"] = "us-west-2" # or your AWS region
+
+# Non-streaming response
+response = litellm.responses(
+ model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ max_output_tokens=100
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="AWS Bedrock Streaming Response"
+import litellm
+import os
+
+# Set AWS credentials
+os.environ["AWS_ACCESS_KEY_ID"] = "your-access-key-id"
+os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret-access-key"
+os.environ["AWS_REGION_NAME"] = "us-west-2" # or your AWS region
+
+# Streaming response
+response = litellm.responses(
+ model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+
+#### Non-streaming
+```python showLineNumbers title="Google AI Studio Non-streaming Response"
+import litellm
+import os
+
+# Set API key for Google AI Studio
+os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"
+
+# Non-streaming response
+response = litellm.responses(
+ model="gemini/gemini-1.5-flash",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ max_output_tokens=100
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="Google AI Studio Streaming Response"
+import litellm
+import os
+
+# Set API key for Google AI Studio
+os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"
+
+# Streaming response
+response = litellm.responses(
+ model="gemini/gemini-1.5-flash",
input="Tell me a three sentence bedtime story about a unicorn.",
stream=True
)
@@ -115,3 +226,408 @@ for event in response:
+
+### LiteLLM Proxy with OpenAI SDK
+
+First, set up and start your LiteLLM proxy server.
+
+```bash title="Start LiteLLM Proxy Server"
+litellm --config /path/to/config.yaml
+
+# RUNNING on http://0.0.0.0:4000
+```
+
+
+
+
+First, add this to your litellm proxy config.yaml:
+```yaml showLineNumbers title="OpenAI Proxy Configuration"
+model_list:
+ - model_name: openai/o1-pro
+ litellm_params:
+ model: openai/o1-pro
+ api_key: os.environ/OPENAI_API_KEY
+```
+
+#### Non-streaming
+```python showLineNumbers title="OpenAI Proxy Non-streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Non-streaming response
+response = client.responses.create(
+ model="openai/o1-pro",
+ input="Tell me a three sentence bedtime story about a unicorn."
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="OpenAI Proxy Streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Streaming response
+response = client.responses.create(
+ model="openai/o1-pro",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+
+First, add this to your litellm proxy config.yaml:
+```yaml showLineNumbers title="Anthropic Proxy Configuration"
+model_list:
+ - model_name: anthropic/claude-3-5-sonnet-20240620
+ litellm_params:
+ model: anthropic/claude-3-5-sonnet-20240620
+ api_key: os.environ/ANTHROPIC_API_KEY
+```
+
+#### Non-streaming
+```python showLineNumbers title="Anthropic Proxy Non-streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Non-streaming response
+response = client.responses.create(
+ model="anthropic/claude-3-5-sonnet-20240620",
+ input="Tell me a three sentence bedtime story about a unicorn."
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="Anthropic Proxy Streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Streaming response
+response = client.responses.create(
+ model="anthropic/claude-3-5-sonnet-20240620",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+
+First, add this to your litellm proxy config.yaml:
+```yaml showLineNumbers title="Vertex AI Proxy Configuration"
+model_list:
+ - model_name: vertex_ai/gemini-1.5-pro
+ litellm_params:
+ model: vertex_ai/gemini-1.5-pro
+ vertex_project: your-gcp-project-id
+ vertex_location: us-central1
+```
+
+#### Non-streaming
+```python showLineNumbers title="Vertex AI Proxy Non-streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Non-streaming response
+response = client.responses.create(
+ model="vertex_ai/gemini-1.5-pro",
+ input="Tell me a three sentence bedtime story about a unicorn."
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="Vertex AI Proxy Streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Streaming response
+response = client.responses.create(
+ model="vertex_ai/gemini-1.5-pro",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+
+First, add this to your litellm proxy config.yaml:
+```yaml showLineNumbers title="AWS Bedrock Proxy Configuration"
+model_list:
+ - model_name: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
+ litellm_params:
+ model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
+ aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
+ aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
+ aws_region_name: us-west-2
+```
+
+#### Non-streaming
+```python showLineNumbers title="AWS Bedrock Proxy Non-streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Non-streaming response
+response = client.responses.create(
+ model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
+ input="Tell me a three sentence bedtime story about a unicorn."
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="AWS Bedrock Proxy Streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Streaming response
+response = client.responses.create(
+ model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+
+First, add this to your litellm proxy config.yaml:
+```yaml showLineNumbers title="Google AI Studio Proxy Configuration"
+model_list:
+ - model_name: gemini/gemini-1.5-flash
+ litellm_params:
+ model: gemini/gemini-1.5-flash
+ api_key: os.environ/GEMINI_API_KEY
+```
+
+#### Non-streaming
+```python showLineNumbers title="Google AI Studio Proxy Non-streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Non-streaming response
+response = client.responses.create(
+ model="gemini/gemini-1.5-flash",
+ input="Tell me a three sentence bedtime story about a unicorn."
+)
+
+print(response)
+```
+
+#### Streaming
+```python showLineNumbers title="Google AI Studio Proxy Streaming Response"
+from openai import OpenAI
+
+# Initialize client with your proxy URL
+client = OpenAI(
+ base_url="http://localhost:4000", # Your proxy URL
+ api_key="your-api-key" # Your proxy API key
+)
+
+# Streaming response
+response = client.responses.create(
+ model="gemini/gemini-1.5-flash",
+ input="Tell me a three sentence bedtime story about a unicorn.",
+ stream=True
+)
+
+for event in response:
+ print(event)
+```
+
+
+
+
+## Supported Responses API Parameters
+
+| Provider | Supported Parameters |
+|----------|---------------------|
+| `openai` | [All Responses API parameters are supported](https://github.com/BerriAI/litellm/blob/7c3df984da8e4dff9201e4c5353fdc7a2b441831/litellm/llms/openai/responses/transformation.py#L23) |
+| `azure` | [All Responses API parameters are supported](https://github.com/BerriAI/litellm/blob/7c3df984da8e4dff9201e4c5353fdc7a2b441831/litellm/llms/openai/responses/transformation.py#L23) |
+| `anthropic` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
+| `bedrock` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
+| `gemini` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
+| `vertex_ai` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
+| `azure_ai` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
+| All other llm api providers | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
+
+## Load Balancing with Routing Affinity
+
+When using the Responses API with multiple deployments of the same model (e.g., multiple Azure OpenAI endpoints), LiteLLM provides routing affinity for conversations. This ensures that follow-up requests using a `previous_response_id` are routed to the same deployment that generated the original response.
+
+
+#### Example Usage
+
+
+
+
+```python showLineNumbers title="Python SDK with Routing Affinity"
+import litellm
+
+# Set up router with multiple deployments of the same model
+router = litellm.Router(
+ model_list=[
+ {
+ "model_name": "azure-gpt4-turbo",
+ "litellm_params": {
+ "model": "azure/gpt-4-turbo",
+ "api_key": "your-api-key-1",
+ "api_version": "2024-06-01",
+ "api_base": "https://endpoint1.openai.azure.com",
+ },
+ },
+ {
+ "model_name": "azure-gpt4-turbo",
+ "litellm_params": {
+ "model": "azure/gpt-4-turbo",
+ "api_key": "your-api-key-2",
+ "api_version": "2024-06-01",
+ "api_base": "https://endpoint2.openai.azure.com",
+ },
+ },
+ ],
+ optional_pre_call_checks=["responses_api_deployment_check"],
+)
+
+# Initial request
+response = await router.aresponses(
+ model="azure-gpt4-turbo",
+ input="Hello, who are you?",
+ truncation="auto",
+)
+
+# Store the response ID
+response_id = response.id
+
+# Follow-up request - will be automatically routed to the same deployment
+follow_up = await router.aresponses(
+ model="azure-gpt4-turbo",
+ input="Tell me more about yourself",
+ truncation="auto",
+ previous_response_id=response_id # This ensures routing to the same deployment
+)
+```
+
+
+
+
+#### 1. Setup routing affinity on proxy config.yaml
+
+To enable routing affinity for Responses API in your LiteLLM proxy, set `optional_pre_call_checks: ["responses_api_deployment_check"]` in your proxy config.yaml.
+
+```yaml showLineNumbers title="config.yaml with Responses API Routing Affinity"
+model_list:
+ - model_name: azure-gpt4-turbo
+ litellm_params:
+ model: azure/gpt-4-turbo
+ api_key: your-api-key-1
+ api_version: 2024-06-01
+ api_base: https://endpoint1.openai.azure.com
+ - model_name: azure-gpt4-turbo
+ litellm_params:
+ model: azure/gpt-4-turbo
+ api_key: your-api-key-2
+ api_version: 2024-06-01
+ api_base: https://endpoint2.openai.azure.com
+
+router_settings:
+ optional_pre_call_checks: ["responses_api_deployment_check"]
+```
+
+#### 2. Use the OpenAI Python SDK to make requests to LiteLLM Proxy
+
+```python showLineNumbers title="OpenAI Client with Proxy Server"
+from openai import OpenAI
+
+client = OpenAI(
+ base_url="http://localhost:4000",
+ api_key="your-api-key"
+)
+
+# Initial request
+response = client.responses.create(
+ model="azure-gpt4-turbo",
+ input="Hello, who are you?"
+)
+
+response_id = response.id
+
+# Follow-up request - will be automatically routed to the same deployment
+follow_up = client.responses.create(
+ model="azure-gpt4-turbo",
+ input="Tell me more about yourself",
+ previous_response_id=response_id # This ensures routing to the same deployment
+)
+```
+
+
+
diff --git a/docs/my-website/docs/set_keys.md b/docs/my-website/docs/set_keys.md
index 3a5ff08d63..693cf5f7f4 100644
--- a/docs/my-website/docs/set_keys.md
+++ b/docs/my-website/docs/set_keys.md
@@ -188,7 +188,13 @@ Currently implemented for:
- OpenAI (if OPENAI_API_KEY is set)
- Fireworks AI (if FIREWORKS_AI_API_KEY is set)
- LiteLLM Proxy (if LITELLM_PROXY_API_KEY is set)
+- Gemini (if GEMINI_API_KEY is set)
+- XAI (if XAI_API_KEY is set)
+- Anthropic (if ANTHROPIC_API_KEY is set)
+You can also specify a custom provider to check:
+
+**All providers**:
```python
from litellm import get_valid_models
@@ -196,6 +202,14 @@ valid_models = get_valid_models(check_provider_endpoint=True)
print(valid_models)
```
+**Specific provider**:
+```python
+from litellm import get_valid_models
+
+valid_models = get_valid_models(check_provider_endpoint=True, custom_llm_provider="openai")
+print(valid_models)
+```
+
### `validate_environment(model: str)`
This helper tells you if you have all the required environment variables for a model, and if not - what's missing.
diff --git a/docs/my-website/docs/tutorials/msft_sso.md b/docs/my-website/docs/tutorials/msft_sso.md
new file mode 100644
index 0000000000..f7ad6440f2
--- /dev/null
+++ b/docs/my-website/docs/tutorials/msft_sso.md
@@ -0,0 +1,162 @@
+import Image from '@theme/IdealImage';
+
+# Microsoft SSO: Sync Groups, Members with LiteLLM
+
+Sync Microsoft SSO Groups, Members with LiteLLM Teams.
+
+
+
+
+
+
+
+## Prerequisites
+
+- An Azure Entra ID account with administrative access
+- A LiteLLM Enterprise App set up in your Azure Portal
+- Access to Microsoft Entra ID (Azure AD)
+
+
+## Overview of this tutorial
+
+1. Auto-Create Entra ID Groups on LiteLLM Teams
+2. Sync Entra ID Team Memberships
+3. Set default params for new teams and users auto-created on LiteLLM
+
+## 1. Auto-Create Entra ID Groups on LiteLLM Teams
+
+In this step, our goal is to have LiteLLM automatically create a new team on the LiteLLM DB when there is a new Group Added to the LiteLLM Enterprise App on Azure Entra ID.
+
+### 1.1 Create a new group in Entra ID
+
+
+Navigate to [your Azure Portal](https://portal.azure.com/) > Groups > New Group. Create a new group.
+
+
+
+### 1.2 Assign the group to your LiteLLM Enterprise App
+
+On your Azure Portal, navigate to `Enterprise Applications` > Select your litellm app
+
+
+
+
+
+
+Once you've selected your litellm app, click on `Users and Groups` > `Add user/group`
+
+
+
+
+
+Now select the group you created in step 1.1. And add it to the LiteLLM Enterprise App. At this point we have added `Production LLM Evals Group` to the LiteLLM Enterprise App. The next steps is having LiteLLM automatically create the `Production LLM Evals Group` on the LiteLLM DB when a new user signs in.
+
+
+
+
+### 1.3 Sign in to LiteLLM UI via SSO
+
+Sign into the LiteLLM UI via SSO. You should be redirected to the Entra ID SSO page. This SSO sign in flow will trigger LiteLLM to fetch the latest Groups and Members from Azure Entra ID.
+
+
+
+### 1.4 Check the new team on LiteLLM UI
+
+On the LiteLLM UI, Navigate to `Teams`, You should see the new team `Production LLM Evals Group` auto-created on LiteLLM.
+
+
+
+#### How this works
+
+When a SSO user signs in to LiteLLM:
+- LiteLLM automatically fetches the Groups under the LiteLLM Enterprise App
+- It finds the Production LLM Evals Group assigned to the LiteLLM Enterprise App
+- LiteLLM checks if this group's ID exists in the LiteLLM Teams Table
+- Since the ID doesn't exist, LiteLLM automatically creates a new team with:
+ - Name: Production LLM Evals Group
+ - ID: Same as the Entra ID group's ID
+
+## 2. Sync Entra ID Team Memberships
+
+In this step, we will have LiteLLM automatically add a user to the `Production LLM Evals` Team on the LiteLLM DB when a new user is added to the `Production LLM Evals` Group in Entra ID.
+
+### 2.1 Navigate to the `Production LLM Evals` Group in Entra ID
+
+Navigate to the `Production LLM Evals` Group in Entra ID.
+
+
+
+
+### 2.2 Add a member to the group in Entra ID
+
+Select `Members` > `Add members`
+
+In this stage you should add the user you want to add to the `Production LLM Evals` Team.
+
+
+
+
+
+### 2.3 Sign in as the new user on LiteLLM UI
+
+Sign in as the new user on LiteLLM UI. You should be redirected to the Entra ID SSO page. This SSO sign in flow will trigger LiteLLM to fetch the latest Groups and Members from Azure Entra ID. During this step LiteLLM sync it's teams, team members with what is available from Entra ID
+
+
+
+
+
+### 2.4 Check the team membership on LiteLLM UI
+
+On the LiteLLM UI, Navigate to `Teams`, You should see the new team `Production LLM Evals Group`. Since your are now a member of the `Production LLM Evals Group` in Entra ID, you should see the new team `Production LLM Evals Group` on the LiteLLM UI.
+
+
+
+## 3. Set default params for new teams auto-created on LiteLLM
+
+Since litellm auto creates a new team on the LiteLLM DB when there is a new Group Added to the LiteLLM Enterprise App on Azure Entra ID, we can set default params for new teams created.
+
+This allows you to set a default budget, models, etc for new teams created.
+
+### 3.1 Set `default_team_params` on litellm
+
+Navigate to your litellm config file and set the following params
+
+```yaml showLineNumbers title="litellm config with default_team_params"
+litellm_settings:
+ default_team_params: # Default Params to apply when litellm auto creates a team from SSO IDP provider
+ max_budget: 100 # Optional[float], optional): $100 budget for the team
+ budget_duration: 30d # Optional[str], optional): 30 days budget_duration for the team
+ models: ["gpt-3.5-turbo"] # Optional[List[str]], optional): models to be used by the team
+```
+
+### 3.2 Auto-create a new team on LiteLLM
+
+- In this step you should add a new group to the LiteLLM Enterprise App on Azure Entra ID (like we did in step 1.1). We will call this group `Default LiteLLM Prod Team` on Azure Entra ID.
+- Start litellm proxy server with your config
+- Sign into LiteLLM UI via SSO
+- Navigate to `Teams` and you should see the new team `Default LiteLLM Prod Team` auto-created on LiteLLM
+- Note LiteLLM will set the default params for this new team.
+
+
+
+
+## Video Walkthrough
+
+This walks through setting up sso auto-add for **Microsoft Entra ID**
+
+Follow along this video for a walkthrough of how to set this up with Microsoft Entra ID
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/my-website/docs/tutorials/openai_codex.md b/docs/my-website/docs/tutorials/openai_codex.md
new file mode 100644
index 0000000000..bb5af956b0
--- /dev/null
+++ b/docs/my-website/docs/tutorials/openai_codex.md
@@ -0,0 +1,146 @@
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Using LiteLLM with OpenAI Codex
+
+This guide walks you through connecting OpenAI Codex to LiteLLM. Using LiteLLM with Codex allows teams to:
+- Access 100+ LLMs through the Codex interface
+- Use powerful models like Gemini through a familiar interface
+- Track spend and usage with LiteLLM's built-in analytics
+- Control model access with virtual keys
+
+
+
+## Quickstart
+
+:::info
+
+Requires LiteLLM v1.66.3.dev5 and higher
+
+:::
+
+
+Make sure to set up LiteLLM with the [LiteLLM Getting Started Guide](../proxy/docker_quick_start.md).
+
+## 1. Install OpenAI Codex
+
+Install the OpenAI Codex CLI tool globally using npm:
+
+
+
+
+```bash showLineNumbers
+npm i -g @openai/codex
+```
+
+
+
+
+```bash showLineNumbers
+yarn global add @openai/codex
+```
+
+
+
+
+## 2. Start LiteLLM Proxy
+
+
+
+
+```bash showLineNumbers
+docker run \
+ -v $(pwd)/litellm_config.yaml:/app/config.yaml \
+ -p 4000:4000 \
+ ghcr.io/berriai/litellm:main-latest \
+ --config /app/config.yaml
+```
+
+
+
+
+```bash showLineNumbers
+litellm --config /path/to/config.yaml
+```
+
+
+
+
+LiteLLM should now be running on [http://localhost:4000](http://localhost:4000)
+
+## 3. Configure LiteLLM for Model Routing
+
+Ensure your LiteLLM Proxy is properly configured to route to your desired models. Create a `litellm_config.yaml` file with the following content:
+
+```yaml showLineNumbers
+model_list:
+ - model_name: o3-mini
+ litellm_params:
+ model: openai/o3-mini
+ api_key: os.environ/OPENAI_API_KEY
+ - model_name: claude-3-7-sonnet-latest
+ litellm_params:
+ model: anthropic/claude-3-7-sonnet-latest
+ api_key: os.environ/ANTHROPIC_API_KEY
+ - model_name: gemini-2.0-flash
+ litellm_params:
+ model: gemini/gemini-2.0-flash
+ api_key: os.environ/GEMINI_API_KEY
+
+litellm_settings:
+ drop_params: true
+```
+
+This configuration enables routing to specific OpenAI, Anthropic, and Gemini models with explicit names.
+
+## 4. Configure Codex to Use LiteLLM Proxy
+
+Set the required environment variables to point Codex to your LiteLLM Proxy:
+
+```bash
+# Point to your LiteLLM Proxy server
+export OPENAI_BASE_URL=http://0.0.0.0:4000
+
+# Use your LiteLLM API key (if you've set up authentication)
+export OPENAI_API_KEY="sk-1234"
+```
+
+## 5. Run Codex with Gemini
+
+With everything configured, you can now run Codex with Gemini:
+
+```bash showLineNumbers
+codex --model gemini-2.0-flash --full-auto
+```
+
+
+
+The `--full-auto` flag allows Codex to automatically generate code without additional prompting.
+
+## 6. Advanced Options
+
+### Using Different Models
+
+You can use any model configured in your LiteLLM proxy:
+
+```bash
+# Use Claude models
+codex --model claude-3-7-sonnet-latest
+
+# Use Google AI Studio Gemini models
+codex --model gemini/gemini-2.0-flash
+```
+
+## Troubleshooting
+
+- If you encounter connection issues, ensure your LiteLLM Proxy is running and accessible at the specified URL
+- Verify your LiteLLM API key is valid if you're using authentication
+- Check that your model routing configuration is correct
+- For model-specific errors, ensure the model is properly configured in your LiteLLM setup
+
+## Additional Resources
+
+- [LiteLLM Docker Quick Start Guide](../proxy/docker_quick_start.md)
+- [OpenAI Codex GitHub Repository](https://github.com/openai/codex)
+- [LiteLLM Virtual Keys and Authentication](../proxy/virtual_keys.md)
diff --git a/docs/my-website/docs/tutorials/openweb_ui.md b/docs/my-website/docs/tutorials/openweb_ui.md
index ab1e2e121e..b2c1204069 100644
--- a/docs/my-website/docs/tutorials/openweb_ui.md
+++ b/docs/my-website/docs/tutorials/openweb_ui.md
@@ -98,6 +98,5 @@ On the models dropdown select `thinking-anthropic-claude-3-7-sonnet`
-
-
-
+## Additional Resources
+- Running LiteLLM and OpenWebUI on Windows Localhost: A Comprehensive Guide [https://www.tanyongsheng.com/note/running-litellm-and-openwebui-on-windows-localhost-a-comprehensive-guide/](https://www.tanyongsheng.com/note/running-litellm-and-openwebui-on-windows-localhost-a-comprehensive-guide/)
\ No newline at end of file
diff --git a/docs/my-website/docs/tutorials/prompt_caching.md b/docs/my-website/docs/tutorials/prompt_caching.md
new file mode 100644
index 0000000000..bf3d5a8dda
--- /dev/null
+++ b/docs/my-website/docs/tutorials/prompt_caching.md
@@ -0,0 +1,128 @@
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Auto-Inject Prompt Caching Checkpoints
+
+Reduce costs by up to 90% by using LiteLLM to auto-inject prompt caching checkpoints.
+
+
+
+
+## How it works
+
+LiteLLM can automatically inject prompt caching checkpoints into your requests to LLM providers. This allows:
+
+- **Cost Reduction**: Long, static parts of your prompts can be cached to avoid repeated processing
+- **No need to modify your application code**: You can configure the auto-caching behavior in the LiteLLM UI or in the `litellm config.yaml` file.
+
+## Configuration
+
+You need to specify `cache_control_injection_points` in your model configuration. This tells LiteLLM:
+1. Where to add the caching directive (`location`)
+2. Which message to target (`role`)
+
+LiteLLM will then automatically add a `cache_control` directive to the specified messages in your requests:
+
+```json
+"cache_control": {
+ "type": "ephemeral"
+}
+```
+
+## Usage Example
+
+In this example, we'll configure caching for system messages by adding the directive to all messages with `role: system`.
+
+
+
+
+```yaml showLineNumbers title="litellm config.yaml"
+model_list:
+ - model_name: anthropic-auto-inject-cache-system-message
+ litellm_params:
+ model: anthropic/claude-3-5-sonnet-20240620
+ api_key: os.environ/ANTHROPIC_API_KEY
+ cache_control_injection_points:
+ - location: message
+ role: system
+```
+
+
+
+
+On the LiteLLM UI, you can specify the `cache_control_injection_points` in the `Advanced Settings` tab when adding a model.
+
+
+
+
+
+
+## Detailed Example
+
+### 1. Original Request to LiteLLM
+
+In this example, we have a very long, static system message and a varying user message. It's efficient to cache the system message since it rarely changes.
+
+```json
+{
+ "messages": [
+ {
+ "role": "system",
+ "content": [
+ {
+ "type": "text",
+ "text": "You are a helpful assistant. This is a set of very long instructions that you will follow. Here is a legal document that you will use to answer the user's question."
+ }
+ ]
+ },
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": "What is the main topic of this legal document?"
+ }
+ ]
+ }
+ ]
+}
+```
+
+### 2. LiteLLM's Modified Request
+
+LiteLLM auto-injects the caching directive into the system message based on our configuration:
+
+```json
+{
+ "messages": [
+ {
+ "role": "system",
+ "content": [
+ {
+ "type": "text",
+ "text": "You are a helpful assistant. This is a set of very long instructions that you will follow. Here is a legal document that you will use to answer the user's question.",
+ "cache_control": {"type": "ephemeral"}
+ }
+ ]
+ },
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": "What is the main topic of this legal document?"
+ }
+ ]
+ }
+ ]
+}
+```
+
+When the model provider processes this request, it will recognize the caching directive and only process the system message once, caching it for subsequent requests.
+
+
+
+
+
+
diff --git a/docs/my-website/docs/tutorials/scim_litellm.md b/docs/my-website/docs/tutorials/scim_litellm.md
new file mode 100644
index 0000000000..c744abe4b4
--- /dev/null
+++ b/docs/my-website/docs/tutorials/scim_litellm.md
@@ -0,0 +1,74 @@
+
+import Image from '@theme/IdealImage';
+
+# SCIM with LiteLLM
+
+Enables identity providers (Okta, Azure AD, OneLogin, etc.) to automate user and team (group) provisioning, updates, and deprovisioning on LiteLLM.
+
+
+This tutorial will walk you through the steps to connect your IDP to LiteLLM SCIM Endpoints.
+
+### Supported SSO Providers for SCIM
+Below is a list of supported SSO providers for connecting to LiteLLM SCIM Endpoints.
+- Microsoft Entra ID (Azure AD)
+- Okta
+- Google Workspace
+- OneLogin
+- Keycloak
+- Auth0
+
+
+## 1. Get your SCIM Tenant URL and Bearer Token
+
+On LiteLLM, navigate to the Settings > Admin Settings > SCIM. On this page you will create a SCIM Token, this allows your IDP to authenticate to litellm `/scim` endpoints.
+
+
+
+## 2. Connect your IDP to LiteLLM SCIM Endpoints
+
+On your IDP provider, navigate to your SSO application and select `Provisioning` > `New provisioning configuration`.
+
+On this page, paste in your litellm scim tenant url and bearer token.
+
+Once this is pasted in, click on `Test Connection` to ensure your IDP can authenticate to the LiteLLM SCIM endpoints.
+
+
+
+
+## 3. Test SCIM Connection
+
+### 3.1 Assign the group to your LiteLLM Enterprise App
+
+On your IDP Portal, navigate to `Enterprise Applications` > Select your litellm app
+
+
+
+
+
+
+Once you've selected your litellm app, click on `Users and Groups` > `Add user/group`
+
+
+
+
+
+Now select the group you created in step 1.1. And add it to the LiteLLM Enterprise App. At this point we have added `Production LLM Evals Group` to the LiteLLM Enterprise App. The next step is having LiteLLM automatically create the `Production LLM Evals Group` on the LiteLLM DB when a new user signs in.
+
+
+
+
+### 3.2 Sign in to LiteLLM UI via SSO
+
+Sign into the LiteLLM UI via SSO. You should be redirected to the Entra ID SSO page. This SSO sign in flow will trigger LiteLLM to fetch the latest Groups and Members from Azure Entra ID.
+
+
+
+### 3.3 Check the new team on LiteLLM UI
+
+On the LiteLLM UI, Navigate to `Teams`, You should see the new team `Production LLM Evals Group` auto-created on LiteLLM.
+
+
+
+
+
+
diff --git a/docs/my-website/docs/tutorials/tag_management.md b/docs/my-website/docs/tutorials/tag_management.md
new file mode 100644
index 0000000000..9b00db47d1
--- /dev/null
+++ b/docs/my-website/docs/tutorials/tag_management.md
@@ -0,0 +1,145 @@
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# [Beta] Routing based on request metadata
+
+Create routing rules based on request metadata.
+
+## Setup
+
+Add the following to your litellm proxy config yaml file.
+
+```yaml showLineNumbers title="litellm proxy config.yaml"
+router_settings:
+ enable_tag_filtering: True # 👈 Key Change
+```
+
+## 1. Create a tag
+
+On the LiteLLM UI, navigate to Experimental > Tag Management > Create Tag.
+
+Create a tag called `private-data` and only select the allowed models for requests with this tag. Once created, you will see the tag in the Tag Management page.
+
+
+
+
+## 2. Test Tag Routing
+
+Now we will test the tag based routing rules.
+
+### 2.1 Invalid model
+
+This request will fail since we send `tags=private-data` but the model `gpt-4o` is not in the allowed models for the `private-data` tag.
+
+
+
+
+
+Here is an example sending the same request using the OpenAI Python SDK.
+
+
+
+```python showLineNumbers
+from openai import OpenAI
+
+client = OpenAI(
+ api_key="sk-1234",
+ base_url="http://0.0.0.0:4000/v1/"
+)
+
+response = client.chat.completions.create(
+ model="gpt-4o",
+ messages=[
+ {"role": "user", "content": "Hello, how are you?"}
+ ],
+ extra_body={
+ "tags": "private-data"
+ }
+)
+```
+
+
+
+
+```bash
+curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
+-H 'Content-Type: application/json' \
+-H 'Authorization: Bearer sk-1234' \
+-d '{
+ "model": "gpt-4o",
+ "messages": [
+ {
+ "role": "user",
+ "content": "Hello, how are you?"
+ }
+ ],
+ "tags": "private-data"
+}'
+```
+
+
+
+
+
+
+### 2.2 Valid model
+
+This request will succeed since we send `tags=private-data` and the model `us.anthropic.claude-3-7-sonnet-20250219-v1:0` is in the allowed models for the `private-data` tag.
+
+
+
+Here is an example sending the same request using the OpenAI Python SDK.
+
+
+
+
+```python showLineNumbers
+from openai import OpenAI
+
+client = OpenAI(
+ api_key="sk-1234",
+ base_url="http://0.0.0.0:4000/v1/"
+)
+
+response = client.chat.completions.create(
+ model="us.anthropic.claude-3-7-sonnet-20250219-v1:0",
+ messages=[
+ {"role": "user", "content": "Hello, how are you?"}
+ ],
+ extra_body={
+ "tags": "private-data"
+ }
+)
+```
+
+
+
+
+```bash
+curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
+-H 'Content-Type: application/json' \
+-H 'Authorization: Bearer sk-1234' \
+-d '{
+ "model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0",
+ "messages": [
+ {
+ "role": "user",
+ "content": "Hello, how are you?"
+ }
+ ],
+ "tags": "private-data"
+}'
+```
+
+
+
+
+
+
+## Additional Tag Features
+- [Sending tags in request headers](https://docs.litellm.ai/docs/proxy/tag_routing#calling-via-request-header)
+- [Tag based routing](https://docs.litellm.ai/docs/proxy/tag_routing)
+- [Track spend per tag](cost_tracking#-custom-tags)
+- [Setup Budgets per Virtual Key, Team](users)
+
diff --git a/docs/my-website/img/arize.png b/docs/my-website/img/arize.png
new file mode 100644
index 0000000000..45d6dacda9
Binary files /dev/null and b/docs/my-website/img/arize.png differ
diff --git a/docs/my-website/img/auto_prompt_caching.png b/docs/my-website/img/auto_prompt_caching.png
new file mode 100644
index 0000000000..6cd3785512
Binary files /dev/null and b/docs/my-website/img/auto_prompt_caching.png differ
diff --git a/docs/my-website/img/custom_prompt_management.png b/docs/my-website/img/custom_prompt_management.png
new file mode 100644
index 0000000000..2c96e0d116
Binary files /dev/null and b/docs/my-website/img/custom_prompt_management.png differ
diff --git a/docs/my-website/img/deadlock_fix_1.png b/docs/my-website/img/deadlock_fix_1.png
new file mode 100644
index 0000000000..df651f440c
Binary files /dev/null and b/docs/my-website/img/deadlock_fix_1.png differ
diff --git a/docs/my-website/img/deadlock_fix_2.png b/docs/my-website/img/deadlock_fix_2.png
new file mode 100644
index 0000000000..0f139d84e5
Binary files /dev/null and b/docs/my-website/img/deadlock_fix_2.png differ
diff --git a/docs/my-website/img/debug_sso.png b/docs/my-website/img/debug_sso.png
new file mode 100644
index 0000000000..d7dde36892
Binary files /dev/null and b/docs/my-website/img/debug_sso.png differ
diff --git a/docs/my-website/img/enterprise_vs_oss.png b/docs/my-website/img/enterprise_vs_oss.png
new file mode 100644
index 0000000000..f2b58fbc14
Binary files /dev/null and b/docs/my-website/img/enterprise_vs_oss.png differ
diff --git a/docs/my-website/img/entra_create_team.png b/docs/my-website/img/entra_create_team.png
new file mode 100644
index 0000000000..223a897d87
Binary files /dev/null and b/docs/my-website/img/entra_create_team.png differ
diff --git a/docs/my-website/img/hf_filter_inference_providers.png b/docs/my-website/img/hf_filter_inference_providers.png
new file mode 100644
index 0000000000..d4c7188919
Binary files /dev/null and b/docs/my-website/img/hf_filter_inference_providers.png differ
diff --git a/docs/my-website/img/image_handling.png b/docs/my-website/img/image_handling.png
new file mode 100644
index 0000000000..bd56206911
Binary files /dev/null and b/docs/my-website/img/image_handling.png differ
diff --git a/docs/my-website/img/litellm_codex.gif b/docs/my-website/img/litellm_codex.gif
new file mode 100644
index 0000000000..04332b5053
Binary files /dev/null and b/docs/my-website/img/litellm_codex.gif differ
diff --git a/docs/my-website/img/litellm_entra_id.png b/docs/my-website/img/litellm_entra_id.png
new file mode 100644
index 0000000000..4cfbd0747f
Binary files /dev/null and b/docs/my-website/img/litellm_entra_id.png differ
diff --git a/docs/my-website/img/litellm_mcp.png b/docs/my-website/img/litellm_mcp.png
new file mode 100644
index 0000000000..cef822eeb2
Binary files /dev/null and b/docs/my-website/img/litellm_mcp.png differ
diff --git a/docs/my-website/img/managed_files_arch.png b/docs/my-website/img/managed_files_arch.png
new file mode 100644
index 0000000000..e49c47334d
Binary files /dev/null and b/docs/my-website/img/managed_files_arch.png differ
diff --git a/docs/my-website/img/mcp_2.png b/docs/my-website/img/mcp_2.png
new file mode 100644
index 0000000000..98e063efc5
Binary files /dev/null and b/docs/my-website/img/mcp_2.png differ
diff --git a/docs/my-website/img/mcp_ui.png b/docs/my-website/img/mcp_ui.png
new file mode 100644
index 0000000000..6731fba71b
Binary files /dev/null and b/docs/my-website/img/mcp_ui.png differ
diff --git a/docs/my-website/img/msft_auto_team.png b/docs/my-website/img/msft_auto_team.png
new file mode 100644
index 0000000000..a50c5bbfbd
Binary files /dev/null and b/docs/my-website/img/msft_auto_team.png differ
diff --git a/docs/my-website/img/msft_default_settings.png b/docs/my-website/img/msft_default_settings.png
new file mode 100644
index 0000000000..0caa60b1f5
Binary files /dev/null and b/docs/my-website/img/msft_default_settings.png differ
diff --git a/docs/my-website/img/msft_enterprise_app.png b/docs/my-website/img/msft_enterprise_app.png
new file mode 100644
index 0000000000..0a8c849a5c
Binary files /dev/null and b/docs/my-website/img/msft_enterprise_app.png differ
diff --git a/docs/my-website/img/msft_enterprise_assign_group.png b/docs/my-website/img/msft_enterprise_assign_group.png
new file mode 100644
index 0000000000..d43e1c6684
Binary files /dev/null and b/docs/my-website/img/msft_enterprise_assign_group.png differ
diff --git a/docs/my-website/img/msft_enterprise_select_group.png b/docs/my-website/img/msft_enterprise_select_group.png
new file mode 100644
index 0000000000..e49032db9f
Binary files /dev/null and b/docs/my-website/img/msft_enterprise_select_group.png differ
diff --git a/docs/my-website/img/msft_member_1.png b/docs/my-website/img/msft_member_1.png
new file mode 100644
index 0000000000..2fe627f773
Binary files /dev/null and b/docs/my-website/img/msft_member_1.png differ
diff --git a/docs/my-website/img/msft_member_2.png b/docs/my-website/img/msft_member_2.png
new file mode 100644
index 0000000000..9757aa9cea
Binary files /dev/null and b/docs/my-website/img/msft_member_2.png differ
diff --git a/docs/my-website/img/msft_member_3.png b/docs/my-website/img/msft_member_3.png
new file mode 100644
index 0000000000..783a4a1dd8
Binary files /dev/null and b/docs/my-website/img/msft_member_3.png differ
diff --git a/docs/my-website/img/msft_sso_sign_in.png b/docs/my-website/img/msft_sso_sign_in.png
new file mode 100644
index 0000000000..43c5173295
Binary files /dev/null and b/docs/my-website/img/msft_sso_sign_in.png differ
diff --git a/docs/my-website/img/prevent_deadlocks.jpg b/docs/my-website/img/prevent_deadlocks.jpg
new file mode 100644
index 0000000000..2807f327d1
Binary files /dev/null and b/docs/my-website/img/prevent_deadlocks.jpg differ
diff --git a/docs/my-website/img/realtime_api.png b/docs/my-website/img/realtime_api.png
new file mode 100644
index 0000000000..798525278c
Binary files /dev/null and b/docs/my-website/img/realtime_api.png differ
diff --git a/docs/my-website/img/release_notes/chat_metrics.png b/docs/my-website/img/release_notes/chat_metrics.png
new file mode 100644
index 0000000000..2e45392cd6
Binary files /dev/null and b/docs/my-website/img/release_notes/chat_metrics.png differ
diff --git a/docs/my-website/img/release_notes/mcp_ui.png b/docs/my-website/img/release_notes/mcp_ui.png
new file mode 100644
index 0000000000..8f4cd4ea19
Binary files /dev/null and b/docs/my-website/img/release_notes/mcp_ui.png differ
diff --git a/docs/my-website/img/release_notes/new_activity_tab.png b/docs/my-website/img/release_notes/new_activity_tab.png
new file mode 100644
index 0000000000..e8cea22a90
Binary files /dev/null and b/docs/my-website/img/release_notes/new_activity_tab.png differ
diff --git a/docs/my-website/img/release_notes/new_tag_usage.png b/docs/my-website/img/release_notes/new_tag_usage.png
new file mode 100644
index 0000000000..4188cbc245
Binary files /dev/null and b/docs/my-website/img/release_notes/new_tag_usage.png differ
diff --git a/docs/my-website/img/release_notes/new_team_usage.png b/docs/my-website/img/release_notes/new_team_usage.png
new file mode 100644
index 0000000000..5fea2506d9
Binary files /dev/null and b/docs/my-website/img/release_notes/new_team_usage.png differ
diff --git a/docs/my-website/img/release_notes/new_team_usage_highlight.jpg b/docs/my-website/img/release_notes/new_team_usage_highlight.jpg
new file mode 100644
index 0000000000..05dbf4b918
Binary files /dev/null and b/docs/my-website/img/release_notes/new_team_usage_highlight.jpg differ
diff --git a/docs/my-website/img/release_notes/spend_by_model.jpg b/docs/my-website/img/release_notes/spend_by_model.jpg
new file mode 100644
index 0000000000..2584949eff
Binary files /dev/null and b/docs/my-website/img/release_notes/spend_by_model.jpg differ
diff --git a/docs/my-website/img/release_notes/sso_sync.png b/docs/my-website/img/release_notes/sso_sync.png
new file mode 100644
index 0000000000..a7bf6b838b
Binary files /dev/null and b/docs/my-website/img/release_notes/sso_sync.png differ
diff --git a/docs/my-website/img/release_notes/tag_management.png b/docs/my-website/img/release_notes/tag_management.png
new file mode 100644
index 0000000000..eca7b8cbb1
Binary files /dev/null and b/docs/my-website/img/release_notes/tag_management.png differ
diff --git a/docs/my-website/img/release_notes/team_model_add.png b/docs/my-website/img/release_notes/team_model_add.png
new file mode 100644
index 0000000000..f548469846
Binary files /dev/null and b/docs/my-website/img/release_notes/team_model_add.png differ
diff --git a/docs/my-website/img/release_notes/ui_usage.png b/docs/my-website/img/release_notes/ui_usage.png
new file mode 100644
index 0000000000..ac39ffb918
Binary files /dev/null and b/docs/my-website/img/release_notes/ui_usage.png differ
diff --git a/docs/my-website/img/release_notes/unified_responses_api_rn.png b/docs/my-website/img/release_notes/unified_responses_api_rn.png
new file mode 100644
index 0000000000..60ede0e211
Binary files /dev/null and b/docs/my-website/img/release_notes/unified_responses_api_rn.png differ
diff --git a/docs/my-website/img/scim_0.png b/docs/my-website/img/scim_0.png
new file mode 100644
index 0000000000..265271b78c
Binary files /dev/null and b/docs/my-website/img/scim_0.png differ
diff --git a/docs/my-website/img/scim_1.png b/docs/my-website/img/scim_1.png
new file mode 100644
index 0000000000..c6d64b5d11
Binary files /dev/null and b/docs/my-website/img/scim_1.png differ
diff --git a/docs/my-website/img/scim_2.png b/docs/my-website/img/scim_2.png
new file mode 100644
index 0000000000..c96cf9f0b5
Binary files /dev/null and b/docs/my-website/img/scim_2.png differ
diff --git a/docs/my-website/img/scim_3.png b/docs/my-website/img/scim_3.png
new file mode 100644
index 0000000000..5ecd3906bd
Binary files /dev/null and b/docs/my-website/img/scim_3.png differ
diff --git a/docs/my-website/img/scim_4.png b/docs/my-website/img/scim_4.png
new file mode 100644
index 0000000000..b4b484418c
Binary files /dev/null and b/docs/my-website/img/scim_4.png differ
diff --git a/docs/my-website/img/scim_integration.png b/docs/my-website/img/scim_integration.png
new file mode 100644
index 0000000000..2cfeb872bf
Binary files /dev/null and b/docs/my-website/img/scim_integration.png differ
diff --git a/docs/my-website/img/tag_create.png b/docs/my-website/img/tag_create.png
new file mode 100644
index 0000000000..d515b3a9f4
Binary files /dev/null and b/docs/my-website/img/tag_create.png differ
diff --git a/docs/my-website/img/tag_invalid.png b/docs/my-website/img/tag_invalid.png
new file mode 100644
index 0000000000..e12f7197b1
Binary files /dev/null and b/docs/my-website/img/tag_invalid.png differ
diff --git a/docs/my-website/img/tag_valid.png b/docs/my-website/img/tag_valid.png
new file mode 100644
index 0000000000..3b6e121d12
Binary files /dev/null and b/docs/my-website/img/tag_valid.png differ
diff --git a/docs/my-website/img/ui_auto_prompt_caching.png b/docs/my-website/img/ui_auto_prompt_caching.png
new file mode 100644
index 0000000000..e6f48e48d0
Binary files /dev/null and b/docs/my-website/img/ui_auto_prompt_caching.png differ
diff --git a/docs/my-website/package-lock.json b/docs/my-website/package-lock.json
index 6c07e67d91..e6f20d567b 100644
--- a/docs/my-website/package-lock.json
+++ b/docs/my-website/package-lock.json
@@ -2148,9 +2148,10 @@
}
},
"node_modules/@babel/runtime": {
- "version": "7.26.0",
- "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.26.0.tgz",
- "integrity": "sha512-FDSOghenHTiToteC/QRlv2q3DhPZ/oOXTBoirfWNx1Cx3TMVcGWQtMMmQcSvb/JjpNeGzx8Pq/b4fKEJuWm1sw==",
+ "version": "7.27.0",
+ "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.27.0.tgz",
+ "integrity": "sha512-VtPOkrdPHZsKc/clNqyi9WUA8TINkZ4cGk63UUE3u4pmB2k+ZMQRDuIOagv8UVd6j7k0T3+RRIb7beKTebNbcw==",
+ "license": "MIT",
"dependencies": {
"regenerator-runtime": "^0.14.0"
},
@@ -12454,9 +12455,10 @@
}
},
"node_modules/http-proxy-middleware": {
- "version": "2.0.7",
- "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.7.tgz",
- "integrity": "sha512-fgVY8AV7qU7z/MmXJ/rxwbrtQH4jBQ9m7kp3llF0liB7glmFeVZFBepQb32T3y8n8k2+AEYuMPCpinYW+/CuRA==",
+ "version": "2.0.9",
+ "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.9.tgz",
+ "integrity": "sha512-c1IyJYLYppU574+YI7R4QyX2ystMtVXZwIdzazUIPIJsHuWNd+mho2j+bKoHftndicGj9yh+xjd+l0yj7VeT1Q==",
+ "license": "MIT",
"dependencies": {
"@types/http-proxy": "^1.17.8",
"http-proxy": "^1.18.1",
@@ -12559,9 +12561,10 @@
}
},
"node_modules/image-size": {
- "version": "1.1.1",
- "resolved": "https://registry.npmjs.org/image-size/-/image-size-1.1.1.tgz",
- "integrity": "sha512-541xKlUw6jr/6gGuk92F+mYM5zaFAc5ahphvkqvNe2bQ6gVBkd6bfrmVJ2t4KDAfikAYZyIqTnktX3i6/aQDrQ==",
+ "version": "1.2.1",
+ "resolved": "https://registry.npmjs.org/image-size/-/image-size-1.2.1.tgz",
+ "integrity": "sha512-rH+46sQJ2dlwfjfhCyNx5thzrv+dtmBIhPHk0zgRUukHzZ/kRueTJXoYYsclBaKcSMBWuGbOFXtioLpzTb5euw==",
+ "license": "MIT",
"dependencies": {
"queue": "6.0.2"
},
diff --git a/docs/my-website/release_notes/v1.55.10/index.md b/docs/my-website/release_notes/v1.55.10/index.md
index 7f9839c2b5..2b5ce75cf0 100644
--- a/docs/my-website/release_notes/v1.55.10/index.md
+++ b/docs/my-website/release_notes/v1.55.10/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.55.8-stable/index.md b/docs/my-website/release_notes/v1.55.8-stable/index.md
index 7e82e94747..38c78eb537 100644
--- a/docs/my-website/release_notes/v1.55.8-stable/index.md
+++ b/docs/my-website/release_notes/v1.55.8-stable/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.56.1/index.md b/docs/my-website/release_notes/v1.56.1/index.md
index 7c4ccc74ea..74f3606b90 100644
--- a/docs/my-website/release_notes/v1.56.1/index.md
+++ b/docs/my-website/release_notes/v1.56.1/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.56.3/index.md b/docs/my-website/release_notes/v1.56.3/index.md
index 95205633ea..3d996ba5b8 100644
--- a/docs/my-website/release_notes/v1.56.3/index.md
+++ b/docs/my-website/release_notes/v1.56.3/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.56.4/index.md b/docs/my-website/release_notes/v1.56.4/index.md
index 93f8725632..bf9cc2d94e 100644
--- a/docs/my-website/release_notes/v1.56.4/index.md
+++ b/docs/my-website/release_notes/v1.56.4/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.57.3/index.md b/docs/my-website/release_notes/v1.57.3/index.md
index 3bee71a8e1..ab1154a0a8 100644
--- a/docs/my-website/release_notes/v1.57.3/index.md
+++ b/docs/my-website/release_notes/v1.57.3/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.57.7/index.md b/docs/my-website/release_notes/v1.57.7/index.md
index ce987baf77..4da2402efa 100644
--- a/docs/my-website/release_notes/v1.57.7/index.md
+++ b/docs/my-website/release_notes/v1.57.7/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.57.8-stable/index.md b/docs/my-website/release_notes/v1.57.8-stable/index.md
index d37a7b9ff8..78fe13f2ed 100644
--- a/docs/my-website/release_notes/v1.57.8-stable/index.md
+++ b/docs/my-website/release_notes/v1.57.8-stable/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
@@ -38,7 +38,7 @@ hide_table_of_contents: false
2. OpenAI Moderations - `omni-moderation-latest` support. [Start Here](https://docs.litellm.ai/docs/moderation)
3. Azure O1 - fake streaming support. This ensures if a `stream=true` is passed, the response is streamed. [Start Here](https://docs.litellm.ai/docs/providers/azure)
4. Anthropic - non-whitespace char stop sequence handling - [PR](https://github.com/BerriAI/litellm/pull/7484)
-5. Azure OpenAI - support entrata id username + password based auth. [Start Here](https://docs.litellm.ai/docs/providers/azure#entrata-id---use-tenant_id-client_id-client_secret)
+5. Azure OpenAI - support Entra ID username + password based auth. [Start Here](https://docs.litellm.ai/docs/providers/azure#entra-id---use-tenant_id-client_id-client_secret)
6. LM Studio - embedding route support. [Start Here](https://docs.litellm.ai/docs/providers/lm-studio)
7. WatsonX - ZenAPIKeyAuth support. [Start Here](https://docs.litellm.ai/docs/providers/watsonx)
diff --git a/docs/my-website/release_notes/v1.59.0/index.md b/docs/my-website/release_notes/v1.59.0/index.md
index 5343ba49ad..2699e42020 100644
--- a/docs/my-website/release_notes/v1.59.0/index.md
+++ b/docs/my-website/release_notes/v1.59.0/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.59.8-stable/index.md b/docs/my-website/release_notes/v1.59.8-stable/index.md
index fa9825fb66..023f284ad5 100644
--- a/docs/my-website/release_notes/v1.59.8-stable/index.md
+++ b/docs/my-website/release_notes/v1.59.8-stable/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.61.20-stable/index.md b/docs/my-website/release_notes/v1.61.20-stable/index.md
index 132c1aa318..5012e2aa90 100644
--- a/docs/my-website/release_notes/v1.61.20-stable/index.md
+++ b/docs/my-website/release_notes/v1.61.20-stable/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.63.0/index.md b/docs/my-website/release_notes/v1.63.0/index.md
index e74a2f9b86..ab74b11b4d 100644
--- a/docs/my-website/release_notes/v1.63.0/index.md
+++ b/docs/my-website/release_notes/v1.63.0/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.63.11-stable/index.md b/docs/my-website/release_notes/v1.63.11-stable/index.md
index 55fefb737a..882747a07b 100644
--- a/docs/my-website/release_notes/v1.63.11-stable/index.md
+++ b/docs/my-website/release_notes/v1.63.11-stable/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
@@ -26,13 +26,18 @@ This release is primarily focused on:
- UI - Credential Management, re-use credentials when adding new models
- UI - Test Connection to LLM Provider before adding a model
-:::info
+## Known Issues
+- 🚨 Known issue on Azure OpenAI - We don't recommend upgrading if you use Azure OpenAI. This version failed our Azure OpenAI load test
-This release will be live on 03/16/2025
-:::
+## Docker Run LiteLLM Proxy
-
+```
+docker run
+-e STORE_MODEL_IN_DB=True
+-p 4000:4000
+ghcr.io/berriai/litellm:main-v1.63.11-stable
+```
## Demo Instance
diff --git a/docs/my-website/release_notes/v1.63.14/index.md b/docs/my-website/release_notes/v1.63.14/index.md
new file mode 100644
index 0000000000..ff2630468c
--- /dev/null
+++ b/docs/my-website/release_notes/v1.63.14/index.md
@@ -0,0 +1,131 @@
+---
+title: v1.63.14-stable
+slug: v1.63.14-stable
+date: 2025-03-22T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+tags: [credential management, thinking content, responses api, snowflake]
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+
+These are the changes since `v1.63.11-stable`.
+
+This release brings:
+- LLM Translation Improvements (MCP Support and Bedrock Application Profiles)
+- Perf improvements for Usage-based Routing
+- Streaming guardrail support via websockets
+- Azure OpenAI client perf fix (from previous release)
+
+## Docker Run LiteLLM Proxy
+
+```
+docker run
+-e STORE_MODEL_IN_DB=True
+-p 4000:4000
+ghcr.io/berriai/litellm:main-v1.63.14-stable.patch1
+```
+
+## Demo Instance
+
+Here's a Demo Instance to test changes:
+- Instance: https://demo.litellm.ai/
+- Login Credentials:
+ - Username: admin
+ - Password: sk-1234
+
+
+
+## New Models / Updated Models
+
+- Azure gpt-4o - fixed pricing to latest global pricing - [PR](https://github.com/BerriAI/litellm/pull/9361)
+- O1-Pro - add pricing + model information - [PR](https://github.com/BerriAI/litellm/pull/9397)
+- Azure AI - mistral 3.1 small pricing added - [PR](https://github.com/BerriAI/litellm/pull/9453)
+- Azure - gpt-4.5-preview pricing added - [PR](https://github.com/BerriAI/litellm/pull/9453)
+
+
+
+## LLM Translation
+
+1. **New LLM Features**
+
+- Bedrock: Support bedrock application inference profiles [Docs](https://docs.litellm.ai/docs/providers/bedrock#bedrock-application-inference-profile)
+ - Infer aws region from bedrock application profile id - (`arn:aws:bedrock:us-east-1:...`)
+- Ollama - support calling via `/v1/completions` [Get Started](../../docs/providers/ollama#using-ollama-fim-on-v1completions)
+- Bedrock - support `us.deepseek.r1-v1:0` model name [Docs](../../docs/providers/bedrock#supported-aws-bedrock-models)
+- OpenRouter - `OPENROUTER_API_BASE` env var support [Docs](../../docs/providers/openrouter.md)
+- Azure - add audio model parameter support - [Docs](../../docs/providers/azure#azure-audio-model)
+- OpenAI - PDF File support [Docs](../../docs/completion/document_understanding#openai-file-message-type)
+- OpenAI - o1-pro Responses API streaming support [Docs](../../docs/response_api.md#streaming)
+- [BETA] MCP - Use MCP Tools with LiteLLM SDK [Docs](../../docs/mcp)
+
+2. **Bug Fixes**
+
+- Voyage: prompt token on embedding tracking fix - [PR](https://github.com/BerriAI/litellm/commit/56d3e75b330c3c3862dc6e1c51c1210e48f1068e)
+- Sagemaker - Fix ‘Too little data for declared Content-Length’ error - [PR](https://github.com/BerriAI/litellm/pull/9326)
+- OpenAI-compatible models - fix issue when calling openai-compatible models w/ custom_llm_provider set - [PR](https://github.com/BerriAI/litellm/pull/9355)
+- VertexAI - Embedding ‘outputDimensionality’ support - [PR](https://github.com/BerriAI/litellm/commit/437dbe724620675295f298164a076cbd8019d304)
+- Anthropic - return consistent json response format on streaming/non-streaming - [PR](https://github.com/BerriAI/litellm/pull/9437)
+
+## Spend Tracking Improvements
+
+- `litellm_proxy/` - support reading litellm response cost header from proxy, when using client sdk
+- Reset Budget Job - fix budget reset error on keys/teams/users [PR](https://github.com/BerriAI/litellm/pull/9329)
+- Streaming - Prevents final chunk w/ usage from being ignored (impacted bedrock streaming + cost tracking) [PR](https://github.com/BerriAI/litellm/pull/9314)
+
+
+## UI
+
+1. Users Page
+ - Feature: Control default internal user settings [PR](https://github.com/BerriAI/litellm/pull/9328)
+2. Icons:
+ - Feature: Replace external "artificialanalysis.ai" icons by local svg [PR](https://github.com/BerriAI/litellm/pull/9374)
+3. Sign In/Sign Out
+ - Fix: Default login when `default_user_id` user does not exist in DB [PR](https://github.com/BerriAI/litellm/pull/9395)
+
+
+## Logging Integrations
+
+- Support post-call guardrails for streaming responses [Get Started](../../docs/proxy/guardrails/custom_guardrail#1-write-a-customguardrail-class)
+- Arize [Get Started](../../docs/observability/arize_integration)
+ - fix invalid package import [PR](https://github.com/BerriAI/litellm/pull/9338)
+ - migrate to using standardloggingpayload for metadata, ensures spans land successfully [PR](https://github.com/BerriAI/litellm/pull/9338)
+ - fix logging to just log the LLM I/O [PR](https://github.com/BerriAI/litellm/pull/9353)
+ - Dynamic API Key/Space param support [Get Started](../../docs/observability/arize_integration#pass-arize-spacekey-per-request)
+- StandardLoggingPayload - Log litellm_model_name in payload. Allows knowing what the model sent to API provider was [Get Started](../../docs/proxy/logging_spec#standardlogginghiddenparams)
+- Prompt Management - Allow building custom prompt management integration [Get Started](../../docs/proxy/custom_prompt_management.md)
+
+## Performance / Reliability improvements
+
+- Redis Caching - add 5s default timeout, prevents hanging redis connection from impacting llm calls [PR](https://github.com/BerriAI/litellm/commit/db92956ae33ed4c4e3233d7e1b0c7229817159bf)
+- Allow disabling all spend updates / writes to DB - patch to allow disabling all spend updates to DB with a flag [PR](https://github.com/BerriAI/litellm/pull/9331)
+- Azure OpenAI - correctly re-use azure openai client, fixes perf issue from previous Stable release [PR](https://github.com/BerriAI/litellm/commit/f2026ef907c06d94440930917add71314b901413)
+- Azure OpenAI - uses litellm.ssl_verify on Azure/OpenAI clients [PR](https://github.com/BerriAI/litellm/commit/f2026ef907c06d94440930917add71314b901413)
+- Usage-based routing - Wildcard model support [Get Started](../../docs/proxy/usage_based_routing#wildcard-model-support)
+- Usage-based routing - Support batch writing increments to redis - reduces latency to same as ‘simple-shuffle’ [PR](https://github.com/BerriAI/litellm/pull/9357)
+- Router - show reason for model cooldown on ‘no healthy deployments available error’ [PR](https://github.com/BerriAI/litellm/pull/9438)
+- Caching - add max value limit to an item in in-memory cache (1MB) - prevents OOM errors on large image url’s being sent through proxy [PR](https://github.com/BerriAI/litellm/pull/9448)
+
+
+## General Improvements
+
+- Passthrough Endpoints - support returning api-base on pass-through endpoints Response Headers [Docs](../../docs/proxy/response_headers#litellm-specific-headers)
+- SSL - support reading ssl security level from env var - Allows user to specify lower security settings [Get Started](../../docs/guides/security_settings)
+- Credentials - only poll Credentials table when `STORE_MODEL_IN_DB` is True [PR](https://github.com/BerriAI/litellm/pull/9376)
+- Image URL Handling - new architecture doc on image url handling [Docs](../../docs/proxy/image_handling)
+- OpenAI - bump to pip install "openai==1.68.2" [PR](https://github.com/BerriAI/litellm/commit/e85e3bc52a9de86ad85c3dbb12d87664ee567a5a)
+- Gunicorn - security fix - bump gunicorn==23.0.0 [PR](https://github.com/BerriAI/litellm/commit/7e9fc92f5c7fea1e7294171cd3859d55384166eb)
+
+
+## Complete Git Diff
+
+[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.11-stable...v1.63.14.rc)
\ No newline at end of file
diff --git a/docs/my-website/release_notes/v1.63.2-stable/index.md b/docs/my-website/release_notes/v1.63.2-stable/index.md
index 0c359452dc..3d47e02ac1 100644
--- a/docs/my-website/release_notes/v1.63.2-stable/index.md
+++ b/docs/my-website/release_notes/v1.63.2-stable/index.md
@@ -6,7 +6,7 @@ authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
- image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
diff --git a/docs/my-website/release_notes/v1.65.0-stable/index.md b/docs/my-website/release_notes/v1.65.0-stable/index.md
new file mode 100644
index 0000000000..3696f5023c
--- /dev/null
+++ b/docs/my-website/release_notes/v1.65.0-stable/index.md
@@ -0,0 +1,160 @@
+---
+title: v1.65.0-stable - Model Context Protocol
+slug: v1.65.0-stable
+date: 2025-03-30T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+tags: [mcp, custom_prompt_management]
+hide_table_of_contents: false
+---
+import Image from '@theme/IdealImage';
+
+v1.65.0-stable is live now. Here are the key highlights of this release:
+- **MCP Support**: Support for adding and using MCP servers on the LiteLLM proxy.
+- **UI view total usage after 1M+ logs**: You can now view usage analytics after crossing 1M+ logs in DB.
+
+## Model Context Protocol (MCP)
+
+This release introduces support for centrally adding MCP servers on LiteLLM. This allows you to add MCP server endpoints and your developers can `list` and `call` MCP tools through LiteLLM.
+
+Read more about MCP [here](https://docs.litellm.ai/docs/mcp).
+
+
+
+ Expose and use MCP servers through LiteLLM
+
+
+## UI view total usage after 1M+ logs
+
+This release brings the ability to view total usage analytics even after exceeding 1M+ logs in your database. We've implemented a scalable architecture that stores only aggregate usage data, resulting in significantly more efficient queries and reduced database CPU utilization.
+
+
+
+
+ View total usage after 1M+ logs
+
+
+
+- How this works:
+ - We now aggregate usage data into a dedicated DailyUserSpend table, significantly reducing query load and CPU usage even beyond 1M+ logs.
+
+- Daily Spend Breakdown API:
+
+ - Retrieve granular daily usage data (by model, provider, and API key) with a single endpoint.
+ Example Request:
+
+ ```shell title="Daily Spend Breakdown API" showLineNumbers
+ curl -L -X GET 'http://localhost:4000/user/daily/activity?start_date=2025-03-20&end_date=2025-03-27' \
+ -H 'Authorization: Bearer sk-...'
+ ```
+
+ ```json title="Daily Spend Breakdown API Response" showLineNumbers
+ {
+ "results": [
+ {
+ "date": "2025-03-27",
+ "metrics": {
+ "spend": 0.0177072,
+ "prompt_tokens": 111,
+ "completion_tokens": 1711,
+ "total_tokens": 1822,
+ "api_requests": 11
+ },
+ "breakdown": {
+ "models": {
+ "gpt-4o-mini": {
+ "spend": 1.095e-05,
+ "prompt_tokens": 37,
+ "completion_tokens": 9,
+ "total_tokens": 46,
+ "api_requests": 1
+ },
+ "providers": { "openai": { ... }, "azure_ai": { ... } },
+ "api_keys": { "3126b6eaf1...": { ... } }
+ }
+ }
+ ],
+ "metadata": {
+ "total_spend": 0.7274667,
+ "total_prompt_tokens": 280990,
+ "total_completion_tokens": 376674,
+ "total_api_requests": 14
+ }
+ }
+ ```
+
+
+
+
+## New Models / Updated Models
+- Support for Vertex AI gemini-2.0-flash-lite & Google AI Studio gemini-2.0-flash-lite [PR](https://github.com/BerriAI/litellm/pull/9523)
+- Support for Vertex AI Fine-Tuned LLMs [PR](https://github.com/BerriAI/litellm/pull/9542)
+- Nova Canvas image generation support [PR](https://github.com/BerriAI/litellm/pull/9525)
+- OpenAI gpt-4o-transcribe support [PR](https://github.com/BerriAI/litellm/pull/9517)
+- Added new Vertex AI text embedding model [PR](https://github.com/BerriAI/litellm/pull/9476)
+
+## LLM Translation
+- OpenAI Web Search Tool Call Support [PR](https://github.com/BerriAI/litellm/pull/9465)
+- Vertex AI topLogprobs support [PR](https://github.com/BerriAI/litellm/pull/9518)
+- Support for sending images and video to Vertex AI multimodal embedding [Doc](https://docs.litellm.ai/docs/providers/vertex#multi-modal-embeddings)
+- Support litellm.api_base for Vertex AI + Gemini across completion, embedding, image_generation [PR](https://github.com/BerriAI/litellm/pull/9516)
+- Bug fix for returning `response_cost` when using litellm python SDK with LiteLLM Proxy [PR](https://github.com/BerriAI/litellm/commit/6fd18651d129d606182ff4b980e95768fc43ca3d)
+- Support for `max_completion_tokens` on Mistral API [PR](https://github.com/BerriAI/litellm/pull/9606)
+- Refactored Vertex AI passthrough routes - fixes unpredictable behaviour with auto-setting default_vertex_region on router model add [PR](https://github.com/BerriAI/litellm/pull/9467)
+
+## Spend Tracking Improvements
+- Log 'api_base' on spend logs [PR](https://github.com/BerriAI/litellm/pull/9509)
+- Support for Gemini audio token cost tracking [PR](https://github.com/BerriAI/litellm/pull/9535)
+- Fixed OpenAI audio input token cost tracking [PR](https://github.com/BerriAI/litellm/pull/9535)
+
+## UI
+
+### Model Management
+- Allowed team admins to add/update/delete models on UI [PR](https://github.com/BerriAI/litellm/pull/9572)
+- Added render supports_web_search on model hub [PR](https://github.com/BerriAI/litellm/pull/9469)
+
+### Request Logs
+- Show API base and model ID on request logs [PR](https://github.com/BerriAI/litellm/pull/9572)
+- Allow viewing keyinfo on request logs [PR](https://github.com/BerriAI/litellm/pull/9568)
+
+### Usage Tab
+- Added Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows [PR](https://github.com/BerriAI/litellm/pull/9538)
+- Connected UI to "LiteLLM_DailyUserSpend" spend table [PR](https://github.com/BerriAI/litellm/pull/9603)
+
+## Logging Integrations
+- Fixed StandardLoggingPayload for GCS Pub Sub Logging Integration [PR](https://github.com/BerriAI/litellm/pull/9508)
+- Track `litellm_model_name` on `StandardLoggingPayload` [Docs](https://docs.litellm.ai/docs/proxy/logging_spec#standardlogginghiddenparams)
+
+## Performance / Reliability Improvements
+- LiteLLM Redis semantic caching implementation [PR](https://github.com/BerriAI/litellm/pull/9356)
+- Gracefully handle exceptions when DB is having an outage [PR](https://github.com/BerriAI/litellm/pull/9533)
+- Allow Pods to startup + passing /health/readiness when allow_requests_on_db_unavailable: True and DB is down [PR](https://github.com/BerriAI/litellm/pull/9569)
+
+
+## General Improvements
+- Support for exposing MCP tools on litellm proxy [PR](https://github.com/BerriAI/litellm/pull/9426)
+- Support discovering Gemini, Anthropic, xAI models by calling their /v1/model endpoint [PR](https://github.com/BerriAI/litellm/pull/9530)
+- Fixed route check for non-proxy admins on JWT auth [PR](https://github.com/BerriAI/litellm/pull/9454)
+- Added baseline Prisma database migrations [PR](https://github.com/BerriAI/litellm/pull/9565)
+- View all wildcard models on /model/info [PR](https://github.com/BerriAI/litellm/pull/9572)
+
+
+## Security
+- Bumped next from 14.2.21 to 14.2.25 in UI dashboard [PR](https://github.com/BerriAI/litellm/pull/9458)
+
+## Complete Git Diff
+
+[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.14-stable.patch1...v1.65.0-stable)
diff --git a/docs/my-website/release_notes/v1.65.0/index.md b/docs/my-website/release_notes/v1.65.0/index.md
new file mode 100644
index 0000000000..84276c997d
--- /dev/null
+++ b/docs/my-website/release_notes/v1.65.0/index.md
@@ -0,0 +1,34 @@
+---
+title: v1.65.0 - Team Model Add - update
+slug: v1.65.0
+date: 2025-03-28T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+tags: [management endpoints, team models, ui]
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+
+v1.65.0 updates the `/model/new` endpoint to prevent non-team admins from creating team models.
+
+This means that only proxy admins or team admins can create team models.
+
+## Additional Changes
+
+- Allows team admins to call `/model/update` to update team models.
+- Allows team admins to call `/model/delete` to delete team models.
+- Introduces new `user_models_only` param to `/v2/model/info` - only return models added by this user.
+
+
+These changes enable team admins to add and manage models for their team on the LiteLLM UI + API.
+
+
+
\ No newline at end of file
diff --git a/docs/my-website/release_notes/v1.65.4-stable/index.md b/docs/my-website/release_notes/v1.65.4-stable/index.md
new file mode 100644
index 0000000000..872024a47a
--- /dev/null
+++ b/docs/my-website/release_notes/v1.65.4-stable/index.md
@@ -0,0 +1,176 @@
+---
+title: v1.65.4-stable
+slug: v1.65.4-stable
+date: 2025-04-05T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+tags: []
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run
+-e STORE_MODEL_IN_DB=True
+-p 4000:4000
+ghcr.io/berriai/litellm:main-v1.65.4-stable
+```
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.65.4.post1
+```
+
+
+
+v1.65.4-stable is live. Here are the improvements since v1.65.0-stable.
+
+## Key Highlights
+- **Preventing DB Deadlocks**: Fixes a high-traffic issue when multiple instances were writing to the DB at the same time.
+- **New Usage Tab**: Enables viewing spend by model and customizing date range
+
+Let's dive in.
+
+### Preventing DB Deadlocks
+
+
+
+This release fixes the DB deadlocking issue that users faced in high traffic (10K+ RPS). This is great because it enables user/key/team spend tracking works at that scale.
+
+Read more about the new architecture [here](https://docs.litellm.ai/docs/proxy/db_deadlocks)
+
+
+### New Usage Tab
+
+
+
+The new Usage tab now brings the ability to track daily spend by model. This makes it easier to catch any spend tracking or token counting errors, when combined with the ability to view successful requests, and token usage.
+
+To test this out, just go to Experimental > New Usage > Activity.
+
+
+## New Models / Updated Models
+
+1. Databricks - claude-3-7-sonnet cost tracking [PR](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/model_prices_and_context_window.json#L10350)
+2. VertexAI - `gemini-2.5-pro-exp-03-25` cost tracking [PR](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/model_prices_and_context_window.json#L4492)
+3. VertexAI - `gemini-2.0-flash` cost tracking [PR](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/model_prices_and_context_window.json#L4689)
+4. Groq - add whisper ASR models to model cost map [PR](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/model_prices_and_context_window.json#L3324)
+5. IBM - Add watsonx/ibm/granite-3-8b-instruct to model cost map [PR](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/model_prices_and_context_window.json#L91)
+6. Google AI Studio - add gemini/gemini-2.5-pro-preview-03-25 to model cost map [PR](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/model_prices_and_context_window.json#L4850)
+
+## LLM Translation
+1. Vertex AI - Support anyOf param for OpenAI json schema translation [Get Started](https://docs.litellm.ai/docs/providers/vertex#json-schema)
+2. Anthropic- response_format + thinking param support (works across Anthropic API, Bedrock, Vertex) [Get Started](https://docs.litellm.ai/docs/reasoning_content)
+3. Anthropic - if thinking token is specified and max tokens is not - ensure max token to anthropic is higher than thinking tokens (works across Anthropic API, Bedrock, Vertex) [PR](https://github.com/BerriAI/litellm/pull/9594)
+4. Bedrock - latency optimized inference support [Get Started](https://docs.litellm.ai/docs/providers/bedrock#usage---latency-optimized-inference)
+5. Sagemaker - handle special tokens + multibyte character code in response [Get Started](https://docs.litellm.ai/docs/providers/aws_sagemaker)
+6. MCP - add support for using SSE MCP servers [Get Started](https://docs.litellm.ai/docs/mcp#usage)
+8. Anthropic - new `litellm.messages.create` interface for calling Anthropic `/v1/messages` via passthrough [Get Started](https://docs.litellm.ai/docs/anthropic_unified#usage)
+11. Anthropic - support ‘file’ content type in message param (works across Anthropic API, Bedrock, Vertex) [Get Started](https://docs.litellm.ai/docs/providers/anthropic#usage---pdf)
+12. Anthropic - map openai 'reasoning_effort' to anthropic 'thinking' param (works across Anthropic API, Bedrock, Vertex) [Get Started](https://docs.litellm.ai/docs/providers/anthropic#usage---thinking--reasoning_content)
+13. Google AI Studio (Gemini) - [BETA] `/v1/files` upload support [Get Started](../../docs/providers/google_ai_studio/files)
+14. Azure - fix o-series tool calling [Get Started](../../docs/providers/azure#tool-calling--function-calling)
+15. Unified file id - [ALPHA] allow calling multiple providers with same file id [PR](https://github.com/BerriAI/litellm/pull/9718)
+ - This is experimental, and not recommended for production use.
+ - We plan to have a production-ready implementation by next week.
+16. Google AI Studio (Gemini) - return logprobs [PR](https://github.com/BerriAI/litellm/pull/9713)
+17. Anthropic - Support prompt caching for Anthropic tool calls [Get Started](https://docs.litellm.ai/docs/completion/prompt_caching)
+18. OpenRouter - unwrap extra body on open router calls [PR](https://github.com/BerriAI/litellm/pull/9747)
+19. VertexAI - fix credential caching issue [PR](https://github.com/BerriAI/litellm/pull/9756)
+20. XAI - filter out 'name' param for XAI [PR](https://github.com/BerriAI/litellm/pull/9761)
+21. Gemini - image generation output support [Get Started](../../docs/providers/gemini#image-generation)
+22. Databricks - support claude-3-7-sonnet w/ thinking + response_format [Get Started](../../docs/providers/databricks#usage---thinking--reasoning_content)
+
+## Spend Tracking Improvements
+1. Reliability fix - Check sent and received model for cost calculation [PR](https://github.com/BerriAI/litellm/pull/9669)
+2. Vertex AI - Multimodal embedding cost tracking [Get Started](https://docs.litellm.ai/docs/providers/vertex#multi-modal-embeddings), [PR](https://github.com/BerriAI/litellm/pull/9623)
+
+## Management Endpoints / UI
+
+
+
+1. New Usage Tab
+ - Report 'total_tokens' + report success/failure calls
+ - Remove double bars on scroll
+ - Ensure ‘daily spend’ chart ordered from earliest to latest date
+ - showing spend per model per day
+ - show key alias on usage tab
+ - Allow non-admins to view their activity
+ - Add date picker to new usage tab
+2. Virtual Keys Tab
+ - remove 'default key' on user signup
+ - fix showing user models available for personal key creation
+3. Test Key Tab
+ - Allow testing image generation models
+4. Models Tab
+ - Fix bulk adding models
+ - support reusable credentials for passthrough endpoints
+ - Allow team members to see team models
+5. Teams Tab
+ - Fix json serialization error on update team metadata
+6. Request Logs Tab
+ - Add reasoning_content token tracking across all providers on streaming
+7. API
+ - return key alias on /user/daily/activity [Get Started](../../docs/proxy/cost_tracking#daily-spend-breakdown-api)
+8. SSO
+ - Allow assigning SSO users to teams on MSFT SSO [PR](https://github.com/BerriAI/litellm/pull/9745)
+
+## Logging / Guardrail Integrations
+
+1. Console Logs - Add json formatting for uncaught exceptions [PR](https://github.com/BerriAI/litellm/pull/9619)
+2. Guardrails - AIM Guardrails support for virtual key based policies [Get Started](../../docs/proxy/guardrails/aim_security)
+3. Logging - fix completion start time tracking [PR](https://github.com/BerriAI/litellm/pull/9688)
+4. Prometheus
+ - Allow adding authentication on Prometheus /metrics endpoints [PR](https://github.com/BerriAI/litellm/pull/9766)
+ - Distinguish LLM Provider Exception vs. LiteLLM Exception in metric naming [PR](https://github.com/BerriAI/litellm/pull/9760)
+ - Emit operational metrics for new DB Transaction architecture [PR](https://github.com/BerriAI/litellm/pull/9719)
+
+## Performance / Loadbalancing / Reliability improvements
+1. Preventing Deadlocks
+ - Reduce DB Deadlocks by storing spend updates in Redis and then committing to DB [PR](https://github.com/BerriAI/litellm/pull/9608)
+ - Ensure no deadlocks occur when updating DailyUserSpendTransaction [PR](https://github.com/BerriAI/litellm/pull/9690)
+ - High Traffic fix - ensure new DB + Redis architecture accurately tracks spend [PR](https://github.com/BerriAI/litellm/pull/9673)
+ - Use Redis for PodLock Manager instead of PG (ensures no deadlocks occur) [PR](https://github.com/BerriAI/litellm/pull/9715)
+ - v2 DB Deadlock Reduction Architecture – Add Max Size for In-Memory Queue + Backpressure Mechanism [PR](https://github.com/BerriAI/litellm/pull/9759)
+
+2. Prisma Migrations [Get Started](../../docs/proxy/prod#9-use-prisma-migrate-deploy)
+ - connects litellm proxy to litellm's prisma migration files
+ - Handle db schema updates from new `litellm-proxy-extras` sdk
+3. Redis - support password for sync sentinel clients [PR](https://github.com/BerriAI/litellm/pull/9622)
+4. Fix "Circular reference detected" error when max_parallel_requests = 0 [PR](https://github.com/BerriAI/litellm/pull/9671)
+5. Code QA - Ban hardcoded numbers [PR](https://github.com/BerriAI/litellm/pull/9709)
+
+## Helm
+1. fix: wrong indentation of ttlSecondsAfterFinished in chart [PR](https://github.com/BerriAI/litellm/pull/9611)
+
+## General Proxy Improvements
+1. Fix - only apply service_account_settings.enforced_params on service accounts [PR](https://github.com/BerriAI/litellm/pull/9683)
+2. Fix - handle metadata null on `/chat/completion` [PR](https://github.com/BerriAI/litellm/issues/9717)
+3. Fix - Move daily user transaction logging outside of 'disable_spend_logs' flag, as they’re unrelated [PR](https://github.com/BerriAI/litellm/pull/9772)
+
+## Demo
+
+Try this on the demo instance [today](https://docs.litellm.ai/docs/proxy/demo)
+
+## Complete Git Diff
+
+See the complete git diff since v1.65.0-stable, [here](https://github.com/BerriAI/litellm/releases/tag/v1.65.4-stable)
+
diff --git a/docs/my-website/release_notes/v1.66.0-stable/index.md b/docs/my-website/release_notes/v1.66.0-stable/index.md
new file mode 100644
index 0000000000..939322e031
--- /dev/null
+++ b/docs/my-website/release_notes/v1.66.0-stable/index.md
@@ -0,0 +1,197 @@
+---
+title: v1.66.0-stable - Realtime API Cost Tracking
+slug: v1.66.0-stable
+date: 2025-04-12T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+tags: ["sso", "unified_file_id", "cost_tracking", "security"]
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run
+-e STORE_MODEL_IN_DB=True
+-p 4000:4000
+ghcr.io/berriai/litellm:main-v1.66.0-stable
+```
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.66.0.post1
+```
+
+
+
+v1.66.0-stable is live now, here are the key highlights of this release
+
+## Key Highlights
+- **Realtime API Cost Tracking**: Track cost of realtime API calls
+- **Microsoft SSO Auto-sync**: Auto-sync groups and group members from Azure Entra ID to LiteLLM
+- **xAI grok-3**: Added support for `xai/grok-3` models
+- **Security Fixes**: Fixed [CVE-2025-0330](https://www.cve.org/CVERecord?id=CVE-2025-0330) and [CVE-2024-6825](https://www.cve.org/CVERecord?id=CVE-2024-6825) vulnerabilities
+
+Let's dive in.
+
+## Realtime API Cost Tracking
+
+
+
+
+This release adds Realtime API logging + cost tracking.
+- **Logging**: LiteLLM now logs the complete response from realtime calls to all logging integrations (DB, S3, Langfuse, etc.)
+- **Cost Tracking**: You can now set 'base_model' and custom pricing for realtime models. [Custom Pricing](../../docs/proxy/custom_pricing)
+- **Budgets**: Your key/user/team budgets now work for realtime models as well.
+
+Start [here](https://docs.litellm.ai/docs/realtime)
+
+
+
+## Microsoft SSO Auto-sync
+
+
+
+ Auto-sync groups and members from Azure Entra ID to LiteLLM
+
+
+This release adds support for auto-syncing groups and members on Microsoft Entra ID with LiteLLM. This means that LiteLLM proxy administrators can spend less time managing teams and members and LiteLLM handles the following:
+
+- Auto-create teams that exist on Microsoft Entra ID
+- Sync team members on Microsoft Entra ID with LiteLLM teams
+
+Get started with this [here](https://docs.litellm.ai/docs/tutorials/msft_sso)
+
+
+## New Models / Updated Models
+
+- **xAI**
+ 1. Added reasoning_effort support for `xai/grok-3-mini-beta` [Get Started](https://docs.litellm.ai/docs/providers/xai#reasoning-usage)
+ 2. Added cost tracking for `xai/grok-3` models [PR](https://github.com/BerriAI/litellm/pull/9920)
+
+- **Hugging Face**
+ 1. Added inference providers support [Get Started](https://docs.litellm.ai/docs/providers/huggingface#serverless-inference-providers)
+
+- **Azure**
+ 1. Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893)
+
+- **VertexAI**
+ 1. Added enterpriseWebSearch tool support [Get Started](https://docs.litellm.ai/docs/providers/vertex#grounding---web-search)
+ 2. Moved to only passing keys accepted by the Vertex AI response schema [PR](https://github.com/BerriAI/litellm/pull/8992)
+
+- **Google AI Studio**
+ 1. Added cost tracking for `gemini-2.5-pro` [PR](https://github.com/BerriAI/litellm/pull/9837)
+ 2. Fixed pricing for 'gemini/gemini-2.5-pro-preview-03-25' [PR](https://github.com/BerriAI/litellm/pull/9896)
+ 3. Fixed handling file_data being passed in [PR](https://github.com/BerriAI/litellm/pull/9786)
+
+- **Azure**
+ 1. Updated Azure Phi-4 pricing [PR](https://github.com/BerriAI/litellm/pull/9862)
+ 2. Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893)
+
+- **Databricks**
+ 1. Removed reasoning_effort from parameters [PR](https://github.com/BerriAI/litellm/pull/9811)
+ 2. Fixed custom endpoint check for Databricks [PR](https://github.com/BerriAI/litellm/pull/9925)
+
+- **General**
+ 1. Added litellm.supports_reasoning() util to track if an llm supports reasoning [Get Started](https://docs.litellm.ai/docs/providers/anthropic#reasoning)
+ 2. Function Calling - Handle pydantic base model in message tool calls, handle tools = [], and support fake streaming on tool calls for meta.llama3-3-70b-instruct-v1:0 [PR](https://github.com/BerriAI/litellm/pull/9774)
+ 3. LiteLLM Proxy - Allow passing `thinking` param to litellm proxy via client sdk [PR](https://github.com/BerriAI/litellm/pull/9386)
+ 4. Fixed correctly translating 'thinking' param for litellm [PR](https://github.com/BerriAI/litellm/pull/9904)
+
+
+## Spend Tracking Improvements
+- **OpenAI, Azure**
+ 1. Realtime API Cost tracking with token usage metrics in spend logs [Get Started](https://docs.litellm.ai/docs/realtime)
+- **Anthropic**
+ 1. Fixed Claude Haiku cache read pricing per token [PR](https://github.com/BerriAI/litellm/pull/9834)
+ 2. Added cost tracking for Claude responses with base_model [PR](https://github.com/BerriAI/litellm/pull/9897)
+ 3. Fixed Anthropic prompt caching cost calculation and trimmed logged message in db [PR](https://github.com/BerriAI/litellm/pull/9838)
+- **General**
+ 1. Added token tracking and log usage object in spend logs [PR](https://github.com/BerriAI/litellm/pull/9843)
+ 2. Handle custom pricing at deployment level [PR](https://github.com/BerriAI/litellm/pull/9855)
+
+
+## Management Endpoints / UI
+
+- **Test Key Tab**
+ 1. Added rendering of Reasoning content, ttft, usage metrics on test key page [PR](https://github.com/BerriAI/litellm/pull/9931)
+
+
+
+ View input, output, reasoning tokens, ttft metrics.
+
+- **Tag / Policy Management**
+ 1. Added Tag/Policy Management. Create routing rules based on request metadata. This allows you to enforce that requests with `tags="private"` only go to specific models. [Get Started](https://docs.litellm.ai/docs/tutorials/tag_management)
+
+
+
+
+
+ Create and manage tags.
+
+- **Redesigned Login Screen**
+ 1. Polished login screen [PR](https://github.com/BerriAI/litellm/pull/9778)
+- **Microsoft SSO Auto-Sync**
+ 1. Added debug route to allow admins to debug SSO JWT fields [PR](https://github.com/BerriAI/litellm/pull/9835)
+ 2. Added ability to use MSFT Graph API to assign users to teams [PR](https://github.com/BerriAI/litellm/pull/9865)
+ 3. Connected litellm to Azure Entra ID Enterprise Application [PR](https://github.com/BerriAI/litellm/pull/9872)
+ 4. Added ability for admins to set `default_team_params` for when litellm SSO creates default teams [PR](https://github.com/BerriAI/litellm/pull/9895)
+ 5. Fixed MSFT SSO to use correct field for user email [PR](https://github.com/BerriAI/litellm/pull/9886)
+ 6. Added UI support for setting Default Team setting when litellm SSO auto creates teams [PR](https://github.com/BerriAI/litellm/pull/9918)
+- **UI Bug Fixes**
+ 1. Prevented team, key, org, model numerical values changing on scrolling [PR](https://github.com/BerriAI/litellm/pull/9776)
+ 2. Instantly reflect key and team updates in UI [PR](https://github.com/BerriAI/litellm/pull/9825)
+
+## Logging / Guardrail Improvements
+
+- **Prometheus**
+ 1. Emit Key and Team Budget metrics on a cron job schedule [Get Started](https://docs.litellm.ai/docs/proxy/prometheus#initialize-budget-metrics-on-startup)
+
+## Security Fixes
+
+- Fixed [CVE-2025-0330](https://www.cve.org/CVERecord?id=CVE-2025-0330) - Leakage of Langfuse API keys in team exception handling [PR](https://github.com/BerriAI/litellm/pull/9830)
+- Fixed [CVE-2024-6825](https://www.cve.org/CVERecord?id=CVE-2024-6825) - Remote code execution in post call rules [PR](https://github.com/BerriAI/litellm/pull/9826)
+
+## Helm
+
+- Added service annotations to litellm-helm chart [PR](https://github.com/BerriAI/litellm/pull/9840)
+- Added extraEnvVars to the helm deployment [PR](https://github.com/BerriAI/litellm/pull/9292)
+
+## Demo
+
+Try this on the demo instance [today](https://docs.litellm.ai/docs/proxy/demo)
+
+## Complete Git Diff
+
+See the complete git diff since v1.65.4-stable, [here](https://github.com/BerriAI/litellm/releases/tag/v1.66.0-stable)
+
+
diff --git a/docs/my-website/release_notes/v1.67.0-stable/index.md b/docs/my-website/release_notes/v1.67.0-stable/index.md
new file mode 100644
index 0000000000..cb7938fce5
--- /dev/null
+++ b/docs/my-website/release_notes/v1.67.0-stable/index.md
@@ -0,0 +1,153 @@
+---
+title: v1.67.0-stable - SCIM Integration
+slug: v1.67.0-stable
+date: 2025-04-19T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+tags: ["sso", "unified_file_id", "cost_tracking", "security"]
+hide_table_of_contents: false
+---
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Key Highlights
+
+- **SCIM Integration**: Enables identity providers (Okta, Azure AD, OneLogin, etc.) to automate user and team (group) provisioning, updates, and deprovisioning
+- **Team and Tag based usage tracking**: You can now see usage and spend by team and tag at 1M+ spend logs.
+- **Unified Responses API**: Support for calling Anthropic, Gemini, Groq, etc. via OpenAI's new Responses API.
+
+Let's dive in.
+
+## SCIM Integration
+
+
+
+This release adds SCIM support to LiteLLM. This allows your SSO provider (Okta, Azure AD, etc) to automatically create/delete users, teams, and memberships on LiteLLM. This means that when you remove a team on your SSO provider, your SSO provider will automatically delete the corresponding team on LiteLLM.
+
+[Read more](../../docs/tutorials/scim_litellm)
+## Team and Tag based usage tracking
+
+
+
+
+This release improves team and tag based usage tracking at 1m+ spend logs, making it easy to monitor your LLM API Spend in production. This covers:
+
+- View **daily spend** by teams + tags
+- View **usage / spend by key**, within teams
+- View **spend by multiple tags**
+- Allow **internal users** to view spend of teams they're a member of
+
+[Read more](#management-endpoints--ui)
+
+## Unified Responses API
+
+This release allows you to call Azure OpenAI, Anthropic, AWS Bedrock, and Google Vertex AI models via the POST /v1/responses endpoint on LiteLLM. This means you can now use popular tools like [OpenAI Codex](https://docs.litellm.ai/docs/tutorials/openai_codex) with your own models.
+
+
+
+
+[Read more](https://docs.litellm.ai/docs/response_api)
+
+
+## New Models / Updated Models
+
+- **OpenAI**
+ 1. gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, o3, o3-mini, o4-mini pricing - [Get Started](../../docs/providers/openai#usage), [PR](https://github.com/BerriAI/litellm/pull/9990)
+ 2. o4 - correctly map o4 to openai o_series model
+- **Azure AI**
+ 1. Phi-4 output cost per token fix - [PR](https://github.com/BerriAI/litellm/pull/9880)
+ 2. Responses API support [Get Started](../../docs/providers/azure#azure-responses-api),[PR](https://github.com/BerriAI/litellm/pull/10116)
+- **Anthropic**
+ 1. redacted message thinking support - [Get Started](../../docs/providers/anthropic#usage---thinking--reasoning_content),[PR](https://github.com/BerriAI/litellm/pull/10129)
+- **Cohere**
+ 1. `/v2/chat` Passthrough endpoint support w/ cost tracking - [Get Started](../../docs/pass_through/cohere), [PR](https://github.com/BerriAI/litellm/pull/9997)
+- **Azure**
+ 1. Support azure tenant_id/client_id env vars - [Get Started](../../docs/providers/azure#entra-id---use-tenant_id-client_id-client_secret), [PR](https://github.com/BerriAI/litellm/pull/9993)
+ 2. Fix response_format check for 2025+ api versions - [PR](https://github.com/BerriAI/litellm/pull/9993)
+ 3. Add gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, o3, o3-mini, o4-mini pricing
+- **VLLM**
+ 1. Files - Support 'file' message type for VLLM video url's - [Get Started](../../docs/providers/vllm#send-video-url-to-vllm), [PR](https://github.com/BerriAI/litellm/pull/10129)
+ 2. Passthrough - new `/vllm/` passthrough endpoint support [Get Started](../../docs/pass_through/vllm), [PR](https://github.com/BerriAI/litellm/pull/10002)
+- **Mistral**
+ 1. new `/mistral` passthrough endpoint support [Get Started](../../docs/pass_through/mistral), [PR](https://github.com/BerriAI/litellm/pull/10002)
+- **AWS**
+ 1. New mapped bedrock regions - [PR](https://github.com/BerriAI/litellm/pull/9430)
+- **VertexAI / Google AI Studio**
+ 1. Gemini - Response format - Retain schema field ordering for google gemini and vertex by specifying propertyOrdering - [Get Started](../../docs/providers/vertex#json-schema), [PR](https://github.com/BerriAI/litellm/pull/9828)
+ 2. Gemini-2.5-flash - return reasoning content [Google AI Studio](../../docs/providers/gemini#usage---thinking--reasoning_content), [Vertex AI](../../docs/providers/vertex#thinking--reasoning_content)
+ 3. Gemini-2.5-flash - pricing + model information [PR](https://github.com/BerriAI/litellm/pull/10125)
+ 4. Passthrough - new `/vertex_ai/discovery` route - enables calling AgentBuilder API routes [Get Started](../../docs/pass_through/vertex_ai#supported-api-endpoints), [PR](https://github.com/BerriAI/litellm/pull/10084)
+- **Fireworks AI**
+ 1. return tool calling responses in `tool_calls` field (fireworks incorrectly returns this as a json str in content) [PR](https://github.com/BerriAI/litellm/pull/10130)
+- **Triton**
+ 1. Remove fixed remove bad_words / stop words from `/generate` call - [Get Started](../../docs/providers/triton-inference-server#triton-generate---chat-completion), [PR](https://github.com/BerriAI/litellm/pull/10163)
+- **Other**
+ 1. Support for all litellm providers on Responses API (works with Codex) - [Get Started](../../docs/tutorials/openai_codex), [PR](https://github.com/BerriAI/litellm/pull/10132)
+ 2. Fix combining multiple tool calls in streaming response - [Get Started](../../docs/completion/stream#helper-function), [PR](https://github.com/BerriAI/litellm/pull/10040)
+
+
+## Spend Tracking Improvements
+
+- **Cost Control** - inject cache control points in prompt for cost reduction [Get Started](../../docs/tutorials/prompt_caching), [PR](https://github.com/BerriAI/litellm/pull/10000)
+- **Spend Tags** - spend tags in headers - support x-litellm-tags even if tag based routing not enabled [Get Started](../../docs/proxy/request_headers#litellm-headers), [PR](https://github.com/BerriAI/litellm/pull/10000)
+- **Gemini-2.5-flash** - support cost calculation for reasoning tokens [PR](https://github.com/BerriAI/litellm/pull/10141)
+
+## Management Endpoints / UI
+- **Users**
+ 1. Show created_at and updated_at on users page - [PR](https://github.com/BerriAI/litellm/pull/10033)
+- **Virtual Keys**
+ 1. Filter by key alias - https://github.com/BerriAI/litellm/pull/10085
+- **Usage Tab**
+
+ 1. Team based usage
+
+ - New `LiteLLM_DailyTeamSpend` Table for aggregate team based usage logging - [PR](https://github.com/BerriAI/litellm/pull/10039)
+
+ - New Team based usage dashboard + new `/team/daily/activity` API - [PR](https://github.com/BerriAI/litellm/pull/10081)
+ - Return team alias on /team/daily/activity API - [PR](https://github.com/BerriAI/litellm/pull/10157)
+ - allow internal user view spend for teams they belong to - [PR](https://github.com/BerriAI/litellm/pull/10157)
+ - allow viewing top keys by team - [PR](https://github.com/BerriAI/litellm/pull/10157)
+
+
+
+ 2. Tag Based Usage
+ - New `LiteLLM_DailyTagSpend` Table for aggregate tag based usage logging - [PR](https://github.com/BerriAI/litellm/pull/10071)
+ - Restrict to only Proxy Admins - [PR](https://github.com/BerriAI/litellm/pull/10157)
+ - allow viewing top keys by tag
+ - Return tags passed in request (i.e. dynamic tags) on `/tag/list` API - [PR](https://github.com/BerriAI/litellm/pull/10157)
+
+ 3. Track prompt caching metrics in daily user, team, tag tables - [PR](https://github.com/BerriAI/litellm/pull/10029)
+ 4. Show usage by key (on all up, team, and tag usage dashboards) - [PR](https://github.com/BerriAI/litellm/pull/10157)
+ 5. swap old usage with new usage tab
+- **Models**
+ 1. Make columns resizable/hideable - [PR](https://github.com/BerriAI/litellm/pull/10119)
+- **API Playground**
+ 1. Allow internal user to call api playground - [PR](https://github.com/BerriAI/litellm/pull/10157)
+- **SCIM**
+ 1. Add LiteLLM SCIM Integration for Team and User management - [Get Started](../../docs/tutorials/scim_litellm), [PR](https://github.com/BerriAI/litellm/pull/10072)
+
+
+## Logging / Guardrail Integrations
+- **GCS**
+ 1. Fix gcs pub sub logging with env var GCS_PROJECT_ID - [Get Started](../../docs/observability/gcs_bucket_integration#usage), [PR](https://github.com/BerriAI/litellm/pull/10042)
+- **AIM**
+ 1. Add litellm call id passing to Aim guardrails on pre and post-hooks calls - [Get Started](../../docs/proxy/guardrails/aim_security), [PR](https://github.com/BerriAI/litellm/pull/10021)
+- **Azure blob storage**
+ 1. Ensure logging works in high throughput scenarios - [Get Started](../../docs/proxy/logging#azure-blob-storage), [PR](https://github.com/BerriAI/litellm/pull/9962)
+
+## General Proxy Improvements
+
+- **Support setting `litellm.modify_params` via env var** [PR](https://github.com/BerriAI/litellm/pull/9964)
+- **Model Discovery** - Check provider’s `/models` endpoints when calling proxy’s `/v1/models` endpoint - [Get Started](../../docs/proxy/model_discovery), [PR](https://github.com/BerriAI/litellm/pull/9958)
+- **`/utils/token_counter`** - fix retrieving custom tokenizer for db models - [Get Started](../../docs/proxy/configs#set-custom-tokenizer), [PR](https://github.com/BerriAI/litellm/pull/10047)
+- **Prisma migrate** - handle existing columns in db table - [PR](https://github.com/BerriAI/litellm/pull/10138)
+
diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js
index e8029560b0..c8096a21bb 100644
--- a/docs/my-website/sidebars.js
+++ b/docs/my-website/sidebars.js
@@ -53,7 +53,7 @@ const sidebars = {
{
type: "category",
label: "Architecture",
- items: ["proxy/architecture", "proxy/db_info", "router_architecture", "proxy/user_management_heirarchy", "proxy/jwt_auth_arch"],
+ items: ["proxy/architecture", "proxy/db_info", "proxy/db_deadlocks", "router_architecture", "proxy/user_management_heirarchy", "proxy/jwt_auth_arch", "proxy/image_handling"],
},
{
type: "link",
@@ -69,6 +69,7 @@ const sidebars = {
"proxy/clientside_auth",
"proxy/request_headers",
"proxy/response_headers",
+ "proxy/model_discovery",
],
},
{
@@ -101,6 +102,7 @@ const sidebars = {
"proxy/admin_ui_sso",
"proxy/self_serve",
"proxy/public_teams",
+ "tutorials/scim_litellm",
"proxy/custom_sso",
"proxy/ui_credentials",
"proxy/ui_logs"
@@ -137,15 +139,17 @@ const sidebars = {
label: "[Beta] Guardrails",
items: [
"proxy/guardrails/quick_start",
- "proxy/guardrails/aim_security",
- "proxy/guardrails/aporia_api",
- "proxy/guardrails/bedrock",
- "proxy/guardrails/guardrails_ai",
- "proxy/guardrails/lakera_ai",
- "proxy/guardrails/pii_masking_v2",
- "proxy/guardrails/secret_detection",
- "proxy/guardrails/custom_guardrail",
- "prompt_injection"
+ ...[
+ "proxy/guardrails/aim_security",
+ "proxy/guardrails/aporia_api",
+ "proxy/guardrails/bedrock",
+ "proxy/guardrails/guardrails_ai",
+ "proxy/guardrails/lakera_ai",
+ "proxy/guardrails/pii_masking_v2",
+ "proxy/guardrails/secret_detection",
+ "proxy/guardrails/custom_guardrail",
+ "proxy/guardrails/prompt_injection",
+ ].sort(),
],
},
{
@@ -186,7 +190,15 @@ const sidebars = {
"providers/azure_ai",
"providers/aiml",
"providers/vertex",
- "providers/gemini",
+
+ {
+ type: "category",
+ label: "Google AI Studio",
+ items: [
+ "providers/gemini",
+ "providers/google_ai_studio/files",
+ ]
+ },
"providers/anthropic",
"providers/aws_sagemaker",
"providers/bedrock",
@@ -244,7 +256,9 @@ const sidebars = {
"exception_mapping",
"completion/provider_specific_params",
"guides/finetuned_models",
+ "guides/security_settings",
"completion/audio",
+ "completion/web_search",
"completion/document_understanding",
"completion/vision",
"completion/json_mode",
@@ -294,6 +308,7 @@ const sidebars = {
"text_completion",
"embedding/supported_embedding",
"anthropic_unified",
+ "mcp",
{
type: "category",
label: "/images",
@@ -318,6 +333,8 @@ const sidebars = {
"pass_through/vertex_ai",
"pass_through/google_ai_studio",
"pass_through/cohere",
+ "pass_through/vllm",
+ "pass_through/mistral",
"pass_through/openai_passthrough",
"pass_through/anthropic_completion",
"pass_through/bedrock",
@@ -328,7 +345,15 @@ const sidebars = {
},
"rerank",
"assistants",
- "files_endpoints",
+
+ {
+ type: "category",
+ label: "/files",
+ items: [
+ "files_endpoints",
+ "proxy/litellm_managed_files",
+ ],
+ },
"batches",
"realtime",
"fine_tuning",
@@ -366,8 +391,12 @@ const sidebars = {
],
},
{
- type: "doc",
- id: "proxy/prompt_management"
+ type: "category",
+ label: "[Beta] Prompt Management",
+ items: [
+ "proxy/prompt_management",
+ "proxy/custom_prompt_management"
+ ],
},
{
type: "category",
@@ -383,9 +412,10 @@ const sidebars = {
type: "category",
label: "Logging & Observability",
items: [
+ "observability/agentops_integration",
+ "observability/langfuse_integration",
"observability/lunary_integration",
"observability/mlflow",
- "observability/langfuse_integration",
"observability/gcs_bucket_integration",
"observability/langsmith_integration",
"observability/literalai_integration",
@@ -393,6 +423,7 @@ const sidebars = {
"observability/logfire_integration",
"observability/argilla",
"observability/arize_integration",
+ "observability/phoenix_integration",
"debugging/local_debugging",
"observability/raw_request_response",
"observability/custom_callback",
@@ -418,6 +449,10 @@ const sidebars = {
label: "Tutorials",
items: [
"tutorials/openweb_ui",
+ "tutorials/openai_codex",
+ "tutorials/msft_sso",
+ "tutorials/prompt_caching",
+ "tutorials/tag_management",
'tutorials/litellm_proxy_aporia',
{
type: "category",
diff --git a/docs/my-website/src/components/TransformRequestPlayground.tsx b/docs/my-website/src/components/TransformRequestPlayground.tsx
new file mode 100644
index 0000000000..8f22e5e198
--- /dev/null
+++ b/docs/my-website/src/components/TransformRequestPlayground.tsx
@@ -0,0 +1,161 @@
+import React, { useState } from 'react';
+import styles from './transform_request.module.css';
+
+const DEFAULT_REQUEST = {
+ "model": "bedrock/gpt-4",
+ "messages": [
+ {
+ "role": "system",
+ "content": "You are a helpful assistant."
+ },
+ {
+ "role": "user",
+ "content": "Explain quantum computing in simple terms"
+ }
+ ],
+ "temperature": 0.7,
+ "max_tokens": 500,
+ "stream": true
+};
+
+type ViewMode = 'split' | 'request' | 'transformed';
+
+const TransformRequestPlayground: React.FC = () => {
+ const [request, setRequest] = useState(JSON.stringify(DEFAULT_REQUEST, null, 2));
+ const [transformedRequest, setTransformedRequest] = useState('');
+ const [viewMode, setViewMode] = useState('split');
+
+ const handleTransform = async () => {
+ try {
+ // Here you would make the actual API call to transform the request
+ // For now, we'll just set a sample response
+ const sampleResponse = `curl -X POST \\
+ https://api.openai.com/v1/chat/completions \\
+ -H 'Authorization: Bearer sk-xxx' \\
+ -H 'Content-Type: application/json' \\
+ -d '{
+ "model": "gpt-4",
+ "messages": [
+ {
+ "role": "system",
+ "content": "You are a helpful assistant."
+ }
+ ],
+ "temperature": 0.7
+ }'`;
+ setTransformedRequest(sampleResponse);
+ } catch (error) {
+ console.error('Error transforming request:', error);
+ }
+ };
+
+ const handleCopy = () => {
+ navigator.clipboard.writeText(transformedRequest);
+ };
+
+ const renderContent = () => {
+ switch (viewMode) {
+ case 'request':
+ return (
+
+
+
Original Request
+
The request you would send to LiteLLM /chat/completions endpoint.
+
+
-
diff --git a/ci_cd/baseline_db.py b/ci_cd/baseline_db.py
new file mode 100644
index 0000000000..ecc080abed
--- /dev/null
+++ b/ci_cd/baseline_db.py
@@ -0,0 +1,60 @@
+import subprocess
+from pathlib import Path
+from datetime import datetime
+
+
+def create_baseline():
+ """Create baseline migration in deploy/migrations"""
+ try:
+ # Get paths
+ root_dir = Path(__file__).parent.parent
+ deploy_dir = root_dir / "deploy"
+ migrations_dir = deploy_dir / "migrations"
+ schema_path = root_dir / "schema.prisma"
+
+ # Create migrations directory
+ migrations_dir.mkdir(parents=True, exist_ok=True)
+
+ # Create migration_lock.toml if it doesn't exist
+ lock_file = migrations_dir / "migration_lock.toml"
+ if not lock_file.exists():
+ lock_file.write_text('provider = "postgresql"\n')
+
+ # Create timestamp-based migration directory
+ timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+ migration_dir = migrations_dir / f"{timestamp}_baseline"
+ migration_dir.mkdir(parents=True, exist_ok=True)
+
+ # Generate migration SQL
+ result = subprocess.run(
+ [
+ "prisma",
+ "migrate",
+ "diff",
+ "--from-empty",
+ "--to-schema-datamodel",
+ str(schema_path),
+ "--script",
+ ],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+
+ # Write the SQL to migration.sql
+ migration_file = migration_dir / "migration.sql"
+ migration_file.write_text(result.stdout)
+
+ print(f"Created baseline migration in {migration_dir}")
+ return True
+
+ except subprocess.CalledProcessError as e:
+ print(f"Error running prisma command: {e.stderr}")
+ return False
+ except Exception as e:
+ print(f"Error creating baseline migration: {str(e)}")
+ return False
+
+
+if __name__ == "__main__":
+ create_baseline()
diff --git a/ci_cd/publish-proxy-extras.sh b/ci_cd/publish-proxy-extras.sh
new file mode 100644
index 0000000000..6c83d1f921
--- /dev/null
+++ b/ci_cd/publish-proxy-extras.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# Exit on error
+set -e
+
+echo "🚀 Building and publishing litellm-proxy-extras"
+
+# Navigate to litellm-proxy-extras directory
+cd "$(dirname "$0")/../litellm-proxy-extras"
+
+# Build the package
+echo "📦 Building package..."
+poetry build
+
+# Publish to PyPI
+echo "🌎 Publishing to PyPI..."
+poetry publish
+
+echo "✅ Done! Package published successfully"
\ No newline at end of file
diff --git a/ci_cd/run_migration.py b/ci_cd/run_migration.py
new file mode 100644
index 0000000000..b11a38395c
--- /dev/null
+++ b/ci_cd/run_migration.py
@@ -0,0 +1,95 @@
+import os
+import subprocess
+from pathlib import Path
+from datetime import datetime
+import testing.postgresql
+import shutil
+
+
+def create_migration(migration_name: str = None):
+ """
+ Create a new migration SQL file in the migrations directory by comparing
+ current database state with schema
+
+ Args:
+ migration_name (str): Name for the migration
+ """
+ try:
+ # Get paths
+ root_dir = Path(__file__).parent.parent
+ migrations_dir = root_dir / "litellm-proxy-extras" / "litellm_proxy_extras" / "migrations"
+ schema_path = root_dir / "schema.prisma"
+
+ # Create temporary PostgreSQL database
+ with testing.postgresql.Postgresql() as postgresql:
+ db_url = postgresql.url()
+
+ # Create temporary migrations directory next to schema.prisma
+ temp_migrations_dir = schema_path.parent / "migrations"
+
+ try:
+ # Copy existing migrations to temp directory
+ if temp_migrations_dir.exists():
+ shutil.rmtree(temp_migrations_dir)
+ shutil.copytree(migrations_dir, temp_migrations_dir)
+
+ # Apply existing migrations to temp database
+ os.environ["DATABASE_URL"] = db_url
+ subprocess.run(
+ ["prisma", "migrate", "deploy", "--schema", str(schema_path)],
+ check=True,
+ )
+
+ # Generate diff between current database and schema
+ result = subprocess.run(
+ [
+ "prisma",
+ "migrate",
+ "diff",
+ "--from-url",
+ db_url,
+ "--to-schema-datamodel",
+ str(schema_path),
+ "--script",
+ ],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+
+ if result.stdout.strip():
+ # Generate timestamp and create migration directory
+ timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+ migration_name = migration_name or "unnamed_migration"
+ migration_dir = migrations_dir / f"{timestamp}_{migration_name}"
+ migration_dir.mkdir(parents=True, exist_ok=True)
+
+ # Write the SQL to migration.sql
+ migration_file = migration_dir / "migration.sql"
+ migration_file.write_text(result.stdout)
+
+ print(f"Created migration in {migration_dir}")
+ return True
+ else:
+ print("No schema changes detected. Migration not needed.")
+ return False
+
+ finally:
+ # Clean up: remove temporary migrations directory
+ if temp_migrations_dir.exists():
+ shutil.rmtree(temp_migrations_dir)
+
+ except subprocess.CalledProcessError as e:
+ print(f"Error generating migration: {e.stderr}")
+ return False
+ except Exception as e:
+ print(f"Error creating migration: {str(e)}")
+ return False
+
+
+if __name__ == "__main__":
+ # If running directly, can optionally pass migration name as argument
+ import sys
+
+ migration_name = sys.argv[1] if len(sys.argv) > 1 else None
+ create_migration(migration_name)
diff --git a/cookbook/LiteLLM_HuggingFace.ipynb b/cookbook/LiteLLM_HuggingFace.ipynb
index 3a9a0785be..d608c2675a 100644
--- a/cookbook/LiteLLM_HuggingFace.ipynb
+++ b/cookbook/LiteLLM_HuggingFace.ipynb
@@ -6,8 +6,9 @@
"id": "9dKM5k8qsMIj"
},
"source": [
- "## LiteLLM HuggingFace\n",
- "Docs for huggingface: https://docs.litellm.ai/docs/providers/huggingface"
+ "## LiteLLM Hugging Face\n",
+ "\n",
+ "Docs for huggingface: https://docs.litellm.ai/docs/providers/huggingface\n"
]
},
{
@@ -27,23 +28,18 @@
"id": "yp5UXRqtpu9f"
},
"source": [
- "## Hugging Face Free Serverless Inference API\n",
- "Read more about the Free Serverless Inference API here: https://huggingface.co/docs/api-inference.\n",
+ "## Serverless Inference Providers\n",
"\n",
- "In order to use litellm to call Serverless Inference API:\n",
+ "Read more about Inference Providers here: https://huggingface.co/blog/inference-providers.\n",
"\n",
- "* Browse Serverless Inference compatible models here: https://huggingface.co/models?inference=warm&pipeline_tag=text-generation.\n",
- "* Copy the model name from hugging face\n",
- "* Set `model = \"huggingface/
-You need to tell LiteLLM when you're calling Huggingface.
-This is done by adding the "huggingface/" prefix to `model`, example `completion(model="huggingface/