@@ -32,9 +32,9 @@ LiteLLM manages:
- Set Budgets & Rate limits per project, api key, model [OpenAI Proxy Server](https://docs.litellm.ai/docs/simple_proxy)
[**Jump to OpenAI Proxy Docs**](https://github.com/BerriAI/litellm?tab=readme-ov-file#openai-proxy---docs) -[**Jump to Supported LLM Providers**](https://github.com/BerriAI/litellm?tab=readme-ov-file#supported-provider-docs) +[**Jump to Supported LLM Providers**](https://github.com/BerriAI/litellm?tab=readme-ov-file#supported-providers-docs) -🚨 **Stable Release:** v1.34.1 +🚨 **Stable Release:** Use docker images with: `main-stable` tag. These run through 12 hr load tests (1k req./min). Support for more providers. Missing a provider or LLM Platform, raise a [feature request](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+). @@ -128,7 +128,9 @@ response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content # OpenAI Proxy - ([Docs](https://docs.litellm.ai/docs/simple_proxy)) -Set Budgets & Rate limits across multiple projects +Track spend + Load Balance across multiple projects + +[Hosted Proxy (Preview)](https://docs.litellm.ai/docs/hosted) The proxy provides: @@ -205,7 +207,7 @@ curl 'http://0.0.0.0:4000/key/generate' \ | [aws - bedrock](https://docs.litellm.ai/docs/providers/bedrock) | ✅ | ✅ | ✅ | ✅ | ✅ | | [google - vertex_ai [Gemini]](https://docs.litellm.ai/docs/providers/vertex) | ✅ | ✅ | ✅ | ✅ | | [google - palm](https://docs.litellm.ai/docs/providers/palm) | ✅ | ✅ | ✅ | ✅ | -| [google AI Studio - gemini](https://docs.litellm.ai/docs/providers/gemini) | ✅ | | ✅ | | | +| [google AI Studio - gemini](https://docs.litellm.ai/docs/providers/gemini) | ✅ | ✅ | ✅ | ✅ | | | [mistral ai api](https://docs.litellm.ai/docs/providers/mistral) | ✅ | ✅ | ✅ | ✅ | ✅ | | [cloudflare AI Workers](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | ✅ | ✅ | ✅ | | [cohere](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | ✅ | ✅ | @@ -220,7 +222,7 @@ curl 'http://0.0.0.0:4000/key/generate' \ | [nlp_cloud](https://docs.litellm.ai/docs/providers/nlp_cloud) | ✅ | ✅ | ✅ | ✅ | | [aleph alpha](https://docs.litellm.ai/docs/providers/aleph_alpha) | ✅ | ✅ | ✅ | ✅ | | [petals](https://docs.litellm.ai/docs/providers/petals) | ✅ | ✅ | ✅ | ✅ | -| [ollama](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | ✅ | +| [ollama](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | ✅ | ✅ | | [deepinfra](https://docs.litellm.ai/docs/providers/deepinfra) | ✅ | ✅ | ✅ | ✅ | | [perplexity-ai](https://docs.litellm.ai/docs/providers/perplexity) | ✅ | ✅ | ✅ | ✅ | | [Groq AI](https://docs.litellm.ai/docs/providers/groq) | ✅ | ✅ | ✅ | ✅ | diff --git a/cookbook/Proxy_Batch_Users.ipynb b/cookbook/Proxy_Batch_Users.ipynb new file mode 100644 index 000000000..70521f5ab --- /dev/null +++ b/cookbook/Proxy_Batch_Users.ipynb @@ -0,0 +1,204 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "680oRk1af-xJ" + }, + "source": [ + "# Environment Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "X7TgJFn8f88p" + }, + "outputs": [], + "source": [ + "import csv\n", + "from typing import Optional\n", + "import httpx, json\n", + "import asyncio\n", + "\n", + "proxy_base_url = \"http://0.0.0.0:4000\" # 👈 SET TO PROXY URL\n", + "master_key = \"sk-1234\" # 👈 SET TO PROXY MASTER KEY" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rauw8EOhgBz5" + }, + "outputs": [], + "source": [ + "## GLOBAL HTTP CLIENT ## - faster http calls\n", + "class HTTPHandler:\n", + " def __init__(self, concurrent_limit=1000):\n", + " # Create a client with a connection pool\n", + " self.client = httpx.AsyncClient(\n", + " limits=httpx.Limits(\n", + " max_connections=concurrent_limit,\n", + " max_keepalive_connections=concurrent_limit,\n", + " )\n", + " )\n", + "\n", + " async def close(self):\n", + " # Close the client when you're done with it\n", + " await self.client.aclose()\n", + "\n", + " async def get(\n", + " self, url: str, params: Optional[dict] = None, headers: Optional[dict] = None\n", + " ):\n", + " response = await self.client.get(url, params=params, headers=headers)\n", + " return response\n", + "\n", + " async def post(\n", + " self,\n", + " url: str,\n", + " data: Optional[dict] = None,\n", + " params: Optional[dict] = None,\n", + " headers: Optional[dict] = None,\n", + " ):\n", + " try:\n", + " response = await self.client.post(\n", + " url, data=data, params=params, headers=headers\n", + " )\n", + " return response\n", + " except Exception as e:\n", + " raise e\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7LXN8zaLgOie" + }, + "source": [ + "# Import Sheet\n", + "\n", + "\n", + "Format: | ID | Name | Max Budget |" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oiED0usegPGf" + }, + "outputs": [], + "source": [ + "async def import_sheet():\n", + " tasks = []\n", + " http_client = HTTPHandler()\n", + " with open('my-batch-sheet.csv', 'r') as file:\n", + " csv_reader = csv.DictReader(file)\n", + " for row in csv_reader:\n", + " task = create_user(client=http_client, user_id=row['ID'], max_budget=row['Max Budget'], user_name=row['Name'])\n", + " tasks.append(task)\n", + " # print(f\"ID: {row['ID']}, Name: {row['Name']}, Max Budget: {row['Max Budget']}\")\n", + "\n", + " keys = await asyncio.gather(*tasks)\n", + "\n", + " with open('my-batch-sheet_new.csv', 'w', newline='') as new_file:\n", + " fieldnames = ['ID', 'Name', 'Max Budget', 'keys']\n", + " csv_writer = csv.DictWriter(new_file, fieldnames=fieldnames)\n", + " csv_writer.writeheader()\n", + "\n", + " with open('my-batch-sheet.csv', 'r') as file:\n", + " csv_reader = csv.DictReader(file)\n", + " for i, row in enumerate(csv_reader):\n", + " row['keys'] = keys[i] # Add the 'keys' value from the corresponding task result\n", + " csv_writer.writerow(row)\n", + "\n", + " await http_client.close()\n", + "\n", + "asyncio.run(import_sheet())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "E7M0Li_UgJeZ" + }, + "source": [ + "# Create Users + Keys\n", + "\n", + "- Creates a user\n", + "- Creates a key with max budget" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NZudRFujf7j-" + }, + "outputs": [], + "source": [ + "\n", + "async def create_key_with_alias(client: HTTPHandler, user_id: str, max_budget: float):\n", + " global proxy_base_url\n", + " if not proxy_base_url.endswith(\"/\"):\n", + " proxy_base_url += \"/\"\n", + " url = proxy_base_url + \"key/generate\"\n", + "\n", + " # call /key/generate\n", + " print(\"CALLING /KEY/GENERATE\")\n", + " response = await client.post(\n", + " url=url,\n", + " headers={\"Authorization\": f\"Bearer {master_key}\"},\n", + " data=json.dumps({\n", + " \"user_id\": user_id,\n", + " \"key_alias\": f\"{user_id}-key\",\n", + " \"max_budget\": max_budget # 👈 KEY CHANGE: SETS MAX BUDGET PER KEY\n", + " })\n", + " )\n", + " print(f\"response: {response.text}\")\n", + " return response.json()[\"key\"]\n", + "\n", + "async def create_user(client: HTTPHandler, user_id: str, max_budget: float, user_name: str):\n", + " \"\"\"\n", + " - call /user/new\n", + " - create key for user\n", + " \"\"\"\n", + " global proxy_base_url\n", + " if not proxy_base_url.endswith(\"/\"):\n", + " proxy_base_url += \"/\"\n", + " url = proxy_base_url + \"user/new\"\n", + "\n", + " # call /user/new\n", + " await client.post(\n", + " url=url,\n", + " headers={\"Authorization\": f\"Bearer {master_key}\"},\n", + " data=json.dumps({\n", + " \"user_id\": user_id,\n", + " \"user_alias\": user_name,\n", + " \"auto_create_key\": False,\n", + " # \"max_budget\": max_budget # 👈 [OPTIONAL] Sets max budget per user (if you want to set a max budget across keys)\n", + " })\n", + " )\n", + "\n", + " # create key for user\n", + " return await create_key_with_alias(client=client, user_id=user_id, max_budget=max_budget)\n" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/cookbook/benchmark/readme.md b/cookbook/benchmark/readme.md index e573f76c2..a543d9101 100644 --- a/cookbook/benchmark/readme.md +++ b/cookbook/benchmark/readme.md @@ -87,6 +87,7 @@ | command-light | cohere | 0.00003 | | command-medium-beta | cohere | 0.00003 | | command-xlarge-beta | cohere | 0.00003 | +| command-r-plus| cohere | 0.000018 | | j2-ultra | ai21 | 0.00003 | | ai21.j2-ultra-v1 | bedrock | 0.0000376 | | gpt-4-1106-preview | openai | 0.00004 | diff --git a/cookbook/misc/config.yaml b/cookbook/misc/config.yaml new file mode 100644 index 000000000..d1d06eb58 --- /dev/null +++ b/cookbook/misc/config.yaml @@ -0,0 +1,73 @@ +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: azure/chatgpt-v-2 + api_base: https://openai-gpt-4-test-v-1.openai.azure.com/ + api_version: "2023-05-15" + api_key: os.environ/AZURE_API_KEY # The `os.environ/` prefix tells litellm to read this from the env. See https://docs.litellm.ai/docs/simple_proxy#load-api-keys-from-vault + - model_name: gpt-3.5-turbo-large + litellm_params: + model: "gpt-3.5-turbo-1106" + api_key: os.environ/OPENAI_API_KEY + rpm: 480 + timeout: 300 + stream_timeout: 60 + - model_name: gpt-4 + litellm_params: + model: azure/chatgpt-v-2 + api_base: https://openai-gpt-4-test-v-1.openai.azure.com/ + api_version: "2023-05-15" + api_key: os.environ/AZURE_API_KEY # The `os.environ/` prefix tells litellm to read this from the env. See https://docs.litellm.ai/docs/simple_proxy#load-api-keys-from-vault + rpm: 480 + timeout: 300 + stream_timeout: 60 + - model_name: sagemaker-completion-model + litellm_params: + model: sagemaker/berri-benchmarking-Llama-2-70b-chat-hf-4 + input_cost_per_second: 0.000420 + - model_name: text-embedding-ada-002 + litellm_params: + model: azure/azure-embedding-model + api_key: os.environ/AZURE_API_KEY + api_base: https://openai-gpt-4-test-v-1.openai.azure.com/ + api_version: "2023-05-15" + model_info: + mode: embedding + base_model: text-embedding-ada-002 + - model_name: dall-e-2 + litellm_params: + model: azure/ + api_version: 2023-06-01-preview + api_base: https://openai-gpt-4-test-v-1.openai.azure.com/ + api_key: os.environ/AZURE_API_KEY + - model_name: openai-dall-e-3 + litellm_params: + model: dall-e-3 + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_key: fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + +litellm_settings: + drop_params: True + # max_budget: 100 + # budget_duration: 30d + num_retries: 5 + request_timeout: 600 + telemetry: False + context_window_fallbacks: [{"gpt-3.5-turbo": ["gpt-3.5-turbo-large"]}] + +general_settings: + master_key: sk-1234 # [OPTIONAL] Use to enforce auth on proxy. See - https://docs.litellm.ai/docs/proxy/virtual_keys + store_model_in_db: True + proxy_budget_rescheduler_min_time: 60 + proxy_budget_rescheduler_max_time: 64 + proxy_batch_write_at: 1 + # database_url: "postgresql://
- \n
- apple \n
- banana \n
@@ -97,7 +97,7 @@ response = completion(
print(response)
```
## Ollama Models
-Ollama supported models: https://github.com/jmorganca/ollama
+Ollama supported models: https://github.com/ollama/ollama
| Model Name | Function Call |
|----------------------|-----------------------------------------------------------------------------------
diff --git a/docs/my-website/docs/providers/openai.md b/docs/my-website/docs/providers/openai.md
index aa42847b1..a8fe541fc 100644
--- a/docs/my-website/docs/providers/openai.md
+++ b/docs/my-website/docs/providers/openai.md
@@ -1,5 +1,8 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
# OpenAI
-LiteLLM supports OpenAI Chat + Text completion and embedding calls.
+LiteLLM supports OpenAI Chat + Embedding calls.
### Required API Keys
@@ -22,6 +25,132 @@ response = completion(
)
```
+### Usage - LiteLLM Proxy Server
+
+Here's how to call OpenAI models with the LiteLLM Proxy Server
+
+### 1. Save key in your environment
+
+```bash
+export OPENAI_API_KEY=""
+```
+
+### 2. Start the proxy
+
+
## Pre-requisites
-* `pip install google-cloud-aiplatform`
+* `pip install google-cloud-aiplatform` (pre-installed on proxy docker image)
* Authentication:
* run `gcloud auth application-default login` See [Google Cloud Docs](https://cloud.google.com/docs/authentication/external/set-up-adc)
- * Alternatively you can set `application_default_credentials.json`
+ * Alternatively you can set `GOOGLE_APPLICATION_CREDENTIALS`
+ Here's how: [**Jump to Code**](#extra)
+
+ - Create a service account on GCP
+ - Export the credentials as a json
+ - load the json and json.dump the json as a string
+ - store the json string in your environment as `GOOGLE_APPLICATION_CREDENTIALS`
## Sample Usage
```python
@@ -123,6 +130,100 @@ Here's how to use Vertex AI with the LiteLLM Proxy Server
+## Specifying Safety Settings
+In certain use-cases you may need to make calls to the models and pass [safety settigns](https://ai.google.dev/docs/safety_setting_gemini) different from the defaults. To do so, simple pass the `safety_settings` argument to `completion` or `acompletion`. For example:
+
+
+