-Go directly to code: [Getting Started Notebook](https://colab.research.google.com/drive/1gR3pY-JzDZahzpVdbGBtrNGDBmzUNJaJ?usp=sharing)
+Demo - https://litellm.ai/playground \
+Read the docs - https://docs.litellm.ai/docs/
-### Installation
+## quick start
```
pip install litellm
```
-### Usage
-
```python
from litellm import completion
@@ -41,13 +43,37 @@ response = completion(model="gpt-3.5-turbo", messages=messages)
response = completion("command-nightly", messages)
```
-Need Help / Support : [see troubleshooting](https://litellm.readthedocs.io/en/latest/troubleshoot)
+Code Sample: [Getting Started Notebook](https://colab.research.google.com/drive/1gR3pY-JzDZahzpVdbGBtrNGDBmzUNJaJ?usp=sharing)
-## Why did we build liteLLM
+Stable version
+
+```
+pip install litellm==0.1.345
+```
+
+## Streaming Queries
+
+liteLLM supports streaming the model response back, pass `stream=True` to get a streaming iterator in response.
+Streaming is supported for OpenAI, Azure, Anthropic, Huggingface models
+
+```python
+response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+for chunk in response:
+ print(chunk['choices'][0]['delta'])
+
+# claude 2
+result = completion('claude-2', messages, stream=True)
+for chunk in result:
+ print(chunk['choices'][0]['delta'])
+```
+
+# support / talk with founders
+
+- [Our calendar 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
+- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
+
+# why did we build this
- **Need for simplicity**: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI, Cohere
-
-## Support
-
-- [Meet with us 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
-- Contact us at ishaan@berri.ai / krrish@berri.ai
diff --git a/docs/my-website/docs/observability/supabase_integration.md b/docs/my-website/docs/observability/supabase_integration.md
index 6ae4f65da..d9fbc2b5a 100644
--- a/docs/my-website/docs/observability/supabase_integration.md
+++ b/docs/my-website/docs/observability/supabase_integration.md
@@ -22,11 +22,13 @@ create table
messages json null default '{}'::json,
response json null default '{}'::json,
end_user text null default ''::text,
+ status text null default ''::text,
error json null default '{}'::json,
response_time real null default '0'::real,
total_cost real null,
additional_details json null default '{}'::json,
- constraint request_logs_pkey primary key (id)
+ litellm_call_id text unique,
+ primary key (id)
) tablespace pg_default;
```
diff --git a/docs/my-website/docusaurus.config.js b/docs/my-website/docusaurus.config.js
index e1e5cbb01..4af0e6f3b 100644
--- a/docs/my-website/docusaurus.config.js
+++ b/docs/my-website/docusaurus.config.js
@@ -6,8 +6,9 @@ const darkCodeTheme = require('prism-react-renderer/themes/dracula');
/** @type {import('@docusaurus/types').Config} */
const config = {
- title: 'LiteLLM',
+ title: 'liteLLM',
tagline: 'Simplify LLM API Calls',
+ favicon: '/img/favicon.ico',
// Set the production url of your site here
url: 'https://litellm.vercel.app/',
@@ -80,35 +81,27 @@ const config = {
{
title: 'Community',
items: [
- {
- label: 'Stack Overflow',
- href: 'https://stackoverflow.com/questions/tagged/docusaurus',
- },
{
label: 'Discord',
- href: 'https://discordapp.com/invite/docusaurus',
+ href: 'https://discord.com/invite/wuPM9dRgDw',
},
{
label: 'Twitter',
- href: 'https://twitter.com/docusaurus',
+ href: 'https://twitter.com/LiteLLM',
},
],
},
{
title: 'More',
items: [
- {
- label: 'Blog',
- to: '/blog',
- },
{
label: 'GitHub',
- href: 'https://github.com/facebook/docusaurus',
+ href: 'https://github.com/BerriAI/litellm/',
},
],
},
],
- copyright: `Copyright © ${new Date().getFullYear()} My Project, Inc. Built with Docusaurus.`,
+ copyright: `Copyright © ${new Date().getFullYear()} liteLLM`,
},
prism: {
theme: lightCodeTheme,
diff --git a/docs/my-website/index.md b/docs/my-website/index.md
new file mode 100644
index 000000000..7d0698afe
--- /dev/null
+++ b/docs/my-website/index.md
@@ -0,0 +1,25 @@
+---
+slug: welcome
+title: Welcome
+authors: [slorber, yangshun]
+tags: [facebook, hello, docusaurus]
+---
+
+[Docusaurus blogging features](https://docusaurus.io/docs/blog) are powered by the [blog plugin](https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-content-blog).
+
+Simply add Markdown files (or folders) to the `blog` directory.
+
+Regular blog authors can be added to `authors.yml`.
+
+The blog post date can be extracted from filenames, such as:
+
+- `2019-05-30-welcome.md`
+- `2019-05-30-welcome/index.md`
+
+A blog post folder can be convenient to co-locate blog post images:
+
+
+
+The blog supports tags as well!
+
+**And if you don't want a blog**: just delete this directory, and use `blog: false` in your Docusaurus config.
\ No newline at end of file
diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js
index 09978e440..4b5344382 100644
--- a/docs/my-website/sidebars.js
+++ b/docs/my-website/sidebars.js
@@ -21,14 +21,15 @@ const sidebars = {
'index',
{
type: 'category',
- label: 'completion_function',
- items: ['completion/input', 'completion/supported','completion/output'],
+ label: 'Completion()',
+ items: ['completion/input','completion/output'],
},
{
type: 'category',
- label: 'embedding_function',
+ label: 'Embedding()',
items: ['embedding/supported_embedding'],
},
+ 'completion/supported',
{
type: 'category',
label: 'Tutorials',
@@ -37,6 +38,7 @@ const sidebars = {
'token_usage',
'stream',
'secret',
+ 'caching',
{
type: 'category',
label: 'Logging & Observability',
diff --git a/docs/my-website/src/pages/index.md b/docs/my-website/src/pages/index.md
index 4ca470335..57d23215d 100644
--- a/docs/my-website/src/pages/index.md
+++ b/docs/my-website/src/pages/index.md
@@ -1,23 +1,27 @@
-# 🚅 litellm
-a light 100 line package to simplify calling OpenAI, Azure, Cohere, Anthropic APIs
+# *🚅 litellm*
+[](https://pypi.org/project/litellm/)
+[](https://pypi.org/project/litellm/0.1.1/)
+[](https://dl.circleci.com/status-badge/redirect/gh/BerriAI/litellm/tree/main)
+
+[](https://github.com/BerriAI/litellm)
-###### litellm manages:
-* Calling all LLM APIs using the OpenAI format - `completion(model, messages)`
-* Consistent output for all LLM APIs, text responses will always be available at `['choices'][0]['message']['content']`
-* Consistent Exceptions for all LLM APIs, we map RateLimit, Context Window, and Authentication Error exceptions across all providers to their OpenAI equivalents. [see Code](https://github.com/BerriAI/litellm/blob/ba1079ff6698ef238c5c7f771dd2b698ec76f8d9/litellm/utils.py#L250)
+[](https://discord.gg/wuPM9dRgDw)
-###### observability:
-* Logging - see exactly what the raw model request/response is by plugging in your own function `completion(.., logger_fn=your_logging_fn)` and/or print statements from the package `litellm.set_verbose=True`
-* Callbacks - automatically send your data to Helicone, Sentry, Posthog, Slack - `litellm.success_callbacks`, `litellm.failure_callbacks` [see Callbacks](https://litellm.readthedocs.io/en/latest/advanced/)
+a light package to simplify calling OpenAI, Azure, Cohere, Anthropic, Huggingface API Endpoints. It manages:
+- translating inputs to the provider's completion and embedding endpoints
+- guarantees [consistent output](https://litellm.readthedocs.io/en/latest/output/), text responses will always be available at `['choices'][0]['message']['content']`
+- exception mapping - common exceptions across providers are mapped to the [OpenAI exception types](https://help.openai.com/en/articles/6897213-openai-library-error-types-guidance)
+# usage
+
-## Quick Start
-Go directly to code: [Getting Started Notebook](https://colab.research.google.com/drive/1gR3pY-JzDZahzpVdbGBtrNGDBmzUNJaJ?usp=sharing)
-### Installation
+Demo - https://litellm.ai/playground \
+Read the docs - https://docs.litellm.ai/docs/
+
+## quick start
```
pip install litellm
```
-### Usage
```python
from litellm import completion
@@ -33,11 +37,32 @@ response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion("command-nightly", messages)
```
-Need Help / Support : [see troubleshooting](https://litellm.readthedocs.io/en/latest/troubleshoot)
+Code Sample: [Getting Started Notebook](https://colab.research.google.com/drive/1gR3pY-JzDZahzpVdbGBtrNGDBmzUNJaJ?usp=sharing)
-## Why did we build liteLLM
+Stable version
+```
+pip install litellm==0.1.345
+```
+
+## Streaming Queries
+liteLLM supports streaming the model response back, pass `stream=True` to get a streaming iterator in response.
+Streaming is supported for OpenAI, Azure, Anthropic, Huggingface models
+```python
+response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+for chunk in response:
+ print(chunk['choices'][0]['delta'])
+
+# claude 2
+result = completion('claude-2', messages, stream=True)
+for chunk in result:
+ print(chunk['choices'][0]['delta'])
+```
+
+# support / talk with founders
+- [Our calendar 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
+- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
+
+# why did we build this
- **Need for simplicity**: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI, Cohere
-
-## Support
-* [Meet with us 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
-* Contact us at ishaan@berri.ai / krrish@berri.ai
diff --git a/docs/my-website/static/img/favicon.ico b/docs/my-website/static/img/favicon.ico
index c01d54bcd..88caa2b83 100644
Binary files a/docs/my-website/static/img/favicon.ico and b/docs/my-website/static/img/favicon.ico differ
diff --git a/litellm/__init__.py b/litellm/__init__.py
index 6b8240678..7cbb0e996 100644
--- a/litellm/__init__.py
+++ b/litellm/__init__.py
@@ -1,50 +1,120 @@
import threading
-success_callback = []
-failure_callback = []
-set_verbose=False
-telemetry=True
-max_tokens = 256 # OpenAI Defaults
+from typing import Callable, List, Optional
+input_callback: List[str] = []
+success_callback: List[str] = []
+failure_callback: List[str] = []
+set_verbose = False
+telemetry = True
+max_tokens = 256 # OpenAI Defaults
retry = True
-api_key = None
-openai_key = None
-azure_key = None
-anthropic_key = None
-replicate_key = None
-cohere_key = None
-openrouter_key = None
-huggingface_key = None
-vertex_project = None
-vertex_location = None
-
-hugging_api_token = None
+api_key: Optional[str] = None
+openai_key: Optional[str] = None
+azure_key: Optional[str] = None
+anthropic_key: Optional[str] = None
+replicate_key: Optional[str] = None
+cohere_key: Optional[str] = None
+openrouter_key: Optional[str] = None
+huggingface_key: Optional[str] = None
+vertex_project: Optional[str] = None
+vertex_location: Optional[str] = None
+hugging_api_token: Optional[str] = None
+togetherai_api_key: Optional[str] = None
+caching = False
+caching_with_models = False # if you want the caching key to be model + prompt
model_cost = {
- "gpt-3.5-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-35-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002}, # azure model name
- "gpt-3.5-turbo-0613": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-0301": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-35-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004}, # azure model name
- "gpt-3.5-turbo-16k-0613": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-4": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-0613": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-32k": {"max_tokens": 8000, "input_cost_per_token": 0.00006, "output_cost_per_token": 0.00012},
- "claude-instant-1": {"max_tokens": 100000, "input_cost_per_token": 0.00000163, "output_cost_per_token": 0.00000551},
- "claude-2": {"max_tokens": 100000, "input_cost_per_token": 0.00001102, "output_cost_per_token": 0.00003268},
- "text-bison-001": {"max_tokens": 8192, "input_cost_per_token": 0.000004, "output_cost_per_token": 0.000004},
- "chat-bison-001": {"max_tokens": 4096, "input_cost_per_token": 0.000002, "output_cost_per_token": 0.000002},
- "command-nightly": {"max_tokens": 4096, "input_cost_per_token": 0.000015, "output_cost_per_token": 0.000015},
+ "gpt-3.5-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-35-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ }, # azure model name
+ "gpt-3.5-turbo-0613": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-0301": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-35-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ }, # azure model name
+ "gpt-3.5-turbo-16k-0613": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-4": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-0613": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-32k": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.00006,
+ "output_cost_per_token": 0.00012,
+ },
+ "claude-instant-1": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00000163,
+ "output_cost_per_token": 0.00000551,
+ },
+ "claude-2": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00001102,
+ "output_cost_per_token": 0.00003268,
+ },
+ "text-bison-001": {
+ "max_tokens": 8192,
+ "input_cost_per_token": 0.000004,
+ "output_cost_per_token": 0.000004,
+ },
+ "chat-bison-001": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000002,
+ "output_cost_per_token": 0.000002,
+ },
+ "command-nightly": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000015,
+ "output_cost_per_token": 0.000015,
+ },
}
+
####### THREAD-SPECIFIC DATA ###################
class MyLocal(threading.local):
def __init__(self):
self.user = "Hello World"
+
_thread_context = MyLocal()
+
+
def identify(event_details):
# Store user in thread local data
if "user" in event_details:
_thread_context.user = event_details["user"]
+
+
####### ADDITIONAL PARAMS ################### configurable params if you use proxy models like Helicone, map spend to org id, etc.
api_base = None
headers = None
@@ -55,60 +125,48 @@ config_path = None
secret_manager_client = None
####### COMPLETION MODELS ###################
open_ai_chat_completion_models = [
- "gpt-4",
- "gpt-4-0613",
- "gpt-4-32k",
- "gpt-4-32k-0613",
- #################
- "gpt-3.5-turbo",
- "gpt-3.5-turbo-16k",
- "gpt-3.5-turbo-0613",
- "gpt-3.5-turbo-16k-0613",
-]
-open_ai_text_completion_models = [
- 'text-davinci-003'
+ "gpt-4",
+ "gpt-4-0613",
+ "gpt-4-32k",
+ "gpt-4-32k-0613",
+ #################
+ "gpt-3.5-turbo",
+ "gpt-3.5-turbo-16k",
+ "gpt-3.5-turbo-0613",
+ "gpt-3.5-turbo-16k-0613",
]
+open_ai_text_completion_models = ["text-davinci-003"]
cohere_models = [
- 'command-nightly',
- "command",
- "command-light",
- "command-medium-beta",
- "command-xlarge-beta"
+ "command-nightly",
+ "command",
+ "command-light",
+ "command-medium-beta",
+ "command-xlarge-beta",
]
-anthropic_models = [
- "claude-2",
- "claude-instant-1",
- "claude-instant-1.2"
-]
+anthropic_models = ["claude-2", "claude-instant-1", "claude-instant-1.2"]
replicate_models = [
"replicate/"
-] # placeholder, to make sure we accept any replicate model in our model_list
+] # placeholder, to make sure we accept any replicate model in our model_list
openrouter_models = [
- 'google/palm-2-codechat-bison',
- 'google/palm-2-chat-bison',
- 'openai/gpt-3.5-turbo',
- 'openai/gpt-3.5-turbo-16k',
- 'openai/gpt-4-32k',
- 'anthropic/claude-2',
- 'anthropic/claude-instant-v1',
- 'meta-llama/llama-2-13b-chat',
- 'meta-llama/llama-2-70b-chat'
+ "google/palm-2-codechat-bison",
+ "google/palm-2-chat-bison",
+ "openai/gpt-3.5-turbo",
+ "openai/gpt-3.5-turbo-16k",
+ "openai/gpt-4-32k",
+ "anthropic/claude-2",
+ "anthropic/claude-instant-v1",
+ "meta-llama/llama-2-13b-chat",
+ "meta-llama/llama-2-70b-chat",
]
-vertex_chat_models = [
- "chat-bison",
- "chat-bison@001"
-]
+vertex_chat_models = ["chat-bison", "chat-bison@001"]
-vertex_text_models = [
- "text-bison",
- "text-bison@001"
-]
+vertex_text_models = ["text-bison", "text-bison@001"]
huggingface_models = [
"meta-llama/Llama-2-7b-hf",
@@ -123,24 +181,56 @@ huggingface_models = [
"meta-llama/Llama-2-13b-chat",
"meta-llama/Llama-2-70b",
"meta-llama/Llama-2-70b-chat",
-] # these have been tested on extensively. But by default all text2text-generation and text-generation models are supported by liteLLM. - https://docs.litellm.ai/docs/completion/supported
+] # these have been tested on extensively. But by default all text2text-generation and text-generation models are supported by liteLLM. - https://docs.litellm.ai/docs/completion/supported
-ai21_models = [
- "j2-ultra",
- "j2-mid",
- "j2-light"
+ai21_models = ["j2-ultra", "j2-mid", "j2-light"]
+
+model_list = (
+ open_ai_chat_completion_models
+ + open_ai_text_completion_models
+ + cohere_models
+ + anthropic_models
+ + replicate_models
+ + openrouter_models
+ + huggingface_models
+ + vertex_chat_models
+ + vertex_text_models
+ + ai21_models
+)
+
+provider_list = [
+ "openai",
+ "cohere",
+ "anthropic",
+ "replicate",
+ "huggingface",
+ "together_ai",
+ "openrouter",
+ "vertex_ai",
+ "ai21",
]
-
-model_list = open_ai_chat_completion_models + open_ai_text_completion_models + cohere_models + anthropic_models + replicate_models + openrouter_models + huggingface_models + vertex_chat_models + vertex_text_models + ai21_models
-
-
####### EMBEDDING MODELS ###################
-open_ai_embedding_models = [
- 'text-embedding-ada-002'
-]
+open_ai_embedding_models = ["text-embedding-ada-002"]
from .timeout import timeout
-from .utils import client, logging, exception_type, get_optional_params, modify_integration, token_counter, cost_per_token, completion_cost, load_test_model, get_litellm_params
-from .main import * # Import all the symbols from main.py
+from .testing import *
+from .utils import (
+ client,
+ exception_type,
+ get_optional_params,
+ modify_integration,
+ token_counter,
+ cost_per_token,
+ completion_cost,
+ get_litellm_params,
+ Logging
+)
+from .main import * # type: ignore
from .integrations import *
-from openai.error import AuthenticationError, InvalidRequestError, RateLimitError, ServiceUnavailableError, OpenAIError
\ No newline at end of file
+from openai.error import (
+ AuthenticationError,
+ InvalidRequestError,
+ RateLimitError,
+ ServiceUnavailableError,
+ OpenAIError,
+)
diff --git a/litellm/__pycache__/__init__.cpython-311.pyc b/litellm/__pycache__/__init__.cpython-311.pyc
index 3a9b2741d..c998bff4a 100644
Binary files a/litellm/__pycache__/__init__.cpython-311.pyc and b/litellm/__pycache__/__init__.cpython-311.pyc differ
diff --git a/litellm/__pycache__/main.cpython-311.pyc b/litellm/__pycache__/main.cpython-311.pyc
index afc8469fd..f9e652468 100644
Binary files a/litellm/__pycache__/main.cpython-311.pyc and b/litellm/__pycache__/main.cpython-311.pyc differ
diff --git a/litellm/__pycache__/timeout.cpython-311.pyc b/litellm/__pycache__/timeout.cpython-311.pyc
index 09f976993..68f0223aa 100644
Binary files a/litellm/__pycache__/timeout.cpython-311.pyc and b/litellm/__pycache__/timeout.cpython-311.pyc differ
diff --git a/litellm/__pycache__/utils.cpython-311.pyc b/litellm/__pycache__/utils.cpython-311.pyc
index 92c3af615..05731d21d 100644
Binary files a/litellm/__pycache__/utils.cpython-311.pyc and b/litellm/__pycache__/utils.cpython-311.pyc differ
diff --git a/litellm/exceptions.py b/litellm/exceptions.py
new file mode 100644
index 000000000..7b48a343d
--- /dev/null
+++ b/litellm/exceptions.py
@@ -0,0 +1,62 @@
+## LiteLLM versions of the OpenAI Exception Types
+from openai.error import (
+ AuthenticationError,
+ InvalidRequestError,
+ RateLimitError,
+ ServiceUnavailableError,
+ OpenAIError,
+)
+
+
+class AuthenticationError(AuthenticationError): # type: ignore
+ def __init__(self, message, llm_provider):
+ self.status_code = 401
+ self.message = message
+ self.llm_provider = llm_provider
+ super().__init__(
+ self.message
+ ) # Call the base class constructor with the parameters it needs
+
+
+class InvalidRequestError(InvalidRequestError): # type: ignore
+ def __init__(self, message, model, llm_provider):
+ self.status_code = 400
+ self.message = message
+ self.model = model
+ self.llm_provider = llm_provider
+ super().__init__(
+ self.message, f"{self.model}"
+ ) # Call the base class constructor with the parameters it needs
+
+
+class RateLimitError(RateLimitError): # type: ignore
+ def __init__(self, message, llm_provider):
+ self.status_code = 429
+ self.message = message
+ self.llm_provider = llm_provider
+ super().__init__(
+ self.message
+ ) # Call the base class constructor with the parameters it needs
+
+
+class ServiceUnavailableError(ServiceUnavailableError): # type: ignore
+ def __init__(self, message, llm_provider):
+ self.status_code = 500
+ self.message = message
+ self.llm_provider = llm_provider
+ super().__init__(
+ self.message
+ ) # Call the base class constructor with the parameters it needs
+
+
+class OpenAIError(OpenAIError): # type: ignore
+ def __init__(self, original_exception):
+ self.status_code = original_exception.http_status
+ super().__init__(
+ http_body=original_exception.http_body,
+ http_status=original_exception.http_status,
+ json_body=original_exception.json_body,
+ headers=original_exception.headers,
+ code=original_exception.code,
+ )
+ self.llm_provider = "openai"
diff --git a/litellm/integrations/__init__.py b/litellm/integrations/__init__.py
index b9742821a..b6e690fd5 100644
--- a/litellm/integrations/__init__.py
+++ b/litellm/integrations/__init__.py
@@ -1 +1 @@
-from . import *
\ No newline at end of file
+from . import *
diff --git a/litellm/integrations/__pycache__/__init__.cpython-311.pyc b/litellm/integrations/__pycache__/__init__.cpython-311.pyc
index e951c50a3..5bd4bfeb8 100644
Binary files a/litellm/integrations/__pycache__/__init__.cpython-311.pyc and b/litellm/integrations/__pycache__/__init__.cpython-311.pyc differ
diff --git a/litellm/integrations/__pycache__/aispend.cpython-311.pyc b/litellm/integrations/__pycache__/aispend.cpython-311.pyc
index 9e2d468cb..111b4eba1 100644
Binary files a/litellm/integrations/__pycache__/aispend.cpython-311.pyc and b/litellm/integrations/__pycache__/aispend.cpython-311.pyc differ
diff --git a/litellm/integrations/__pycache__/berrispend.cpython-311.pyc b/litellm/integrations/__pycache__/berrispend.cpython-311.pyc
index 87b3f5e36..616f227eb 100644
Binary files a/litellm/integrations/__pycache__/berrispend.cpython-311.pyc and b/litellm/integrations/__pycache__/berrispend.cpython-311.pyc differ
diff --git a/litellm/integrations/__pycache__/helicone.cpython-311.pyc b/litellm/integrations/__pycache__/helicone.cpython-311.pyc
index 03de753b4..972c339ed 100644
Binary files a/litellm/integrations/__pycache__/helicone.cpython-311.pyc and b/litellm/integrations/__pycache__/helicone.cpython-311.pyc differ
diff --git a/litellm/integrations/__pycache__/supabase.cpython-311.pyc b/litellm/integrations/__pycache__/supabase.cpython-311.pyc
index c3f60037e..26cdede8f 100644
Binary files a/litellm/integrations/__pycache__/supabase.cpython-311.pyc and b/litellm/integrations/__pycache__/supabase.cpython-311.pyc differ
diff --git a/litellm/integrations/aispend.py b/litellm/integrations/aispend.py
index 6723a6227..2015d45dd 100644
--- a/litellm/integrations/aispend.py
+++ b/litellm/integrations/aispend.py
@@ -1,53 +1,121 @@
#### What this does ####
-# On success + failure, log events to aispend.io
+# On success + failure, log events to aispend.io
import dotenv, os
import requests
-dotenv.load_dotenv() # Loading env variables using dotenv
+
+dotenv.load_dotenv() # Loading env variables using dotenv
import traceback
import datetime
model_cost = {
- "gpt-3.5-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-35-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002}, # azure model name
- "gpt-3.5-turbo-0613": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-0301": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-35-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004}, # azure model name
- "gpt-3.5-turbo-16k-0613": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-4": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-0613": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-32k": {"max_tokens": 8000, "input_cost_per_token": 0.00006, "output_cost_per_token": 0.00012},
- "claude-instant-1": {"max_tokens": 100000, "input_cost_per_token": 0.00000163, "output_cost_per_token": 0.00000551},
- "claude-2": {"max_tokens": 100000, "input_cost_per_token": 0.00001102, "output_cost_per_token": 0.00003268},
- "text-bison-001": {"max_tokens": 8192, "input_cost_per_token": 0.000004, "output_cost_per_token": 0.000004},
- "chat-bison-001": {"max_tokens": 4096, "input_cost_per_token": 0.000002, "output_cost_per_token": 0.000002},
- "command-nightly": {"max_tokens": 4096, "input_cost_per_token": 0.000015, "output_cost_per_token": 0.000015},
+ "gpt-3.5-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-35-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ }, # azure model name
+ "gpt-3.5-turbo-0613": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-0301": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-35-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ }, # azure model name
+ "gpt-3.5-turbo-16k-0613": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-4": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-0613": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-32k": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.00006,
+ "output_cost_per_token": 0.00012,
+ },
+ "claude-instant-1": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00000163,
+ "output_cost_per_token": 0.00000551,
+ },
+ "claude-2": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00001102,
+ "output_cost_per_token": 0.00003268,
+ },
+ "text-bison-001": {
+ "max_tokens": 8192,
+ "input_cost_per_token": 0.000004,
+ "output_cost_per_token": 0.000004,
+ },
+ "chat-bison-001": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000002,
+ "output_cost_per_token": 0.000002,
+ },
+ "command-nightly": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000015,
+ "output_cost_per_token": 0.000015,
+ },
}
+
class AISpendLogger:
# Class variables or attributes
def __init__(self):
# Instance variables
self.account_id = os.getenv("AISPEND_ACCOUNT_ID")
self.api_key = os.getenv("AISPEND_API_KEY")
-
+
def price_calculator(self, model, response_obj, start_time, end_time):
# try and find if the model is in the model_cost map
# else default to the average of the costs
prompt_tokens_cost_usd_dollar = 0
completion_tokens_cost_usd_dollar = 0
if model in model_cost:
- prompt_tokens_cost_usd_dollar = model_cost[model]["input_cost_per_token"] * response_obj["usage"]["prompt_tokens"]
- completion_tokens_cost_usd_dollar = model_cost[model]["output_cost_per_token"] * response_obj["usage"]["completion_tokens"]
- elif "replicate" in model:
+ prompt_tokens_cost_usd_dollar = (
+ model_cost[model]["input_cost_per_token"]
+ * response_obj["usage"]["prompt_tokens"]
+ )
+ completion_tokens_cost_usd_dollar = (
+ model_cost[model]["output_cost_per_token"]
+ * response_obj["usage"]["completion_tokens"]
+ )
+ elif "replicate" in model:
# replicate models are charged based on time
# llama 2 runs on an nvidia a100 which costs $0.0032 per second - https://replicate.com/replicate/llama-2-70b-chat
- model_run_time = end_time - start_time # assuming time in seconds
+ model_run_time = end_time - start_time # assuming time in seconds
cost_usd_dollar = model_run_time * 0.0032
prompt_tokens_cost_usd_dollar = cost_usd_dollar / 2
completion_tokens_cost_usd_dollar = cost_usd_dollar / 2
else:
- # calculate average input cost
+ # calculate average input cost
input_cost_sum = 0
output_cost_sum = 0
for model in model_cost:
@@ -55,37 +123,52 @@ class AISpendLogger:
output_cost_sum += model_cost[model]["output_cost_per_token"]
avg_input_cost = input_cost_sum / len(model_cost.keys())
avg_output_cost = output_cost_sum / len(model_cost.keys())
- prompt_tokens_cost_usd_dollar = model_cost[model]["input_cost_per_token"] * response_obj["usage"]["prompt_tokens"]
- completion_tokens_cost_usd_dollar = model_cost[model]["output_cost_per_token"] * response_obj["usage"]["completion_tokens"]
+ prompt_tokens_cost_usd_dollar = (
+ model_cost[model]["input_cost_per_token"]
+ * response_obj["usage"]["prompt_tokens"]
+ )
+ completion_tokens_cost_usd_dollar = (
+ model_cost[model]["output_cost_per_token"]
+ * response_obj["usage"]["completion_tokens"]
+ )
return prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar
-
+
def log_event(self, model, response_obj, start_time, end_time, print_verbose):
# Method definition
try:
- print_verbose(f"AISpend Logging - Enters logging function for model {model}")
+ print_verbose(
+ f"AISpend Logging - Enters logging function for model {model}"
+ )
url = f"https://aispend.io/api/v1/accounts/{self.account_id}/data"
headers = {
- 'Authorization': f'Bearer {self.api_key}',
- 'Content-Type': 'application/json'
+ "Authorization": f"Bearer {self.api_key}",
+ "Content-Type": "application/json",
}
- response_timestamp = datetime.datetime.fromtimestamp(int(response_obj["created"])).strftime('%Y-%m-%d')
+ response_timestamp = datetime.datetime.fromtimestamp(
+ int(response_obj["created"])
+ ).strftime("%Y-%m-%d")
- prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar = self.price_calculator(model, response_obj, start_time, end_time)
+ (
+ prompt_tokens_cost_usd_dollar,
+ completion_tokens_cost_usd_dollar,
+ ) = self.price_calculator(model, response_obj, start_time, end_time)
prompt_tokens_cost_usd_cent = prompt_tokens_cost_usd_dollar * 100
completion_tokens_cost_usd_cent = completion_tokens_cost_usd_dollar * 100
- data = [{
- "requests": 1,
- "requests_context": 1,
- "context_tokens": response_obj["usage"]["prompt_tokens"],
- "requests_generated": 1,
- "generated_tokens": response_obj["usage"]["completion_tokens"],
- "recorded_date": response_timestamp,
- "model_id": response_obj["model"],
- "generated_tokens_cost_usd_cent": prompt_tokens_cost_usd_cent,
- "context_tokens_cost_usd_cent": completion_tokens_cost_usd_cent
- }]
+ data = [
+ {
+ "requests": 1,
+ "requests_context": 1,
+ "context_tokens": response_obj["usage"]["prompt_tokens"],
+ "requests_generated": 1,
+ "generated_tokens": response_obj["usage"]["completion_tokens"],
+ "recorded_date": response_timestamp,
+ "model_id": response_obj["model"],
+ "generated_tokens_cost_usd_cent": prompt_tokens_cost_usd_cent,
+ "context_tokens_cost_usd_cent": completion_tokens_cost_usd_cent,
+ }
+ ]
print_verbose(f"AISpend Logging - final data object: {data}")
except:
diff --git a/litellm/integrations/berrispend.py b/litellm/integrations/berrispend.py
index 1742bfed7..7d91ffca7 100644
--- a/litellm/integrations/berrispend.py
+++ b/litellm/integrations/berrispend.py
@@ -1,52 +1,120 @@
#### What this does ####
-# On success + failure, log events to aispend.io
+# On success + failure, log events to aispend.io
import dotenv, os
import requests
-dotenv.load_dotenv() # Loading env variables using dotenv
+
+dotenv.load_dotenv() # Loading env variables using dotenv
import traceback
import datetime
model_cost = {
- "gpt-3.5-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-35-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002}, # azure model name
- "gpt-3.5-turbo-0613": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-0301": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-35-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004}, # azure model name
- "gpt-3.5-turbo-16k-0613": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-4": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-0613": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-32k": {"max_tokens": 8000, "input_cost_per_token": 0.00006, "output_cost_per_token": 0.00012},
- "claude-instant-1": {"max_tokens": 100000, "input_cost_per_token": 0.00000163, "output_cost_per_token": 0.00000551},
- "claude-2": {"max_tokens": 100000, "input_cost_per_token": 0.00001102, "output_cost_per_token": 0.00003268},
- "text-bison-001": {"max_tokens": 8192, "input_cost_per_token": 0.000004, "output_cost_per_token": 0.000004},
- "chat-bison-001": {"max_tokens": 4096, "input_cost_per_token": 0.000002, "output_cost_per_token": 0.000002},
- "command-nightly": {"max_tokens": 4096, "input_cost_per_token": 0.000015, "output_cost_per_token": 0.000015},
+ "gpt-3.5-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-35-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ }, # azure model name
+ "gpt-3.5-turbo-0613": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-0301": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-35-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ }, # azure model name
+ "gpt-3.5-turbo-16k-0613": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-4": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-0613": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-32k": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.00006,
+ "output_cost_per_token": 0.00012,
+ },
+ "claude-instant-1": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00000163,
+ "output_cost_per_token": 0.00000551,
+ },
+ "claude-2": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00001102,
+ "output_cost_per_token": 0.00003268,
+ },
+ "text-bison-001": {
+ "max_tokens": 8192,
+ "input_cost_per_token": 0.000004,
+ "output_cost_per_token": 0.000004,
+ },
+ "chat-bison-001": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000002,
+ "output_cost_per_token": 0.000002,
+ },
+ "command-nightly": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000015,
+ "output_cost_per_token": 0.000015,
+ },
}
+
class BerriSpendLogger:
# Class variables or attributes
def __init__(self):
# Instance variables
self.account_id = os.getenv("BERRISPEND_ACCOUNT_ID")
-
+
def price_calculator(self, model, response_obj, start_time, end_time):
# try and find if the model is in the model_cost map
# else default to the average of the costs
prompt_tokens_cost_usd_dollar = 0
completion_tokens_cost_usd_dollar = 0
if model in model_cost:
- prompt_tokens_cost_usd_dollar = model_cost[model]["input_cost_per_token"] * response_obj["usage"]["prompt_tokens"]
- completion_tokens_cost_usd_dollar = model_cost[model]["output_cost_per_token"] * response_obj["usage"]["completion_tokens"]
- elif "replicate" in model:
+ prompt_tokens_cost_usd_dollar = (
+ model_cost[model]["input_cost_per_token"]
+ * response_obj["usage"]["prompt_tokens"]
+ )
+ completion_tokens_cost_usd_dollar = (
+ model_cost[model]["output_cost_per_token"]
+ * response_obj["usage"]["completion_tokens"]
+ )
+ elif "replicate" in model:
# replicate models are charged based on time
# llama 2 runs on an nvidia a100 which costs $0.0032 per second - https://replicate.com/replicate/llama-2-70b-chat
- model_run_time = end_time - start_time # assuming time in seconds
+ model_run_time = end_time - start_time # assuming time in seconds
cost_usd_dollar = model_run_time * 0.0032
prompt_tokens_cost_usd_dollar = cost_usd_dollar / 2
completion_tokens_cost_usd_dollar = cost_usd_dollar / 2
else:
- # calculate average input cost
+ # calculate average input cost
input_cost_sum = 0
output_cost_sum = 0
for model in model_cost:
@@ -54,42 +122,59 @@ class BerriSpendLogger:
output_cost_sum += model_cost[model]["output_cost_per_token"]
avg_input_cost = input_cost_sum / len(model_cost.keys())
avg_output_cost = output_cost_sum / len(model_cost.keys())
- prompt_tokens_cost_usd_dollar = model_cost[model]["input_cost_per_token"] * response_obj["usage"]["prompt_tokens"]
- completion_tokens_cost_usd_dollar = model_cost[model]["output_cost_per_token"] * response_obj["usage"]["completion_tokens"]
+ prompt_tokens_cost_usd_dollar = (
+ model_cost[model]["input_cost_per_token"]
+ * response_obj["usage"]["prompt_tokens"]
+ )
+ completion_tokens_cost_usd_dollar = (
+ model_cost[model]["output_cost_per_token"]
+ * response_obj["usage"]["completion_tokens"]
+ )
return prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar
-
- def log_event(self, model, messages, response_obj, start_time, end_time, print_verbose):
+
+ def log_event(
+ self, model, messages, response_obj, start_time, end_time, print_verbose
+ ):
# Method definition
try:
- print_verbose(f"BerriSpend Logging - Enters logging function for model {model}")
+ print_verbose(
+ f"BerriSpend Logging - Enters logging function for model {model}"
+ )
url = f"https://berrispend.berri.ai/spend"
- headers = {
- 'Content-Type': 'application/json'
- }
+ headers = {"Content-Type": "application/json"}
- prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar = self.price_calculator(model, response_obj, start_time, end_time)
- total_cost = prompt_tokens_cost_usd_dollar + completion_tokens_cost_usd_dollar
+ (
+ prompt_tokens_cost_usd_dollar,
+ completion_tokens_cost_usd_dollar,
+ ) = self.price_calculator(model, response_obj, start_time, end_time)
+ total_cost = (
+ prompt_tokens_cost_usd_dollar + completion_tokens_cost_usd_dollar
+ )
- response_time = (end_time-start_time).total_seconds()
+ response_time = (end_time - start_time).total_seconds()
if "response" in response_obj:
- data = [{
- "response_time": response_time,
- "model_id": response_obj["model"],
- "total_cost": total_cost,
- "messages": messages,
- "response": response_obj['choices'][0]['message']['content'],
- "account_id": self.account_id
- }]
+ data = [
+ {
+ "response_time": response_time,
+ "model_id": response_obj["model"],
+ "total_cost": total_cost,
+ "messages": messages,
+ "response": response_obj["choices"][0]["message"]["content"],
+ "account_id": self.account_id,
+ }
+ ]
elif "error" in response_obj:
- data = [{
- "response_time": response_time,
- "model_id": response_obj["model"],
- "total_cost": total_cost,
- "messages": messages,
- "error": response_obj['error'],
- "account_id": self.account_id
- }]
+ data = [
+ {
+ "response_time": response_time,
+ "model_id": response_obj["model"],
+ "total_cost": total_cost,
+ "messages": messages,
+ "error": response_obj["error"],
+ "account_id": self.account_id,
+ }
+ ]
print_verbose(f"BerriSpend Logging - final data object: {data}")
response = requests.post(url, headers=headers, json=data)
diff --git a/litellm/integrations/helicone.py b/litellm/integrations/helicone.py
index 9e74b246f..f9dff85db 100644
--- a/litellm/integrations/helicone.py
+++ b/litellm/integrations/helicone.py
@@ -2,19 +2,24 @@
# On success, logs events to Helicone
import dotenv, os
import requests
-dotenv.load_dotenv() # Loading env variables using dotenv
+
+dotenv.load_dotenv() # Loading env variables using dotenv
import traceback
+
+
class HeliconeLogger:
# Class variables or attributes
helicone_model_list = ["gpt", "claude"]
+
def __init__(self):
# Instance variables
self.provider_url = "https://api.openai.com/v1"
- self.key = os.getenv('HELICONE_API_KEY')
+ self.key = os.getenv("HELICONE_API_KEY")
def claude_mapping(self, model, messages, response_obj):
from anthropic import HUMAN_PROMPT, AI_PROMPT
- prompt = f"{HUMAN_PROMPT}"
+
+ prompt = f"{HUMAN_PROMPT}"
for message in messages:
if "role" in message:
if message["role"] == "user":
@@ -26,48 +31,84 @@ class HeliconeLogger:
prompt += f"{AI_PROMPT}"
claude_provider_request = {"model": model, "prompt": prompt}
- claude_response_obj = {"completion": response_obj['choices'][0]['message']['content'], "model": model, "stop_reason": "stop_sequence"}
+ claude_response_obj = {
+ "completion": response_obj["choices"][0]["message"]["content"],
+ "model": model,
+ "stop_reason": "stop_sequence",
+ }
return claude_provider_request, claude_response_obj
-
- def log_success(self, model, messages, response_obj, start_time, end_time, print_verbose):
+
+ def log_success(
+ self, model, messages, response_obj, start_time, end_time, print_verbose
+ ):
# Method definition
try:
- print_verbose(f"Helicone Logging - Enters logging function for model {model}")
- model = model if any(accepted_model in model for accepted_model in self.helicone_model_list) else "gpt-3.5-turbo"
+ print_verbose(
+ f"Helicone Logging - Enters logging function for model {model}"
+ )
+ model = (
+ model
+ if any(
+ accepted_model in model
+ for accepted_model in self.helicone_model_list
+ )
+ else "gpt-3.5-turbo"
+ )
provider_request = {"model": model, "messages": messages}
- if "claude" in model:
- provider_request, response_obj = self.claude_mapping(model=model, messages=messages, response_obj=response_obj)
+ if "claude" in model:
+ provider_request, response_obj = self.claude_mapping(
+ model=model, messages=messages, response_obj=response_obj
+ )
providerResponse = {
- "json": response_obj,
- "headers": {"openai-version": "2020-10-01"},
- "status": 200
+ "json": response_obj,
+ "headers": {"openai-version": "2020-10-01"},
+ "status": 200,
}
# Code to be executed
url = "https://api.hconeai.com/oai/v1/log"
headers = {
- 'Authorization': f'Bearer {self.key}',
- 'Content-Type': 'application/json'
+ "Authorization": f"Bearer {self.key}",
+ "Content-Type": "application/json",
}
start_time_seconds = int(start_time.timestamp())
- start_time_milliseconds = int((start_time.timestamp() - start_time_seconds) * 1000)
+ start_time_milliseconds = int(
+ (start_time.timestamp() - start_time_seconds) * 1000
+ )
end_time_seconds = int(end_time.timestamp())
- end_time_milliseconds = int((end_time.timestamp() - end_time_seconds) * 1000)
+ end_time_milliseconds = int(
+ (end_time.timestamp() - end_time_seconds) * 1000
+ )
data = {
- "providerRequest": {"url": self.provider_url, "json": provider_request, "meta": {"Helicone-Auth": f"Bearer {self.key}"}},
+ "providerRequest": {
+ "url": self.provider_url,
+ "json": provider_request,
+ "meta": {"Helicone-Auth": f"Bearer {self.key}"},
+ },
"providerResponse": providerResponse,
- "timing": {"startTime": {"seconds": start_time_seconds, "milliseconds": start_time_milliseconds}, "endTime": {"seconds": end_time_seconds, "milliseconds": end_time_milliseconds}} # {"seconds": .., "milliseconds": ..}
+ "timing": {
+ "startTime": {
+ "seconds": start_time_seconds,
+ "milliseconds": start_time_milliseconds,
+ },
+ "endTime": {
+ "seconds": end_time_seconds,
+ "milliseconds": end_time_milliseconds,
+ },
+ }, # {"seconds": .., "milliseconds": ..}
}
response = requests.post(url, headers=headers, json=data)
if response.status_code == 200:
print_verbose("Helicone Logging - Success!")
else:
- print_verbose(f"Helicone Logging - Error Request was not successful. Status Code: {response.status_code}")
+ print_verbose(
+ f"Helicone Logging - Error Request was not successful. Status Code: {response.status_code}"
+ )
print_verbose(f"Helicone Logging - Error {response.text}")
except:
# traceback.print_exc()
print_verbose(f"Helicone Logging Error - {traceback.format_exc()}")
- pass
\ No newline at end of file
+ pass
diff --git a/litellm/integrations/litedebugger.py b/litellm/integrations/litedebugger.py
new file mode 100644
index 000000000..a42692a3e
--- /dev/null
+++ b/litellm/integrations/litedebugger.py
@@ -0,0 +1,74 @@
+import requests, traceback, json
+class LiteDebugger:
+ def __init__(self):
+ self.api_url = "https://api.litellm.ai/debugger"
+ pass
+
+ def input_log_event(self, model, messages, end_user, litellm_call_id, print_verbose):
+ try:
+ print_verbose(
+ f"LiteLLMDebugger: Logging - Enters input logging function for model {model}"
+ )
+ litellm_data_obj = {
+ "model": model,
+ "messages": messages,
+ "end_user": end_user,
+ "status": "initiated",
+ "litellm_call_id": litellm_call_id
+ }
+ response = requests.post(url=self.api_url, headers={"content-type": "application/json"}, data=json.dumps(litellm_data_obj))
+ print_verbose(f"LiteDebugger: api response - {response.text}")
+ except:
+ print_verbose(f"LiteDebugger: Logging Error - {traceback.format_exc()}")
+ pass
+
+ def log_event(self, model,
+ messages,
+ end_user,
+ response_obj,
+ start_time,
+ end_time,
+ litellm_call_id,
+ print_verbose,):
+ try:
+ print_verbose(
+ f"LiteLLMDebugger: Logging - Enters input logging function for model {model}"
+ )
+ total_cost = 0 # [TODO] implement cost tracking
+ response_time = (end_time - start_time).total_seconds()
+ if "choices" in response_obj:
+ litellm_data_obj = {
+ "response_time": response_time,
+ "model": response_obj["model"],
+ "total_cost": total_cost,
+ "messages": messages,
+ "response": response_obj["choices"][0]["message"]["content"],
+ "end_user": end_user,
+ "litellm_call_id": litellm_call_id,
+ "status": "success"
+ }
+ print_verbose(
+ f"LiteDebugger: Logging - final data object: {litellm_data_obj}"
+ )
+ response = requests.post(url=self.api_url, headers={"content-type": "application/json"}, data=json.dumps(litellm_data_obj))
+ elif "error" in response_obj:
+ if "Unable to map your input to a model." in response_obj["error"]:
+ total_cost = 0
+ litellm_data_obj = {
+ "response_time": response_time,
+ "model": response_obj["model"],
+ "total_cost": total_cost,
+ "messages": messages,
+ "error": response_obj["error"],
+ "end_user": end_user,
+ "litellm_call_id": litellm_call_id,
+ "status": "failure"
+ }
+ print_verbose(
+ f"LiteDebugger: Logging - final data object: {litellm_data_obj}"
+ )
+ response = requests.post(url=self.api_url, headers={"content-type": "application/json"}, data=json.dumps(litellm_data_obj))
+ print_verbose(f"LiteDebugger: api response - {response.text}")
+ except:
+ print_verbose(f"LiteDebugger: Logging Error - {traceback.format_exc()}")
+ pass
\ No newline at end of file
diff --git a/litellm/integrations/supabase.py b/litellm/integrations/supabase.py
index 1ac28763f..3ea63f5c6 100644
--- a/litellm/integrations/supabase.py
+++ b/litellm/integrations/supabase.py
@@ -3,31 +3,94 @@
import dotenv, os
import requests
-dotenv.load_dotenv() # Loading env variables using dotenv
+
+dotenv.load_dotenv() # Loading env variables using dotenv
import traceback
import datetime, subprocess, sys
model_cost = {
- "gpt-3.5-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-35-turbo": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002}, # azure model name
- "gpt-3.5-turbo-0613": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-0301": {"max_tokens": 4000, "input_cost_per_token": 0.0000015, "output_cost_per_token": 0.000002},
- "gpt-3.5-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-35-turbo-16k": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004}, # azure model name
- "gpt-3.5-turbo-16k-0613": {"max_tokens": 16000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.000004},
- "gpt-4": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-0613": {"max_tokens": 8000, "input_cost_per_token": 0.000003, "output_cost_per_token": 0.00006},
- "gpt-4-32k": {"max_tokens": 8000, "input_cost_per_token": 0.00006, "output_cost_per_token": 0.00012},
- "claude-instant-1": {"max_tokens": 100000, "input_cost_per_token": 0.00000163, "output_cost_per_token": 0.00000551},
- "claude-2": {"max_tokens": 100000, "input_cost_per_token": 0.00001102, "output_cost_per_token": 0.00003268},
- "text-bison-001": {"max_tokens": 8192, "input_cost_per_token": 0.000004, "output_cost_per_token": 0.000004},
- "chat-bison-001": {"max_tokens": 4096, "input_cost_per_token": 0.000002, "output_cost_per_token": 0.000002},
- "command-nightly": {"max_tokens": 4096, "input_cost_per_token": 0.000015, "output_cost_per_token": 0.000015},
+ "gpt-3.5-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-35-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ }, # azure model name
+ "gpt-3.5-turbo-0613": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-0301": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002,
+ },
+ "gpt-3.5-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-35-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ }, # azure model name
+ "gpt-3.5-turbo-16k-0613": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004,
+ },
+ "gpt-4": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-0613": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006,
+ },
+ "gpt-4-32k": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.00006,
+ "output_cost_per_token": 0.00012,
+ },
+ "claude-instant-1": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00000163,
+ "output_cost_per_token": 0.00000551,
+ },
+ "claude-2": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00001102,
+ "output_cost_per_token": 0.00003268,
+ },
+ "text-bison-001": {
+ "max_tokens": 8192,
+ "input_cost_per_token": 0.000004,
+ "output_cost_per_token": 0.000004,
+ },
+ "chat-bison-001": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000002,
+ "output_cost_per_token": 0.000002,
+ },
+ "command-nightly": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000015,
+ "output_cost_per_token": 0.000015,
+ },
}
+
class Supabase:
# Class variables or attributes
supabase_table_name = "request_logs"
+
def __init__(self):
# Instance variables
self.supabase_url = os.getenv("SUPABASE_URL")
@@ -35,9 +98,11 @@ class Supabase:
try:
import supabase
except ImportError:
- subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'supabase'])
+ subprocess.check_call([sys.executable, "-m", "pip", "install", "supabase"])
import supabase
- self.supabase_client = supabase.create_client(self.supabase_url, self.supabase_key)
+ self.supabase_client = supabase.create_client(
+ self.supabase_url, self.supabase_key
+ )
def price_calculator(self, model, response_obj, start_time, end_time):
# try and find if the model is in the model_cost map
@@ -45,17 +110,23 @@ class Supabase:
prompt_tokens_cost_usd_dollar = 0
completion_tokens_cost_usd_dollar = 0
if model in model_cost:
- prompt_tokens_cost_usd_dollar = model_cost[model]["input_cost_per_token"] * response_obj["usage"]["prompt_tokens"]
- completion_tokens_cost_usd_dollar = model_cost[model]["output_cost_per_token"] * response_obj["usage"]["completion_tokens"]
- elif "replicate" in model:
+ prompt_tokens_cost_usd_dollar = (
+ model_cost[model]["input_cost_per_token"]
+ * response_obj["usage"]["prompt_tokens"]
+ )
+ completion_tokens_cost_usd_dollar = (
+ model_cost[model]["output_cost_per_token"]
+ * response_obj["usage"]["completion_tokens"]
+ )
+ elif "replicate" in model:
# replicate models are charged based on time
# llama 2 runs on an nvidia a100 which costs $0.0032 per second - https://replicate.com/replicate/llama-2-70b-chat
- model_run_time = end_time - start_time # assuming time in seconds
+ model_run_time = end_time - start_time # assuming time in seconds
cost_usd_dollar = model_run_time * 0.0032
prompt_tokens_cost_usd_dollar = cost_usd_dollar / 2
completion_tokens_cost_usd_dollar = cost_usd_dollar / 2
else:
- # calculate average input cost
+ # calculate average input cost
input_cost_sum = 0
output_cost_sum = 0
for model in model_cost:
@@ -63,41 +134,104 @@ class Supabase:
output_cost_sum += model_cost[model]["output_cost_per_token"]
avg_input_cost = input_cost_sum / len(model_cost.keys())
avg_output_cost = output_cost_sum / len(model_cost.keys())
- prompt_tokens_cost_usd_dollar = model_cost[model]["input_cost_per_token"] * response_obj["usage"]["prompt_tokens"]
- completion_tokens_cost_usd_dollar = model_cost[model]["output_cost_per_token"] * response_obj["usage"]["completion_tokens"]
+ prompt_tokens_cost_usd_dollar = (
+ model_cost[model]["input_cost_per_token"]
+ * response_obj["usage"]["prompt_tokens"]
+ )
+ completion_tokens_cost_usd_dollar = (
+ model_cost[model]["output_cost_per_token"]
+ * response_obj["usage"]["completion_tokens"]
+ )
return prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar
-
- def log_event(self, model, messages, end_user, response_obj, start_time, end_time, print_verbose):
+
+ def input_log_event(self, model, messages, end_user, litellm_call_id, print_verbose):
try:
- print_verbose(f"Supabase Logging - Enters logging function for model {model}, response_obj: {response_obj}")
+ print_verbose(
+ f"Supabase Logging - Enters input logging function for model {model}"
+ )
+ supabase_data_obj = {
+ "model": model,
+ "messages": messages,
+ "end_user": end_user,
+ "status": "initiated",
+ "litellm_call_id": litellm_call_id
+ }
+ data, count = (
+ self.supabase_client.table(self.supabase_table_name)
+ .insert(supabase_data_obj)
+ .execute()
+ )
+ print(f"data: {data}")
+ except:
+ print_verbose(f"Supabase Logging Error - {traceback.format_exc()}")
+ pass
- prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar = self.price_calculator(model, response_obj, start_time, end_time)
- total_cost = prompt_tokens_cost_usd_dollar + completion_tokens_cost_usd_dollar
+ def log_event(
+ self,
+ model,
+ messages,
+ end_user,
+ response_obj,
+ start_time,
+ end_time,
+ litellm_call_id,
+ print_verbose,
+ ):
+ try:
+ print_verbose(
+ f"Supabase Logging - Enters logging function for model {model}, response_obj: {response_obj}"
+ )
- response_time = (end_time-start_time).total_seconds()
+ (
+ prompt_tokens_cost_usd_dollar,
+ completion_tokens_cost_usd_dollar,
+ ) = self.price_calculator(model, response_obj, start_time, end_time)
+ total_cost = (
+ prompt_tokens_cost_usd_dollar + completion_tokens_cost_usd_dollar
+ )
+
+ response_time = (end_time - start_time).total_seconds()
if "choices" in response_obj:
supabase_data_obj = {
"response_time": response_time,
"model": response_obj["model"],
- "total_cost": total_cost,
+ "total_cost": total_cost,
"messages": messages,
- "response": response_obj['choices'][0]['message']['content'],
- "end_user": end_user
+ "response": response_obj["choices"][0]["message"]["content"],
+ "end_user": end_user,
+ "litellm_call_id": litellm_call_id,
+ "status": "success"
}
- print_verbose(f"Supabase Logging - final data object: {supabase_data_obj}")
- data, count = self.supabase_client.table(self.supabase_table_name).insert(supabase_data_obj).execute()
+ print_verbose(
+ f"Supabase Logging - final data object: {supabase_data_obj}"
+ )
+ data, count = (
+ self.supabase_client.table(self.supabase_table_name)
+ .upsert(supabase_data_obj)
+ .execute()
+ )
elif "error" in response_obj:
+ if "Unable to map your input to a model." in response_obj["error"]:
+ total_cost = 0
supabase_data_obj = {
"response_time": response_time,
"model": response_obj["model"],
- "total_cost": total_cost,
+ "total_cost": total_cost,
"messages": messages,
- "error": response_obj['error'],
- "end_user": end_user
+ "error": response_obj["error"],
+ "end_user": end_user,
+ "litellm_call_id": litellm_call_id,
+ "status": "failure"
}
- print_verbose(f"Supabase Logging - final data object: {supabase_data_obj}")
- data, count = self.supabase_client.table(self.supabase_table_name).insert(supabase_data_obj).execute()
-
+ print_verbose(
+ f"Supabase Logging - final data object: {supabase_data_obj}"
+ )
+ data, count = (
+ self.supabase_client.table(self.supabase_table_name)
+ .upsert(supabase_data_obj)
+ .execute()
+ )
+
except:
# traceback.print_exc()
print_verbose(f"Supabase Logging Error - {traceback.format_exc()}")
diff --git a/litellm/llms/__init__.py b/litellm/llms/__init__.py
index b9742821a..b6e690fd5 100644
--- a/litellm/llms/__init__.py
+++ b/litellm/llms/__init__.py
@@ -1 +1 @@
-from . import *
\ No newline at end of file
+from . import *
diff --git a/litellm/llms/anthropic.py b/litellm/llms/anthropic.py
index c617a0ae0..fecc655f2 100644
--- a/litellm/llms/anthropic.py
+++ b/litellm/llms/anthropic.py
@@ -1,59 +1,78 @@
import os, json
from enum import Enum
import requests
-from litellm import logging
-import time
+import time
from typing import Callable
+from litellm.utils import ModelResponse
+
class AnthropicConstants(Enum):
HUMAN_PROMPT = "\n\nHuman:"
AI_PROMPT = "\n\nAssistant:"
+
class AnthropicError(Exception):
def __init__(self, status_code, message):
self.status_code = status_code
self.message = message
- super().__init__(self.message) # Call the base class constructor with the parameters it needs
+ super().__init__(
+ self.message
+ ) # Call the base class constructor with the parameters it needs
-class AnthropicLLM:
-
- def __init__(self, encoding, default_max_tokens_to_sample, api_key=None):
+
+class AnthropicLLM:
+ def __init__(self, encoding, default_max_tokens_to_sample, logging_obj, api_key=None):
self.encoding = encoding
self.default_max_tokens_to_sample = default_max_tokens_to_sample
self.completion_url = "https://api.anthropic.com/v1/complete"
+ self.api_key = api_key
+ self.logging_obj = logging_obj
self.validate_environment(api_key=api_key)
-
- def validate_environment(self, api_key): # set up the environment required to run the model
- # set the api key
- try:
- self.api_key = os.getenv("ANTHROPIC_API_KEY") if "ANTHROPIC_API_KEY" in os.environ else api_key
- if self.api_key == None:
- raise Exception
-
- self.headers = {
- "accept": "application/json",
- "anthropic-version": "2023-06-01",
- "content-type": "application/json",
- "x-api-key": self.api_key
- }
- except:
- raise ValueError("Missing Anthropic API Key - A call is being made to anthropic but no key is set either in the environment variables or via params")
- pass
+ def validate_environment(
+ self, api_key
+ ): # set up the environment required to run the model
+ # set the api key
+ if self.api_key == None:
+ raise ValueError(
+ "Missing Anthropic API Key - A call is being made to anthropic but no key is set either in the environment variables or via params"
+ )
+ self.api_key = api_key
+ self.headers = {
+ "accept": "application/json",
+ "anthropic-version": "2023-06-01",
+ "content-type": "application/json",
+ "x-api-key": self.api_key,
+ }
- def completion(self, model: str, messages: list, model_response: dict, print_verbose: Callable, optional_params=None, litellm_params=None, logger_fn=None): # logic for parsing in - calling - parsing out model completion calls
+ def completion(
+ self,
+ model: str,
+ messages: list,
+ model_response: ModelResponse,
+ print_verbose: Callable,
+ optional_params=None,
+ litellm_params=None,
+ logger_fn=None,
+ ): # logic for parsing in - calling - parsing out model completion calls
model = model
prompt = f"{AnthropicConstants.HUMAN_PROMPT.value}"
for message in messages:
if "role" in message:
if message["role"] == "user":
- prompt += f"{AnthropicConstants.HUMAN_PROMPT.value}{message['content']}"
+ prompt += (
+ f"{AnthropicConstants.HUMAN_PROMPT.value}{message['content']}"
+ )
else:
- prompt += f"{AnthropicConstants.AI_PROMPT.value}{message['content']}"
+ prompt += (
+ f"{AnthropicConstants.AI_PROMPT.value}{message['content']}"
+ )
else:
prompt += f"{AnthropicConstants.HUMAN_PROMPT.value}{message['content']}"
prompt += f"{AnthropicConstants.AI_PROMPT.value}"
- if "max_tokens" in optional_params and optional_params["max_tokens"] != float('inf'):
+ if "max_tokens" in optional_params and optional_params["max_tokens"] != float(
+ "inf"
+ ):
max_tokens = optional_params["max_tokens"]
else:
max_tokens = self.default_max_tokens_to_sample
@@ -61,39 +80,51 @@ class AnthropicLLM:
"model": model,
"prompt": prompt,
"max_tokens_to_sample": max_tokens,
- **optional_params
+ **optional_params,
}
## LOGGING
- logging(model=model, input=prompt, additional_args={"litellm_params": litellm_params, "optional_params": optional_params}, logger_fn=logger_fn)
+ self.logging_obj.pre_call(input=prompt, api_key=self.api_key, additional_args={"complete_input_dict": data})
## COMPLETION CALL
- response = requests.post(self.completion_url, headers=self.headers, data=json.dumps(data))
+ response = requests.post(
+ self.completion_url, headers=self.headers, data=json.dumps(data)
+ )
if "stream" in optional_params and optional_params["stream"] == True:
return response.iter_lines()
else:
## LOGGING
- logging(model=model, input=prompt, additional_args={"litellm_params": litellm_params, "optional_params": optional_params, "original_response": response.text}, logger_fn=logger_fn)
+ self.logging_obj.post_call(input=prompt, api_key=self.api_key, original_response=response.text, additional_args={"complete_input_dict": data})
print_verbose(f"raw model_response: {response.text}")
## RESPONSE OBJECT
completion_response = response.json()
if "error" in completion_response:
- raise AnthropicError(message=completion_response["error"], status_code=response.status_code)
+ raise AnthropicError(
+ message=completion_response["error"],
+ status_code=response.status_code,
+ )
else:
- model_response["choices"][0]["message"]["content"] = completion_response["completion"]
-
+ model_response["choices"][0]["message"][
+ "content"
+ ] = completion_response["completion"]
+
## CALCULATING USAGE
- prompt_tokens = len(self.encoding.encode(prompt)) ##[TODO] use the anthropic tokenizer here
- completion_tokens = len(self.encoding.encode(model_response["choices"][0]["message"]["content"])) ##[TODO] use the anthropic tokenizer here
-
-
+ prompt_tokens = len(
+ self.encoding.encode(prompt)
+ ) ##[TODO] use the anthropic tokenizer here
+ completion_tokens = len(
+ self.encoding.encode(model_response["choices"][0]["message"]["content"])
+ ) ##[TODO] use the anthropic tokenizer here
+
model_response["created"] = time.time()
model_response["model"] = model
model_response["usage"] = {
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
- "total_tokens": prompt_tokens + completion_tokens
- }
+ "total_tokens": prompt_tokens + completion_tokens,
+ }
return model_response
-
- def embedding(): # logic for parsing in - calling - parsing out model embedding calls
- pass
\ No newline at end of file
+
+ def embedding(
+ self,
+ ): # logic for parsing in - calling - parsing out model embedding calls
+ pass
diff --git a/litellm/llms/base.py b/litellm/llms/base.py
index 368df9624..2c05a00c1 100644
--- a/litellm/llms/base.py
+++ b/litellm/llms/base.py
@@ -1,11 +1,16 @@
## This is a template base class to be used for adding new LLM providers via API calls
-class BaseLLM():
- def validate_environment(): # set up the environment required to run the model
- pass
- def completion(): # logic for parsing in - calling - parsing out model completion calls
+class BaseLLM:
+ def validate_environment(self): # set up the environment required to run the model
pass
- def embedding(): # logic for parsing in - calling - parsing out model embedding calls
- pass
\ No newline at end of file
+ def completion(
+ self,
+ ): # logic for parsing in - calling - parsing out model completion calls
+ pass
+
+ def embedding(
+ self,
+ ): # logic for parsing in - calling - parsing out model embedding calls
+ pass
diff --git a/litellm/llms/huggingface_restapi.py b/litellm/llms/huggingface_restapi.py
index 50bb9451a..624fb4f05 100644
--- a/litellm/llms/huggingface_restapi.py
+++ b/litellm/llms/huggingface_restapi.py
@@ -2,39 +2,60 @@
import os, json
from enum import Enum
import requests
-from litellm import logging
-import time
+import time
from typing import Callable
+from litellm.utils import ModelResponse
+from typing import Optional
+
class HuggingfaceError(Exception):
def __init__(self, status_code, message):
self.status_code = status_code
self.message = message
- super().__init__(self.message) # Call the base class constructor with the parameters it needs
+ super().__init__(
+ self.message
+ ) # Call the base class constructor with the parameters it needs
-class HuggingfaceRestAPILLM():
- def __init__(self, encoding, api_key=None) -> None:
+
+class HuggingfaceRestAPILLM:
+ def __init__(self, encoding, logging_obj, api_key=None) -> None:
self.encoding = encoding
+ self.logging_obj = logging_obj
self.validate_environment(api_key=api_key)
- def validate_environment(self, api_key): # set up the environment required to run the model
+ def validate_environment(
+ self, api_key
+ ): # set up the environment required to run the model
self.headers = {
"content-type": "application/json",
}
# get the api key if it exists in the environment or is passed in, but don't require it
- self.api_key = os.getenv("HF_TOKEN") if "HF_TOKEN" in os.environ else api_key
+ self.api_key = api_key
if self.api_key != None:
- self.headers["Authorization"] = f"Bearer {self.api_key}"
+ self.headers["Authorization"] = f"Bearer {self.api_key}"
- def completion(self, model: str, messages: list, custom_api_base: str, model_response: dict, print_verbose: Callable, optional_params=None, litellm_params=None, logger_fn=None): # logic for parsing in - calling - parsing out model completion calls
+ def completion(
+ self,
+ model: str,
+ messages: list,
+ custom_api_base: str,
+ model_response: ModelResponse,
+ print_verbose: Callable,
+ optional_params=None,
+ litellm_params=None,
+ logger_fn=None,
+ ): # logic for parsing in - calling - parsing out model completion calls
+ completion_url: str = ""
if custom_api_base:
completion_url = custom_api_base
elif "HF_API_BASE" in os.environ:
- completion_url = os.getenv("HF_API_BASE")
+ completion_url = os.getenv("HF_API_BASE", "")
else:
completion_url = f"https://api-inference.huggingface.co/models/{model}"
prompt = ""
- if "meta-llama" in model and "chat" in model: # use the required special tokens for meta-llama - https://huggingface.co/blog/llama2#how-to-prompt-llama-2
+ if (
+ "meta-llama" in model and "chat" in model
+ ): # use the required special tokens for meta-llama - https://huggingface.co/blog/llama2#how-to-prompt-llama-2
prompt = "