phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-26 19:24:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Fix wrong import and use space_id instead of space_key for Arize integration

Browse source
This commit is contained in:
Nate Mar 2025-03-17 20:37:28 -07:00
parent f8ac675321
commit 0dfc21e80a
11 changed files with 221 additions and 186 deletions
Download patch file Download diff file Expand all files Collapse all files

7
cookbook/logging_observability/LiteLLM_Arize.ipynb vendored
View file

@ -105,11 +105,16 @@
"import os\n", "import os\n",
"from getpass import getpass\n", "from getpass import getpass\n",
"\n", "\n",
"os.environ[\"ARIZE_SPACE_KEY\"] = getpass(\"Enter your Arize space key: \")\n", "os.environ[\"ARIZE_SPACE_ID\"] = getpass(\"Enter your Arize space id: \")\n",
"os.environ[\"ARIZE_API_KEY\"] = getpass(\"Enter your Arize API key: \")\n", "os.environ[\"ARIZE_API_KEY\"] = getpass(\"Enter your Arize API key: \")\n",
"os.environ['OPENAI_API_KEY']= getpass(\"Enter your OpenAI API key: \")" "os.environ['OPENAI_API_KEY']= getpass(\"Enter your OpenAI API key: \")"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},

16
docs/my-website/docs/observability/arize_integration.md
View file

@ -11,12 +11,12 @@ https://github.com/BerriAI/litellm
::: :::
## Pre-Requisites ## Pre-Requisites
Make an account on [Arize AI](https://app.arize.com/auth/login) Make an account on [Arize AI](https://app.arize.com/auth/login)
## Quick Start ## Quick Start
Use just 2 lines of code, to instantly log your responses **across all providers** with arize Use just 2 lines of code, to instantly log your responses **across all providers** with arize
You can also use the instrumentor option instead of the callback, which you can find [here](https://docs.arize.com/arize/llm-tracing/tracing-integrations-auto/litellm). You can also use the instrumentor option instead of the callback, which you can find [here](https://docs.arize.com/arize/llm-tracing/tracing-integrations-auto/litellm).
@ -24,11 +24,12 @@ You can also use the instrumentor option instead of the callback, which you can
```python ```python
litellm.callbacks = ["arize"] litellm.callbacks = ["arize"]
``` ```
```python ```python
import litellm import litellm
import os import os
os.environ["ARIZE_SPACE_KEY"] = "" os.environ["ARIZE_SPACE_ID"] = ""
os.environ["ARIZE_API_KEY"] = "" os.environ["ARIZE_API_KEY"] = ""
# LLM API Keys # LLM API Keys
@ -48,7 +49,6 @@ response = litellm.completion(
### Using with LiteLLM Proxy ### Using with LiteLLM Proxy
```yaml ```yaml
model_list: model_list:
- model_name: gpt-4 - model_name: gpt-4
@ -61,10 +61,10 @@ litellm_settings:
callbacks: ["arize"] callbacks: ["arize"]
environment_variables: environment_variables:
ARIZE_SPACE_KEY: "d0*****" ARIZE_SPACE_ID: "d0*****"
ARIZE_API_KEY: "141a****" ARIZE_API_KEY: "141a****"
ARIZE_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize GRPC api endpoint ARIZE_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize GRPC api endpoint
ARIZE_HTTP_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize HTTP api endpoint. Set either this or ARIZE_ENDPOINT ARIZE_HTTP_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize HTTP api endpoint. Set either this or ARIZE_ENDPOINT
``` ```
## Support & Talk to Founders ## Support & Talk to Founders

2
docs/my-website/docs/proxy/config_settings.md
View file

@ -302,7 +302,7 @@ router_settings:
| AISPEND_API_KEY | API Key for AI Spend | AISPEND_API_KEY | API Key for AI Spend
| ALLOWED_EMAIL_DOMAINS | List of email domains allowed for access | ALLOWED_EMAIL_DOMAINS | List of email domains allowed for access
| ARIZE_API_KEY | API key for Arize platform integration | ARIZE_API_KEY | API key for Arize platform integration
| ARIZE_SPACE_KEY | Space key for Arize platform | ARIZE_SPACE_ID | Space key for Arize platform
| ARGILLA_BATCH_SIZE | Batch size for Argilla logging | ARGILLA_BATCH_SIZE | Batch size for Argilla logging
| ARGILLA_API_KEY | API key for Argilla platform | ARGILLA_API_KEY | API key for Argilla platform
| ARGILLA_SAMPLING_RATE | Sampling rate for Argilla logging | ARGILLA_SAMPLING_RATE | Sampling rate for Argilla logging

228
docs/my-website/docs/proxy/logging.md
View file

@ -17,8 +17,6 @@ Log Proxy input, output, and exceptions using:
- DynamoDB - DynamoDB
- etc. - etc.
## Getting the LiteLLM Call ID ## Getting the LiteLLM Call ID
LiteLLM generates a unique `call_id` for each request. This `call_id` can be LiteLLM generates a unique `call_id` for each request. This `call_id` can be
@ -52,18 +50,17 @@ A number of these headers could be useful for troubleshooting, but the
`x-litellm-call-id` is the one that is most useful for tracking a request across `x-litellm-call-id` is the one that is most useful for tracking a request across
components in your system, including in logging tools. components in your system, including in logging tools.
## Logging Features ## Logging Features
### Conditional Logging by Virtual Keys, Teams ### Conditional Logging by Virtual Keys, Teams
Use this to: Use this to:
1. Conditionally enable logging for some virtual keys/teams 1. Conditionally enable logging for some virtual keys/teams
2. Set different logging providers for different virtual keys/teams 2. Set different logging providers for different virtual keys/teams
[👉 **Get Started** - Team/Key Based Logging](team_logging) [👉 **Get Started** - Team/Key Based Logging](team_logging)
### Redacting UserAPIKeyInfo ### Redacting UserAPIKeyInfo
Redact information about the user api key (hashed token, user_id, team id, etc.), from logs. Redact information about the user api key (hashed token, user_id, team id, etc.), from logs.
@ -76,7 +73,6 @@ litellm_settings:
redact_user_api_key_info: true redact_user_api_key_info: true
``` ```
### Redact Messages, Response Content ### Redact Messages, Response Content
Set `litellm.turn_off_message_logging=True` This will prevent the messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tracked. Set `litellm.turn_off_message_logging=True` This will prevent the messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tracked.
@ -86,6 +82,7 @@ Set `litellm.turn_off_message_logging=True` This will prevent the messages and r
<TabItem value="global" label="Global"> <TabItem value="global" label="Global">
**1. Setup config.yaml ** **1. Setup config.yaml **
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -97,6 +94,7 @@ litellm_settings:
``` ```
**2. Send request** **2. Send request**
```shell ```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
@ -111,8 +109,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}' }'
``` ```
</TabItem> </TabItem>
<TabItem value="dynamic" label="Per Request"> <TabItem value="dynamic" label="Per Request">
@ -170,13 +166,11 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
<Image img={require('../../img/message_redaction_spend_logs.png')} /> <Image img={require('../../img/message_redaction_spend_logs.png')} />
### Disable Message Redaction ### Disable Message Redaction
If you have `litellm.turn_on_message_logging` turned on, you can override it for specific requests by If you have `litellm.turn_on_message_logging` turned on, you can override it for specific requests by
setting a request header `LiteLLM-Disable-Message-Redaction: true`. setting a request header `LiteLLM-Disable-Message-Redaction: true`.
```shell ```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
@ -192,7 +186,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}' }'
``` ```
### Turn off all tracking/logging ### Turn off all tracking/logging
For some use cases, you may want to turn off all tracking/logging. You can do this by passing `no-log=True` in the request body. For some use cases, you may want to turn off all tracking/logging. You can do this by passing `no-log=True` in the request body.
@ -205,6 +198,7 @@ Disable this by setting `global_disable_no_log_param:true` in your config.yaml f
litellm_settings: litellm_settings:
global_disable_no_log_param: True global_disable_no_log_param: True
``` ```
::: :::
<Tabs> <Tabs>
@ -268,7 +262,6 @@ print(response)
LiteLLM.Info: "no-log request, skipping logging" LiteLLM.Info: "no-log request, skipping logging"
``` ```
## What gets logged? ## What gets logged?
Found under `kwargs["standard_logging_object"]`. This is a standard payload, logged for every response. Found under `kwargs["standard_logging_object"]`. This is a standard payload, logged for every response.
@ -431,10 +424,8 @@ print(response)
Set `tags` as part of your request body Set `tags` as part of your request body
<Tabs> <Tabs>
<TabItem value="openai" label="OpenAI Python v1.0.0+"> <TabItem value="openai" label="OpenAI Python v1.0.0+">
```python ```python
@ -462,6 +453,7 @@ response = client.chat.completions.create(
print(response) print(response)
``` ```
</TabItem> </TabItem>
<TabItem value="Curl" label="Curl Request"> <TabItem value="Curl" label="Curl Request">
@ -486,6 +478,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
} }
}' }'
``` ```
</TabItem> </TabItem>
<TabItem value="langchain" label="Langchain"> <TabItem value="langchain" label="Langchain">
@ -528,28 +521,26 @@ print(response)
</TabItem> </TabItem>
</Tabs> </Tabs>
### LiteLLM Tags - `cache_hit`, `cache_key` ### LiteLLM Tags - `cache_hit`, `cache_key`
Use this if you want to control which LiteLLM-specific fields are logged as tags by the LiteLLM proxy. By default LiteLLM Proxy logs no LiteLLM-specific fields Use this if you want to control which LiteLLM-specific fields are logged as tags by the LiteLLM proxy. By default LiteLLM Proxy logs no LiteLLM-specific fields
| LiteLLM specific field | Description | Example Value | | LiteLLM specific field | Description | Example Value |
|------------------------|-------------------------------------------------------|------------------------------------------------| | ------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------- |
| `cache_hit` | Indicates whether a cache hit occured (True) or not (False) | `true`, `false` | | `cache_hit` | Indicates whether a cache hit occured (True) or not (False) | `true`, `false` |
| `cache_key` | The Cache key used for this request | `d2b758c****`| | `cache_key` | The Cache key used for this request | `d2b758c****` |
| `proxy_base_url` | The base URL for the proxy server, the value of env var `PROXY_BASE_URL` on your server | `https://proxy.example.com`| | `proxy_base_url` | The base URL for the proxy server, the value of env var `PROXY_BASE_URL` on your server | `https://proxy.example.com` |
| `user_api_key_alias` | An alias for the LiteLLM Virtual Key.| `prod-app1` | | `user_api_key_alias` | An alias for the LiteLLM Virtual Key. | `prod-app1` |
| `user_api_key_user_id` | The unique ID associated with a user's API key. | `user_123`, `user_456` | | `user_api_key_user_id` | The unique ID associated with a user's API key. | `user_123`, `user_456` |
| `user_api_key_user_email` | The email associated with a user's API key. | `user@example.com`, `admin@example.com` | | `user_api_key_user_email` | The email associated with a user's API key. | `user@example.com`, `admin@example.com` |
| `user_api_key_team_alias` | An alias for a team associated with an API key. | `team_alpha`, `dev_team` | | `user_api_key_team_alias` | An alias for a team associated with an API key. | `team_alpha`, `dev_team` |
**Usage** **Usage**
Specify `langfuse_default_tags` to control what litellm fields get logged on Langfuse Specify `langfuse_default_tags` to control what litellm fields get logged on Langfuse
Example config.yaml Example config.yaml
```yaml ```yaml
model_list: model_list:
- model_name: gpt-4 - model_name: gpt-4
@ -562,7 +553,18 @@ litellm_settings:
success_callback: ["langfuse"] success_callback: ["langfuse"]
# 👇 Key Change # 👇 Key Change
langfuse_default_tags: ["cache_hit", "cache_key", "proxy_base_url", "user_api_key_alias", "user_api_key_user_id", "user_api_key_user_email", "user_api_key_team_alias", "semantic-similarity", "proxy_base_url"] langfuse_default_tags:
[
"cache_hit",
"cache_key",
"proxy_base_url",
"user_api_key_alias",
"user_api_key_user_id",
"user_api_key_user_email",
"user_api_key_team_alias",
"semantic-similarity",
"proxy_base_url",
]
``` ```
### View POST sent from LiteLLM to provider ### View POST sent from LiteLLM to provider
@ -730,30 +732,30 @@ This is the Span from OTEL Logging
```json ```json
{ {
"name": "litellm-acompletion", "name": "litellm-acompletion",
"context": { "context": {
"trace_id": "0x8d354e2346060032703637a0843b20a3", "trace_id": "0x8d354e2346060032703637a0843b20a3",
"span_id": "0xd8d3476a2eb12724", "span_id": "0xd8d3476a2eb12724",
"trace_state": "[]" "trace_state": "[]"
}, },
"kind": "SpanKind.INTERNAL", "kind": "SpanKind.INTERNAL",
"parent_id": null, "parent_id": null,
"start_time": "2024-06-04T19:46:56.415888Z", "start_time": "2024-06-04T19:46:56.415888Z",
"end_time": "2024-06-04T19:46:56.790278Z", "end_time": "2024-06-04T19:46:56.790278Z",
"status": { "status": {
"status_code": "OK" "status_code": "OK"
}, },
"attributes": {
"model": "llama3-8b-8192"
},
"events": [],
"links": [],
"resource": {
"attributes": { "attributes": {
"model": "llama3-8b-8192" "service.name": "litellm"
}, },
"events": [], "schema_url": ""
"links": [], }
"resource": {
"attributes": {
"service.name": "litellm"
},
"schema_url": ""
}
} }
``` ```
@ -965,6 +967,7 @@ callback_settings:
``` ```
### Traceparent Header ### Traceparent Header
##### Context propagation across Services `Traceparent HTTP Header` ##### Context propagation across Services `Traceparent HTTP Header`
❓ Use this when you want to **pass information about the incoming request in a distributed tracing system** ❓ Use this when you want to **pass information about the incoming request in a distributed tracing system**
@ -1042,25 +1045,23 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
::: :::
| Property | Details |
| Property | Details | | ---------------------------- | -------------------------------------------------------------- |
|----------|---------| | Description | Log LLM Input/Output to cloud storage buckets |
| Description | Log LLM Input/Output to cloud storage buckets | | Load Test Benchmarks | [Benchmarks](https://docs.litellm.ai/docs/benchmarks) |
| Load Test Benchmarks | [Benchmarks](https://docs.litellm.ai/docs/benchmarks) |
| Google Docs on Cloud Storage | [Google Cloud Storage](https://cloud.google.com/storage?hl=en) | | Google Docs on Cloud Storage | [Google Cloud Storage](https://cloud.google.com/storage?hl=en) |
#### Usage #### Usage
1. Add `gcs_bucket` to LiteLLM Config.yaml 1. Add `gcs_bucket` to LiteLLM Config.yaml
```yaml ```yaml
model_list: model_list:
- litellm_params: - litellm_params:
api_base: https://exampleopenaiendpoint-production.up.railway.app/ api_base: https://exampleopenaiendpoint-production.up.railway.app/
api_key: my-fake-key api_key: my-fake-key
model: openai/my-fake-model model: openai/my-fake-model
model_name: fake-openai-endpoint model_name: fake-openai-endpoint
litellm_settings: litellm_settings:
callbacks: ["gcs_bucket"] # 👈 KEY CHANGE # 👈 KEY CHANGE callbacks: ["gcs_bucket"] # 👈 KEY CHANGE # 👈 KEY CHANGE
@ -1096,7 +1097,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
' '
``` ```
#### Expected Logs on GCS Buckets #### Expected Logs on GCS Buckets
<Image img={require('../../img/gcs_bucket.png')} /> <Image img={require('../../img/gcs_bucket.png')} />
@ -1105,7 +1105,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
[**The standard logging object is logged on GCS Bucket**](../proxy/logging_spec) [**The standard logging object is logged on GCS Bucket**](../proxy/logging_spec)
#### Getting `service_account.json` from Google Cloud Console #### Getting `service_account.json` from Google Cloud Console
1. Go to [Google Cloud Console](https://console.cloud.google.com/) 1. Go to [Google Cloud Console](https://console.cloud.google.com/)
@ -1115,8 +1114,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
5. Click on 'Keys' -> Add Key -> Create New Key -> JSON 5. Click on 'Keys' -> Add Key -> Create New Key -> JSON
6. Save the JSON file and add the path to `GCS_PATH_SERVICE_ACCOUNT` 6. Save the JSON file and add the path to `GCS_PATH_SERVICE_ACCOUNT`
## Google Cloud Storage - PubSub Topic ## Google Cloud Storage - PubSub Topic
Log LLM Logs/SpendLogs to [Google Cloud Storage PubSub Topic](https://cloud.google.com/pubsub/docs/reference/rest) Log LLM Logs/SpendLogs to [Google Cloud Storage PubSub Topic](https://cloud.google.com/pubsub/docs/reference/rest)
@ -1127,26 +1124,25 @@ Log LLM Logs/SpendLogs to [Google Cloud Storage PubSub Topic](https://cloud.goog
::: :::
| Property | Details |
| Property | Details | | ----------- | ------------------------------------------------------------------ |
|----------|---------|
| Description | Log LiteLLM `SpendLogs Table` to Google Cloud Storage PubSub Topic | | Description | Log LiteLLM `SpendLogs Table` to Google Cloud Storage PubSub Topic |
When to use `gcs_pubsub`? When to use `gcs_pubsub`?
- If your LiteLLM Database has crossed 1M+ spend logs and you want to send `SpendLogs` to a PubSub Topic that can be consumed by GCS BigQuery - If your LiteLLM Database has crossed 1M+ spend logs and you want to send `SpendLogs` to a PubSub Topic that can be consumed by GCS BigQuery
#### Usage #### Usage
1. Add `gcs_pubsub` to LiteLLM Config.yaml 1. Add `gcs_pubsub` to LiteLLM Config.yaml
```yaml ```yaml
model_list: model_list:
- litellm_params: - litellm_params:
api_base: https://exampleopenaiendpoint-production.up.railway.app/ api_base: https://exampleopenaiendpoint-production.up.railway.app/
api_key: my-fake-key api_key: my-fake-key
model: openai/my-fake-model model: openai/my-fake-model
model_name: fake-openai-endpoint model_name: fake-openai-endpoint
litellm_settings: litellm_settings:
callbacks: ["gcs_pubsub"] # 👈 KEY CHANGE # 👈 KEY CHANGE callbacks: ["gcs_pubsub"] # 👈 KEY CHANGE # 👈 KEY CHANGE
@ -1182,8 +1178,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
' '
``` ```
## s3 Buckets ## s3 Buckets
We will use the `--config` to set We will use the `--config` to set
@ -1276,17 +1270,15 @@ Log LLM Logs to [Azure Data Lake Storage](https://learn.microsoft.com/en-us/azur
::: :::
| Property | Details |
| Property | Details | | ------------------------------- | --------------------------------------------------------------------------------------------------------------- |
|----------|---------| | Description | Log LLM Input/Output to Azure Blob Storag (Bucket) |
| Description | Log LLM Input/Output to Azure Blob Storag (Bucket) |
| Azure Docs on Data Lake Storage | [Azure Data Lake Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) | | Azure Docs on Data Lake Storage | [Azure Data Lake Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) |
#### Usage #### Usage
1. Add `azure_storage` to LiteLLM Config.yaml 1. Add `azure_storage` to LiteLLM Config.yaml
```yaml ```yaml
model_list: model_list:
- model_name: fake-openai-endpoint - model_name: fake-openai-endpoint
@ -1339,7 +1331,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
' '
``` ```
#### Expected Logs on Azure Data Lake Storage #### Expected Logs on Azure Data Lake Storage
<Image img={require('../../img/azure_blob.png')} /> <Image img={require('../../img/azure_blob.png')} />
@ -1348,11 +1339,10 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
[**The standard logging object is logged on Azure Data Lake Storage**](../proxy/logging_spec) [**The standard logging object is logged on Azure Data Lake Storage**](../proxy/logging_spec)
## DataDog ## DataDog
LiteLLM Supports logging to the following Datdog Integrations: LiteLLM Supports logging to the following Datdog Integrations:
- `datadog` [Datadog Logs](https://docs.datadoghq.com/logs/) - `datadog` [Datadog Logs](https://docs.datadoghq.com/logs/)
- `datadog_llm_observability` [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) - `datadog_llm_observability` [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/)
- `ddtrace-run` [Datadog Tracing](#datadog-tracing) - `ddtrace-run` [Datadog Tracing](#datadog-tracing)
@ -1447,26 +1437,29 @@ docker run \
LiteLLM supports customizing the following Datadog environment variables LiteLLM supports customizing the following Datadog environment variables
| Environment Variable | Description | Default Value | Required | | Environment Variable | Description | Default Value | Required |
|---------------------|-------------|---------------|----------| | -------------------- | ------------------------------------------------------------- | ---------------- | -------- |
| `DD_API_KEY` | Your Datadog API key for authentication | None | ✅ Yes | | `DD_API_KEY` | Your Datadog API key for authentication | None | ✅ Yes |
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") | None | ✅ Yes | | `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") | None | ✅ Yes |
| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No | | `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No |
| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No | | `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No |
| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No | | `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No |
| `DD_VERSION` | Version tag for your logs | "unknown" | ❌ No | | `DD_VERSION` | Version tag for your logs | "unknown" | ❌ No |
| `HOSTNAME` | Hostname tag for your logs | "" | ❌ No | | `HOSTNAME` | Hostname tag for your logs | "" | ❌ No |
| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No | | `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No |
## Lunary ## Lunary
#### Step1: Install dependencies and set your environment variables #### Step1: Install dependencies and set your environment variables
Install the dependencies Install the dependencies
```shell ```shell
pip install litellm lunary pip install litellm lunary
``` ```
Get you Lunary public key from from https://app.lunary.ai/settings Get you Lunary public key from from https://app.lunary.ai/settings
```shell ```shell
export LUNARY_PUBLIC_KEY="<your-public-key>" export LUNARY_PUBLIC_KEY="<your-public-key>"
``` ```
@ -1484,6 +1477,7 @@ litellm_settings:
``` ```
#### Step 3: Start the LiteLLM proxy #### Step 3: Start the LiteLLM proxy
```shell ```shell
litellm --config config.yaml litellm --config config.yaml
``` ```
@ -1510,8 +1504,8 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
## MLflow ## MLflow
#### Step1: Install dependencies #### Step1: Install dependencies
Install the dependencies. Install the dependencies.
```shell ```shell
@ -1531,6 +1525,7 @@ litellm_settings:
``` ```
#### Step 3: Start the LiteLLM proxy #### Step 3: Start the LiteLLM proxy
```shell ```shell
litellm --config config.yaml litellm --config config.yaml
``` ```
@ -1559,8 +1554,6 @@ Run the following command to start MLflow UI and review recorded traces.
mlflow ui mlflow ui
``` ```
## Custom Callback Class [Async] ## Custom Callback Class [Async]
Use this when you want to run custom callbacks in `python` Use this when you want to run custom callbacks in `python`
@ -1685,7 +1678,6 @@ model_list:
litellm_settings: litellm_settings:
callbacks: custom_callbacks.proxy_handler_instance # sets litellm.callbacks = [proxy_handler_instance] callbacks: custom_callbacks.proxy_handler_instance # sets litellm.callbacks = [proxy_handler_instance]
``` ```
#### Step 3 - Start proxy + test request #### Step 3 - Start proxy + test request
@ -1783,7 +1775,7 @@ class MyCustomHandler(CustomLogger):
**Expected Output** **Expected Output**
```json ```json
{'mode': 'embedding', 'input_cost_per_token': 0.002} { "mode": "embedding", "input_cost_per_token": 0.002 }
``` ```
##### Logging responses from proxy ##### Logging responses from proxy
@ -1966,7 +1958,6 @@ environment_variables:
LANGSMITH_BASE_URL: "https://api.smith.langchain.com" # (Optional - only needed if you have a custom Langsmith instance) LANGSMITH_BASE_URL: "https://api.smith.langchain.com" # (Optional - only needed if you have a custom Langsmith instance)
``` ```
2. Start Proxy 2. Start Proxy
``` ```
@ -1989,10 +1980,10 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
} }
' '
``` ```
Expect to see your log on Langfuse Expect to see your log on Langfuse
<Image img={require('../../img/langsmith_new.png')} /> <Image img={require('../../img/langsmith_new.png')} />
## Arize AI ## Arize AI
1. Set `success_callback: ["arize"]` on litellm config.yaml 1. Set `success_callback: ["arize"]` on litellm config.yaml
@ -2009,10 +2000,10 @@ litellm_settings:
callbacks: ["arize"] callbacks: ["arize"]
environment_variables: environment_variables:
ARIZE_SPACE_KEY: "d0*****" ARIZE_SPACE_ID: "d0*****"
ARIZE_API_KEY: "141a****" ARIZE_API_KEY: "141a****"
ARIZE_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize GRPC api endpoint ARIZE_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize GRPC api endpoint
ARIZE_HTTP_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize HTTP api endpoint. Set either this or ARIZE_ENDPOINT ARIZE_HTTP_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize HTTP api endpoint. Set either this or ARIZE_ENDPOINT
``` ```
2. Start Proxy 2. Start Proxy
@ -2037,10 +2028,10 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
} }
' '
``` ```
Expect to see your log on Langfuse Expect to see your log on Langfuse
<Image img={require('../../img/langsmith_new.png')} /> <Image img={require('../../img/langsmith_new.png')} />
## Langtrace ## Langtrace
1. Set `success_callback: ["langtrace"]` on litellm config.yaml 1. Set `success_callback: ["langtrace"]` on litellm config.yaml
@ -2057,7 +2048,7 @@ litellm_settings:
callbacks: ["langtrace"] callbacks: ["langtrace"]
environment_variables: environment_variables:
LANGTRACE_API_KEY: "141a****" LANGTRACE_API_KEY: "141a****"
``` ```
2. Start Proxy 2. Start Proxy
@ -2110,11 +2101,11 @@ export GALILEO_PASSWORD=""
```yaml ```yaml
model_list: model_list:
- litellm_params: - litellm_params:
api_base: https://exampleopenaiendpoint-production.up.railway.app/ api_base: https://exampleopenaiendpoint-production.up.railway.app/
api_key: my-fake-key api_key: my-fake-key
model: openai/my-fake-model model: openai/my-fake-model
model_name: fake-openai-endpoint model_name: fake-openai-endpoint
litellm_settings: litellm_settings:
success_callback: ["galileo"] # 👈 KEY CHANGE success_callback: ["galileo"] # 👈 KEY CHANGE
@ -2163,11 +2154,11 @@ export OPENMETER_API_KEY=""
```yaml ```yaml
model_list: model_list:
- litellm_params: - litellm_params:
api_base: https://openai-function-calling-workers.tasslexyz.workers.dev/ api_base: https://openai-function-calling-workers.tasslexyz.workers.dev/
api_key: my-fake-key api_key: my-fake-key
model: openai/my-fake-model model: openai/my-fake-model
model_name: fake-openai-endpoint model_name: fake-openai-endpoint
litellm_settings: litellm_settings:
success_callback: ["openmeter"] # 👈 KEY CHANGE success_callback: ["openmeter"] # 👈 KEY CHANGE
@ -2413,7 +2404,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}' }'
``` ```
<!-- ## (BETA) Moderation with Azure Content Safety <!-- ## (BETA) Moderation with Azure Content Safety
Note: This page is for logging callbacks and this is a moderation service. Commenting until we found a better location for this. Note: This page is for logging callbacks and this is a moderation service. Commenting until we found a better location for this.

2
litellm/integrations/arize/_utils.py
View file

@ -12,7 +12,7 @@ else:
def set_attributes(span: Span, kwargs, response_obj): def set_attributes(span: Span, kwargs, response_obj):
from openinference.semconv.trace import ( from litellm.integrations._types.open_inference import (
MessageAttributes, MessageAttributes,
OpenInferenceSpanKindValues, OpenInferenceSpanKindValues,
SpanAttributes, SpanAttributes,

8
litellm/integrations/arize/arize.py
View file

@ -40,11 +40,11 @@ class ArizeLogger:
Raises: Raises:
ValueError: If required environment variables are not set. ValueError: If required environment variables are not set.
""" """
space_key = os.environ.get("ARIZE_SPACE_KEY") space_id = os.environ.get("ARIZE_SPACE_ID")
api_key = os.environ.get("ARIZE_API_KEY") api_key = os.environ.get("ARIZE_API_KEY")
if not space_key: if not space_id:
raise ValueError("ARIZE_SPACE_KEY not found in environment variables") raise ValueError("ARIZE_SPACE_ID not found in environment variables")
if not api_key: if not api_key:
raise ValueError("ARIZE_API_KEY not found in environment variables") raise ValueError("ARIZE_API_KEY not found in environment variables")
@ -65,7 +65,7 @@ class ArizeLogger:
endpoint = "https://otlp.arize.com/v1" endpoint = "https://otlp.arize.com/v1"
return ArizeConfig( return ArizeConfig(
space_key=space_key, space_id=space_id,
api_key=api_key, api_key=api_key,
protocol=protocol, protocol=protocol,
endpoint=endpoint, endpoint=endpoint,

6
litellm/litellm_core_utils/litellm_logging.py
View file

@ -2654,7 +2654,7 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
) )
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = ( os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = (
f"space_key={arize_config.space_key},api_key={arize_config.api_key}" f"space_id={arize_config.space_id},api_key={arize_config.api_key}"
) )
for callback in _in_memory_loggers: for callback in _in_memory_loggers:
if ( if (
@ -2899,8 +2899,8 @@ def get_custom_logger_compatible_class( # noqa: PLR0915
elif logging_integration == "arize": elif logging_integration == "arize":
from litellm.integrations.opentelemetry import OpenTelemetry from litellm.integrations.opentelemetry import OpenTelemetry
if "ARIZE_SPACE_KEY" not in os.environ: if "ARIZE_SPACE_ID" not in os.environ:
raise ValueError("ARIZE_SPACE_KEY not found in environment variables") raise ValueError("ARIZE_SPACE_ID not found in environment variables")
if "ARIZE_API_KEY" not in os.environ: if "ARIZE_API_KEY" not in os.environ:
raise ValueError("ARIZE_API_KEY not found in environment variables") raise ValueError("ARIZE_API_KEY not found in environment variables")
for callback in _in_memory_loggers: for callback in _in_memory_loggers:

4
litellm/types/integrations/arize.py
View file

@ -1,4 +1,4 @@
from typing import TYPE_CHECKING, Literal, Any from typing import TYPE_CHECKING, Literal, Any, Optional
from pydantic import BaseModel from pydantic import BaseModel
@ -8,7 +8,7 @@ else:
Protocol = Any Protocol = Any
class ArizeConfig(BaseModel): class ArizeConfig(BaseModel):
space_key: str space_id: str
api_key: str api_key: str
protocol: Protocol protocol: Protocol
endpoint: str endpoint: str

44
tests/local_testing/test_arize_ai.py
View file

@ -1,11 +1,13 @@
import asyncio import asyncio
import logging import logging
from litellm import Choices
import pytest import pytest
from dotenv import load_dotenv from dotenv import load_dotenv
import litellm import litellm
from litellm._logging import verbose_logger, verbose_proxy_logger from litellm._logging import verbose_logger, verbose_proxy_logger
from litellm.integrations._types.open_inference import SpanAttributes
from litellm.integrations.arize.arize import ArizeConfig, ArizeLogger from litellm.integrations.arize.arize import ArizeConfig, ArizeLogger
load_dotenv() load_dotenv()
@ -32,7 +34,7 @@ async def test_async_otel_callback():
@pytest.fixture @pytest.fixture
def mock_env_vars(monkeypatch): def mock_env_vars(monkeypatch):
monkeypatch.setenv("ARIZE_SPACE_KEY", "test_space_key") monkeypatch.setenv("ARIZE_SPACE_ID", "test_space_id")
monkeypatch.setenv("ARIZE_API_KEY", "test_api_key") monkeypatch.setenv("ARIZE_API_KEY", "test_api_key")
@ -42,7 +44,7 @@ def test_get_arize_config(mock_env_vars):
""" """
config = ArizeLogger.get_arize_config() config = ArizeLogger.get_arize_config()
assert isinstance(config, ArizeConfig) assert isinstance(config, ArizeConfig)
assert config.space_key == "test_space_key" assert config.space_id == "test_space_id"
assert config.api_key == "test_api_key" assert config.api_key == "test_api_key"
assert config.endpoint == "https://otlp.arize.com/v1" assert config.endpoint == "https://otlp.arize.com/v1"
assert config.protocol == "otlp_grpc" assert config.protocol == "otlp_grpc"
@ -58,3 +60,41 @@ def test_get_arize_config_with_endpoints(mock_env_vars, monkeypatch):
config = ArizeLogger.get_arize_config() config = ArizeLogger.get_arize_config()
assert config.endpoint == "grpc://test.endpoint" assert config.endpoint == "grpc://test.endpoint"
assert config.protocol == "otlp_grpc" assert config.protocol == "otlp_grpc"
def test_arize_set_attributes():
"""
Test setting attributes for Arize
"""
from unittest.mock import MagicMock
from litellm.types.utils import ModelResponse
span = MagicMock()
kwargs = {
"role": "user",
"content": "simple arize test",
"model": "gpt-4o",
"messages": [{"role": "user", "content": "basic arize test"}],
"litellm_params": {"metadata": {"key": "value"}},
"standard_logging_object": {"model_parameters": {"user": "test_user"}}
}
response_obj = ModelResponse(usage={"total_tokens": 100, "completion_tokens": 60, "prompt_tokens": 40},
choices=[Choices(message={"role": "assistant", "content": "response content"})])
ArizeLogger.set_arize_attributes(span, kwargs, response_obj)
assert span.set_attribute.call_count == 14
span.set_attribute.assert_any_call(SpanAttributes.METADATA, str({"key": "value"}))
span.set_attribute.assert_any_call(SpanAttributes.LLM_MODEL_NAME, "gpt-4o")
span.set_attribute.assert_any_call(SpanAttributes.OPENINFERENCE_SPAN_KIND, "LLM")
span.set_attribute.assert_any_call(SpanAttributes.INPUT_VALUE, "basic arize test")
span.set_attribute.assert_any_call("llm.input_messages.0.message.role", "user")
span.set_attribute.assert_any_call("llm.input_messages.0.message.content", "basic arize test")
span.set_attribute.assert_any_call(SpanAttributes.LLM_INVOCATION_PARAMETERS, '{"user": "test_user"}')
span.set_attribute.assert_any_call(SpanAttributes.USER_ID, "test_user")
span.set_attribute.assert_any_call(SpanAttributes.OUTPUT_VALUE, "response content")
span.set_attribute.assert_any_call("llm.output_messages.0.message.role", "assistant")
span.set_attribute.assert_any_call("llm.output_messages.0.message.content", "response content")
span.set_attribute.assert_any_call(SpanAttributes.LLM_TOKEN_COUNT_TOTAL, 100)
span.set_attribute.assert_any_call(SpanAttributes.LLM_TOKEN_COUNT_COMPLETION, 60)
span.set_attribute.assert_any_call(SpanAttributes.LLM_TOKEN_COUNT_PROMPT, 40)

0
tests/logging_callback_tests/test_arize_logging.py Normal file
View file

2
tests/logging_callback_tests/test_unit_tests_init_callbacks.py
View file

@ -89,7 +89,7 @@ expected_env_vars = {
"OPIK_API_KEY": "opik_api_key", "OPIK_API_KEY": "opik_api_key",
"LANGTRACE_API_KEY": "langtrace_api_key", "LANGTRACE_API_KEY": "langtrace_api_key",
"LOGFIRE_TOKEN": "logfire_token", "LOGFIRE_TOKEN": "logfire_token",
"ARIZE_SPACE_KEY": "arize_space_key", "ARIZE_SPACE_ID": "arize_space_id",
"ARIZE_API_KEY": "arize_api_key", "ARIZE_API_KEY": "arize_api_key",
"PHOENIX_API_KEY": "phoenix_api_key", "PHOENIX_API_KEY": "phoenix_api_key",
"ARGILLA_API_KEY": "argilla_api_key", "ARGILLA_API_KEY": "argilla_api_key",