phoenix/litellm
1
0
Fork 0
forked from phoenix/litellm-mirror
Code Pull requests Activity

Merge pull request #4651 from msabramo/docs-logging-cleanup

Browse source 
Docs: Miscellaneous cleanup of `docs/my-website/docs/proxy/logging.md`
This commit is contained in:
Krish Dholakia 2024-07-11 21:52:20 -07:00 • committed by GitHub
commit 79d6b69d1c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 65 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

116
docs/my-website/docs/proxy/logging.md
View file

@ -1,28 +1,21 @@
# 🪢 Logging
Log Proxy input, output, and exceptions using:
- Langfuse
- OpenTelemetry
- Custom Callbacks
- DataDog
- DynamoDB
- s3 Bucket
- etc.
import Image from '@theme/IdealImage'; import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs'; import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem'; import TabItem from '@theme/TabItem';
# 🪢 Logging - Langfuse, OpenTelemetry, Custom Callbacks, DataDog, s3 Bucket, Sentry, Athina, Azure Content-Safety
Log Proxy Input, Output, Exceptions using Langfuse, OpenTelemetry, Custom Callbacks, DataDog, DynamoDB, s3 Bucket
## Table of Contents
- [Logging to Langfuse](#logging-proxy-inputoutput---langfuse)
- [Logging with OpenTelemetry (OpenTelemetry)](#logging-proxy-inputoutput-in-opentelemetry-format)
- [Async Custom Callbacks](#custom-callback-class-async)
- [Async Custom Callback APIs](#custom-callback-apis-async)
- [Logging to Galileo](#logging-llm-io-to-galileo)
- [Logging to OpenMeter](#logging-proxy-inputoutput---langfuse)
- [Logging to s3 Buckets](#logging-proxy-inputoutput---s3-buckets)
- [Logging to DataDog](#logging-proxy-inputoutput---datadog)
- [Logging to DynamoDB](#logging-proxy-inputoutput---dynamodb)
- [Logging to Sentry](#logging-proxy-inputoutput---sentry)
- [Logging to Athina](#logging-proxy-inputoutput-athina)
- [(BETA) Moderation with Azure Content-Safety](#moderation-with-azure-content-safety)
## Logging Proxy Input/Output - Langfuse ## Logging Proxy Input/Output - Langfuse
We will use the `--config` to set `litellm.success_callback = ["langfuse"]` this will log all successfull LLM calls to langfuse. Make sure to set `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` in your environment We will use the `--config` to set `litellm.success_callback = ["langfuse"]` this will log all successfull LLM calls to langfuse. Make sure to set `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` in your environment
**Step 1** Install langfuse **Step 1** Install langfuse
@ -32,6 +25,7 @@ pip install langfuse>=2.0.0
``` ```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` **Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -42,6 +36,7 @@ litellm_settings:
``` ```
**Step 3**: Set required env variables for logging to langfuse **Step 3**: Set required env variables for logging to langfuse
```shell ```shell
export LANGFUSE_PUBLIC_KEY="pk_kk" export LANGFUSE_PUBLIC_KEY="pk_kk"
export LANGFUSE_SECRET_KEY="sk_ss" export LANGFUSE_SECRET_KEY="sk_ss"
@ -52,11 +47,13 @@ export LANGFUSE_HOST="https://xxx.langfuse.com"
**Step 4**: Start the proxy, make a test request **Step 4**: Start the proxy, make a test request
Start proxy Start proxy
```shell ```shell
litellm --config config.yaml --debug litellm --config config.yaml --debug
``` ```
Test Request Test Request
``` ```
litellm --test litellm --test
``` ```
@ -67,7 +64,6 @@ Expected output on Langfuse
### Logging Metadata to Langfuse ### Logging Metadata to Langfuse
<Tabs> <Tabs>
<TabItem value="Curl" label="Curl Request"> <TabItem value="Curl" label="Curl Request">
@ -93,6 +89,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
} }
}' }'
``` ```
</TabItem> </TabItem>
<TabItem value="openai" label="OpenAI v1.0.0+"> <TabItem value="openai" label="OpenAI v1.0.0+">
@ -126,6 +123,7 @@ response = client.chat.completions.create(
print(response) print(response)
``` ```
</TabItem> </TabItem>
<TabItem value="langchain" label="Langchain"> <TabItem value="langchain" label="Langchain">
@ -168,7 +166,6 @@ print(response)
</TabItem> </TabItem>
</Tabs> </Tabs>
### Team based Logging to Langfuse ### Team based Logging to Langfuse
**Example:** **Example:**
@ -257,6 +254,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
} }
}' }'
``` ```
</TabItem> </TabItem>
<TabItem value="openai" label="OpenAI v1.0.0+"> <TabItem value="openai" label="OpenAI v1.0.0+">
@ -287,6 +285,7 @@ response = client.chat.completions.create(
print(response) print(response)
``` ```
</TabItem> </TabItem>
<TabItem value="langchain" label="Langchain"> <TabItem value="langchain" label="Langchain">
@ -332,7 +331,6 @@ You will see `raw_request` in your Langfuse Metadata. This is the RAW CURL comma
<Image img={require('../../img/debug_langfuse.png')} /> <Image img={require('../../img/debug_langfuse.png')} />
## Logging Proxy Input/Output in OpenTelemetry format ## Logging Proxy Input/Output in OpenTelemetry format
:::info :::info
@ -348,10 +346,8 @@ OTEL_SERVICE_NAME=<your-service-name>` # default="litellm"
<Tabs> <Tabs>
<TabItem value="Console Exporter" label="Log to console"> <TabItem value="Console Exporter" label="Log to console">
**Step 1:** Set callbacks and env vars **Step 1:** Set callbacks and env vars
Add the following to your env Add the following to your env
@ -367,7 +363,6 @@ litellm_settings:
callbacks: ["otel"] callbacks: ["otel"]
``` ```
**Step 2**: Start the proxy, make a test request **Step 2**: Start the proxy, make a test request
Start proxy Start proxy
@ -427,7 +422,6 @@ This is the Span from OTEL Logging
</TabItem> </TabItem>
<TabItem value="Honeycomb" label="Log to Honeycomb"> <TabItem value="Honeycomb" label="Log to Honeycomb">
#### Quick Start - Log to Honeycomb #### Quick Start - Log to Honeycomb
@ -449,7 +443,6 @@ litellm_settings:
callbacks: ["otel"] callbacks: ["otel"]
``` ```
**Step 2**: Start the proxy, make a test request **Step 2**: Start the proxy, make a test request
Start proxy Start proxy
@ -474,10 +467,8 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}' }'
``` ```
</TabItem> </TabItem>
<TabItem value="otel-col" label="Log to OTEL HTTP Collector"> <TabItem value="otel-col" label="Log to OTEL HTTP Collector">
#### Quick Start - Log to OTEL Collector #### Quick Start - Log to OTEL Collector
@ -499,7 +490,6 @@ litellm_settings:
callbacks: ["otel"] callbacks: ["otel"]
``` ```
**Step 2**: Start the proxy, make a test request **Step 2**: Start the proxy, make a test request
Start proxy Start proxy
@ -526,7 +516,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
</TabItem> </TabItem>
<TabItem value="otel-col-grpc" label="Log to OTEL GRPC Collector"> <TabItem value="otel-col-grpc" label="Log to OTEL GRPC Collector">
#### Quick Start - Log to OTEL GRPC Collector #### Quick Start - Log to OTEL GRPC Collector
@ -548,7 +537,6 @@ litellm_settings:
callbacks: ["otel"] callbacks: ["otel"]
``` ```
**Step 2**: Start the proxy, make a test request **Step 2**: Start the proxy, make a test request
Start proxy Start proxy
@ -573,7 +561,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}' }'
``` ```
</TabItem> </TabItem>
<TabItem value="traceloop" label="Log to Traceloop Cloud"> <TabItem value="traceloop" label="Log to Traceloop Cloud">
@ -596,7 +583,6 @@ environment_variables:
TRACELOOP_API_KEY: "XXXXX" TRACELOOP_API_KEY: "XXXXX"
``` ```
**Step 3**: Start the proxy, make a test request **Step 3**: Start the proxy, make a test request
Start proxy Start proxy
@ -632,11 +618,15 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
❓ Use this when you want to **pass information about the incoming request in a distributed tracing system** ❓ Use this when you want to **pass information about the incoming request in a distributed tracing system**
✅ Key change: Pass the **`traceparent` header** in your requests. [Read more about traceparent headers here](https://uptrace.dev/opentelemetry/opentelemetry-traceparent.html#what-is-traceparent-header) ✅ Key change: Pass the **`traceparent` header** in your requests. [Read more about traceparent headers here](https://uptrace.dev/opentelemetry/opentelemetry-traceparent.html#what-is-traceparent-header)
```curl ```curl
traceparent: 00-80e1afed08e019fc1110464cfa66635c-7a085853722dc6d2-01 traceparent: 00-80e1afed08e019fc1110464cfa66635c-7a085853722dc6d2-01
``` ```
Example Usage Example Usage
1. Make Request to LiteLLM Proxy with `traceparent` header 1. Make Request to LiteLLM Proxy with `traceparent` header
```python ```python
import openai import openai
import uuid import uuid
@ -660,7 +650,6 @@ response = client.chat.completions.create(
) )
print(response) print(response)
``` ```
```shell ```shell
@ -674,12 +663,12 @@ Search for Trace=`80e1afed08e019fc1110464cfa66635c` on your OTEL Collector
<Image img={require('../../img/otel_parent.png')} /> <Image img={require('../../img/otel_parent.png')} />
## Custom Callback Class [Async] ## Custom Callback Class [Async]
Use this when you want to run custom callbacks in `python` Use this when you want to run custom callbacks in `python`
#### Step 1 - Create your custom `litellm` callback class #### Step 1 - Create your custom `litellm` callback class
We use `litellm.integrations.custom_logger` for this, **more details about litellm custom callbacks [here](https://docs.litellm.ai/docs/observability/custom_callback)** We use `litellm.integrations.custom_logger` for this, **more details about litellm custom callbacks [here](https://docs.litellm.ai/docs/observability/custom_callback)**
Define your custom callback class in a python file. Define your custom callback class in a python file.
@ -782,16 +771,17 @@ proxy_handler_instance = MyCustomHandler()
``` ```
#### Step 2 - Pass your custom callback class in `config.yaml` #### Step 2 - Pass your custom callback class in `config.yaml`
We pass the custom callback class defined in **Step1** to the config.yaml. We pass the custom callback class defined in **Step1** to the config.yaml.
Set `callbacks` to `python_filename.logger_instance_name` Set `callbacks` to `python_filename.logger_instance_name`
In the config below, we pass In the config below, we pass
- python_filename: `custom_callbacks.py` - python_filename: `custom_callbacks.py`
- logger_instance_name: `proxy_handler_instance`. This is defined in Step 1 - logger_instance_name: `proxy_handler_instance`. This is defined in Step 1
`callbacks: custom_callbacks.proxy_handler_instance` `callbacks: custom_callbacks.proxy_handler_instance`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -804,6 +794,7 @@ litellm_settings:
``` ```
#### Step 3 - Start proxy + test request #### Step 3 - Start proxy + test request
```shell ```shell
litellm --config proxy_config.yaml litellm --config proxy_config.yaml
``` ```
@ -825,6 +816,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
``` ```
#### Resulting Log on Proxy #### Resulting Log on Proxy
```shell ```shell
On Success On Success
Model: gpt-3.5-turbo, Model: gpt-3.5-turbo,
@ -877,7 +869,6 @@ class MyCustomHandler(CustomLogger):
"max_tokens": 10 "max_tokens": 10
} }
} }
``` ```
#### Logging `model_info` set in config.yaml #### Logging `model_info` set in config.yaml
@ -895,11 +886,13 @@ class MyCustomHandler(CustomLogger):
``` ```
**Expected Output** **Expected Output**
```json ```json
{'mode': 'embedding', 'input_cost_per_token': 0.002} {'mode': 'embedding', 'input_cost_per_token': 0.002}
``` ```
### Logging responses from proxy ### Logging responses from proxy
Both `/chat/completions` and `/embeddings` responses are available as `response_obj` Both `/chat/completions` and `/embeddings` responses are available as `response_obj`
**Note: for `/chat/completions`, both `stream=True` and `non stream` responses are available as `response_obj`** **Note: for `/chat/completions`, both `stream=True` and `non stream` responses are available as `response_obj`**
@ -913,6 +906,7 @@ class MyCustomHandler(CustomLogger):
``` ```
**Expected Output /chat/completion [for both `stream` and `non-stream` responses]** **Expected Output /chat/completion [for both `stream` and `non-stream` responses]**
```json ```json
ModelResponse( ModelResponse(
id='chatcmpl-8Tfu8GoMElwOZuj2JlHBhNHG01PPo', id='chatcmpl-8Tfu8GoMElwOZuj2JlHBhNHG01PPo',
@ -939,6 +933,7 @@ ModelResponse(
``` ```
**Expected Output /embeddings** **Expected Output /embeddings**
```json ```json
{ {
'model': 'ada', 'model': 'ada',
@ -958,7 +953,6 @@ ModelResponse(
} }
``` ```
## Custom Callback APIs [Async] ## Custom Callback APIs [Async]
:::info :::info
@ -968,10 +962,12 @@ This is an Enterprise only feature [Get Started with Enterprise here](https://gi
::: :::
Use this if you: Use this if you:
- Want to use custom callbacks written in a non Python programming language - Want to use custom callbacks written in a non Python programming language
- Want your callbacks to run on a different microservice - Want your callbacks to run on a different microservice
#### Step 1. Create your generic logging API endpoint #### Step 1. Create your generic logging API endpoint
Set up a generic API endpoint that can receive data in JSON format. The data will be included within a "data" field. Set up a generic API endpoint that can receive data in JSON format. The data will be included within a "data" field.
Your server should support the following Request format: Your server should support the following Request format:
@ -1034,11 +1030,8 @@ async def log_event(request: Request):
if __name__ == "__main__": if __name__ == "__main__":
import uvicorn import uvicorn
uvicorn.run(app, host="127.0.0.1", port=4000) uvicorn.run(app, host="127.0.0.1", port=4000)
``` ```
#### Step 2. Set your `GENERIC_LOGGER_ENDPOINT` to the endpoint + route we should send callback logs to #### Step 2. Set your `GENERIC_LOGGER_ENDPOINT` to the endpoint + route we should send callback logs to
```shell ```shell
@ -1048,6 +1041,7 @@ os.environ["GENERIC_LOGGER_ENDPOINT"] = "http://localhost:4000/log-event"
#### Step 3. Create a `config.yaml` file and set `litellm_settings`: `success_callback` = ["generic"] #### Step 3. Create a `config.yaml` file and set `litellm_settings`: `success_callback` = ["generic"]
Example litellm proxy config.yaml Example litellm proxy config.yaml
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -1059,8 +1053,8 @@ litellm_settings:
Start the LiteLLM Proxy and make a test request to verify the logs reached your callback API Start the LiteLLM Proxy and make a test request to verify the logs reached your callback API
## Logging LLM IO to Galileo ## Logging LLM IO to Galileo
[BETA] [BETA]
Log LLM I/O on [www.rungalileo.io](https://www.rungalileo.io/) Log LLM I/O on [www.rungalileo.io](https://www.rungalileo.io/)
@ -1083,6 +1077,7 @@ export GALILEO_PASSWORD=""
### Quick Start ### Quick Start
1. Add to Config.yaml 1. Add to Config.yaml
```yaml ```yaml
model_list: model_list:
- litellm_params: - litellm_params:
@ -1118,7 +1113,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
' '
``` ```
🎉 That's it - Expect to see your Logs on your Galileo Dashboard 🎉 That's it - Expect to see your Logs on your Galileo Dashboard
## Logging Proxy Cost + Usage - OpenMeter ## Logging Proxy Cost + Usage - OpenMeter
@ -1136,6 +1130,7 @@ export OPENMETER_API_KEY=""
### Quick Start ### Quick Start
1. Add to Config.yaml 1. Add to Config.yaml
```yaml ```yaml
model_list: model_list:
- litellm_params: - litellm_params:
@ -1171,13 +1166,14 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
' '
``` ```
<Image img={require('../../img/openmeter_img_2.png')} /> <Image img={require('../../img/openmeter_img_2.png')} />
## Logging Proxy Input/Output - DataDog ## Logging Proxy Input/Output - DataDog
We will use the `--config` to set `litellm.success_callback = ["datadog"]` this will log all successfull LLM calls to DataDog We will use the `--config` to set `litellm.success_callback = ["datadog"]` this will log all successfull LLM calls to DataDog
**Step 1**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` **Step 1**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -1197,6 +1193,7 @@ DD_SITE="us5.datadoghq.com" # your datadog base url
**Step 3**: Start the proxy, make a test request **Step 3**: Start the proxy, make a test request
Start proxy Start proxy
```shell ```shell
litellm --config config.yaml --debug litellm --config config.yaml --debug
``` ```
@ -1224,10 +1221,10 @@ Expected output on Datadog
<Image img={require('../../img/dd_small1.png')} /> <Image img={require('../../img/dd_small1.png')} />
## Logging Proxy Input/Output - s3 Buckets ## Logging Proxy Input/Output - s3 Buckets
We will use the `--config` to set We will use the `--config` to set
- `litellm.success_callback = ["s3"]` - `litellm.success_callback = ["s3"]`
This will log all successfull LLM calls to s3 Bucket This will log all successfull LLM calls to s3 Bucket
@ -1241,6 +1238,7 @@ AWS_REGION_NAME = ""
``` ```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` **Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -1260,11 +1258,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request **Step 3**: Start the proxy, make a test request
Start proxy Start proxy
```shell ```shell
litellm --config config.yaml --debug litellm --config config.yaml --debug
``` ```
Test Request Test Request
```shell ```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
@ -1284,6 +1284,7 @@ Your logs should be available on the specified s3 Bucket
## Logging Proxy Input/Output - DynamoDB ## Logging Proxy Input/Output - DynamoDB
We will use the `--config` to set We will use the `--config` to set
- `litellm.success_callback = ["dynamodb"]` - `litellm.success_callback = ["dynamodb"]`
- `litellm.dynamodb_table_name = "your-table-name"` - `litellm.dynamodb_table_name = "your-table-name"`
@ -1298,6 +1299,7 @@ AWS_REGION_NAME = ""
``` ```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` **Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -1311,11 +1313,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request **Step 3**: Start the proxy, make a test request
Start proxy Start proxy
```shell ```shell
litellm --config config.yaml --debug litellm --config config.yaml --debug
``` ```
Test Request Test Request
```shell ```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
@ -1403,19 +1407,18 @@ Your logs should be available on DynamoDB
} }
``` ```
## Logging Proxy Input/Output - Sentry ## Logging Proxy Input/Output - Sentry
If api calls fail (llm/database) you can log those to Sentry: If api calls fail (llm/database) you can log those to Sentry:
**Step 1** Install Sentry **Step 1** Install Sentry
```shell ```shell
pip install --upgrade sentry-sdk pip install --upgrade sentry-sdk
``` ```
**Step 2**: Save your Sentry_DSN and add `litellm_settings`: `failure_callback` **Step 2**: Save your Sentry_DSN and add `litellm_settings`: `failure_callback`
```shell ```shell
export SENTRY_DSN="your-sentry-dsn" export SENTRY_DSN="your-sentry-dsn"
``` ```
@ -1435,11 +1438,13 @@ general_settings:
**Step 3**: Start the proxy, make a test request **Step 3**: Start the proxy, make a test request
Start proxy Start proxy
```shell ```shell
litellm --config config.yaml --debug litellm --config config.yaml --debug
``` ```
Test Request Test Request
``` ```
litellm --test litellm --test
``` ```
@ -1457,6 +1462,7 @@ ATHINA_API_KEY = "your-athina-api-key"
``` ```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` **Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -1469,11 +1475,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request **Step 3**: Start the proxy, make a test request
Start proxy Start proxy
```shell ```shell
litellm --config config.yaml --debug litellm --config config.yaml --debug
``` ```
Test Request Test Request
``` ```
curl --location 'http://0.0.0.0:4000/chat/completions' \ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
@ -1505,6 +1513,7 @@ AZURE_CONTENT_SAFETY_KEY = "<your-azure-content-safety-key>"
``` ```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` **Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -1520,11 +1529,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request **Step 3**: Start the proxy, make a test request
Start proxy Start proxy
```shell ```shell
litellm --config config.yaml --debug litellm --config config.yaml --debug
``` ```
Test Request Test Request
``` ```
curl --location 'http://0.0.0.0:4000/chat/completions' \ curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
@ -1540,7 +1551,8 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
``` ```
An HTTP 400 error will be returned if the content is detected with a value greater than the threshold set in the `config.yaml`. An HTTP 400 error will be returned if the content is detected with a value greater than the threshold set in the `config.yaml`.
The details of the response will describe : The details of the response will describe:
- The `source` : input text or llm generated text - The `source` : input text or llm generated text
- The `category` : the category of the content that triggered the moderation - The `category` : the category of the content that triggered the moderation
- The `severity` : the severity from 0 to 10 - The `severity` : the severity from 0 to 10

2
litellm/proxy/proxy_server.py
View file

@ -2796,7 +2796,7 @@ async def chat_completion(
## LOGGING OBJECT ## - initialize logging object for logging success/failure events for call ## LOGGING OBJECT ## - initialize logging object for logging success/failure events for call
## IMPORTANT Note: - initialize this before running pre-call checks. Ensures we log rejected requests to langfuse. ## IMPORTANT Note: - initialize this before running pre-call checks. Ensures we log rejected requests to langfuse.
data["litellm_call_id"] = str(uuid.uuid4()) data["litellm_call_id"] = request.headers.get('x-litellm-call-id', str(uuid.uuid4()))
logging_obj, data = litellm.utils.function_setup( logging_obj, data = litellm.utils.function_setup(
original_function="acompletion", original_function="acompletion",
rules_obj=litellm.utils.Rules(), rules_obj=litellm.utils.Rules(),