diff --git a/docs/my-website/docs/proxy/logging.md b/docs/my-website/docs/proxy/logging.md index 5aa78f73d..8bf0fcee2 100644 --- a/docs/my-website/docs/proxy/logging.md +++ b/docs/my-website/docs/proxy/logging.md @@ -7,10 +7,17 @@ import TabItem from '@theme/TabItem'; Log Proxy Input, Output, Exceptions using Custom Callbacks, Langfuse, OpenTelemetry, LangFuse, DynamoDB, s3 Bucket +- [Async Custom Callbacks](#custom-callback-class-async) +- [Logging to Langfuse](#logging-proxy-inputoutput---langfuse) +- [Logging to s3 Buckets](#logging-proxy-inputoutput---s3-buckets) +- [Logging to DynamoDB](#logging-proxy-inputoutput---dynamodb) +- [Logging to Sentry](#logging-proxy-inputoutput---sentry) +- [Logging to Traceloop (OpenTelemetry)](#opentelemetry---traceloop) + ## Custom Callback Class [Async] Use this when you want to run custom callbacks in `python` -### Step 1 - Create your custom `litellm` callback class +#### Step 1 - Create your custom `litellm` callback class We use `litellm.integrations.custom_logger` for this, **more details about litellm custom callbacks [here](https://docs.litellm.ai/docs/observability/custom_callback)** Define your custom callback class in a python file. @@ -112,7 +119,7 @@ proxy_handler_instance = MyCustomHandler() # need to set litellm.callbacks = [proxy_handler_instance] # on the proxy ``` -### Step 2 - Pass your custom callback class in `config.yaml` +#### Step 2 - Pass your custom callback class in `config.yaml` We pass the custom callback class defined in **Step1** to the config.yaml. Set `callbacks` to `python_filename.logger_instance_name` @@ -134,7 +141,7 @@ litellm_settings: ``` -### Step 3 - Start proxy + test request +#### Step 3 - Start proxy + test request ```shell litellm --config proxy_config.yaml ``` @@ -167,7 +174,7 @@ On Success Proxy Metadata: {'user_api_key': None, 'headers': Headers({'host': '0.0.0.0:8000', 'user-agent': 'curl/7.88.1', 'accept': '*/*', 'authorization': 'Bearer sk-1234', 'content-length': '199', 'content-type': 'application/x-www-form-urlencoded'}), 'model_group': 'gpt-3.5-turbo', 'deployment': 'gpt-3.5-turbo-ModelID-gpt-3.5-turbo'} ``` -### Logging Proxy Request Object, Header, Url +#### Logging Proxy Request Object, Header, Url Here's how you can access the `url`, `headers`, `request body` sent to the proxy for each request @@ -211,7 +218,7 @@ class MyCustomHandler(CustomLogger): ``` -### Logging `model_info` set in config.yaml +#### Logging `model_info` set in config.yaml Here is how to log the `model_info` set in your proxy `config.yaml`. Information on setting `model_info` on [config.yaml](https://docs.litellm.ai/docs/proxy/configs) @@ -428,176 +435,6 @@ print(response) - -## OpenTelemetry - Traceloop - -Traceloop allows you to log LLM Input/Output in the OpenTelemetry format - -We will use the `--config` to set `litellm.success_callback = ["traceloop"]` this will log all successfull LLM calls to traceloop - -**Step 1** Install traceloop-sdk and set Traceloop API key - -```shell -pip install traceloop-sdk -U -``` - -Traceloop outputs standard OpenTelemetry data that can be connected to your observability stack. Send standard OpenTelemetry from LiteLLM Proxy to [Traceloop](https://www.traceloop.com/docs/openllmetry/integrations/traceloop), [Dynatrace](https://www.traceloop.com/docs/openllmetry/integrations/dynatrace), [Datadog](https://www.traceloop.com/docs/openllmetry/integrations/datadog) -, [New Relic](https://www.traceloop.com/docs/openllmetry/integrations/newrelic), [Honeycomb](https://www.traceloop.com/docs/openllmetry/integrations/honeycomb), [Grafana Tempo](https://www.traceloop.com/docs/openllmetry/integrations/grafana), [Splunk](https://www.traceloop.com/docs/openllmetry/integrations/splunk), [OpenTelemetry Collector](https://www.traceloop.com/docs/openllmetry/integrations/otel-collector) - -**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` -```yaml -model_list: - - model_name: gpt-3.5-turbo - litellm_params: - model: gpt-3.5-turbo -litellm_settings: - success_callback: ["traceloop"] -``` - -**Step 3**: Start the proxy, make a test request - -Start proxy -```shell -litellm --config config.yaml --debug -``` - -Test Request -``` -curl --location 'http://0.0.0.0:8000/chat/completions' \ - --header 'Content-Type: application/json' \ - --data ' { - "model": "gpt-3.5-turbo", - "messages": [ - { - "role": "user", - "content": "what llm are you" - } - ] - }' -``` - - - - - - ## Logging Proxy Input/Output - s3 Buckets We will use the `--config` to set @@ -815,3 +652,52 @@ Test Request ``` litellm --test ``` + +## Logging Proxy Input/Output Traceloop (OpenTelemetry) + +Traceloop allows you to log LLM Input/Output in the OpenTelemetry format + +We will use the `--config` to set `litellm.success_callback = ["traceloop"]` this will log all successfull LLM calls to traceloop + +**Step 1** Install traceloop-sdk and set Traceloop API key + +```shell +pip install traceloop-sdk -U +``` + +Traceloop outputs standard OpenTelemetry data that can be connected to your observability stack. Send standard OpenTelemetry from LiteLLM Proxy to [Traceloop](https://www.traceloop.com/docs/openllmetry/integrations/traceloop), [Dynatrace](https://www.traceloop.com/docs/openllmetry/integrations/dynatrace), [Datadog](https://www.traceloop.com/docs/openllmetry/integrations/datadog) +, [New Relic](https://www.traceloop.com/docs/openllmetry/integrations/newrelic), [Honeycomb](https://www.traceloop.com/docs/openllmetry/integrations/honeycomb), [Grafana Tempo](https://www.traceloop.com/docs/openllmetry/integrations/grafana), [Splunk](https://www.traceloop.com/docs/openllmetry/integrations/splunk), [OpenTelemetry Collector](https://www.traceloop.com/docs/openllmetry/integrations/otel-collector) + +**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback` +```yaml +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: gpt-3.5-turbo +litellm_settings: + success_callback: ["traceloop"] +``` + +**Step 3**: Start the proxy, make a test request + +Start proxy +```shell +litellm --config config.yaml --debug +``` + +Test Request +``` +curl --location 'http://0.0.0.0:8000/chat/completions' \ + --header 'Content-Type: application/json' \ + --data ' { + "model": "gpt-3.5-turbo", + "messages": [ + { + "role": "user", + "content": "what llm are you" + } + ] + }' +``` + +