diff --git a/docs/my-website/docs/enterprise.md b/docs/my-website/docs/enterprise.md index 2d45ea3ea..875aec57f 100644 --- a/docs/my-website/docs/enterprise.md +++ b/docs/my-website/docs/enterprise.md @@ -13,7 +13,8 @@ This covers: - ✅ [**Audit Logs with retention policy**](../docs/proxy/enterprise.md#audit-logs) - ✅ [**JWT-Auth**](../docs/proxy/token_auth.md) - ✅ [**Control available public, private routes**](../docs/proxy/enterprise.md#control-available-public-private-routes) -- ✅ [**Prompt Injection Detection**](#prompt-injection-detection-lakeraai) +- ✅ [**Guardrails, Content Moderation, PII Masking, Secret/API Key Masking**](../docs/proxy/enterprise.md#prompt-injection-detection---lakeraai) +- ✅ [**Prompt Injection Detection**](../docs/proxy/enterprise.md#prompt-injection-detection---lakeraai) - ✅ [**Invite Team Members to access `/spend` Routes**](../docs/proxy/cost_tracking#allowing-non-proxy-admins-to-access-spend-endpoints) - ✅ **Feature Prioritization** - ✅ **Custom Integrations** diff --git a/docs/my-website/docs/proxy/enterprise.md b/docs/my-website/docs/proxy/enterprise.md index 40a5261cd..9fff879e5 100644 --- a/docs/my-website/docs/proxy/enterprise.md +++ b/docs/my-website/docs/proxy/enterprise.md @@ -15,10 +15,10 @@ Features: - ✅ [Audit Logs](#audit-logs) - ✅ [Tracking Spend for Custom Tags](#tracking-spend-for-custom-tags) - ✅ [Control available public, private routes](#control-available-public-private-routes) -- ✅ [Enforce Required Params for LLM Requests (ex. Reject requests missing ["metadata"]["generation_name"])](#enforce-required-params-for-llm-requests) -- ✅ [Content Moderation with LLM Guard, LlamaGuard, Google Text Moderations](#content-moderation) +- ✅ [Content Moderation with LLM Guard, LlamaGuard, Secret Detection, Google Text Moderations](#content-moderation) - ✅ [Prompt Injection Detection (with LakeraAI API)](#prompt-injection-detection---lakeraai) - ✅ [Custom Branding + Routes on Swagger Docs](#swagger-docs---custom-routes--branding) +- ✅ [Enforce Required Params for LLM Requests (ex. Reject requests missing ["metadata"]["generation_name"])](#enforce-required-params-for-llm-requests) - ✅ Reject calls from Blocked User list - ✅ Reject calls (incoming / outgoing) with Banned Keywords (e.g. competitors) @@ -495,7 +495,98 @@ curl --request POST \ ## Content Moderation -#### Content Moderation with LLM Guard +### Content Moderation - Secret Detection +❓ Use this to REDACT API Keys, Secrets sent in requests to an LLM. + +Example if you want to redact the value of `OPENAI_API_KEY` in the following request + +#### Incoming Request + +```json +{ + "messages": [ + { + "role": "user", + "content": "Hey, how's it going, API_KEY = 'sk_1234567890abcdef'", + } + ] +} +``` + +#### Request after Moderation + +```json +{ + "messages": [ + { + "role": "user", + "content": "Hey, how's it going, API_KEY = '[REDACTED]'", + } + ] +} +``` + +**Usage** + +**Step 1** Add this to your config.yaml + +```yaml +litellm_settings: + callbacks: ["hide_secrets"] +``` + +**Step 2** Run litellm proxy with `--detailed_debug` to see the server logs + +``` +litellm --config config.yaml --detailed_debug +``` + +**Step 3** Test it with request + +Send this request +```shell +curl --location 'http://localhost:4000/chat/completions' \ + --header 'Authorization: Bearer sk-1234' \ + --header 'Content-Type: application/json' \ + --data '{ + "model": "llama3", + "messages": [ + { + "role": "user", + "content": "what is the value of my open ai key? openai_api_key=sk-1234998222" + } + ] +}' +``` + + +Expect to see the following warning on your litellm server logs + +```shell +LiteLLM Proxy:WARNING: secret_detection.py:88 - Detected and redacted secrets in message: ['Secret Keyword'] +``` + + +You can also see the raw request sent from litellm to the API Provider +```json +POST Request Sent from LiteLLM: +curl -X POST \ +https://api.groq.com/openai/v1/ \ +-H 'Authorization: Bearer gsk_mySVchjY********************************************' \ +-d { + "model": "llama3-8b-8192", + "messages": [ + { + "role": "user", + "content": "what is the time today, openai_api_key=[REDACTED]" + } + ], + "stream": false, + "extra_body": {} +} +``` + +### Content Moderation with LLM Guard Set the LLM Guard API Base in your environment @@ -630,7 +721,7 @@ curl --location 'http://0.0.0.0:4000/v1/chat/completions' \ -#### Content Moderation with LlamaGuard +### Content Moderation with LlamaGuard Currently works with Sagemaker's LlamaGuard endpoint. @@ -664,7 +755,7 @@ callbacks: ["llamaguard_moderations"] -#### Content Moderation with Google Text Moderation +### Content Moderation with Google Text Moderation Requires your GOOGLE_APPLICATION_CREDENTIALS to be set in your .env (same as VertexAI).