mirror of
https://github.com/BerriAI/litellm.git
synced 2025-04-27 19:54:13 +00:00
docs(enterprise.md): update docs to turn on/off llm guard per key
This commit is contained in:
parent
c10f1d2f25
commit
7b38dbbbe9
1 changed files with 66 additions and 26 deletions
|
@ -12,9 +12,9 @@ Features here are behind a commercial license in our `/enterprise` folder. [**Se
|
||||||
:::
|
:::
|
||||||
|
|
||||||
Features:
|
Features:
|
||||||
|
- ✅ Content Moderation with LLM Guard
|
||||||
- ✅ Content Moderation with LlamaGuard
|
- ✅ Content Moderation with LlamaGuard
|
||||||
- ✅ Content Moderation with Google Text Moderations
|
- ✅ Content Moderation with Google Text Moderations
|
||||||
- ✅ Content Moderation with LLM Guard
|
|
||||||
- ✅ Reject calls from Blocked User list
|
- ✅ Reject calls from Blocked User list
|
||||||
- ✅ Reject calls (incoming / outgoing) with Banned Keywords (e.g. competitors)
|
- ✅ Reject calls (incoming / outgoing) with Banned Keywords (e.g. competitors)
|
||||||
- ✅ Don't log/store specific requests (eg confidential LLM requests)
|
- ✅ Don't log/store specific requests (eg confidential LLM requests)
|
||||||
|
@ -23,6 +23,71 @@ Features:
|
||||||
|
|
||||||
|
|
||||||
## Content Moderation
|
## Content Moderation
|
||||||
|
### Content Moderation with LLM Guard
|
||||||
|
|
||||||
|
Set the LLM Guard API Base in your environment
|
||||||
|
|
||||||
|
```env
|
||||||
|
LLM_GUARD_API_BASE = "http://0.0.0.0:8192" # deployed llm guard api
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `llmguard_moderations` as a callback
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
litellm_settings:
|
||||||
|
callbacks: ["llmguard_moderations"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Now you can easily test it
|
||||||
|
|
||||||
|
- Make a regular /chat/completion call
|
||||||
|
|
||||||
|
- Check your proxy logs for any statement with `LLM Guard:`
|
||||||
|
|
||||||
|
Expected results:
|
||||||
|
|
||||||
|
```
|
||||||
|
LLM Guard: Received response - {"sanitized_prompt": "hello world", "is_valid": true, "scanners": { "Regex": 0.0 }}
|
||||||
|
```
|
||||||
|
#### Turn on/off per key
|
||||||
|
|
||||||
|
**1. Update config**
|
||||||
|
```yaml
|
||||||
|
litellm_settings:
|
||||||
|
callbacks: ["llmguard_moderations"]
|
||||||
|
llm_guard_mode: "key-specific"
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Create new key**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl --location 'http://localhost:4000/key/generate' \
|
||||||
|
--header 'Authorization: Bearer sk-1234' \
|
||||||
|
--header 'Content-Type: application/json' \
|
||||||
|
--data '{
|
||||||
|
"models": ["fake-openai-endpoint"],
|
||||||
|
"permissions": {
|
||||||
|
"enable_llm_guard_check": true # 👈 KEY CHANGE
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
|
||||||
|
# Returns {..'key': 'my-new-key'}
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Test it!**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl --location 'http://0.0.0.0:4000/v1/chat/completions' \
|
||||||
|
--header 'Content-Type: application/json' \
|
||||||
|
--header 'Authorization: Bearer my-new-key' \ # 👈 TEST KEY
|
||||||
|
--data '{"model": "fake-openai-endpoint", "messages": [
|
||||||
|
{"role": "system", "content": "Be helpful"},
|
||||||
|
{"role": "user", "content": "What do you know?"}
|
||||||
|
]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
### Content Moderation with LlamaGuard
|
### Content Moderation with LlamaGuard
|
||||||
|
|
||||||
Currently works with Sagemaker's LlamaGuard endpoint.
|
Currently works with Sagemaker's LlamaGuard endpoint.
|
||||||
|
@ -55,32 +120,7 @@ callbacks: ["llamaguard_moderations"]
|
||||||
llamaguard_unsafe_content_categories: /path/to/llamaguard_prompt.txt
|
llamaguard_unsafe_content_categories: /path/to/llamaguard_prompt.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### Content Moderation with LLM Guard
|
|
||||||
|
|
||||||
Set the LLM Guard API Base in your environment
|
|
||||||
|
|
||||||
```env
|
|
||||||
LLM_GUARD_API_BASE = "http://0.0.0.0:8192" # deployed llm guard api
|
|
||||||
```
|
|
||||||
|
|
||||||
Add `llmguard_moderations` as a callback
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
litellm_settings:
|
|
||||||
callbacks: ["llmguard_moderations"]
|
|
||||||
```
|
|
||||||
|
|
||||||
Now you can easily test it
|
|
||||||
|
|
||||||
- Make a regular /chat/completion call
|
|
||||||
|
|
||||||
- Check your proxy logs for any statement with `LLM Guard:`
|
|
||||||
|
|
||||||
Expected results:
|
|
||||||
|
|
||||||
```
|
|
||||||
LLM Guard: Received response - {"sanitized_prompt": "hello world", "is_valid": true, "scanners": { "Regex": 0.0 }}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Content Moderation with Google Text Moderation
|
### Content Moderation with Google Text Moderation
|
||||||
|
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue