mirror of
https://github.com/BerriAI/litellm.git
synced 2025-04-27 11:43:54 +00:00
docs(enterprise.md): update docs to turn on/off llm guard per key
This commit is contained in:
parent
c10f1d2f25
commit
7b38dbbbe9
1 changed files with 66 additions and 26 deletions
|
@ -12,9 +12,9 @@ Features here are behind a commercial license in our `/enterprise` folder. [**Se
|
|||
:::
|
||||
|
||||
Features:
|
||||
- ✅ Content Moderation with LLM Guard
|
||||
- ✅ Content Moderation with LlamaGuard
|
||||
- ✅ Content Moderation with Google Text Moderations
|
||||
- ✅ Content Moderation with LLM Guard
|
||||
- ✅ Reject calls from Blocked User list
|
||||
- ✅ Reject calls (incoming / outgoing) with Banned Keywords (e.g. competitors)
|
||||
- ✅ Don't log/store specific requests (eg confidential LLM requests)
|
||||
|
@ -23,6 +23,71 @@ Features:
|
|||
|
||||
|
||||
## Content Moderation
|
||||
### Content Moderation with LLM Guard
|
||||
|
||||
Set the LLM Guard API Base in your environment
|
||||
|
||||
```env
|
||||
LLM_GUARD_API_BASE = "http://0.0.0.0:8192" # deployed llm guard api
|
||||
```
|
||||
|
||||
Add `llmguard_moderations` as a callback
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
callbacks: ["llmguard_moderations"]
|
||||
```
|
||||
|
||||
Now you can easily test it
|
||||
|
||||
- Make a regular /chat/completion call
|
||||
|
||||
- Check your proxy logs for any statement with `LLM Guard:`
|
||||
|
||||
Expected results:
|
||||
|
||||
```
|
||||
LLM Guard: Received response - {"sanitized_prompt": "hello world", "is_valid": true, "scanners": { "Regex": 0.0 }}
|
||||
```
|
||||
#### Turn on/off per key
|
||||
|
||||
**1. Update config**
|
||||
```yaml
|
||||
litellm_settings:
|
||||
callbacks: ["llmguard_moderations"]
|
||||
llm_guard_mode: "key-specific"
|
||||
```
|
||||
|
||||
**2. Create new key**
|
||||
|
||||
```bash
|
||||
curl --location 'http://localhost:4000/key/generate' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"models": ["fake-openai-endpoint"],
|
||||
"permissions": {
|
||||
"enable_llm_guard_check": true # 👈 KEY CHANGE
|
||||
}
|
||||
}'
|
||||
|
||||
# Returns {..'key': 'my-new-key'}
|
||||
```
|
||||
|
||||
**2. Test it!**
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:4000/v1/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header 'Authorization: Bearer my-new-key' \ # 👈 TEST KEY
|
||||
--data '{"model": "fake-openai-endpoint", "messages": [
|
||||
{"role": "system", "content": "Be helpful"},
|
||||
{"role": "user", "content": "What do you know?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
### Content Moderation with LlamaGuard
|
||||
|
||||
Currently works with Sagemaker's LlamaGuard endpoint.
|
||||
|
@ -55,32 +120,7 @@ callbacks: ["llamaguard_moderations"]
|
|||
llamaguard_unsafe_content_categories: /path/to/llamaguard_prompt.txt
|
||||
```
|
||||
|
||||
### Content Moderation with LLM Guard
|
||||
|
||||
Set the LLM Guard API Base in your environment
|
||||
|
||||
```env
|
||||
LLM_GUARD_API_BASE = "http://0.0.0.0:8192" # deployed llm guard api
|
||||
```
|
||||
|
||||
Add `llmguard_moderations` as a callback
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
callbacks: ["llmguard_moderations"]
|
||||
```
|
||||
|
||||
Now you can easily test it
|
||||
|
||||
- Make a regular /chat/completion call
|
||||
|
||||
- Check your proxy logs for any statement with `LLM Guard:`
|
||||
|
||||
Expected results:
|
||||
|
||||
```
|
||||
LLM Guard: Received response - {"sanitized_prompt": "hello world", "is_valid": true, "scanners": { "Regex": 0.0 }}
|
||||
```
|
||||
|
||||
### Content Moderation with Google Text Moderation
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue