phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-27 19:54:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs updates based on feedback

Browse source
This commit is contained in:
Ishaan Jaff 2024-08-19 15:36:10 -07:00
parent b7d4031f89
commit d2f4535cf5
1 changed files with 31 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

50
docs/my-website/docs/tutorials/litellm_proxy_aporia.md
View file

@ -34,6 +34,7 @@ Add the `Toxicity - Response` to your Post LLM API Call project
## 2. Define Guardrails on your LiteLLM config.yaml ## 2. Define Guardrails on your LiteLLM config.yaml
- Define your guardrails under the `guardrails` section and set `pre_call_guardrails` and `post_call_guardrails`
```yaml ```yaml
model_list: model_list:
- model_name: gpt-3.5-turbo - model_name: gpt-3.5-turbo
@ -41,17 +42,30 @@ model_list:
model: openai/gpt-3.5-turbo model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY api_key: os.environ/OPENAI_API_KEY
litellm_settings:
pre_call_guardrails: ["aporia-example-1"] # run guardrail before making LLM API call
post_call_guardrails: ["aporia-example-2"] # run guardrail after making LLM API call
during_call_guardrails: []
guardrails: guardrails:
pre_call: # guardrail only runs on input before LLM API call - guardrail_name: "aporia-pre-guard"
guardrail: "aporia" # supported values ["aporia", "bedrock", "lakera"] litellm_params:
api_key: os.environ/APORIA_API_KEY_1 guardrail: aporia # supported values: "aporia", "bedrock", "lakera"
api_base: os.environ/APORIA_API_BASE_1 api_key: os.environ/APORIA_API_KEY_1
post_call: # guardrail only runs on output after LLM API call api_base: os.environ/APORIA_API_BASE_1
guardrail: "aporia" # supported values ["aporia", "bedrock", "lakera"] - guardrail_name: "aporia-post-guard"
api_key: os.environ/APORIA_API_KEY_2 litellm_params:
api_base: os.environ/APORIA_API_BASE_2 guardrail: aporia # supported values: "aporia", "bedrock", "lakera"
api_key: os.environ/APORIA_API_KEY_2
api_base: os.environ/APORIA_API_BASE_2
``` ```
### Supported Event Hooks
- pre_call_guardrails: `Optional[List[str]]` Run **before** LLM call, on **input**
- post_call_guardrails: `Optional[List[str]]` Run **after** LLM call, on **input & output**
- during_call_guardrails: `Optional[List[str]]` Run **during** LLM call, on **input**
## 3. Start LiteLLM Gateway ## 3. Start LiteLLM Gateway
@ -62,7 +76,7 @@ litellm --config config.yaml --detailed_debug
## 4. Test request ## 4. Test request
<Tabs> <Tabs>
<TabItem label="Fails Guardrail" value = "not-allowed"> <TabItem label="Unsuccessful call" value = "not-allowed">
Expect this to fail since since `ishaan@berri.ai` in the request is PII Expect this to fail since since `ishaan@berri.ai` in the request is PII
@ -80,7 +94,7 @@ curl -i http://localhost:4000/v1/chat/completions \
</TabItem> </TabItem>
<TabItem label="Success" value = "allowed"> <TabItem label="Successful Call " value = "allowed">
```shell ```shell
curl -i http://localhost:4000/v1/chat/completions \ curl -i http://localhost:4000/v1/chat/completions \
@ -102,9 +116,9 @@ curl -i http://localhost:4000/v1/chat/completions \
## Advanced ## Advanced
### Control Guardrails per Project (API Key) ### Control Guardrails per Project (API Key)
Use this to control what guardrail/s run per project. In this tutorial we only want the following guardrails to run for 1 project Use this to control what guardrails run per project. In this tutorial we only want the following guardrails to run for 1 project
- pre_call: aporia - `pre_call_guardrails`: ["aporia"]
- post_call: aporia - `post_call_guardrails`: ["aporia"]
**Step 1** Create Key with guardrail settings **Step 1** Create Key with guardrail settings
@ -116,9 +130,8 @@ curl -X POST 'http://0.0.0.0:4000/key/generate' \
-H 'Authorization: Bearer sk-1234' \ -H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \ -H 'Content-Type: application/json' \
-D '{ -D '{
"guardrails": { "pre_call_guardrails": ["aporia"],
"pre_call": ["aporia"], "post_call_guardrails": ["aporia"]
"post_call": ["aporia"]
} }
}' }'
``` ```
@ -132,9 +145,8 @@ curl --location 'http://0.0.0.0:4000/key/update' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
--data '{ --data '{
"key": "sk-jNm1Zar7XfNdZXp49Z1kSQ", "key": "sk-jNm1Zar7XfNdZXp49Z1kSQ",
"guardrails": { "pre_call_guardrails": ["aporia"],
"pre_call": ["aporia"], "post_call_guardrails": ["aporia"]
"post_call": ["aporia"]
} }
}' }'
``` ```