docs - add pass through routes on litelm proxy

This commit is contained in:
Ishaan Jaff 2024-06-28 15:55:29 -07:00
parent 8f2931937a
commit ac066462df
2 changed files with 99 additions and 1 deletions

View file

@ -0,0 +1,97 @@
# ➡️ Create Pass Through Endpoints
Add pass through routes to LiteLLM Proxy
**Example:** Add a route `/v1/rerank` that forwards requests to `https://api.cohere.com/v1/rerank` through LiteLLM Proxy
💡 This allows making the following Request to LiteLLM Proxy
```shell
curl --request POST \
--url http://localhost:4000/v1/rerank \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{
"model": "rerank-english-v3.0",
"query": "What is the capital of the United States?",
"top_n": 3,
"documents": ["Carson City is the capital city of the American state of Nevada."]
}'
```
## Tutorial - Setup Cohere Re-Rank Endpoint on LiteLLM Proxy
**Step 1** Define pass through routes on [litellm config.yaml](configs.md)
```yaml
general_settings:
master_key: sk-1234
pass_through_endpoints:
- path: "/v1/rerank" # route you want to add to LiteLLM Proxy Server
target: "https://api.cohere.com/v1/rerank" # URL this route should forward requests to
headers: # headers to forward to this URL
Authorization: "bearer os.environ/COHERE_API_KEY" # (Optional) Auth Header to forward to your Endpoint
content-type: application/json # (Optional) Extra Headers to pass to this endpoint
accept: application/json
```
**Step 2** Start Proxy Server in detailed_debug mode
```shell
litellm --config config.yaml
```
**Step 3** Make Request to pass through endpoint
```shell
curl --request POST \
--url http://localhost:4000/v1/rerank \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{
"model": "rerank-english-v3.0",
"query": "What is the capital of the United States?",
"top_n": 3,
"documents": ["Carson City is the capital city of the American state of Nevada.",
"The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
"Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.",
"Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.",
"Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."]
}'
```
🎉 **Expected Response**
This request got forwarded from LiteLLM Proxy -> Defined Target URL (with headers)
```shell
{
"id": "37103a5b-8cfb-48d3-87c7-da288bedd429",
"results": [
{
"index": 2,
"relevance_score": 0.999071
},
{
"index": 4,
"relevance_score": 0.7867867
},
{
"index": 0,
"relevance_score": 0.32713068
}
],
"meta": {
"api_version": {
"version": "1"
},
"billed_units": {
"search_units": 1
}
}
}
```

View file

@ -48,6 +48,7 @@ const sidebars = {
"proxy/billing",
"proxy/user_keys",
"proxy/virtual_keys",
"proxy/token_auth",
"proxy/alerting",
{
type: "category",
@ -56,11 +57,11 @@ const sidebars = {
},
"proxy/ui",
"proxy/prometheus",
"proxy/pass_through",
"proxy/email",
"proxy/multiple_admins",
"proxy/team_based_routing",
"proxy/customer_routing",
"proxy/token_auth",
{
type: "category",
label: "Extra Load Balancing",