forked from phoenix/litellm-mirror
docs - add pass through routes on litelm proxy
This commit is contained in:
parent
8f2931937a
commit
ac066462df
2 changed files with 99 additions and 1 deletions
97
docs/my-website/docs/proxy/pass_through.md
Normal file
97
docs/my-website/docs/proxy/pass_through.md
Normal file
|
@ -0,0 +1,97 @@
|
|||
# ➡️ Create Pass Through Endpoints
|
||||
|
||||
Add pass through routes to LiteLLM Proxy
|
||||
|
||||
**Example:** Add a route `/v1/rerank` that forwards requests to `https://api.cohere.com/v1/rerank` through LiteLLM Proxy
|
||||
|
||||
|
||||
💡 This allows making the following Request to LiteLLM Proxy
|
||||
```shell
|
||||
curl --request POST \
|
||||
--url http://localhost:4000/v1/rerank \
|
||||
--header 'accept: application/json' \
|
||||
--header 'content-type: application/json' \
|
||||
--data '{
|
||||
"model": "rerank-english-v3.0",
|
||||
"query": "What is the capital of the United States?",
|
||||
"top_n": 3,
|
||||
"documents": ["Carson City is the capital city of the American state of Nevada."]
|
||||
}'
|
||||
```
|
||||
|
||||
## Tutorial - Setup Cohere Re-Rank Endpoint on LiteLLM Proxy
|
||||
|
||||
**Step 1** Define pass through routes on [litellm config.yaml](configs.md)
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
pass_through_endpoints:
|
||||
- path: "/v1/rerank" # route you want to add to LiteLLM Proxy Server
|
||||
target: "https://api.cohere.com/v1/rerank" # URL this route should forward requests to
|
||||
headers: # headers to forward to this URL
|
||||
Authorization: "bearer os.environ/COHERE_API_KEY" # (Optional) Auth Header to forward to your Endpoint
|
||||
content-type: application/json # (Optional) Extra Headers to pass to this endpoint
|
||||
accept: application/json
|
||||
```
|
||||
|
||||
**Step 2** Start Proxy Server in detailed_debug mode
|
||||
|
||||
```shell
|
||||
litellm --config config.yaml
|
||||
```
|
||||
**Step 3** Make Request to pass through endpoint
|
||||
|
||||
```shell
|
||||
curl --request POST \
|
||||
--url http://localhost:4000/v1/rerank \
|
||||
--header 'accept: application/json' \
|
||||
--header 'content-type: application/json' \
|
||||
--data '{
|
||||
"model": "rerank-english-v3.0",
|
||||
"query": "What is the capital of the United States?",
|
||||
"top_n": 3,
|
||||
"documents": ["Carson City is the capital city of the American state of Nevada.",
|
||||
"The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
|
||||
"Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.",
|
||||
"Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.",
|
||||
"Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."]
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
🎉 **Expected Response**
|
||||
|
||||
This request got forwarded from LiteLLM Proxy -> Defined Target URL (with headers)
|
||||
|
||||
```shell
|
||||
{
|
||||
"id": "37103a5b-8cfb-48d3-87c7-da288bedd429",
|
||||
"results": [
|
||||
{
|
||||
"index": 2,
|
||||
"relevance_score": 0.999071
|
||||
},
|
||||
{
|
||||
"index": 4,
|
||||
"relevance_score": 0.7867867
|
||||
},
|
||||
{
|
||||
"index": 0,
|
||||
"relevance_score": 0.32713068
|
||||
}
|
||||
],
|
||||
"meta": {
|
||||
"api_version": {
|
||||
"version": "1"
|
||||
},
|
||||
"billed_units": {
|
||||
"search_units": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
|
@ -48,6 +48,7 @@ const sidebars = {
|
|||
"proxy/billing",
|
||||
"proxy/user_keys",
|
||||
"proxy/virtual_keys",
|
||||
"proxy/token_auth",
|
||||
"proxy/alerting",
|
||||
{
|
||||
type: "category",
|
||||
|
@ -56,11 +57,11 @@ const sidebars = {
|
|||
},
|
||||
"proxy/ui",
|
||||
"proxy/prometheus",
|
||||
"proxy/pass_through",
|
||||
"proxy/email",
|
||||
"proxy/multiple_admins",
|
||||
"proxy/team_based_routing",
|
||||
"proxy/customer_routing",
|
||||
"proxy/token_auth",
|
||||
{
|
||||
type: "category",
|
||||
label: "Extra Load Balancing",
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue