forked from phoenix/litellm-mirror
docs - add pass through routes on litelm proxy
This commit is contained in:
parent
8f2931937a
commit
ac066462df
2 changed files with 99 additions and 1 deletions
97
docs/my-website/docs/proxy/pass_through.md
Normal file
97
docs/my-website/docs/proxy/pass_through.md
Normal file
|
@ -0,0 +1,97 @@
|
||||||
|
# ➡️ Create Pass Through Endpoints
|
||||||
|
|
||||||
|
Add pass through routes to LiteLLM Proxy
|
||||||
|
|
||||||
|
**Example:** Add a route `/v1/rerank` that forwards requests to `https://api.cohere.com/v1/rerank` through LiteLLM Proxy
|
||||||
|
|
||||||
|
|
||||||
|
💡 This allows making the following Request to LiteLLM Proxy
|
||||||
|
```shell
|
||||||
|
curl --request POST \
|
||||||
|
--url http://localhost:4000/v1/rerank \
|
||||||
|
--header 'accept: application/json' \
|
||||||
|
--header 'content-type: application/json' \
|
||||||
|
--data '{
|
||||||
|
"model": "rerank-english-v3.0",
|
||||||
|
"query": "What is the capital of the United States?",
|
||||||
|
"top_n": 3,
|
||||||
|
"documents": ["Carson City is the capital city of the American state of Nevada."]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tutorial - Setup Cohere Re-Rank Endpoint on LiteLLM Proxy
|
||||||
|
|
||||||
|
**Step 1** Define pass through routes on [litellm config.yaml](configs.md)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
general_settings:
|
||||||
|
master_key: sk-1234
|
||||||
|
pass_through_endpoints:
|
||||||
|
- path: "/v1/rerank" # route you want to add to LiteLLM Proxy Server
|
||||||
|
target: "https://api.cohere.com/v1/rerank" # URL this route should forward requests to
|
||||||
|
headers: # headers to forward to this URL
|
||||||
|
Authorization: "bearer os.environ/COHERE_API_KEY" # (Optional) Auth Header to forward to your Endpoint
|
||||||
|
content-type: application/json # (Optional) Extra Headers to pass to this endpoint
|
||||||
|
accept: application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2** Start Proxy Server in detailed_debug mode
|
||||||
|
|
||||||
|
```shell
|
||||||
|
litellm --config config.yaml
|
||||||
|
```
|
||||||
|
**Step 3** Make Request to pass through endpoint
|
||||||
|
|
||||||
|
```shell
|
||||||
|
curl --request POST \
|
||||||
|
--url http://localhost:4000/v1/rerank \
|
||||||
|
--header 'accept: application/json' \
|
||||||
|
--header 'content-type: application/json' \
|
||||||
|
--data '{
|
||||||
|
"model": "rerank-english-v3.0",
|
||||||
|
"query": "What is the capital of the United States?",
|
||||||
|
"top_n": 3,
|
||||||
|
"documents": ["Carson City is the capital city of the American state of Nevada.",
|
||||||
|
"The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
|
||||||
|
"Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.",
|
||||||
|
"Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.",
|
||||||
|
"Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
🎉 **Expected Response**
|
||||||
|
|
||||||
|
This request got forwarded from LiteLLM Proxy -> Defined Target URL (with headers)
|
||||||
|
|
||||||
|
```shell
|
||||||
|
{
|
||||||
|
"id": "37103a5b-8cfb-48d3-87c7-da288bedd429",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"index": 2,
|
||||||
|
"relevance_score": 0.999071
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"index": 4,
|
||||||
|
"relevance_score": 0.7867867
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"index": 0,
|
||||||
|
"relevance_score": 0.32713068
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"meta": {
|
||||||
|
"api_version": {
|
||||||
|
"version": "1"
|
||||||
|
},
|
||||||
|
"billed_units": {
|
||||||
|
"search_units": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -48,6 +48,7 @@ const sidebars = {
|
||||||
"proxy/billing",
|
"proxy/billing",
|
||||||
"proxy/user_keys",
|
"proxy/user_keys",
|
||||||
"proxy/virtual_keys",
|
"proxy/virtual_keys",
|
||||||
|
"proxy/token_auth",
|
||||||
"proxy/alerting",
|
"proxy/alerting",
|
||||||
{
|
{
|
||||||
type: "category",
|
type: "category",
|
||||||
|
@ -56,11 +57,11 @@ const sidebars = {
|
||||||
},
|
},
|
||||||
"proxy/ui",
|
"proxy/ui",
|
||||||
"proxy/prometheus",
|
"proxy/prometheus",
|
||||||
|
"proxy/pass_through",
|
||||||
"proxy/email",
|
"proxy/email",
|
||||||
"proxy/multiple_admins",
|
"proxy/multiple_admins",
|
||||||
"proxy/team_based_routing",
|
"proxy/team_based_routing",
|
||||||
"proxy/customer_routing",
|
"proxy/customer_routing",
|
||||||
"proxy/token_auth",
|
|
||||||
{
|
{
|
||||||
type: "category",
|
type: "category",
|
||||||
label: "Extra Load Balancing",
|
label: "Extra Load Balancing",
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue