mirror of
https://github.com/BerriAI/litellm.git
synced 2025-04-26 11:14:04 +00:00
cfb6890b9f
* docs(sidebar.js): docs for support model access groups for wildcard routes * feat(key_management_endpoints.py): add check if user is premium_user when adding model access group for wildcard route * refactor(docs/): make control model access a root-level doc in proxy sidebar easier to discover how to control model access on litellm * docs: more cleanup * feat(fireworks_ai/): add document inlining support Enables user to call non-vision models with images/pdfs/etc. * test(test_fireworks_ai_translation.py): add unit testing for fireworks ai transform inline helper util * docs(docs/): add document inlining details to fireworks ai docs * feat(fireworks_ai/): allow user to dynamically disable auto add transform inline allows client-side disabling of this feature for proxy users * feat(fireworks_ai/): return 'supports_vision' and 'supports_pdf_input' true on all fireworks ai models now true as fireworks ai supports document inlining * test: fix tests * fix(router.py): add unit testing for _is_model_access_group_for_wildcard_route
143 lines
No EOL
4 KiB
Markdown
143 lines
No EOL
4 KiB
Markdown
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
|
|
# Provider specific Wildcard routing
|
|
|
|
**Proxy all models from a provider**
|
|
|
|
Use this if you want to **proxy all models from a specific provider without defining them on the config.yaml**
|
|
|
|
## Step 1. Define provider specific routing
|
|
|
|
<Tabs>
|
|
<TabItem value="sdk" label="SDK">
|
|
|
|
```python
|
|
from litellm import Router
|
|
|
|
router = Router(
|
|
model_list=[
|
|
{
|
|
"model_name": "anthropic/*",
|
|
"litellm_params": {
|
|
"model": "anthropic/*",
|
|
"api_key": os.environ["ANTHROPIC_API_KEY"]
|
|
}
|
|
},
|
|
{
|
|
"model_name": "groq/*",
|
|
"litellm_params": {
|
|
"model": "groq/*",
|
|
"api_key": os.environ["GROQ_API_KEY"]
|
|
}
|
|
},
|
|
{
|
|
"model_name": "fo::*:static::*", # all requests matching this pattern will be routed to this deployment, example: model="fo::hi::static::hi" will be routed to deployment: "openai/fo::*:static::*"
|
|
"litellm_params": {
|
|
"model": "openai/fo::*:static::*",
|
|
"api_key": os.environ["OPENAI_API_KEY"]
|
|
}
|
|
}
|
|
]
|
|
)
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="proxy" label="PROXY">
|
|
|
|
**Step 1** - define provider specific routing on config.yaml
|
|
```yaml
|
|
model_list:
|
|
# provider specific wildcard routing
|
|
- model_name: "anthropic/*"
|
|
litellm_params:
|
|
model: "anthropic/*"
|
|
api_key: os.environ/ANTHROPIC_API_KEY
|
|
- model_name: "groq/*"
|
|
litellm_params:
|
|
model: "groq/*"
|
|
api_key: os.environ/GROQ_API_KEY
|
|
- model_name: "fo::*:static::*" # all requests matching this pattern will be routed to this deployment, example: model="fo::hi::static::hi" will be routed to deployment: "openai/fo::*:static::*"
|
|
litellm_params:
|
|
model: "openai/fo::*:static::*"
|
|
api_key: os.environ/OPENAI_API_KEY
|
|
```
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
## [PROXY-Only] Step 2 - Run litellm proxy
|
|
|
|
```shell
|
|
$ litellm --config /path/to/config.yaml
|
|
```
|
|
|
|
## Step 3 - Test it
|
|
|
|
<Tabs>
|
|
<TabItem value="sdk" label="SDK">
|
|
|
|
```python
|
|
from litellm import Router
|
|
|
|
router = Router(model_list=...)
|
|
|
|
# Test with `anthropic/` - all models with `anthropic/` prefix will get routed to `anthropic/*`
|
|
resp = completion(model="anthropic/claude-3-sonnet-20240229", messages=[{"role": "user", "content": "Hello, Claude!"}])
|
|
print(resp)
|
|
|
|
# Test with `groq/` - all models with `groq/` prefix will get routed to `groq/*`
|
|
resp = completion(model="groq/llama3-8b-8192", messages=[{"role": "user", "content": "Hello, Groq!"}])
|
|
print(resp)
|
|
|
|
# Test with `fo::*::static::*` - all requests matching this pattern will be routed to `openai/fo::*:static::*`
|
|
resp = completion(model="fo::hi::static::hi", messages=[{"role": "user", "content": "Hello, Claude!"}])
|
|
print(resp)
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="proxy" label="PROXY">
|
|
|
|
Test with `anthropic/` - all models with `anthropic/` prefix will get routed to `anthropic/*`
|
|
```bash
|
|
curl http://localhost:4000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer sk-1234" \
|
|
-d '{
|
|
"model": "anthropic/claude-3-sonnet-20240229",
|
|
"messages": [
|
|
{"role": "user", "content": "Hello, Claude!"}
|
|
]
|
|
}'
|
|
```
|
|
|
|
Test with `groq/` - all models with `groq/` prefix will get routed to `groq/*`
|
|
```shell
|
|
curl http://localhost:4000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer sk-1234" \
|
|
-d '{
|
|
"model": "groq/llama3-8b-8192",
|
|
"messages": [
|
|
{"role": "user", "content": "Hello, Claude!"}
|
|
]
|
|
}'
|
|
```
|
|
|
|
Test with `fo::*::static::*` - all requests matching this pattern will be routed to `openai/fo::*:static::*`
|
|
```shell
|
|
curl http://localhost:4000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer sk-1234" \
|
|
-d '{
|
|
"model": "fo::hi::static::hi",
|
|
"messages": [
|
|
{"role": "user", "content": "Hello, Claude!"}
|
|
]
|
|
}'
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
|
|
## [[PROXY-Only] Control Wildcard Model Access](./proxy/model_access#-control-access-on-wildcard-models) |