forked from phoenix/litellm-mirror
(docs) config
This commit is contained in:
parent
9493aeff68
commit
cc955fca89
1 changed files with 10 additions and 23 deletions
|
@ -589,7 +589,7 @@ Curl Command
|
||||||
curl --location 'http://0.0.0.0:8000/chat/completions' \
|
curl --location 'http://0.0.0.0:8000/chat/completions' \
|
||||||
--header 'Content-Type: application/json' \
|
--header 'Content-Type: application/json' \
|
||||||
--data ' {
|
--data ' {
|
||||||
"model": "gpt-4-team1",
|
"model": "zephyr-alpha",
|
||||||
"messages": [
|
"messages": [
|
||||||
{
|
{
|
||||||
"role": "user",
|
"role": "user",
|
||||||
|
@ -599,24 +599,7 @@ curl --location 'http://0.0.0.0:8000/chat/completions' \
|
||||||
}
|
}
|
||||||
'
|
'
|
||||||
```
|
```
|
||||||
**Setting model name**
|
|
||||||
```python
|
|
||||||
import openai
|
|
||||||
openai.api_base = "http://0.0.0.0:8000"
|
|
||||||
|
|
||||||
completion = openai.chat.completions.create(model="zephyr-alpha", messages=[{"role": "user", "content": "Hello world"}])
|
|
||||||
print(completion.choices[0].message.content)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Setting API Base with model name**
|
|
||||||
If you're repo only let's you specify api base, then you can add the model name to the api base passed in -
|
|
||||||
```python
|
|
||||||
import openai
|
|
||||||
openai.api_base = "http://0.0.0.0:8000/openai/deployments/zephyr-alpha/chat/completions" # zephyr-alpha will be used
|
|
||||||
|
|
||||||
completion = openai.chat.completions.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hello world"}])
|
|
||||||
print(completion.choices[0].message.content)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Save Model-specific params (API Base, API Keys, Temperature, Headers etc.)
|
### Save Model-specific params (API Base, API Keys, Temperature, Headers etc.)
|
||||||
You can use the config to save model-specific information like api_base, api_key, temperature, max_tokens, etc.
|
You can use the config to save model-specific information like api_base, api_key, temperature, max_tokens, etc.
|
||||||
|
@ -656,15 +639,19 @@ Set a model alias for your deployments.
|
||||||
|
|
||||||
In the `config.yaml` the model_name parameter is the user-facing name to use for your deployment.
|
In the `config.yaml` the model_name parameter is the user-facing name to use for your deployment.
|
||||||
|
|
||||||
E.g.: If we want to save a Huggingface TGI Mistral-7b deployment, as 'mistral-7b' for our users, we might save it as:
|
In the config below requests with `model=gpt-4` will route to `ollama/zephyr`
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
model_list:
|
model_list:
|
||||||
- model_name: mistral-7b # ALIAS
|
- model_name: text-davinci-003
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: huggingface/mistralai/Mistral-7B-Instruct-v0.1 # ACTUAL NAME
|
model: ollama/zephyr
|
||||||
api_key: your_huggingface_api_key # [OPTIONAL] if deployed on huggingface inference endpoints
|
- model_name: gpt-4
|
||||||
api_base: your_api_base # url where model is deployed
|
litellm_params:
|
||||||
|
model: ollama/llama2
|
||||||
|
- model_name: gpt-3.5-turbo
|
||||||
|
litellm_params:
|
||||||
|
model: ollama/llama2
|
||||||
```
|
```
|
||||||
|
|
||||||
### Set Custom Prompt Templates
|
### Set Custom Prompt Templates
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue