forked from phoenix/litellm-mirror
fix(proxy_server): cleaning up print statements
This commit is contained in:
parent
d0e7c3016e
commit
d9ddd7b338
6 changed files with 661 additions and 29 deletions
|
@ -1,3 +1,5 @@
|
|||
import Image from '@theme/IdealImage';
|
||||
|
||||
# Customize Prompt Templates on OpenAI-Compatible server
|
||||
|
||||
**You will learn:** How to set a custom prompt template on our OpenAI compatible server.
|
||||
|
@ -11,15 +13,23 @@ $ litellm --model huggingface/codellama/CodeLlama-34b-Instruct-hf --api_base htt
|
|||
# OpenAI compatible server running on http://0.0.0.0/8000
|
||||
```
|
||||
|
||||
By default this will follow [our formatting](https://github.com/BerriAI/litellm/blob/9932371f883c55fd0f3142f91d9c40279e8fe241/litellm/llms/prompt_templates/factory.py#L10) for CodeLlama (based on the [Huggingface's documentation](https://huggingface.co/blog/codellama#conversational-instructions)).
|
||||
In a new shell, run:
|
||||
```shell
|
||||
$ litellm --test
|
||||
```
|
||||
This will send a test request to our endpoint.
|
||||
|
||||
```python
|
||||
[INST] <<SYS>>
|
||||
{{ system_prompt }}
|
||||
<</SYS>>
|
||||
|
||||
{{ user_msg_1 }} [/INST] {{ model_answer_1 }} [INST] {{ user_msg_2 }} [/INST]
|
||||
Now, let's see what got sent to huggingface. Run:
|
||||
```shell
|
||||
$ litellm --logs
|
||||
```
|
||||
This will return the most recent log (by default logs are stored in a local file called 'api_logs.json').
|
||||
|
||||
As we can see, this is the formatting sent to huggingface:
|
||||
|
||||
<Image img={require('../../img/codellama_input.png')} />
|
||||
-
|
||||
This follows [our formatting](https://github.com/BerriAI/litellm/blob/9932371f883c55fd0f3142f91d9c40279e8fe241/litellm/llms/prompt_templates/factory.py#L10) for CodeLlama (based on the [Huggingface's documentation](https://huggingface.co/blog/codellama#conversational-instructions)).
|
||||
|
||||
But this lacks BOS(`<s>`) and EOS(`</s>`) tokens.
|
||||
|
||||
|
@ -31,19 +41,22 @@ Our litellm server accepts prompt templates as part of a config file. You can sa
|
|||
|
||||
For now, let's just create a simple config file with our prompt template, and tell our server about it.
|
||||
|
||||
Create a file called `litellm_config.toml`, and here's how we'll format it:
|
||||
Create a file called `litellm_config.toml`:
|
||||
|
||||
```toml
|
||||
[model.<your-model-name>]
|
||||
// our prompt template
|
||||
```shell
|
||||
$ touch litellm_config.toml
|
||||
```
|
||||
|
||||
We want to add:
|
||||
* BOS (`<s>`) tokens at the start of every System and Human message
|
||||
* EOS (`</s>`) tokens at the end of every assistant message.
|
||||
|
||||
This is what it looks like:
|
||||
```toml
|
||||
Let's open our file in our terminal:
|
||||
```shell
|
||||
$ vi litellm_config.toml
|
||||
```
|
||||
|
||||
paste our prompt template:
|
||||
```shell
|
||||
[model."huggingface/codellama/CodeLlama-34b-Instruct-hf".prompt_template]
|
||||
MODEL_SYSTEM_MESSAGE_START_TOKEN = "<s>[INST] <<SYS>>\n]"
|
||||
MODEL_SYSTEM_MESSAGE_END_TOKEN = "\n<</SYS>>\n [/INST]\n"
|
||||
|
@ -55,11 +68,30 @@ MODEL_ASSISTANT_MESSAGE_START_TOKEN = ""
|
|||
MODEL_ASSISTANT_MESSAGE_END_TOKEN = "</s>"
|
||||
```
|
||||
|
||||
## Step 3: Save template
|
||||
save our file (in vim):
|
||||
```shell
|
||||
:wq
|
||||
```
|
||||
|
||||
## Step 3: Run new template
|
||||
|
||||
Let's save our custom template to our litellm server by running:
|
||||
```shell
|
||||
litellm --config -f ./litellm_config.toml
|
||||
$ litellm --config -f ./litellm_config.toml
|
||||
```
|
||||
LiteLLM will save a copy of this file in it's package, so it can persist these settings across restarts.
|
||||
|
||||
Re-start our server:
|
||||
```shell
|
||||
$ litellm --model huggingface/codellama/CodeLlama-34b-Instruct-hf --api_base https://my-endpoint.com
|
||||
```
|
||||
|
||||
LiteLLM will save a copy of this file in it's package, so it can persist these settings across restarts.
|
||||
In a new shell, run:
|
||||
```shell
|
||||
$ litellm --test
|
||||
```
|
||||
|
||||
See our new input prompt to Huggingface!
|
||||
<Image img={require('../../img/codellama_formatted_input.png')} />
|
||||
|
||||
Congratulations 🎉
|
BIN
docs/my-website/img/codellama_formatted_input.png
Normal file
BIN
docs/my-website/img/codellama_formatted_input.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 5.2 KiB |
BIN
docs/my-website/img/codellama_input.png
Normal file
BIN
docs/my-website/img/codellama_input.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 21 KiB |
Loading…
Add table
Add a link
Reference in a new issue