fix(proxy_server): cleaning up print statements

This commit is contained in:
Krrish Dholakia 2023-10-13 20:16:31 -07:00
parent d0e7c3016e
commit d9ddd7b338
6 changed files with 661 additions and 29 deletions

View file

@ -1,3 +1,5 @@
import Image from '@theme/IdealImage';
# Customize Prompt Templates on OpenAI-Compatible server
**You will learn:** How to set a custom prompt template on our OpenAI compatible server.
@ -11,15 +13,23 @@ $ litellm --model huggingface/codellama/CodeLlama-34b-Instruct-hf --api_base htt
# OpenAI compatible server running on http://0.0.0.0/8000
```
By default this will follow [our formatting](https://github.com/BerriAI/litellm/blob/9932371f883c55fd0f3142f91d9c40279e8fe241/litellm/llms/prompt_templates/factory.py#L10) for CodeLlama (based on the [Huggingface's documentation](https://huggingface.co/blog/codellama#conversational-instructions)).
In a new shell, run:
```shell
$ litellm --test
```
This will send a test request to our endpoint.
```python
[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>
{{ user_msg_1 }} [/INST] {{ model_answer_1 }} [INST] {{ user_msg_2 }} [/INST]
Now, let's see what got sent to huggingface. Run:
```shell
$ litellm --logs
```
This will return the most recent log (by default logs are stored in a local file called 'api_logs.json').
As we can see, this is the formatting sent to huggingface:
<Image img={require('../../img/codellama_input.png')} />
-
This follows [our formatting](https://github.com/BerriAI/litellm/blob/9932371f883c55fd0f3142f91d9c40279e8fe241/litellm/llms/prompt_templates/factory.py#L10) for CodeLlama (based on the [Huggingface's documentation](https://huggingface.co/blog/codellama#conversational-instructions)).
But this lacks BOS(`<s>`) and EOS(`</s>`) tokens.
@ -31,19 +41,22 @@ Our litellm server accepts prompt templates as part of a config file. You can sa
For now, let's just create a simple config file with our prompt template, and tell our server about it.
Create a file called `litellm_config.toml`, and here's how we'll format it:
Create a file called `litellm_config.toml`:
```toml
[model.<your-model-name>]
// our prompt template
```shell
$ touch litellm_config.toml
```
We want to add:
* BOS (`<s>`) tokens at the start of every System and Human message
* EOS (`</s>`) tokens at the end of every assistant message.
This is what it looks like:
```toml
Let's open our file in our terminal:
```shell
$ vi litellm_config.toml
```
paste our prompt template:
```shell
[model."huggingface/codellama/CodeLlama-34b-Instruct-hf".prompt_template]
MODEL_SYSTEM_MESSAGE_START_TOKEN = "<s>[INST] <<SYS>>\n]"
MODEL_SYSTEM_MESSAGE_END_TOKEN = "\n<</SYS>>\n [/INST]\n"
@ -55,11 +68,30 @@ MODEL_ASSISTANT_MESSAGE_START_TOKEN = ""
MODEL_ASSISTANT_MESSAGE_END_TOKEN = "</s>"
```
## Step 3: Save template
save our file (in vim):
```shell
:wq
```
## Step 3: Run new template
Let's save our custom template to our litellm server by running:
```shell
litellm --config -f ./litellm_config.toml
$ litellm --config -f ./litellm_config.toml
```
LiteLLM will save a copy of this file in it's package, so it can persist these settings across restarts.
Re-start our server:
```shell
$ litellm --model huggingface/codellama/CodeLlama-34b-Instruct-hf --api_base https://my-endpoint.com
```
LiteLLM will save a copy of this file in it's package, so it can persist these settings across restarts.
In a new shell, run:
```shell
$ litellm --test
```
See our new input prompt to Huggingface!
<Image img={require('../../img/codellama_formatted_input.png')} />
Congratulations 🎉

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB