mirror of
https://github.com/BerriAI/litellm.git
synced 2025-04-25 02:34:29 +00:00
update playground tutorial
This commit is contained in:
parent
7819ffbbcc
commit
a598ae4a7e
4 changed files with 197 additions and 0 deletions
|
@ -1 +1,181 @@
|
||||||
# Create your first playground
|
# Create your first playground
|
||||||
|
import Image from '@theme/IdealImage';
|
||||||
|
|
||||||
|
Learn how to build a light version of the demo playground as shown on the website in less than 10 minutes.
|
||||||
|
|
||||||
|
**What we'll build**: We'll build <u>the server</u> and connect it to our template frontend, ending up with a deployed playground by the end!
|
||||||
|
|
||||||
|
|
||||||
|
:::info
|
||||||
|
|
||||||
|
Before you start with this section, make sure you have followed the [environment-setup](./installation) guide. Please note, that this demo relies on you having API keys from at least 1 model provider (E.g. OpenAI).
|
||||||
|
:::
|
||||||
|
|
||||||
|
## 1. Test keys
|
||||||
|
|
||||||
|
Let's make sure our keys are working. Run this script in any environment of your choice (e.g. [Google Colab](https://colab.research.google.com/#create=true)).
|
||||||
|
|
||||||
|
🚨 Don't forget to replace the placeholder key values with your keys!
|
||||||
|
|
||||||
|
```python
|
||||||
|
pip install litellm
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
from litellm import completion
|
||||||
|
|
||||||
|
## set ENV variables
|
||||||
|
os.environ["OPENAI_API_KEY"] = "openai key" ## REPLACE THIS
|
||||||
|
os.environ["COHERE_API_KEY"] = "cohere key" ## REPLACE THIS
|
||||||
|
os.environ["AI21_API_KEY"] = "ai21 key" ## REPLACE THIS
|
||||||
|
|
||||||
|
|
||||||
|
messages = [{ "content": "Hello, how are you?","role": "user"}]
|
||||||
|
|
||||||
|
# openai call
|
||||||
|
response = completion(model="gpt-3.5-turbo", messages=messages)
|
||||||
|
|
||||||
|
# cohere call
|
||||||
|
response = completion("command-nightly", messages)
|
||||||
|
|
||||||
|
# ai21 call
|
||||||
|
response = completion("j2-mid", messages)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Set-up Server
|
||||||
|
|
||||||
|
### 2.1 Spin-up Template
|
||||||
|
Let's build a basic Flask app as our backend server.
|
||||||
|
|
||||||
|
Create a `main.py` file, and put in this starter code.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from flask import Flask, jsonify, request
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
|
||||||
|
# Example route
|
||||||
|
@app.route('/', methods=['GET'])
|
||||||
|
def hello():
|
||||||
|
return jsonify(message="Hello, Flask!")
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
from waitress import serve
|
||||||
|
serve(app, host="0.0.0.0", port=4000, threads=500)
|
||||||
|
```
|
||||||
|
|
||||||
|
Let's test that it's working.
|
||||||
|
|
||||||
|
Start the server:
|
||||||
|
```python
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Run a curl command to test it:
|
||||||
|
```curl
|
||||||
|
curl -X GET localhost:4000
|
||||||
|
```
|
||||||
|
|
||||||
|
This is what you should see
|
||||||
|
|
||||||
|
<Image img={require('../../img/test_python_server_1.png')} alt="python_code_sample_1" />
|
||||||
|
|
||||||
|
### 2.2 Add `completion` route
|
||||||
|
|
||||||
|
Now, let's add a route for our completion calls. This is when we'll add litellm to our server to handle the model requests.
|
||||||
|
|
||||||
|
**Notes**:
|
||||||
|
* 🚨 Don't forget to replace the placeholder key values with your keys!
|
||||||
|
* `completion_with_retries`: LLM API calls can fail in production. This function wraps the normal litellm completion() call with [tenacity](https://tenacity.readthedocs.io/en/latest/) to retry the call in case it fails.
|
||||||
|
|
||||||
|
The snippet we'll add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import os
|
||||||
|
from litellm import completion_with_retries
|
||||||
|
|
||||||
|
## set ENV variables
|
||||||
|
os.environ["OPENAI_API_KEY"] = "openai key" ## REPLACE THIS
|
||||||
|
os.environ["COHERE_API_KEY"] = "cohere key" ## REPLACE THIS
|
||||||
|
os.environ["AI21_API_KEY"] = "ai21 key" ## REPLACE THIS
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/chat/completions', methods=["POST"])
|
||||||
|
def api_completion():
|
||||||
|
data = request.json
|
||||||
|
data["max_tokens"] = 256 # By default let's set max_tokens to 256
|
||||||
|
try:
|
||||||
|
# COMPLETION CALL
|
||||||
|
response = completion_with_retries(**data)
|
||||||
|
except Exception as e:
|
||||||
|
# print the error
|
||||||
|
print(e)
|
||||||
|
return response
|
||||||
|
```
|
||||||
|
|
||||||
|
The complete code:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import os
|
||||||
|
from flask import Flask, jsonify, request
|
||||||
|
from litellm import completion_with_retries
|
||||||
|
|
||||||
|
|
||||||
|
## set ENV variables
|
||||||
|
os.environ["OPENAI_API_KEY"] = "openai key" ## REPLACE THIS
|
||||||
|
os.environ["COHERE_API_KEY"] = "cohere key" ## REPLACE THIS
|
||||||
|
os.environ["AI21_API_KEY"] = "ai21 key" ## REPLACE THIS
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
|
||||||
|
# Example route
|
||||||
|
@app.route('/', methods=['GET'])
|
||||||
|
def hello():
|
||||||
|
return jsonify(message="Hello, Flask!")
|
||||||
|
|
||||||
|
@app.route('/chat/completions', methods=["POST"])
|
||||||
|
def api_completion():
|
||||||
|
data = request.json
|
||||||
|
data["max_tokens"] = 256 # By default let's set max_tokens to 256
|
||||||
|
try:
|
||||||
|
# COMPLETION CALL
|
||||||
|
response = completion_with_retries(**data)
|
||||||
|
except Exception as e:
|
||||||
|
# print the error
|
||||||
|
print(e)
|
||||||
|
|
||||||
|
return response
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
from waitress import serve
|
||||||
|
serve(app, host="0.0.0.0", port=4000, threads=500)
|
||||||
|
```
|
||||||
|
|
||||||
|
Start the server:
|
||||||
|
```python
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Run this curl command to test it:
|
||||||
|
```curl
|
||||||
|
curl -X POST localhost:4000/chat/completions \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{
|
||||||
|
"model": "gpt-3.5-turbo",
|
||||||
|
"messages": [{
|
||||||
|
"content": "Hello, how are you?",
|
||||||
|
"role": "user"
|
||||||
|
}]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
This is what you should see
|
||||||
|
|
||||||
|
<Image img={require('../../img/test_python_server_2.png')} alt="python_code_sample_2" />
|
||||||
|
|
||||||
|
## 3. Connect to our frontend template
|
||||||
|
|
||||||
|
|
||||||
|
## 4. Deploy!
|
||||||
|
|
||||||
|
|
||||||
|
|
17
docs/my-website/docs/tutorials/installation.md
Normal file
17
docs/my-website/docs/tutorials/installation.md
Normal file
|
@ -0,0 +1,17 @@
|
||||||
|
---
|
||||||
|
displayed_sidebar: tutorialSidebar
|
||||||
|
---
|
||||||
|
|
||||||
|
# Set up environment
|
||||||
|
|
||||||
|
Let's get the necessary keys to set up our demo environment.
|
||||||
|
|
||||||
|
## 1. Get your keys
|
||||||
|
|
||||||
|
Every LLM provider needs API keys (e.g. `OPENAI_API_KEY`). For this demo, let's get the API Keys for OpenAI, Cohere, and AI21.
|
||||||
|
|
||||||
|
**OpenAI**: https://platform.openai.com/account/api-keys
|
||||||
|
**Cohere**: https://dashboard.cohere.com/welcome/login?redirect_uri=%2Fapi-keys
|
||||||
|
**AI21**: https://studio.ai21.com/account/api-key
|
||||||
|
|
||||||
|
|
BIN
docs/my-website/img/test_python_server_1.png
Normal file
BIN
docs/my-website/img/test_python_server_1.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 94 KiB |
BIN
docs/my-website/img/test_python_server_2.png
Normal file
BIN
docs/my-website/img/test_python_server_2.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 254 KiB |
Loading…
Add table
Add a link
Reference in a new issue