🚅 LiteLLM - A/B Testing LLMs in Production
Call all LLM APIs using the OpenAI format [Anthropic, Huggingface, Cohere, Azure OpenAI etc.]
LiteLLM allows you to call 100+ LLMs using completion
This template server allows you to define LLMs with their A/B test ratios
```python
llm_dict = {
"gpt-4": 0.2,
"together_ai/togethercomputer/llama-2-70b-chat": 0.4,
"claude-2": 0.2,
"claude-1.2": 0.2
}
```
All models defined can be called with the same Input/Output format using litellm `completion`
```python
from litellm import completion
# SET API KEYS in .env
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion(model="command-nightly", messages=messages)
# anthropic
response = completion(model="claude-2", messages=messages)
```
After calling `completion()` costs and latency can be viewed on the LiteLLM Client UI
### LiteLLM Client UI
## Using LiteLLM A/B Testing Server
# Installation
```
pip install litellm
```
Stable version
```
pip install litellm==0.1.424
```
## Clone LiteLLM Git Repo
```
git clone https://github.com/BerriAI/litellm/
```
## Navigate to LiteLLM-A/B Test Server
```
cd litellm/cookbook/llm-ab-test-server
```
## Run the Server
```
python3 main.py
```
## Set your LLM Configs
Set your LLMs and LLM weights you want to run A/B testing with
# support / talk with founders
- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
- Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
# why did we build this
- **Need for simplicity**: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI, Cohere