litellm-mirror/litellm_server

litellm-server

A simple, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.

Usage

docker run -e PORT=8000 -e OPENAI_API_KEY=<your-openai-key> -p 8000:8000 ghcr.io/berriai/litellm:latest

# UVICORN: OpenAI Proxy running on http://0.0.0.0:8000

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

See how to call Huggingface,Bedrock,TogetherAI,Anthropic, etc.

Endpoints:

• /chat/completions - chat completions endpoint to call 100+ LLMs
• /router/completions - for multiple deployments of the same model (e.g. Azure OpenAI), uses the least used deployment. Learn more
• /models - available models on server

Running Locally

$ git clone https://github.com/BerriAI/litellm.git

$ cd ./litellm/litellm_server

$ uvicorn main:app --host 0.0.0.0 --port 8000

Custom Config

1. Create + Modify router_config.yaml (save your azure/openai/etc. deployment info)

cp ./router_config_template.yaml ./router_config.yaml

2. Build Docker Image

docker build -t litellm_server . --build-arg CONFIG_FILE=./router_config.yaml 

3. Run Docker Image

docker run --name litellm-proxy -e PORT=8000 -p 8000:8000 litellm_server