mirror of https://github.com/BerriAI/litellm.git synced 2025-04-27 11:43:54 +00:00

Krish Dholakia d02b9a111a fix(main.py): fix retries being multiplied when using openai sdk (#7221 )

* fix(main.py): fix retries being multiplied when using openai sdk

Closes https://github.com/BerriAI/litellm/pull/7130

* docs(prompt_management.md): add langfuse prompt management doc

* feat(team_endpoints.py): allow teams to add their own models

Enables teams to call their own finetuned models via the proxy

* test: add better enforcement check testing for `/model/new` now that teams can add their own models

* docs(team_model_add.md): tutorial for allowing teams to add their own models

* test: fix test

2024-12-14 11:56:55 -08:00

1.9 KiB

Raw Blame History

import Image from '@theme/IdealImage';

Prompt Management

LiteLLM supports using Langfuse for prompt management on the proxy.

Quick Start

Add Langfuse as a 'callback' in your config.yaml

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/chatgpt-v-2
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE

litellm_settings:
    callbacks: ["langfuse"] # 👈 KEY CHANGE

Start the proxy

litellm-proxy --config config.yaml

Test it!

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
    "model": "gpt-4",
    "messages": [
        {
            "role": "user",
            "content": "THIS WILL BE IGNORED"
        }
    ],
    "metadata": {
        "langfuse_prompt_id": "value",
        "langfuse_prompt_variables": { # [OPTIONAL]
            "key": "value"
        }
    }
}'

What is 'langfuse_prompt_id'?

langfuse_prompt_id: The ID of the prompt that will be used for the request.

What will the formatted prompt look like?

`/chat/completions` messages

The message will be added to the start of the prompt.

if the Langfuse prompt is a list, it will be added to the start of the messages list (assuming it's an OpenAI compatible message).
if the Langfuse prompt is a string, it will be added as a system message.

if isinstance(compiled_prompt, list):
    data["messages"] = compiled_prompt + data["messages"]
else:
    data["messages"] = [
        {"role": "system", "content": compiled_prompt}
    ] + data["messages"]

`/completions` messages