litellm-mirror/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LLM Ops Stack  - LiteLLM Proxy + Langfuse \n",
    "\n",
    "This notebook demonstrates how to use LiteLLM Proxy with Langfuse \n",
    "- Use LiteLLM Proxy for calling 100+ LLMs in OpenAI format\n",
    "- Use Langfuse for viewing request / response traces \n",
    "\n",
    "\n",
    "In this notebook we will setup LiteLLM Proxy to make requests to OpenAI, Anthropic, Bedrock and automatically log traces to Langfuse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup LiteLLM Proxy\n",
    "\n",
    "### 1.1 Define .env variables \n",
    "Define .env variables on the container that litellm proxy is running on.\n",
    "```bash\n",
    "## LLM API Keys\n",
    "OPENAI_API_KEY=sk-proj-1234567890\n",
    "ANTHROPIC_API_KEY=sk-ant-api03-1234567890\n",
    "AWS_ACCESS_KEY_ID=1234567890\n",
    "AWS_SECRET_ACCESS_KEY=1234567890\n",
    "\n",
    "## Langfuse Logging \n",
    "LANGFUSE_PUBLIC_KEY=\"pk-lf-xxxx9\"\n",
    "LANGFUSE_SECRET_KEY=\"sk-lf-xxxx9\"\n",
    "LANGFUSE_HOST=\"https://us.cloud.langfuse.com\"\n",
    "```\n",
    "\n",
    "\n",
    "### 1.1 Setup LiteLLM Proxy Config yaml \n",
    "```yaml\n",
    "model_list:\n",
    "  - model_name: gpt-4o\n",
    "    litellm_params:\n",
    "      model: openai/gpt-4o\n",
    "      api_key: os.environ/OPENAI_API_KEY\n",
    "  - model_name: claude-3-5-sonnet-20241022\n",
    "    litellm_params:\n",
    "      model: anthropic/claude-3-5-sonnet-20241022\n",
    "      api_key: os.environ/ANTHROPIC_API_KEY\n",
    "  - model_name: us.amazon.nova-micro-v1:0\n",
    "    litellm_params:\n",
    "      model: bedrock/us.amazon.nova-micro-v1:0\n",
    "      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID\n",
    "      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY\n",
    "\n",
    "litellm_settings:\n",
    "  callbacks: [\"langfuse\"]\n",
    "\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Make LLM Requests to LiteLLM Proxy\n",
    "\n",
    "Now we will make our first LLM request to LiteLLM Proxy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Setup Client Side Variables to point to LiteLLM Proxy\n",
    "Set `LITELLM_PROXY_BASE_URL` to the base url of the LiteLLM Proxy and `LITELLM_VIRTUAL_KEY` to the virtual key you want to use for Authentication to LiteLLM Proxy. (Note: In this initial setup you can)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "LITELLM_PROXY_BASE_URL=\"http://0.0.0.0:4000\"\n",
    "LITELLM_VIRTUAL_KEY=\"sk-oXXRa1xxxxxxxxxxx\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatCompletion(id='chatcmpl-B0sq6QkOKNMJ0dwP3x7OoMqk1jZcI', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Langfuse is a platform designed to monitor, observe, and troubleshoot AI and large language model (LLM) applications. It provides features that help developers gain insights into how their AI systems are performing, make debugging easier, and optimize the deployment of models. Langfuse allows for tracking of model interactions, collecting telemetry, and visualizing data, which is crucial for understanding the behavior of AI models in production environments. This kind of tool is particularly useful for developers working with language models who need to ensure reliability and efficiency in their applications.', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739550502, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_523b9b6e5f', usage=CompletionUsage(completion_tokens=109, prompt_tokens=13, total_tokens=122, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import openai\n",
    "client = openai.OpenAI(\n",
    "    api_key=LITELLM_VIRTUAL_KEY,\n",
    "    base_url=LITELLM_PROXY_BASE_URL\n",
    ")\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=\"gpt-4o\",\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"what is Langfuse?\"\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 View Traces on Langfuse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Call Anthropic, Bedrock models "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Advanced - Set Langfuse Trace ID, Tags, Metadata "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## "
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}