{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": { "id": "Ys9n20Es2IzT" }, "source": [ "# Evaluate Multiple LLM Providers with LiteLLM\n", "\n", "\n", "\n", "* Quality Testing\n", "* Load Testing\n", "* Duration Testing\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ZXOXl23PIIP6" }, "outputs": [], "source": [ "!pip install litellm python-dotenv" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "LINuBzXDItq2" }, "outputs": [], "source": [ "import litellm\n", "from litellm import load_test_model, testing_batch_completion\n", "import time" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "EkxMhsWdJdu4" }, "outputs": [], "source": [ "import os \n", "os.environ[\"OPENAI_API_KEY\"] = \"...\"\n", "os.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\n", "os.environ[\"REPLICATE_API_KEY\"] = \"...\"" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "id": "mv5XdnqeW5I_" }, "source": [ "# Quality Test endpoint\n", "\n", "## Test the same prompt across multiple LLM providers\n", "\n", "In this example, let's ask some questions about Paul Graham" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "XpzrR5m4W_Us" }, "outputs": [], "source": [ "models = [\"gpt-3.5-turbo\", \"gpt-3.5-turbo-16k\", \"gpt-4\", \"claude-instant-1\", {\"model\": \"replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781\", \"custom_llm_provider\": \"replicate\"}]\n", "context = \"\"\"Paul Graham (/ɡræm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a \"hacker philosopher\".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016.\"\"\"\n", "prompts = [\"Who is Paul Graham?\", \"What is Paul Graham known for?\" , \"Is paul graham a writer?\" , \"Where does Paul Graham live?\", \"What has Paul Graham done?\"]\n", "messages = [[{\"role\": \"user\", \"content\": context + \"\\n\" + prompt}] for prompt in prompts] # pass in a list of messages we want to test\n", "result = testing_batch_completion(models=models, messages=messages)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "id": "9nzeLySnvIIW" }, "source": [ "## Visualize the data" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 403 }, "id": "X-2n7hdAuVAY", "outputId": "69cc0de1-68e3-4c12-a8ea-314880010d94" }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
Model Name | \n", "claude-instant-1 | \n", "gpt-3.5-turbo-0613 | \n", "gpt-3.5-turbo-16k-0613 | \n", "gpt-4-0613 | \n", "replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781 | \n", "
---|---|---|---|---|---|
Prompt | \n", "\n", " | \n", " | \n", " | \n", " | \n", " |
\\nIs paul graham a writer? | \n", "Yes, Paul Graham is considered a writer in ad... | \n", "Yes, Paul Graham is a writer. He has written s... | \n", "Yes, Paul Graham is a writer. He has authored ... | \n", "Yes, Paul Graham is a writer. He is an essayis... | \n", "Yes, Paul Graham is an author. According to t... | \n", "
\\nWhat has Paul Graham done? | \n", "Paul Graham has made significant contribution... | \n", "Paul Graham has achieved several notable accom... | \n", "Paul Graham has made significant contributions... | \n", "Paul Graham is known for his work on the progr... | \n", "Paul Graham has had a diverse career in compu... | \n", "
\\nWhat is Paul Graham known for? | \n", "Paul Graham is known for several things:\\n\\n-... | \n", "Paul Graham is known for his work on the progr... | \n", "Paul Graham is known for his work on the progr... | \n", "Paul Graham is known for his work on the progr... | \n", "Paul Graham is known for many things, includi... | \n", "
\\nWhere does Paul Graham live? | \n", "Based on the information provided:\\n\\n- Paul ... | \n", "According to the given information, Paul Graha... | \n", "Paul Graham currently lives in England, where ... | \n", "The text does not provide a current place of r... | \n", "Based on the information provided, Paul Graha... | \n", "
\\nWho is Paul Graham? | \n", "Paul Graham is an influential computer scient... | \n", "Paul Graham is an English computer scientist, ... | \n", "Paul Graham is an English computer scientist, ... | \n", "Paul Graham is an English computer scientist, ... | \n", "Paul Graham is an English computer scientist,... | \n", "