diff --git a/docs/quick_start.ipynb b/docs/quick_start.ipynb new file mode 100644 index 000000000..ff8151b7e --- /dev/null +++ b/docs/quick_start.ipynb @@ -0,0 +1,367 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c1e7571c", + "metadata": { + "id": "c1e7571c" + }, + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-llama/llama-stack/blob/main/docs/getting_started.ipynb)\n", + "\n", + "# Llama Stack - Building AI Applications\n", + "\n", + "\"drawing\"\n", + "\n", + "Get started with Llama Stack in minutes!\n", + "\n", + "[Llama Stack](https://github.com/meta-llama/llama-stack) is a stateful service with REST APIs to support the seamless transition of AI applications across different environments. You can build and test using a local server first and deploy to a hosted endpoint for production.\n", + "\n", + "In this guide, we'll walk through how to build a RAG application locally using Llama Stack with [Ollama](https://ollama.com/)\n", + "as the inference [provider](docs/source/providers/index.md#inference) for a Llama Model.\n" + ] + }, + { + "cell_type": "markdown", + "id": "4CV1Q19BDMVw", + "metadata": { + "id": "4CV1Q19BDMVw" + }, + "source": [ + "## Step 1: Install and setup" + ] + }, + { + "cell_type": "markdown", + "id": "K4AvfUAJZOeS", + "metadata": { + "id": "K4AvfUAJZOeS" + }, + "source": [ + "### 1.1. Install uv and test inference with Ollama\n", + "\n", + "We'll install [uv](https://docs.astral.sh/uv/) to setup the Python virtual environment, along with [colab-xterm](https://github.com/InfuseAI/colab-xterm) for running command-line tools, and [Ollama](https://ollama.com/download) as the inference provider." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7a2d7b85", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install uv llama_stack llama-stack-client\n", + "\n", + "## If running on Collab:\n", + "# !pip install colab-xterm\n", + "# %load_ext colabxterm\n", + "\n", + "!curl https://ollama.ai/install.sh | sh" + ] + }, + { + "cell_type": "markdown", + "id": "39fa584b", + "metadata": {}, + "source": [ + "### 1.2. Test inference with Ollama" + ] + }, + { + "cell_type": "markdown", + "id": "3bf81522", + "metadata": {}, + "source": [ + "We’ll now launch a terminal and run inference on a Llama model with Ollama to verify that the model is working correctly." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7e8e0f1", + "metadata": {}, + "outputs": [], + "source": [ + "## If running on Colab:\n", + "# %xterm\n", + "\n", + "## To be ran in the terminal:\n", + "# ollama serve &\n", + "# ollama run llama3.2:3b --keepalive 60m" + ] + }, + { + "cell_type": "markdown", + "id": "f3c5f243", + "metadata": {}, + "source": [ + "If successful, you should see the model respond to a prompt.\n", + "\n", + "...\n", + "```\n", + ">>> hi\n", + "Hello! How can I assist you today?\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "oDUB7M_qe-Gs", + "metadata": { + "id": "oDUB7M_qe-Gs" + }, + "source": [ + "## Step 2: Run the Llama Stack server\n", + "\n", + "In this showcase, we will start a Llama Stack server that is running locally." + ] + }, + { + "cell_type": "markdown", + "id": "732eadc6", + "metadata": {}, + "source": [ + "### 2.1. Setup the Llama Stack Server" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "J2kGed0R5PSf", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "collapsed": true, + "id": "J2kGed0R5PSf", + "outputId": "2478ea60-8d35-48a1-b011-f233831740c5" + }, + "outputs": [], + "source": [ + "import os \n", + "import subprocess\n", + "\n", + "if \"UV_SYSTEM_PYTHON\" in os.environ:\n", + " del os.environ[\"UV_SYSTEM_PYTHON\"]\n", + "\n", + "# this command installs all the dependencies needed for the llama stack server with the ollama inference provider\n", + "!uv run --with llama-stack llama stack build --template ollama --image-type venv --image-name myvenv\n", + "\n", + "def run_llama_stack_server_background():\n", + " log_file = open(\"llama_stack_server.log\", \"w\")\n", + " process = subprocess.Popen(\n", + " f\"uv run --with llama-stack llama stack run ollama --image-type venv --image-name myvenv --env INFERENCE_MODEL=llama3.2:3b\",\n", + " shell=True,\n", + " stdout=log_file,\n", + " stderr=log_file,\n", + " text=True\n", + " )\n", + " \n", + " print(f\"Starting Llama Stack server with PID: {process.pid}\")\n", + " return process\n", + "\n", + "def wait_for_server_to_start():\n", + " import requests\n", + " from requests.exceptions import ConnectionError\n", + " import time\n", + " \n", + " url = \"http://0.0.0.0:8321/v1/health\"\n", + " max_retries = 30\n", + " retry_interval = 1\n", + " \n", + " print(\"Waiting for server to start\", end=\"\")\n", + " for _ in range(max_retries):\n", + " try:\n", + " response = requests.get(url)\n", + " if response.status_code == 200:\n", + " print(\"\\nServer is ready!\")\n", + " return True\n", + " except ConnectionError:\n", + " print(\".\", end=\"\", flush=True)\n", + " time.sleep(retry_interval)\n", + " \n", + " print(\"\\nServer failed to start after\", max_retries * retry_interval, \"seconds\")\n", + " return False\n", + "\n", + "\n", + "# use this helper if needed to kill the server \n", + "def kill_llama_stack_server():\n", + " # Kill any existing llama stack server processes\n", + " os.system(\"ps aux | grep -v grep | grep llama_stack.distribution.server.server | awk '{print $2}' | xargs kill -9\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "c40e9efd", + "metadata": {}, + "source": [ + "### 2.2. Start the Llama Stack Server" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "f779283d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Starting Llama Stack server with PID: 787100\n", + "Waiting for server to start\n", + "Server is ready!\n" + ] + } + ], + "source": [ + "server_process = run_llama_stack_server_background()\n", + "assert wait_for_server_to_start()" + ] + }, + { + "cell_type": "markdown", + "id": "28477c03", + "metadata": {}, + "source": [ + "## Step 3: Run the demo" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "7da71011", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "rag_tool> Ingesting document: https://www.paulgraham.com/greatwork.html\n", + "prompt> How do you do great work?\n", + "\u001b[33minference> \u001b[0m\u001b[33m[k\u001b[0m\u001b[33mnowledge\u001b[0m\u001b[33m_search\u001b[0m\u001b[33m(query\u001b[0m\u001b[33m=\"\u001b[0m\u001b[33mWhat\u001b[0m\u001b[33m is\u001b[0m\u001b[33m the\u001b[0m\u001b[33m key\u001b[0m\u001b[33m to\u001b[0m\u001b[33m doing\u001b[0m\u001b[33m great\u001b[0m\u001b[33m work\u001b[0m\u001b[33m\")]\u001b[0m\u001b[97m\u001b[0m\n", + "\u001b[32mtool_execution> Tool:knowledge_search Args:{'query': 'What is the key to doing great work'}\u001b[0m\n", + "\u001b[32mtool_execution> Tool:knowledge_search Response:[TextContentItem(text='knowledge_search tool found 5 chunks:\\nBEGIN of knowledge_search tool results.\\n', type='text'), TextContentItem(text=\"Result 1:\\nDocument_id:docum\\nContent: work. Doing great work means doing something important\\nso well that you expand people's ideas of what's possible. But\\nthere's no threshold for importance. It's a matter of degree, and\\noften hard to judge at the time anyway.\\n\", type='text'), TextContentItem(text=\"Result 2:\\nDocument_id:docum\\nContent: work. Doing great work means doing something important\\nso well that you expand people's ideas of what's possible. But\\nthere's no threshold for importance. It's a matter of degree, and\\noften hard to judge at the time anyway.\\n\", type='text'), TextContentItem(text=\"Result 3:\\nDocument_id:docum\\nContent: work. Doing great work means doing something important\\nso well that you expand people's ideas of what's possible. But\\nthere's no threshold for importance. It's a matter of degree, and\\noften hard to judge at the time anyway.\\n\", type='text'), TextContentItem(text=\"Result 4:\\nDocument_id:docum\\nContent: work. Doing great work means doing something important\\nso well that you expand people's ideas of what's possible. But\\nthere's no threshold for importance. It's a matter of degree, and\\noften hard to judge at the time anyway.\\n\", type='text'), TextContentItem(text=\"Result 5:\\nDocument_id:docum\\nContent: work. Doing great work means doing something important\\nso well that you expand people's ideas of what's possible. But\\nthere's no threshold for importance. It's a matter of degree, and\\noften hard to judge at the time anyway.\\n\", type='text'), TextContentItem(text='END of knowledge_search tool results.\\n', type='text'), TextContentItem(text='The above results were retrieved to help answer the user\\'s query: \"What is the key to doing great work\". Use them as supporting information only in answering this query.\\n', type='text')]\u001b[0m\n", + "\u001b[33minference> \u001b[0m\u001b[33mDoing\u001b[0m\u001b[33m great\u001b[0m\u001b[33m work\u001b[0m\u001b[33m means\u001b[0m\u001b[33m doing\u001b[0m\u001b[33m something\u001b[0m\u001b[33m important\u001b[0m\u001b[33m so\u001b[0m\u001b[33m well\u001b[0m\u001b[33m that\u001b[0m\u001b[33m you\u001b[0m\u001b[33m expand\u001b[0m\u001b[33m people\u001b[0m\u001b[33m's\u001b[0m\u001b[33m ideas\u001b[0m\u001b[33m of\u001b[0m\u001b[33m what\u001b[0m\u001b[33m's\u001b[0m\u001b[33m possible\u001b[0m\u001b[33m.\u001b[0m\u001b[33m However\u001b[0m\u001b[33m,\u001b[0m\u001b[33m there\u001b[0m\u001b[33m's\u001b[0m\u001b[33m no\u001b[0m\u001b[33m threshold\u001b[0m\u001b[33m for\u001b[0m\u001b[33m importance\u001b[0m\u001b[33m,\u001b[0m\u001b[33m and\u001b[0m\u001b[33m it\u001b[0m\u001b[33m's\u001b[0m\u001b[33m often\u001b[0m\u001b[33m hard\u001b[0m\u001b[33m to\u001b[0m\u001b[33m judge\u001b[0m\u001b[33m at\u001b[0m\u001b[33m the\u001b[0m\u001b[33m time\u001b[0m\u001b[33m anyway\u001b[0m\u001b[33m.\u001b[0m\u001b[33m Great\u001b[0m\u001b[33m work\u001b[0m\u001b[33m is\u001b[0m\u001b[33m a\u001b[0m\u001b[33m matter\u001b[0m\u001b[33m of\u001b[0m\u001b[33m degree\u001b[0m\u001b[33m,\u001b[0m\u001b[33m and\u001b[0m\u001b[33m it\u001b[0m\u001b[33m can\u001b[0m\u001b[33m be\u001b[0m\u001b[33m difficult\u001b[0m\u001b[33m to\u001b[0m\u001b[33m determine\u001b[0m\u001b[33m whether\u001b[0m\u001b[33m someone\u001b[0m\u001b[33m has\u001b[0m\u001b[33m done\u001b[0m\u001b[33m great\u001b[0m\u001b[33m work\u001b[0m\u001b[33m until\u001b[0m\u001b[33m after\u001b[0m\u001b[33m the\u001b[0m\u001b[33m fact\u001b[0m\u001b[33m.\u001b[0m\u001b[97m\u001b[0m\n", + "\u001b[30m\u001b[0m" + ] + } + ], + "source": [ + "from llama_stack_client import Agent, AgentEventLogger, RAGDocument, LlamaStackClient\n", + "\n", + "vector_db_id = \"my_demo_vector_db\"\n", + "client = LlamaStackClient(base_url=\"http://0.0.0.0:8321\")\n", + "\n", + "models = client.models.list()\n", + "\n", + "# Select the first LLM and first embedding models\n", + "model_id = next(m for m in models if m.model_type == \"llm\").identifier\n", + "embedding_model_id = (\n", + " em := next(m for m in models if m.model_type == \"embedding\")\n", + ").identifier\n", + "embedding_dimension = em.metadata[\"embedding_dimension\"]\n", + "\n", + "_ = client.vector_dbs.register(\n", + " vector_db_id=vector_db_id,\n", + " embedding_model=embedding_model_id,\n", + " embedding_dimension=embedding_dimension,\n", + " provider_id=\"faiss\",\n", + ")\n", + "source = \"https://www.paulgraham.com/greatwork.html\"\n", + "print(\"rag_tool> Ingesting document:\", source)\n", + "document = RAGDocument(\n", + " document_id=\"document_1\",\n", + " content=source,\n", + " mime_type=\"text/html\",\n", + " metadata={},\n", + ")\n", + "client.tool_runtime.rag_tool.insert(\n", + " documents=[document],\n", + " vector_db_id=vector_db_id,\n", + " chunk_size_in_tokens=50,\n", + ")\n", + "agent = Agent(\n", + " client,\n", + " model=model_id,\n", + " instructions=\"You are a helpful assistant\",\n", + " tools=[\n", + " {\n", + " \"name\": \"builtin::rag/knowledge_search\",\n", + " \"args\": {\"vector_db_ids\": [vector_db_id]},\n", + " }\n", + " ],\n", + ")\n", + "\n", + "prompt = \"How do you do great work?\"\n", + "print(\"prompt>\", prompt)\n", + "\n", + "response = agent.create_turn(\n", + " messages=[{\"role\": \"user\", \"content\": prompt}],\n", + " session_id=agent.create_session(\"rag_session\"),\n", + " stream=True,\n", + ")\n", + "\n", + "for log in AgentEventLogger().log(response):\n", + " log.print()" + ] + }, + { + "cell_type": "markdown", + "id": "341aaadf", + "metadata": {}, + "source": [ + "Congratulations! You've successfully built your first RAG application using Llama Stack! 🎉🥳" + ] + }, + { + "cell_type": "markdown", + "id": "e88e1185", + "metadata": {}, + "source": [ + "## Next Steps" + ] + }, + { + "cell_type": "markdown", + "id": "bcb73600", + "metadata": {}, + "source": [ + "Now you're ready to dive deeper into Llama Stack!\n", + "- Explore the [Detailed Tutorial](./detailed_tutorial.md).\n", + "- Try the [Getting Started Notebook](https://github.com/meta-llama/llama-stack/blob/main/docs/getting_started.ipynb).\n", + "- Browse more [Notebooks on GitHub](https://github.com/meta-llama/llama-stack/tree/main/docs/notebooks).\n", + "- Learn about Llama Stack [Concepts](../concepts/index.md).\n", + "- Discover how to [Build Llama Stacks](../distributions/index.md).\n", + "- Refer to our [References](../references/index.md) for details on the Llama CLI and Python SDK.\n", + "- Check out the [llama-stack-apps](https://github.com/meta-llama/llama-stack-apps/tree/main/examples) repository for example applications and tutorials." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "T4", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}