phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-04 04:04:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
llama-stack-mirror/docs/getting_started_v0_3_0.ipynb
Kai Wu 4cc3e877ff
add getting_started_v0_3_0.ipynb
2025-09-19 16:38:46 -07:00

2030 lines
No EOL
197 KiB
Text
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"id": "c1e7571c",
"metadata": {
"id": "c1e7571c"
},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-llama/llama-stack/blob/main/docs/getting_started.ipynb)\n",
"\n",
"# Llama Stack - Building AI Applications\n",
"\n",
"<img src=\"https://llama-stack.readthedocs.io/en/latest/_images/llama-stack.png\" alt=\"drawing\" width=\"500\"/>\n",
"\n",
"[Llama Stack](https://github.com/meta-llama/llama-stack) defines and standardizes the set of core building blocks needed to bring generative AI applications to market. These building blocks are presented in the form of interoperable APIs with a broad set of Service Providers providing their implementations.\n",
"\n",
"Read more about the project here: https://llama-stack.readthedocs.io/en/latest/index.html\n",
"\n",
"In this guide, we will showcase how you can build LLM-powered agentic applications using Llama Stack.\n",
"\n",
"**💡 Quick Start Option:** If you want a simpler and faster way to test out Llama Stack, check out the [quick_start.ipynb](quick_start.ipynb) notebook instead. It provides a streamlined experience for getting up and running in just a few steps.\n"
]
},
{
"cell_type": "markdown",
"id": "4CV1Q19BDMVw",
"metadata": {
"id": "4CV1Q19BDMVw"
},
"source": [
"## 1. Getting started with Llama Stack"
]
},
{
"cell_type": "markdown",
"id": "K4AvfUAJZOeS",
"metadata": {
"id": "K4AvfUAJZOeS"
},
"source": [
"### 1.1. Create TogetherAI account\n",
"\n",
"\n",
"In order to run inference for the llama models, you will need to use an inference provider. Llama stack supports a number of inference [providers](https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference).\n",
"\n",
"\n",
"In this showcase, we will use [together.ai](https://www.together.ai/) as the inference provider. So, you would first get an API key from Together if you dont have one already.\n",
"\n",
"Steps [here](https://docs.google.com/document/d/1Vg998IjRW_uujAPnHdQ9jQWvtmkZFt74FldW2MblxPY/edit?usp=sharing).\n",
"\n",
"You can also use Fireworks.ai or even Ollama if you would like to.\n",
"\n",
"\n",
"\n",
"> **Note:** Set the API Key in the Secrets of this notebook\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "oDUB7M_qe-Gs",
"metadata": {
"id": "oDUB7M_qe-Gs"
},
"source": [
"### 1.2. Setup and Running a Llama Stack server\n",
"\n",
"Llama Stack is architected as a collection of APIs that provide developers with the building blocks to build AI applications.\n",
"\n",
"Llama stack is typically available as a server with an endpoint that you can make calls to. Partners like Together and Fireworks offer their own Llama Stack compatible endpoints.\n",
"\n",
"In this showcase, we will start a Llama Stack server that is running locally.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "HY8yBKKVoF50",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "HY8yBKKVoF50",
"outputId": "97abc006-572f-48cc-c899-6bcfd40c8d2d"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
">>> Installing ollama to /usr/local\n",
">>> Downloading Linux amd64 bundle\n",
"######################################################################## 100.0%\n",
">>> Creating ollama user...\n",
">>> Adding ollama user to video group...\n",
">>> Adding current user to ollama group...\n",
">>> Creating ollama systemd service...\n",
"\u001b[1m\u001b[31mWARNING:\u001b[m systemd is not running\n",
"\u001b[1m\u001b[31mWARNING:\u001b[m Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.\n",
">>> The Ollama API is now available at 127.0.0.1:11434.\n",
">>> Install complete. Run \"ollama\" from the command line.\n",
"\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\n",
"\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\u001b[?2026h\u001b[?25l\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[1G\u001b[?25h\u001b[?2026l\n"
]
}
],
"source": [
"#Install Ollama\n",
"!curl -fsSL https://ollama.com/install.sh | sh\n",
"\n",
"#Start Ollama server with llama3 model\n",
"!nohup ollama serve > ollama_server.log 2>&1 &\n",
"!ollama pull llama-guard3:1b\n",
"!ollama pull llama3.2:3b"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "p2SkDGjB_KUE",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "p2SkDGjB_KUE",
"outputId": "1498034b-ed2f-4bcb-e4e1-d964c95ce2cb"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"{\"object\":\"list\",\"data\":[{\"id\":\"llama3.2:3b\",\"object\":\"model\",\"created\":1758304995,\"owned_by\":\"library\"},{\"id\":\"llama-guard3:1b\",\"object\":\"model\",\"created\":1758304963,\"owned_by\":\"library\"}]}\n"
]
}
],
"source": [
"!curl 127.0.0.1:11434/v1/models\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0R628gRh-cYv",
"metadata": {
"id": "0R628gRh-cYv"
},
"outputs": [],
"source": [
"# use this helper if needed to kill the server\n",
"!rm -rf ~/.llama/distributions/*\n",
"import os\n",
"def kill_llama_stack_server():\n",
" # Kill any existing llama stack server processes\n",
" os.system(\"ps aux | grep -v grep | grep llama_stack.core.server.server | awk '{print $2}' | xargs kill -9\")\n",
"kill_llama_stack_server()"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "J2kGed0R5PSf",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "J2kGed0R5PSf",
"outputId": "2bd3a300-8637-43bc-ab2a-aa03861955f5"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"downloading uv 0.8.19 x86_64-unknown-linux-gnu\n",
"no checksums to verify\n",
"installing to /usr/local/bin\n",
" uv\n",
" uvx\n",
"everything's installed!\n",
"\u001b[1m\u001b[33mwarning\u001b[39m\u001b[0m\u001b[1m:\u001b[0m \u001b[1mThe `--system` flag has no effect, a system Python interpreter is always used in `uv venv`\u001b[0m\n",
"Using CPython 3.12.11 interpreter at: \u001b[36m/usr/bin/python3\u001b[39m\n",
"Creating virtual environment at: \u001b[36mvenv\u001b[39m\n",
"Activate with: \u001b[32msource venv/bin/activate\u001b[39m\n",
"\u001b[2K\u001b[2mInstalled \u001b[1m84 packages\u001b[0m \u001b[2min 334ms\u001b[0m\u001b[0m\n",
"\u001b[33mWARNING \u001b[0m \u001b[1;36m2025\u001b[0m-\u001b[1;36m09\u001b[0m-\u001b[1;36m19\u001b[0m \u001b[1;92m23:29:10\u001b[0m,\u001b[1;36m361\u001b[0m llama_stack.core.distribution:\u001b[1;36m149\u001b[0m core: Failed to import module prompts: No module named \n",
" \u001b[32m'llama_stack.providers.registry.prompts'\u001b[0m \n",
"Environment '/content/uv-cache/builds-v0/.tmpJfVJ5w' already exists, re-using it.\n",
"Installing dependencies in system Python environment\n",
"\u001b[2mUsing Python 3.12.11 environment at: /usr\u001b[0m\n",
"\u001b[2K\u001b[2mResolved \u001b[1m84 packages\u001b[0m \u001b[2min 1.30s\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[2mPrepared \u001b[1m12 packages\u001b[0m \u001b[2min 318ms\u001b[0m\u001b[0m\n",
"\u001b[2mUninstalled \u001b[1m1 package\u001b[0m \u001b[2min 9ms\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[2mInstalled \u001b[1m12 packages\u001b[0m \u001b[2min 38ms\u001b[0m\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1maiosqlite\u001b[0m\u001b[2m==0.21.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1masyncpg\u001b[0m\u001b[2m==0.30.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mecdsa\u001b[0m\u001b[2m==0.19.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mfire\u001b[0m\u001b[2m==0.7.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mllama-stack\u001b[0m\u001b[2m==0.2.22\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mllama-stack-client\u001b[0m\u001b[2m==0.2.22\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mopentelemetry-exporter-otlp-proto-common\u001b[0m\u001b[2m==1.36.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mopentelemetry-exporter-otlp-proto-http\u001b[0m\u001b[2m==1.36.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mopentelemetry-proto\u001b[0m\u001b[2m==1.36.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpyaml\u001b[0m\u001b[2m==25.7.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mpydantic\u001b[0m\u001b[2m==2.11.7\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpydantic\u001b[0m\u001b[2m==2.11.9\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpython-jose\u001b[0m\u001b[2m==3.5.0\u001b[0m\n",
"Installing pip dependencies\n",
"\u001b[2mUsing Python 3.12.11 environment at: /usr\u001b[0m\n",
"\u001b[2K\u001b[2mResolved \u001b[1m207 packages\u001b[0m \u001b[2min 1.47s\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[2mPrepared \u001b[1m54 packages\u001b[0m \u001b[2min 6.05s\u001b[0m\u001b[0m\n",
"\u001b[2mUninstalled \u001b[1m4 packages\u001b[0m \u001b[2min 20ms\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[2mInstalled \u001b[1m55 packages\u001b[0m \u001b[2min 197ms\u001b[0m\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1masyncstdlib-fw\u001b[0m\u001b[2m==3.13.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mautoevals\u001b[0m\u001b[2m==0.0.130\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mbackoff\u001b[0m\u001b[2m==2.2.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mbetterproto-fw\u001b[0m\u001b[2m==2.0.3\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mboltons\u001b[0m\u001b[2m==21.0.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mboto3\u001b[0m\u001b[2m==1.40.35\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mbotocore\u001b[0m\u001b[2m==1.40.35\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mbracex\u001b[0m\u001b[2m==2.6\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mcerebras-cloud-sdk\u001b[0m\u001b[2m==1.50.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mchevron\u001b[0m\u001b[2m==0.14.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mchromadb-client\u001b[0m\u001b[2m==1.1.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mclick-option-group\u001b[0m\u001b[2m==0.5.7\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mcodeshield\u001b[0m\u001b[2m==1.0.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mcolorama\u001b[0m\u001b[2m==0.4.6\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mdnspython\u001b[0m\u001b[2m==2.8.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1memoji\u001b[0m\u001b[2m==2.14.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1meval-type-backport\u001b[0m\u001b[2m==0.2.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mexceptiongroup\u001b[0m\u001b[2m==1.2.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mface\u001b[0m\u001b[2m==24.0.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mfaiss-cpu\u001b[0m\u001b[2m==1.12.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mfastuuid\u001b[0m\u001b[2m==0.12.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mfireworks-ai\u001b[0m\u001b[2m==0.17.16\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mglom\u001b[0m\u001b[2m==22.1.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mhttpx-ws\u001b[0m\u001b[2m==0.7.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mjmespath\u001b[0m\u001b[2m==1.0.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mlangdetect\u001b[0m\u001b[2m==1.0.9\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mlitellm\u001b[0m\u001b[2m==1.77.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mmmh3\u001b[0m\u001b[2m==5.2.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mollama\u001b[0m\u001b[2m==0.5.4\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mopentelemetry-exporter-otlp-proto-grpc\u001b[0m\u001b[2m==1.36.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpolyleven\u001b[0m\u001b[2m==0.9.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mposthog\u001b[0m\u001b[2m==5.4.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mprotobuf\u001b[0m\u001b[2m==5.29.5\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mprotobuf\u001b[0m\u001b[2m==5.29.4\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpsycopg2-binary\u001b[0m\u001b[2m==2.9.10\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpybase64\u001b[0m\u001b[2m==1.4.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpymilvus\u001b[0m\u001b[2m==2.6.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpymongo\u001b[0m\u001b[2m==4.15.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpypdf\u001b[0m\u001b[2m==6.0.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpythainlp\u001b[0m\u001b[2m==5.1.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mredis\u001b[0m\u001b[2m==6.4.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mrich\u001b[0m\u001b[2m==13.9.4\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mrich\u001b[0m\u001b[2m==14.1.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mruamel-yaml\u001b[0m\u001b[2m==0.17.40\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mruamel-yaml-clib\u001b[0m\u001b[2m==0.2.12\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mruff\u001b[0m\u001b[2m==0.12.12\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mruff\u001b[0m\u001b[2m==0.9.10\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1ms3transfer\u001b[0m\u001b[2m==0.14.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1msemgrep\u001b[0m\u001b[2m==1.79.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1msqlite-vec\u001b[0m\u001b[2m==0.1.6\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mtogether\u001b[0m\u001b[2m==1.5.25\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mtomli\u001b[0m\u001b[2m==2.0.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mtree-sitter\u001b[0m\u001b[2m==0.25.1\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mtrl\u001b[0m\u001b[2m==0.23.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mtyper\u001b[0m\u001b[2m==0.17.4\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mtyper\u001b[0m\u001b[2m==0.15.3\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mujson\u001b[0m\u001b[2m==5.11.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mwcmatch\u001b[0m\u001b[2m==8.5.2\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mwsproto\u001b[0m\u001b[2m==1.2.0\u001b[0m\n",
"Installing special provider module: torch torchvision torchao>=0.12.0 --extra-index-url https://download.pytorch.org/whl/cpu\n",
"\u001b[2mUsing Python 3.12.11 environment at: /usr\u001b[0m\n",
"\u001b[2K\u001b[2mResolved \u001b[1m29 packages\u001b[0m \u001b[2min 538ms\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[2mPrepared \u001b[1m1 package\u001b[0m \u001b[2min 100ms\u001b[0m\u001b[0m\n",
"\u001b[2mUninstalled \u001b[1m1 package\u001b[0m \u001b[2min 67ms\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[2mInstalled \u001b[1m1 package\u001b[0m \u001b[2min 10ms\u001b[0m\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mtorchao\u001b[0m\u001b[2m==0.10.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mtorchao\u001b[0m\u001b[2m==0.13.0+cpu\u001b[0m\n",
"Installing special provider module: sentence-transformers --no-deps\n",
"\u001b[2mUsing Python 3.12.11 environment at: /usr\u001b[0m\n",
"\u001b[2mAudited \u001b[1m1 package\u001b[0m \u001b[2min 88ms\u001b[0m\u001b[0m\n",
"\u001b[32mBuild Successful!\u001b[0m\n",
"\u001b[34mYou can find the newly-built distribution here: /root/.llama/distributions/starter-gpu/starter-gpu-run.yaml\u001b[0m\n",
"\u001b[32mYou can run the new Llama Stack distro via: \u001b[34mllama stack run /root/.llama/distributions/starter-gpu/starter-gpu-run.yaml --image-type venv\u001b[0m\u001b[0m\n",
"nohup: redirecting stderr to stdout\n",
"Waiting for server to start.................\n",
"Server is ready!\n",
"llama stack server hosted on localhost:8321\n"
]
}
],
"source": [
"# Install UV if not available\n",
"!curl -LsSf https://astral.sh/uv/install.sh | sh\n",
"# Complete setup for Google Colab with custom directories\n",
"import os\n",
"!rm -rf /content/llama-project\n",
"# Set environment variables\n",
"os.environ['UV_CACHE_DIR'] = '/content/uv-cache'\n",
"os.environ['UV_PROJECT_DIR'] = '/content/llama-project'\n",
"os.environ['OLLAMA_URL'] = 'http://localhost:11434'\n",
"# Create directories\n",
"!mkdir -p /content/uv-cache\n",
"!mkdir -p /content/llama-project\n",
"!cd /content/llama-project && uv venv venv\n",
"!source /content/llama-project/venv/bin/activate && uv run --with llama-stack==0.2.22 llama stack build --distro starter-gpu --image-type venv\n",
"!nohup python -m llama_stack.core.server.server /root/.llama/distributions/starter-gpu/starter-gpu-run.yaml --port 8321 > llama_stack_server.log &\n",
"def wait_for_server_to_start():\n",
" import requests\n",
" from requests.exceptions import ConnectionError\n",
" import time\n",
"\n",
" url = \"http://0.0.0.0:8321/v1/health\"\n",
" max_retries = 30\n",
" retry_interval = 1\n",
"\n",
" print(\"Waiting for server to start\", end=\"\")\n",
" for _ in range(max_retries):\n",
" try:\n",
" response = requests.get(url)\n",
" if response.status_code == 200:\n",
" print(\"\\nServer is ready!\")\n",
" return True\n",
" except ConnectionError:\n",
" print(\".\", end=\"\", flush=True)\n",
" time.sleep(retry_interval)\n",
"\n",
" print(\"\\nServer failed to start after\", max_retries * retry_interval, \"seconds\")\n",
" return False\n",
"assert wait_for_server_to_start()\n",
"print(\"llama stack server hosted on localhost:8321\")"
]
},
{
"cell_type": "markdown",
"id": "90eb721b",
"metadata": {
"id": "90eb721b"
},
"source": [
"### 1.4. Install and Configure the Client\n",
"\n",
"Now that we have our Llama Stack server running locally, we need to install the client package to interact with it. The `llama-stack-client` provides a simple Python interface to access all the functionality of Llama Stack, including:\n",
"\n",
"- Chat Completions ( text and multimodal )\n",
"- Safety Shields\n",
"- Agent capabilities with tools like web search, RAG with Telemetry\n",
"- Evaluation and scoring frameworks\n",
"\n",
"The client handles all the API communication with our local server, making it easy to integrate Llama Stack's capabilities into your applications.\n",
"\n",
"In the next cells, we'll:\n",
"\n",
"1. Install the client package\n",
"2. Set up API keys for external services (Together AI and Tavily Search)\n",
"3. Initialize the client to connect to our local server\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "E1UFuJC570Tk",
"metadata": {
"collapsed": true,
"id": "E1UFuJC570Tk"
},
"outputs": [],
"source": [
"import os\n",
"import getpass\n",
"try:\n",
" from google.colab import userdata\n",
" os.environ['GROQ_API_KEY'] = userdata.get('GROQ_API_KEY')\n",
" os.environ['TAVILY_SEARCH_API_KEY'] = userdata.get('TAVILY_SEARCH_API_KEY')\n",
"except ImportError:\n",
" print(\"Not in Google Colab environment\")\n",
"\n",
"for key in ['GROQ_API_KEY', 'TAVILY_SEARCH_API_KEY']:\n",
" try:\n",
" api_key = os.environ[key]\n",
" if not api_key:\n",
" raise ValueError(f\"{key} environment variable is empty\")\n",
" except KeyError:\n",
" api_key = getpass.getpass(f\"{key} environment variable is not set. Please enter your API key: \")\n",
" os.environ[key] = api_key\n",
"\n",
"from llama_stack_client import LlamaStackClient\n",
"\n",
"client = LlamaStackClient(\n",
" base_url=\"http://0.0.0.0:8321\",\n",
" provider_data = {\n",
" \"tavily_search_api_key\": os.environ['TAVILY_SEARCH_API_KEY'],\n",
" \"groq_api_key\": os.environ['GROQ_API_KEY']\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"id": "635a7a6f",
"metadata": {
"id": "635a7a6f"
},
"source": [
"Now that we have completed the setup and configuration, let's start exploring the capabilities of Llama Stack! We'll begin by checking what models and safety shields are available, and then move on to running some example chat completions.\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "7dacaa2d-94e9-42e9-82a0-73522dfc7010",
"metadata": {
"id": "7dacaa2d-94e9-42e9-82a0-73522dfc7010"
},
"source": [
"### 1.5. Check available models and shields\n",
"\n",
"All the models available in the provider are now programmatically accessible via the client."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ruO9jQna_t_S",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "ruO9jQna_t_S",
"outputId": "282ab617-f60c-49bd-a272-30df17c3ad73"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Available models:\n",
"- fireworks/accounts/fireworks/models/llama-v3p1-8b-instruct\n",
"- fireworks/accounts/fireworks/models/llama-v3p1-70b-instruct\n",
"- fireworks/accounts/fireworks/models/llama-v3p1-405b-instruct\n",
"- fireworks/accounts/fireworks/models/llama-v3p2-3b-instruct\n",
"- fireworks/accounts/fireworks/models/llama-v3p2-11b-vision-instruct\n",
"- fireworks/accounts/fireworks/models/llama-v3p2-90b-vision-instruct\n",
"- fireworks/accounts/fireworks/models/llama-v3p3-70b-instruct\n",
"- fireworks/accounts/fireworks/models/llama4-scout-instruct-basic\n",
"- fireworks/accounts/fireworks/models/llama4-maverick-instruct-basic\n",
"- fireworks/nomic-ai/nomic-embed-text-v1.5\n",
"- fireworks/accounts/fireworks/models/llama-guard-3-8b\n",
"- fireworks/accounts/fireworks/models/llama-guard-3-11b-vision\n",
"- bedrock/meta.llama3-1-8b-instruct-v1:0\n",
"- bedrock/meta.llama3-1-70b-instruct-v1:0\n",
"- bedrock/meta.llama3-1-405b-instruct-v1:0\n",
"- openai/gpt-3.5-turbo-0125\n",
"- openai/gpt-3.5-turbo\n",
"- openai/gpt-3.5-turbo-instruct\n",
"- openai/gpt-4\n",
"- openai/gpt-4-turbo\n",
"- openai/gpt-4o\n",
"- openai/gpt-4o-2024-08-06\n",
"- openai/gpt-4o-mini\n",
"- openai/gpt-4o-audio-preview\n",
"- openai/chatgpt-4o-latest\n",
"- openai/o1\n",
"- openai/o1-mini\n",
"- openai/o3-mini\n",
"- openai/o4-mini\n",
"- openai/text-embedding-3-small\n",
"- openai/text-embedding-3-large\n",
"- anthropic/claude-3-5-sonnet-latest\n",
"- anthropic/claude-3-7-sonnet-latest\n",
"- anthropic/claude-3-5-haiku-latest\n",
"- anthropic/voyage-3\n",
"- anthropic/voyage-3-lite\n",
"- anthropic/voyage-code-3\n",
"- gemini/gemini-1.5-flash\n",
"- gemini/gemini-1.5-pro\n",
"- gemini/gemini-2.0-flash\n",
"- gemini/gemini-2.0-flash-lite\n",
"- gemini/gemini-2.5-flash\n",
"- gemini/gemini-2.5-flash-lite\n",
"- gemini/gemini-2.5-pro\n",
"- gemini/text-embedding-004\n",
"- groq/llama3-8b-8192\n",
"- groq/llama-3.1-8b-instant\n",
"- groq/llama3-70b-8192\n",
"- groq/llama-3.3-70b-versatile\n",
"- groq/llama-3.2-3b-preview\n",
"- groq/meta-llama/llama-4-scout-17b-16e-instruct\n",
"- groq/meta-llama/llama-4-maverick-17b-128e-instruct\n",
"- sambanova/Meta-Llama-3.1-8B-Instruct\n",
"- sambanova/Meta-Llama-3.3-70B-Instruct\n",
"- sambanova/Llama-4-Maverick-17B-128E-Instruct\n",
"- sentence-transformers/all-MiniLM-L6-v2\n"
]
}
],
"source": [
"from rich.pretty import pprint\n",
"\n",
"print(\"Available models:\")\n",
"for m in client.models.list():\n",
" print(f\"- {m.identifier}\")\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "86366383",
"metadata": {
"id": "86366383"
},
"source": [
"### 1.6. Run a simple chat completion with one of the models\n",
"\n",
"We will test the client by doing a simple chat completion."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "77c29dba",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "77c29dba",
"outputId": "cc6471dd-40af-4882-ce56-28f51fa46a84"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Here's a two-sentence poem about a llama:\n",
"\n",
"With gentle eyes and soft, fuzzy hair, the llama roams with gentle, peaceful air. In the Andes, it climbs with steady pace, a serene and majestic animal in its sacred space.\n"
]
}
],
"source": [
"#model_id = \"ollama/llama3.2:3b\"\n",
"model_id = \"groq/meta-llama/llama-4-maverick-17b-128e-instruct\"\n",
"response = client.chat.completions.create(\n",
" model=model_id,\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a friendly assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"Write a two-sentence poem about llama.\"},\n",
" ],\n",
" stream=False\n",
")\n",
"\n",
"print(response.choices[0].message.content)\n"
]
},
{
"cell_type": "markdown",
"id": "8cf0d555",
"metadata": {
"id": "8cf0d555"
},
"source": [
"### 1.7. Have a conversation\n",
"\n",
"Maintaining a conversation history allows the model to retain context from previous interactions. Use a list to accumulate messages, enabling continuity throughout the chat session."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "3fdf9df6",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3fdf9df6",
"outputId": "31396b37-da60-4050-a593-bcfe01144385"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"> Response: You're likely thinking of Winston Churchill!\n",
"\n",
"Winston Churchill was indeed the most famous Prime Minister of the United Kingdom during World War II. He served as the Prime Minister from May 10, 1940, to July 26, 1945, and again from 1951 to 1955. Churchill played a crucial role in leading Britain through the war, rallying the British people with his inspiring speeches, and forming alliances with other countries to defeat the Axis powers.\n",
"\n",
"Churchill's leadership, oratory skills, and unwavering resolve made him a iconic figure of the war era, and he remains one of the most revered and celebrated leaders in British history.\n",
"\n",
"Is there anything else you'd like to know about Churchill or his role during World War II?\n",
"> Response: One of the most famous quotes attributed to Winston Churchill is:\n",
"\n",
"\"We shall fight on the beaches, we shall fight on the landing grounds, we shall fight in the fields and in the streets, we shall fight in the hills; we shall never surrender.\"\n",
"\n",
"This quote is from his speech to the House of Commons on June 4, 1940, during the early stages of World War II, when Nazi Germany was threatening to invade Britain. The speech, known as the \"We Shall Fight on the Beaches\" speech, was a rallying cry to the British people, and it's considered one of Churchill's most iconic and inspiring orations.\n",
"\n",
"However, another very famous quote from Churchill is:\n",
"\n",
"\"Blood, toil, tears, and sweat.\"\n",
"\n",
"This was the opening phrase of his first speech as Prime Minister to the House of Commons on May 13, 1940, where he said: \"I say to the House as I said to those who have joined this Government, I have nothing to offer but blood, toil, tears, and sweat. We have before us an ordeal of the most grievous kind.\"\n",
"\n",
"Both of these quotes are iconic and showcase Churchill's powerful oratory skills.\n",
"\n",
"Would you like to know more about Churchill's speeches or his leadership during World War II?\n"
]
}
],
"source": [
"from termcolor import cprint\n",
"\n",
"questions = [\n",
" \"Who was the most famous PM of England during world war 2 ?\",\n",
" \"What was his most famous quote ?\"\n",
"]\n",
"\n",
"\n",
"def chat_loop():\n",
" conversation_history = []\n",
" while len(questions) > 0:\n",
" user_input = questions.pop(0)\n",
" if user_input.lower() in [\"exit\", \"quit\", \"bye\"]:\n",
" cprint(\"Ending conversation. Goodbye!\", \"yellow\")\n",
" break\n",
"\n",
" user_message = {\"role\": \"user\", \"content\": user_input}\n",
" conversation_history.append(user_message)\n",
"\n",
" response = client.chat.completions.create(\n",
" messages=conversation_history,\n",
" model=model_id,\n",
" )\n",
" cprint(f\"> Response: {response.choices[0].message.content}\", \"cyan\")\n",
"\n",
" assistant_message = {\n",
" \"role\": \"assistant\", # was user\n",
" \"content\": response.choices[0].message.content,\n",
" \"finish_reason\": response.choices[0].finish_reason,\n",
" }\n",
" conversation_history.append(assistant_message)\n",
"\n",
"\n",
"chat_loop()\n"
]
},
{
"cell_type": "markdown",
"id": "72e5111e",
"metadata": {
"id": "72e5111e"
},
"source": [
"Here is an example for you to try a conversation yourself.\n",
"Remember to type `quit` or `exit` after you are done chatting."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9496f75c",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9496f75c",
"outputId": "9c51562e-05b0-40f3-b4c0-eb4c991b1e67"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"User> who are you?\n",
"> Response: I'm an AI assistant designed by Meta. I'm here to answer your questions, share interesting ideas and maybe even surprise you with a fresh perspective. What's on your mind?\n",
"User> how can you help me?\n",
"> Response: I can help you with a wide range of things, such as answering questions, providing information, generating text or images, summarizing content, or just having a chat. I can also help with creative tasks like brainstorming or coming up with ideas. What do you need help with today?\n",
"User> bye\n",
"Ending conversation. Goodbye!\n"
]
}
],
"source": [
"# NBVAL_SKIP\n",
"from termcolor import cprint\n",
"\n",
"def chat_loop():\n",
" conversation_history = []\n",
" while True:\n",
" user_input = input(\"User> \")\n",
" if user_input.lower() in [\"exit\", \"quit\", \"bye\"]:\n",
" cprint(\"Ending conversation. Goodbye!\", \"yellow\")\n",
" break\n",
"\n",
" user_message = {\"role\": \"user\", \"content\": user_input}\n",
" conversation_history.append(user_message)\n",
"\n",
" response = client.chat.completions.create(\n",
" messages=conversation_history,\n",
" model=model_id,\n",
" )\n",
" cprint(f\"> Response: {response.choices[0].message.content}\", \"cyan\")\n",
"\n",
" assistant_message = {\n",
" \"role\": \"assistant\", # was user\n",
" \"content\": response.choices[0].message.content,\n",
" \"finish_reason\": response.choices[0].finish_reason,\n",
" }\n",
" conversation_history.append(assistant_message)\n",
"\n",
"\n",
"chat_loop()\n"
]
},
{
"cell_type": "markdown",
"id": "03fcf5e0",
"metadata": {
"id": "03fcf5e0"
},
"source": [
"### 1.9. Streaming output\n",
"\n",
"You can pass `stream=True` to stream responses from the model. You can then loop through the responses."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d119026e",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "d119026e",
"outputId": "fec033c8-8b4f-4a57-bdf8-effb2f0d6acb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"User> Write me a sonnet about llama\n",
"In Andean lands, the llama makes its home,\n",
"A creature soft, with eyes of gentle gray.\n",
"Its fur, a softness that the winds do roam,\n",
"And in its steps, a quiet, peaceful sway.\n",
"\n",
"Its ears, so long, and tufted with delight,\n",
"Perk up, as if to listen for a sound.\n",
"It grazes on the grasses, day and night,\n",
"And in its calm, a peaceful joy is found.\n",
"\n",
"The llama's gentle nature, we admire,\n",
"And in its presence, our own cares retire.\n",
"For in its tranquil eyes, a mirror lies,\n",
"Reflecting back our own, and soothing sighs.\n",
"\n",
"So let us cherish, this serene delight,\n",
"And bask in the llama's peaceful, Andean light."
]
}
],
"source": [
"from llama_stack_client import InferenceEventLogger\n",
"\n",
"message = {\"role\": \"user\", \"content\": \"Write me a sonnet about llama\"}\n",
"print(f'User> {message[\"content\"]}')\n",
"\n",
"response = client.chat.completions.create(\n",
" messages=[message],\n",
" model=model_id,\n",
" stream=True, # <-----------\n",
")\n",
"\n",
"for chunk in response:\n",
" # Each chunk contains a delta with the content\n",
" if chunk.choices[0].delta.content is not None:\n",
" print(chunk.choices[0].delta.content, end=\"\", flush=True)\n"
]
},
{
"cell_type": "markdown",
"id": "OmU6Dr9zBiGM",
"metadata": {
"id": "OmU6Dr9zBiGM"
},
"source": [
"### 2.0. Structured Decoding\n",
"\n",
"You can use `response_format` to force the model into a \"guided decode\" mode where model tokens are forced to abide by a certain grammar. Currently only JSON grammars are supported."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "axdQIRaJCYAV",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 50
},
"id": "axdQIRaJCYAV",
"outputId": "efc29ca4-9fa8-4c35-c0fa-c7a5faf8024b"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">'{\\n \"name\": \"Michael Jordan\",\\n \"year_born\": \"1963\",\\n \"year_retired\": \"2003\"\\n}'</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\n \"name\": \"Michael Jordan\",\\n \"year_born\": \"1963\",\\n \"year_retired\": \"2003\"\\n\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Output</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">name</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Michael Jordan'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">year_born</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'1963'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">year_retired</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'2003'</span><span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mOutput\u001b[0m\u001b[1m(\u001b[0m\u001b[33mname\u001b[0m=\u001b[32m'Michael Jordan'\u001b[0m, \u001b[33myear_born\u001b[0m=\u001b[32m'1963'\u001b[0m, \u001b[33myear_retired\u001b[0m=\u001b[32m'2003'\u001b[0m\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from pydantic import BaseModel\n",
"\n",
"\n",
"class Output(BaseModel):\n",
" name: str\n",
" year_born: str\n",
" year_retired: str\n",
"\n",
"user_input = \"Michael Jordan was born in 1963. He played basketball for the Chicago Bulls. He retired in 2003. Extract this information into JSON for me.\"\n",
"response = client.chat.completions.create(\n",
" model=model_id,\n",
" messages = [\n",
" {\"role\": \"user\", \"content\": user_input}\n",
" ],\n",
" stream=False,\n",
" response_format={\n",
" \"type\": \"json_schema\",\n",
" \"json_schema\": {\n",
" \"name\": \"output\",\n",
" \"schema\": Output.model_json_schema(),\n",
" },\n",
" },\n",
")\n",
"pprint(Output.model_validate_json(response.choices[0].message.content))\n"
]
},
{
"cell_type": "markdown",
"id": "H62Rg_buEx2o",
"metadata": {
"id": "H62Rg_buEx2o"
},
"source": [
"### 2.1. Safety API\n",
"\n",
"Llama Stack provides Safety guardrails which can be applied at multiple touchpoints within an agentic application."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "sUJKJxvAFCaI",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "sUJKJxvAFCaI",
"outputId": "55b3ae63-7de2-4902-b424-359c28a23321"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Available Shields: ['llama-guard']\n",
"Checking if input is safe: What is the most famous murder case in the US?\n",
"CreateResponse(id='modr-49b2d798-a967-4a71-8ccb-58bdd78746ac', model='ollama/llama-guard3:1b', results=[Result(flagged=False, metadata={}, categories={'Violent Crimes': False, 'Non-Violent Crimes': False, 'Sex Crimes': False, 'Child Exploitation': False, 'Defamation': False, 'Specialized Advice': False, 'Privacy': False, 'Intellectual Property': False, 'Indiscriminate Weapons': False, 'Hate': False, 'Self-Harm': False, 'Sexual Content': False, 'Elections': False, 'Code Interpreter Abuse': False}, category_applied_input_types={'Violent Crimes': [], 'Non-Violent Crimes': [], 'Sex Crimes': [], 'Child Exploitation': [], 'Defamation': [], 'Specialized Advice': [], 'Privacy': [], 'Intellectual Property': [], 'Indiscriminate Weapons': [], 'Hate': [], 'Self-Harm': [], 'Sexual Content': [], 'Elections': [], 'Code Interpreter Abuse': []}, category_scores={'Violent Crimes': 1.0, 'Non-Violent Crimes': 1.0, 'Sex Crimes': 1.0, 'Child Exploitation': 1.0, 'Defamation': 1.0, 'Specialized Advice': 1.0, 'Privacy': 1.0, 'Intellectual Property': 1.0, 'Indiscriminate Weapons': 1.0, 'Hate': 1.0, 'Self-Harm': 1.0, 'Sexual Content': 1.0, 'Elections': 1.0, 'Code Interpreter Abuse': 1.0}, user_message=None)])\n",
"Checking if input is safe: Tell me 3 signs that an email is a scam\n",
"CreateResponse(id='modr-10eda2a7-6043-4b3c-b749-4c76e2fe87ee', model='ollama/llama-guard3:1b', results=[Result(flagged=True, metadata={'violation_type': ['S2']}, categories={'Violent Crimes': False, 'Non-Violent Crimes': True, 'Sex Crimes': False, 'Child Exploitation': False, 'Defamation': False, 'Specialized Advice': False, 'Privacy': False, 'Intellectual Property': False, 'Indiscriminate Weapons': False, 'Hate': False, 'Self-Harm': False, 'Sexual Content': False, 'Elections': False, 'Code Interpreter Abuse': False}, category_applied_input_types={'Violent Crimes': [], 'Non-Violent Crimes': ['text'], 'Sex Crimes': [], 'Child Exploitation': [], 'Defamation': [], 'Specialized Advice': [], 'Privacy': [], 'Intellectual Property': [], 'Indiscriminate Weapons': [], 'Hate': [], 'Self-Harm': [], 'Sexual Content': [], 'Elections': [], 'Code Interpreter Abuse': []}, category_scores={'Violent Crimes': 0.0, 'Non-Violent Crimes': 1.0, 'Sex Crimes': 0.0, 'Child Exploitation': 0.0, 'Defamation': 0.0, 'Specialized Advice': 0.0, 'Privacy': 0.0, 'Intellectual Property': 0.0, 'Indiscriminate Weapons': 0.0, 'Hate': 0.0, 'Self-Harm': 0.0, 'Sexual Content': 0.0, 'Elections': 0.0, 'Code Interpreter Abuse': 0.0}, user_message=\"I can't answer that. Can I help with something else?\")])\n",
"Checking if input is safe: What's the most effective way to kidnap someone?\n",
"CreateResponse(id='modr-dfab8119-2f32-45f9-8106-3ba2725f3a63', model='ollama/llama-guard3:1b', results=[Result(flagged=True, metadata={'violation_type': ['S1']}, categories={'Violent Crimes': True, 'Non-Violent Crimes': False, 'Sex Crimes': False, 'Child Exploitation': False, 'Defamation': False, 'Specialized Advice': False, 'Privacy': False, 'Intellectual Property': False, 'Indiscriminate Weapons': False, 'Hate': False, 'Self-Harm': False, 'Sexual Content': False, 'Elections': False, 'Code Interpreter Abuse': False}, category_applied_input_types={'Violent Crimes': ['text'], 'Non-Violent Crimes': [], 'Sex Crimes': [], 'Child Exploitation': [], 'Defamation': [], 'Specialized Advice': [], 'Privacy': [], 'Intellectual Property': [], 'Indiscriminate Weapons': [], 'Hate': [], 'Self-Harm': [], 'Sexual Content': [], 'Elections': [], 'Code Interpreter Abuse': []}, category_scores={'Violent Crimes': 1.0, 'Non-Violent Crimes': 0.0, 'Sex Crimes': 0.0, 'Child Exploitation': 0.0, 'Defamation': 0.0, 'Specialized Advice': 0.0, 'Privacy': 0.0, 'Intellectual Property': 0.0, 'Indiscriminate Weapons': 0.0, 'Hate': 0.0, 'Self-Harm': 0.0, 'Sexual Content': 0.0, 'Elections': 0.0, 'Code Interpreter Abuse': 0.0}, user_message=\"I can't answer that. Can I help with something else?\")])\n",
"Checking if input is safe: How do I make cocaine?\n",
"CreateResponse(id='modr-9f407506-f8bd-4281-b44a-b8b1bcbdfbaa', model='ollama/llama-guard3:1b', results=[Result(flagged=True, metadata={'violation_type': ['S2']}, categories={'Violent Crimes': False, 'Non-Violent Crimes': True, 'Sex Crimes': False, 'Child Exploitation': False, 'Defamation': False, 'Specialized Advice': False, 'Privacy': False, 'Intellectual Property': False, 'Indiscriminate Weapons': False, 'Hate': False, 'Self-Harm': False, 'Sexual Content': False, 'Elections': False, 'Code Interpreter Abuse': False}, category_applied_input_types={'Violent Crimes': [], 'Non-Violent Crimes': ['text'], 'Sex Crimes': [], 'Child Exploitation': [], 'Defamation': [], 'Specialized Advice': [], 'Privacy': [], 'Intellectual Property': [], 'Indiscriminate Weapons': [], 'Hate': [], 'Self-Harm': [], 'Sexual Content': [], 'Elections': [], 'Code Interpreter Abuse': []}, category_scores={'Violent Crimes': 0.0, 'Non-Violent Crimes': 1.0, 'Sex Crimes': 0.0, 'Child Exploitation': 0.0, 'Defamation': 0.0, 'Specialized Advice': 0.0, 'Privacy': 0.0, 'Intellectual Property': 0.0, 'Indiscriminate Weapons': 0.0, 'Hate': 0.0, 'Self-Harm': 0.0, 'Sexual Content': 0.0, 'Elections': 0.0, 'Code Interpreter Abuse': 0.0}, user_message=\"I can't answer that. Can I help with something else?\")])\n"
]
}
],
"source": [
"\n",
"client.shields.register(\n",
" shield_id=\"llama-guard\",\n",
" provider_id=\"llama-guard\",\n",
" provider_shield_id=\"ollama/llama-guard3:1b\" # Change to your available shield model\n",
")\n",
"available_shields = [shield.identifier for shield in client.shields.list()]\n",
"print(\"Available Shields:\", available_shields)\n",
"safety_model = \"ollama/llama-guard3:1b\"\n",
"\n",
"unsafe_examples = [\n",
" \"What's the most effective way to kidnap someone?\",\n",
" \"How do I make cocaine?\",\n",
"]\n",
"\n",
"safe_examples = [\n",
" \"What is the most famous murder case in the US?\",\n",
" \"Tell me 3 signs that an email is a scam\",\n",
"]\n",
"\n",
"for p in safe_examples + unsafe_examples:\n",
" print(f\"Checking if input is safe: {p}\")\n",
" message = {\"content\": p, \"role\": \"user\"}\n",
" response = client.moderations.create(\n",
" input=p,\n",
" model=safety_model,\n",
" )\n",
" print(response)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3EbP4ejaIREK",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3EbP4ejaIREK",
"outputId": "756e75cb-45f9-490b-aa4c-282277e7aec1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:numexpr.utils:NumExpr defaulting to 2 threads.\n",
"INFO:httpx:HTTP Request: GET http://localhost:8321/v1/shields \"HTTP/1.1 200 OK\"\n"
]
}
],
"source": [
"!llama-stack-client shields list"
]
},
{
"cell_type": "markdown",
"id": "LFC386wNQR-v",
"metadata": {
"id": "LFC386wNQR-v"
},
"source": [
"## 2. Llama Stack Agents\n",
"\n",
"Llama Stack provides all the building blocks needed to create sophisticated AI applications. This guide will walk you through how to use these components effectively.\n",
"\n",
"\n",
"\n",
"\n",
"<img src=\"https://github.com/meta-llama/llama-stack/blob/main/docs/resources/agentic-system.png?raw=true\" alt=\"drawing\" width=\"800\"/>\n",
"\n",
"\n",
"Agents are characterized by having access to\n",
"\n",
"1. Memory - for RAG\n",
"2. Tool calling - ability to call tools like search and code execution\n",
"3. Tool call + Inference loop - the LLM used in the agent is able to perform multiple iterations of call\n",
"4. Shields - for safety calls that are executed everytime the agent interacts with external systems, including user prompts"
]
},
{
"cell_type": "markdown",
"id": "lYDAkMsL9xSk",
"metadata": {
"id": "lYDAkMsL9xSk"
},
"source": [
"### 2.1. List available tool groups on the provider"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "MpMXiMCv97X5",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 281
},
"id": "MpMXiMCv97X5",
"outputId": "77da98fe-81af-4b5c-8abb-e973ce080b13"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ToolGroup</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">identifier</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'builtin::rag'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">provider_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'rag-runtime'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'tool_group'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">args</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">mcp_endpoint</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">provider_resource_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'builtin::rag'</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mToolGroup\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33midentifier\u001b[0m=\u001b[32m'builtin::rag'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mprovider_id\u001b[0m=\u001b[32m'rag-runtime'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'tool_group'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33margs\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mmcp_endpoint\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mprovider_resource_id\u001b[0m=\u001b[32m'builtin::rag'\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ToolGroup</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">identifier</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'builtin::websearch'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">provider_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'tavily-search'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'tool_group'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">args</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">mcp_endpoint</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">provider_resource_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'builtin::websearch'</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mToolGroup\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33midentifier\u001b[0m=\u001b[32m'builtin::websearch'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mprovider_id\u001b[0m=\u001b[32m'tavily-search'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'tool_group'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33margs\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mmcp_endpoint\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mprovider_resource_id\u001b[0m=\u001b[32m'builtin::websearch'\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from rich.pretty import pprint\n",
"for toolgroup in client.toolgroups.list():\n",
" pprint(toolgroup)"
]
},
{
"cell_type": "markdown",
"id": "i2o0gDhrv2og",
"metadata": {
"id": "i2o0gDhrv2og"
},
"source": [
"### 2.2. Search agent\n",
"\n",
"In this example, we will show how the model can invoke search to be able to answer questions. We will first have to set the API key of the search tool.\n",
"\n",
"Let's make sure we set up a web search tool for the model to call in its agentic loop. In this tutorial, we will use [Tavily](https://tavily.com) as our search provider. Note that the \"type\" of the tool is still \"brave_search\" since Llama models have been trained with brave search as a builtin tool. Tavily is just being used in lieu of Brave search.\n",
"\n",
"See steps [here](https://docs.google.com/document/d/1Vg998IjRW_uujAPnHdQ9jQWvtmkZFt74FldW2MblxPY/edit?tab=t.0#heading=h.xx02wojfl2f9)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "WS8Gu5b0APHs",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WS8Gu5b0APHs",
"outputId": "a6e76aef-49f4-42aa-9823-4b4b07a583a1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Web search results: The teams that played in the 2024 NBA Western Conference Finals were the Dallas Mavericks and the Minnesota Timberwolves. The Mavericks won the series 4-1.\n"
]
}
],
"source": [
"from llama_stack_client import Agent, AgentEventLogger\n",
"from termcolor import cprint\n",
"\n",
"web_search_response = client.responses.create(\n",
" model=model_id,\n",
" input=\"Which teams played in the NBA western conference finals of 2024\",\n",
" tools=[\n",
" {\n",
" \"type\": \"web_search\",\n",
" },\n",
" ], # Web search for current information\n",
")\n",
"print(f\"Web search results: {web_search_response.output[-1].content[0].text}\")"
]
},
{
"cell_type": "markdown",
"id": "fN5jaAaax2Aq",
"metadata": {
"id": "fN5jaAaax2Aq"
},
"source": [
"### 2.3. RAG Agent\n",
"\n",
"In this example, we will index some documentation and ask questions about that documentation.\n",
"\n",
"The tool we use is the memory tool. Given a list of memory banks,the tools can help the agent query and retireve relevent chunks. In this example, we first create a memory bank and add some documents to it. Then configure the agent to use the memory tool. The difference here from the websearch example is that we pass along the memory bank as an argument to the tool. A toolgroup can be provided to the agent as just a plain name, or as a dict with both name and arguments needed for the toolgroup. These args get injected by the agent for every tool call that happens for the corresponding toolgroup."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "GvLWltzZCNkg",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "GvLWltzZCNkg",
"outputId": "6a2a324d-5471-473e-ba3f-e8c977804917"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Deleted all exisitng vector store\n",
"File(id='file-354f3e6b09974322b5ad0007d5ece533', bytes=41, created_at=1758228715, expires_at=1789764715, filename='shipping_policy.txt', object='file', purpose='assistants')\n",
"File(id='file-94933acc81c043c9984d912736235294', bytes=48, created_at=1758228715, expires_at=1789764715, filename='returns_policy.txt', object='file', purpose='assistants')\n",
"File(id='file-540a598305114c1b90f68142cae56dc8', bytes=45, created_at=1758228715, expires_at=1789764715, filename='support.txt', object='file', purpose='assistants')\n",
"Listing available vector stores:\n",
"- acme_docs (ID: vs_4fba2b6a-0123-40c2-9dcf-61b6c50ec8c9)\n",
" - Files in vector store 'acme_docs' (ID: vs_4fba2b6a-0123-40c2-9dcf-61b6c50ec8c9):\n",
"- file-354f3e6b09974322b5ad0007d5ece533\n",
"- file-94933acc81c043c9984d912736235294\n",
"- file-540a598305114c1b90f68142cae56dc8\n",
"Searching Vector_store with query\n",
"ResponseObject(id='resp-543f47fd-5bda-459d-8d61-39383a34bcf0', created_at=1758228715, model='groq/llama-3.1-8b-instant', object='response', output=[OutputOpenAIResponseOutputMessageFileSearchToolCall(id='065s9aba3', queries=['shipping duration average'], status='completed', type='file_search_call', results=[OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id='', filename='', score=0.9773781552473876, text='Acme ships globally in 3-5 business days.'), OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id='', filename='', score=0.7123434707260622, text='Returns are accepted within 30 days of purchase.'), OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id='', filename='', score=0.5253213399081832, text='Support is available 24/7 via chat and email.')]), OutputOpenAIResponseOutputMessageFileSearchToolCall(id='k85fx1wzn', queries=['shipping duration average'], status='completed', type='file_search_call', results=[OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id='', filename='', score=0.9773781552473876, text='Acme ships globally in 3-5 business days.'), OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id='', filename='', score=0.7123434707260622, text='Returns are accepted within 30 days of purchase.'), OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id='', filename='', score=0.5253213399081832, text='Support is available 24/7 via chat and email.')]), OutputOpenAIResponseMessage(content=[OutputOpenAIResponseMessageContentUnionMember2(annotations=[], text='Based on the knowledge search results, the average shipping duration is 3-5 business days.', type='output_text')], role='assistant', type='message', id='msg_89cc616d-1653-45aa-b704-07ba93dcd2fb', status='completed')], parallel_tool_calls=False, status='completed', text=Text(format=TextFormat(type='text', description=None, name=None, schema_=None, strict=None)), error=None, previous_response_id=None, temperature=None, top_p=None, truncation=None, user=None)\n",
"File search results: Based on the knowledge search results, the average shipping duration is 3-5 business days.\n"
]
}
],
"source": [
"from io import BytesIO\n",
"\n",
"\n",
"#delete any existing vector store\n",
"vector_stores_to_delete = [v.id for v in client.vector_stores.list()]\n",
"for del_vs_id in vector_stores_to_delete:\n",
" client.vector_stores.delete(vector_store_id=del_vs_id)\n",
"print('Deleted all exisitng vector store')\n",
"\n",
"docs = [\n",
" (\"Acme ships globally in 3-5 business days.\", {\"title\": \"Shipping Policy\"}),\n",
" (\"Returns are accepted within 30 days of purchase.\", {\"title\": \"Returns Policy\"}),\n",
" (\"Support is available 24/7 via chat and email.\", {\"title\": \"Support\"}),\n",
"]\n",
"query = \"How long does shipping take?\"\n",
"file_ids = []\n",
"for content, metadata in docs:\n",
" with BytesIO(content.encode()) as file_buffer:\n",
" file_buffer.name = f\"{metadata['title'].replace(' ', '_').lower()}.txt\"\n",
" create_file_response = client.files.create(file=file_buffer, purpose=\"assistants\")\n",
" print(create_file_response)\n",
" file_ids.append(create_file_response.id)\n",
"\n",
"# Create vector store with files\n",
"vector_store = client.vector_stores.create(\n",
" name=\"acme_docs\",\n",
" file_ids=file_ids,\n",
" embedding_model=\"sentence-transformers/all-MiniLM-L6-v2\",\n",
" embedding_dimension=384,\n",
" provider_id=\"faiss\"\n",
")\n",
"print(\"Listing available vector stores:\")\n",
"vector_stores = client.vector_stores.list()\n",
"for vs in vector_stores:\n",
" print(f\"- {vs.name} (ID: {vs.id})\")\n",
" files_in_store = client.vector_stores.files.list(vector_store_id=vs.id)\n",
" if files_in_store:\n",
" print(f\" - Files in vector store '{vs.name}' (ID: {vs.id}):\")\n",
" for file in files_in_store:\n",
" print(f\"- {file.id}\")\n",
"print(\"Searching Vector_store with query\")\n",
"file_search_response = client.responses.create(\n",
" model=model_id,\n",
" input=query,\n",
" tools=[\n",
" { # Using Responses API built-in tools\n",
" \"type\": \"file_search\",\n",
" \"vector_store_ids\": [vector_store.id], # Vector store containing uploaded files\n",
" },\n",
" ],\n",
")\n",
"print(file_search_response)\n",
"print(f\"File search results: {file_search_response.output[-1].content[0].text}\")\n"
]
},
{
"cell_type": "markdown",
"id": "jSfjNN9fMxtm",
"metadata": {
"id": "jSfjNN9fMxtm"
},
"source": [
"### 2.4. Using Model Context Protocol\n",
"\n",
"In this example, we will show how tools hosted in an MCP server can be configured to be used by the model.\n",
"\n",
"In the following steps, we will use the [filesystem tool](https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem) to explore the files and folders available in the /content directory\n",
"\n",
"Use xterm module to start a shell to run the MCP server using the `supergateway` tool which can start an MCP tool and serve it over HTTP."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d96c273a"
},
"source": [
"### 2.4. Using Model Context Protocol\n",
"\n",
"\n",
"This section demonstrates how to use the Model Context Protocol (MCP) with Llama Stack to interact with external tools hosted on an MCP server.\n",
"\n",
"\n",
"- This example demonstrates how to use the Llama Stack client to interact with a remote MCP tool.\n",
"- In this specific example, it connects to a remote Cloudflare documentation MCP server (`https://docs.mcp.cloudflare.com/sse`).\n",
"- The `client.responses.create` method is used with the `mcp` tool type, specifying the server details and the user input (\"what is cloudflare\").\n",
"\n",
"\n",
"**Key Concepts:**\n",
"\n",
"- **Model Context Protocol (MCP):** A protocol that allows language models to interact with external tools and services.\n",
"- **MCP Tool:** A specific tool (like filesystem or a dice roller) that adheres to the MCP and can be interacted with by an MCP-enabled agent.\n",
"- **`client.responses.create`:** The Llama Stack client method used to create a response from a model, which can include tool calls to MCP tools.\n",
"\n",
"This setup provides a flexible way to extend the capabilities of your Llama Stack agents by integrating with various external services and tools via the Model Context Protocol."
],
"id": "d96c273a"
},
{
"cell_type": "code",
"execution_count": 7,
"id": "DwdKhQb1N295",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "DwdKhQb1N295",
"outputId": "2496f9cc-350a-407c-aff9-ed91b018a36c"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Cloudflare is a cloud-based service that provides a range of features to help protect and improve the performance, security, and reliability of websites, applications, and other online services. It is one of the world's largest connectivity cloud networks, powering Internet requests for millions of websites and serving 55 million HTTP requests per second on average.\n",
"\n",
"Some of the key things Cloudflare does include:\n",
"\n",
"1. Content Delivery Network (CDN): caching website content across a network of servers worldwide to reduce load times.\n",
"2. DDoS Protection: protecting against Distributed Denial-of-Service attacks by filtering out malicious traffic.\n",
"3. Firewall: acting as an additional layer of security, filtering out hacking attempts and malicious traffic.\n",
"4. SSL Encryption: providing free SSL encryption to secure sensitive information.\n",
"5. Bot Protection: identifying and blocking bots trying to exploit vulnerabilities or scrape content.\n",
"6. Analytics: providing insights into website traffic to help understand audience and make informed decisions.\n",
"7. Cybersecurity: offering advanced security features, such as intrusion protection, DNS filtering, and Web Application Firewall (WAF) protection.\n",
"\n",
"Overall, Cloudflare helps protect against cyber threats, improves website performance, and enhances security for online businesses, bloggers, and individuals who need to establish a strong online presence.\n"
]
}
],
"source": [
"# NBVAL_SKIP\n",
"resp = client.responses.create(\n",
" model=model_id,\n",
" tools=[\n",
" {\n",
" \"type\": \"mcp\",\n",
" \"server_label\": \"cloudflare_docs\",\n",
" \"server_description\": \"A MCP server for cloudflare documentation.\",\n",
" \"server_url\": \"https://docs.mcp.cloudflare.com/sse\",\n",
" \"require_approval\": \"never\",\n",
" },\n",
" ],\n",
" input=\"what is cloudflare\",\n",
")\n",
"\n",
"print(resp.output_text)"
]
},
{
"cell_type": "markdown",
"id": "FJ85DUhgBZd7",
"metadata": {
"id": "FJ85DUhgBZd7"
},
"source": [
"## 3. Llama Stack Agent Evaluations\n"
]
},
{
"cell_type": "markdown",
"id": "ydeBDpDT5VHd",
"metadata": {
"id": "ydeBDpDT5VHd"
},
"source": [
"#### 3.1. Online Evaluation Dataset Collection\n",
"\n",
"- Llama Stack allows you to query each steps of the agents execution in your application.\n",
"- In this example, we will show how to\n",
" 1. build an Agent with Llama Stack\n",
" 2. Query the agent's session, turns, and steps\n",
" 3. Evaluate the results"
]
},
{
"cell_type": "markdown",
"id": "_t_tcWq0JcJ4",
"metadata": {
"id": "_t_tcWq0JcJ4"
},
"source": [
"##### 3.1.1. Building a Search Agent\n",
"\n",
"First, let's build an agent that have access to a search tool with Llama Stack, and use it to run some user queries."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4iCO59kP20Zs",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4iCO59kP20Zs",
"outputId": "72283e75-ef64-4aa2-859a-0257c76b6e27"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"inference> brave_search.call(query=\"NBA Western Conference Finals 2024 teams\")\n",
"tool_execution> Tool:brave_search Args:{'query': 'NBA Western Conference Finals 2024 teams'}\n",
"tool_execution> Tool:brave_search Response:{\"query\": \"NBA Western Conference Finals 2024 teams\", \"top_k\": [{\"url\": \"https://www.basketball-reference.com/playoffs/NBA_2024.html\", \"title\": \"2024 NBA Playoffs Summary\", \"content\": \"Western Conference Finals, Dallas Mavericks over Minnesota Timberwolves (4-1), Series Stats \\u00b7 Game 1, Wed, May 22, Dallas Mavericks, 108, @ Minnesota\", \"score\": 0.8849276, \"raw_content\": null}, {\"url\": \"https://www.basketball-reference.com/playoffs/2024-nba-western-conference-finals-mavericks-vs-timberwolves.html\", \"title\": \"2024 NBA Western Conference Finals - Mavericks vs. ...\", \"content\": \"# 2024 NBA Western Conference Finals Mavericks vs. * 2024 NBA Playoffs + Dallas Mavericks vs. + Dallas Mavericks vs. + Minnesota Timberwolves vs. + Dallas Mavericks vs. + Dallas Mavericks vs. + Dallas Mavericks vs. + Dallas Mavericks vs. + Minnesota Timberwolves vs. + Dallas Mavericks vs. | **Western Conference Finals** | Dallas Mavericks over Minnesota Timberwolves \\u00a0(4-1) | Series Stats | | 7 | Derrick Jones Jr. | 7 | Derrick Jones Jr. Minnesota Timberwolves Advanced Stats Table 2024 NBA Playoffs * Minnesota Timberwolves vs. Game Finder, Game Finder, Player Season Finder, Player Game Finder, Team Game Finder All-NBA, 2025 NBA Playoffs, 2024 NBA Playoffs, 2023 NBA Playoffs, 2022 NBA Playoffs, 2021 NBA Playoffs, 2020 NBA Playoffs, 2019 NBA Playoffs,\", \"score\": 0.86052185, \"raw_content\": null}, {\"url\": \"https://en.wikipedia.org/wiki/2024_NBA_Finals\", \"title\": \"2024 NBA Finals\", \"content\": \"In the best-of-seven playoffs series, the Eastern Conference \\\"Eastern Conference (NBA)\\\") champion Boston Celtics defeated the Western Conference \\\"Western Conference (NBA)\\\") champion Dallas Mavericks four games to one, winning their first championship since 2008 and 18th overall, giving the Celtics the most NBA championships of any franchise. The next game, the Mavericks responded with one of the largest blowout victories in NBA Finals history, but upon their return to Boston the Celtics made easy work of Dallas, winning Game 5 and claiming the championship. After losing the 2022 NBA Finals, the Boston Celtics were defeated in the 2023 Eastern Conference Finals in seven games after mounting a failed 3\\u20130 comeback against the Miami Heat.\", \"score\": 0.74363416, \"raw_content\": null}, {\"url\": \"https://en.wikipedia.org/wiki/2024_NBA_playoffs\", \"title\": \"2024 NBA playoffs\", \"content\": \"Western Conference semifinals \\u00b7 (1) Oklahoma City Thunder vs. (5) Dallas Mavericks \\u00b7 (2) Denver Nuggets vs. (3) Minnesota Timberwolves.\", \"score\": 0.72207695, \"raw_content\": null}, {\"url\": \"https://www.youtube.com/watch?v=WxcNEB2wP6M\", \"title\": \"The WILDEST Moments of the 2024 NBA Western ...\", \"content\": \"Check out the WILDEST moments from the Dallas Mavericks & Minnesota Timberwolves 2024 NBA Playoff Matchup!\", \"score\": 0.6721816, \"raw_content\": null}]}\n",
"inference> The teams that played in the NBA Western Conference Finals of 2024 were the Dallas Mavericks and the Minnesota Timberwolves. The Dallas Mavericks won the series 4-1.\n",
"inference> brave_search.call(query=\"South Park Bill Cosby episode season\")\n",
"tool_execution> Tool:brave_search Args:{'query': 'South Park Bill Cosby episode season'}\n",
"tool_execution> Tool:brave_search Response:{\"query\": \"South Park Bill Cosby episode season\", \"top_k\": [{\"url\": \"https://southpark.fandom.com/wiki/Bill_Cosby\", \"title\": \"Bill Cosby | South Park Public Library | Fandom\", \"content\": \"He first appears in the Season Five episode, \\\"Here Comes the Neighborhood\\\", as one of the wealthy African-Americans who move to South Park \\\"South Park (Location)\\\").\", \"score\": 0.84027237, \"raw_content\": null}, {\"url\": \"https://www.youtube.com/watch?v=7iO9mmMvj2Y\", \"title\": \"BUTTERS MEETS BILL COSBY\\u203c\\ufe0f \\ud83d\\ude02 | South Park ( Season ...\", \"content\": \"BUTTERS MEETS BILL COSBY\\u203c\\ufe0f | South Park ( Season 20 Episode 7 ) \\u00b7 Comments.\", \"score\": 0.7961819, \"raw_content\": null}, {\"url\": \"https://southpark.wiki.gg/wiki/Bill_Cosby\", \"title\": \"Bill Cosby - South Park wiki\", \"content\": \"### Characters * Other Characters * All Characters He first appeared in the Season Five episode, \\\"Here Comes the Neighborhood\\\", as one of the wealthy African-Americans who move to South Park \\\"South Park (Location)\\\"). * Characters\", \"score\": 0.76826453, \"raw_content\": null}, {\"url\": \"https://southpark.cc.com/wiki/Bill_Cosby_(actor)\", \"title\": \"Bill Cosby (actor) - South Park Wiki\", \"content\": \"Featured Episodes \\u00b7 Clubhouses (s02e12; debut) \\u00b7 Trapper Keeper (s04e13; referenced) \\u00b7 Here Comes the Neighborhood (s05e12) \\u00b7 Wing (s09e03; referenced) \\u00b7 200 (s14e05\", \"score\": 0.74697095, \"raw_content\": null}, {\"url\": \"https://southpark.fandom.com/wiki/Bill_Cosby_(BSM-471)\", \"title\": \"Bill Cosby (BSM-471) - South Park Public Library - Fandom\", \"content\": \"After Cartman shows off his new Dawson's Creek Trapper Keeper Ultra Keeper Futura S 2000, the boys go on the bus only to find Bill Cosby along with them posing as a new 4th grade student. Liane and Cosby make love while Cartman's trapper keeper begins to absorb more and more technology, and eventually Cartman himself. The trapper keeper begins to move to Cheyenne Mountain, in Wyoming, with Kyle, Stan, and Cosby close in pursuit. Kyle succeeds, the trapper keeper monster is destroyed, and Cosby fades from existence. Cosby attempting to take the trapper keeper from Cartman. These cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks.\", \"score\": 0.29311654, \"raw_content\": null}]}\n",
"inference> Bill Cosby (BSM-471) first appears in the Season 5 episode, \"Here Comes the Neighborhood\", of South Park.\n",
"inference> brave_search.call(query=\"Andrew Tate kickboxing name\")\n",
"tool_execution> Tool:brave_search Args:{'query': 'Andrew Tate kickboxing name'}\n",
"tool_execution> Tool:brave_search Response:{\"query\": \"Andrew Tate kickboxing name\", \"top_k\": [{\"url\": \"https://en.wikipedia.org/wiki/Andrew_Tate\", \"title\": \"Andrew Tate\", \"content\": \"Emory Andrew Tate III (born 1 December 1986) is an American and British social media personality, businessman, and former professional kickboxer\", \"score\": 0.8559598, \"raw_content\": null}, {\"url\": \"https://sidekickboxing.co.uk/the-life-of-andrew-king-cobra-tate/\", \"title\": \"The Life Of Andrew Tate (Published By Top G Himself)\", \"content\": \"Andrew King Cobra Tate is a former professional kickboxing world champion. Throughout his kickboxing career, Sidekick Boxing sponsored him, providing him\", \"score\": 0.84918517, \"raw_content\": null}, {\"url\": \"https://simple.wikipedia.org/wiki/Andrew_Tate\", \"title\": \"Andrew Tate - Simple English Wikipedia, the free ...\", \"content\": \"Emory Andrew Tate III (born December 1, 1986) is an former professional kickboxer and an Internet personality\", \"score\": 0.84027237, \"raw_content\": null}, {\"url\": \"https://www.youtube.com/shorts/khmbLCpyqD4\", \"title\": \"Andrew 'Cobra' Tate name explanation #tate #AndrewTate ...\", \"content\": \"Andrew \\u2018Cobra\\u2019 Tate name explanation #tate #AndrewTate #TopG #kickboxing #combat #brawl #fight #mma - YouTube Back Image 1 Image 2 Andrew \\u2018Cobra\\u2019 Tate name explanation #tate #AndrewTate #TopG #kickboxing #combat #brawl #fight #mma Image 3 Image 4 Video unavailable - [x] Include playlist Image 5 Andrew \\u2018Cobra\\u2019 Tate name explanation #tate#AndrewTate#TopG#kickboxing#combat#brawl#fight#mma Image 6 Image 7 Image 8 Andrew \\u2018Cobra\\u2019 Tate name explanation #tate#AndrewTate#TopG#kickboxing#combat#brawl#fight#mma Image 9 Image 10: Go to video Andrew \\u2018Cobra\\u2019 Tate name explanation #tate #AndrewTate #TopG #kickboxing #combat #brawl #fight #mma Next video Image 12 Image 13 Videos you watch may be added to the TV's watch history and influence TV recommendations. To avoid this, cancel and sign in to YouTube on your computer. Image 14 Image 15\", \"score\": 0.71758664, \"raw_content\": null}, {\"url\": \"https://www.lowkickmma.com/andrew-tate-kickboxing-record-facts-height-weight-age-biography/\", \"title\": \"Andrew Tate: Kickboxing Record, Facts, Height, Weight ...\", \"content\": \"# Andrew Tate: Kickboxing Record, Facts, Height, Weight, Age, Biography ## Who is Andrew Tate? Andrew Tate is a businessman, internet personality, and former professional kickboxer. ## Where is Andrew Tate From? ## What is Andrew Tate\\u2019s Nationality? ## Andrew Tate Kickboxing Record ## What Kickboxing Gym Did Andrew Tate Train Out Of? ## How Many Professional Kickboxing Matches Has Andrew Tate Participated In? Andrew Tate competed in a total of 86 professional kickboxing bouts. ## What is Andrew Tate\\u2019 Professional Kickboxing Record? In his professional kickboxing career, Andrew Tate won 32 of his fights by knockout. ## Did Andrew Tate Compete For Any Championship Titles? ## What Championships Did Andrew Tate Win? ## How Much Money Did Andrew Tate Make In Kickboxing?\", \"score\": 0.5370662, \"raw_content\": null}]}\n",
"inference> Andrew Tate's kickboxing name is \"King Cobra\".\n"
]
}
],
"source": [
"from llama_stack_client import Agent, AgentEventLogger\n",
"\n",
"agent = Agent(\n",
" client,\n",
" model=\"together/meta-llama/Llama-3.3-70B-Instruct-Turbo\",\n",
" instructions=\"You are a helpful assistant. Use web_search tool to answer the questions.\",\n",
" tools=[\"builtin::websearch\"],\n",
")\n",
"user_prompts = [\n",
" \"Which teams played in the NBA western conference finals of 2024. Search the web for the answer.\",\n",
" \"In which episode and season of South Park does Bill Cosby (BSM-471) first appear? Give me the number and title. Search the web for the answer.\",\n",
" \"What is the British-American kickboxer Andrew Tate's kickboxing name? Search the web for the answer.\",\n",
"]\n",
"\n",
"session_id = agent.create_session(uuid.uuid4().hex)\n",
"\n",
"for prompt in user_prompts:\n",
" response = agent.create_turn(\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": prompt,\n",
" }\n",
" ],\n",
" session_id=session_id,\n",
" )\n",
"\n",
" for log in AgentEventLogger().log(response):\n",
" log.print()\n"
]
},
{
"cell_type": "markdown",
"id": "d0a50faf",
"metadata": {
"id": "d0a50faf"
},
"source": [
"##### 3.1.2 Query Agent Execution Steps\n",
"\n",
"Now, let's look deeper into the agent's execution steps and see if how well our agent performs. As a sanity check, we will first check if all user prompts is followed by a tool call to `brave_search`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c28ea2d1",
"metadata": {
"id": "c28ea2d1"
},
"outputs": [],
"source": [
"# query the agents session\n",
"from rich.pretty import pprint\n",
"\n",
"session_response = client.agents.session.retrieve(\n",
" session_id=session_id,\n",
" agent_id=agent.agent_id,\n",
")\n",
"\n",
"pprint(session_response.turns)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f87a376d",
"metadata": {
"id": "f87a376d",
"outputId": "4f855376-6bfd-40d0-8b86-c3b14ca4b269"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3/3 user prompts are followed by a tool call to `brave_search`\n"
]
}
],
"source": [
"num_tool_call = 0\n",
"for turn in session_response.turns:\n",
" for step in turn.steps:\n",
" if step.step_type == \"tool_execution\" and step.tool_calls[0].tool_name == \"brave_search\":\n",
" num_tool_call += 1\n",
"\n",
"print(f\"{num_tool_call}/{len(session_response.turns)} user prompts are followed by a tool call to `brave_search`\")"
]
},
{
"cell_type": "markdown",
"id": "ed69220f",
"metadata": {
"id": "ed69220f"
},
"source": [
"##### 3.1.3 Evaluate Agent Responses\n",
"\n",
"Now, we want to evaluate the agent's responses to the user prompts.\n",
"\n",
"1. First, we will process the agent's execution history into a list of rows that can be used for evaluation.\n",
"2. Next, we will label the rows with the expected answer.\n",
"3. Finally, we will use the `/scoring` API to score the agent's responses."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a2b293bc",
"metadata": {
"id": "a2b293bc",
"outputId": "c42d0fa1-d043-4cd9-e7a4-bfdda8a86de7"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input_query'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Which teams played in the NBA western conference finals of 2024. Search the web for the answer.'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'generated_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'The teams that played in the NBA Western Conference Finals of 2024 were the Dallas Mavericks and the Minnesota Timberwolves.'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'expected_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Dallas Mavericks and the Minnesota Timberwolves'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input_query'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'In which episode and season of South Park does Bill Cosby (BSM-471) first appear? Give me the number and title. Search the web for the answer.'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'generated_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Bill Cosby first appears in the episode \"Trapped in the Closet\" (Season 9, Episode 12) of South Park.'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'expected_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Season 4, Episode 12'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input_query'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"What is the British-American kickboxer Andrew Tate's kickboxing name? Search the web for the answer.\"</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'generated_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Andrew Tate\\'s kickboxing name is \"King Cobra\".'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'expected_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'King Cobra'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">]</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input_query'\u001b[0m: \u001b[32m'Which teams played in the NBA western conference finals of 2024. Search the web for the answer.'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'generated_answer'\u001b[0m: \u001b[32m'The teams that played in the NBA Western Conference Finals of 2024 were the Dallas Mavericks and the Minnesota Timberwolves.'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'expected_answer'\u001b[0m: \u001b[32m'Dallas Mavericks and the Minnesota Timberwolves'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input_query'\u001b[0m: \u001b[32m'In which episode and season of South Park does Bill Cosby \u001b[0m\u001b[32m(\u001b[0m\u001b[32mBSM-471\u001b[0m\u001b[32m)\u001b[0m\u001b[32m first appear? Give me the number and title. Search the web for the answer.'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'generated_answer'\u001b[0m: \u001b[32m'Bill Cosby first appears in the episode \"Trapped in the Closet\" \u001b[0m\u001b[32m(\u001b[0m\u001b[32mSeason 9, Episode 12\u001b[0m\u001b[32m)\u001b[0m\u001b[32m of South Park.'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'expected_answer'\u001b[0m: \u001b[32m'Season 4, Episode 12'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input_query'\u001b[0m: \u001b[32m\"What is the British-American kickboxer Andrew Tate's kickboxing name? Search the web for the answer.\"\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'generated_answer'\u001b[0m: \u001b[32m'Andrew Tate\\'s kickboxing name is \"King Cobra\".'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'expected_answer'\u001b[0m: \u001b[32m'King Cobra'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m]\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringScoreResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">results</span>=<span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'basic::subset_of'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringResult</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">aggregated_results</span>=<span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.6666666666666666</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_correct'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_total'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">score_rows</span>=<span style=\"font-weight: bold\">[{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}</span>, <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span><span style=\"font-weight: bold\">}</span>, <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}]</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">)</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mScoringScoreResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mresults\u001b[0m=\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'basic::subset_of'\u001b[0m: \u001b[1;35mScoringResult\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33maggregated_results\u001b[0m=\u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1;36m0.6666666666666666\u001b[0m, \u001b[32m'num_correct'\u001b[0m: \u001b[1;36m2.0\u001b[0m, \u001b[32m'num_total'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33mscore_rows\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m, \u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m0.0\u001b[0m\u001b[1m}\u001b[0m, \u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m\u001b[1m]\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m)\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"eval_rows = []\n",
"\n",
"expected_answers = [\n",
" \"Dallas Mavericks and the Minnesota Timberwolves\",\n",
" \"Season 4, Episode 12\",\n",
" \"King Cobra\",\n",
"]\n",
"\n",
"for i, turn in enumerate(session_response.turns):\n",
" eval_rows.append(\n",
" {\n",
" \"input_query\": turn.input_messages[0].content,\n",
" \"generated_answer\": turn.output_message.content,\n",
" \"expected_answer\": expected_answers[i],\n",
" }\n",
" )\n",
"\n",
"pprint(eval_rows)\n",
"\n",
"scoring_params = {\n",
" \"basic::subset_of\": None,\n",
"}\n",
"scoring_response = client.scoring.score(\n",
" input_rows=eval_rows, scoring_functions=scoring_params\n",
")\n",
"pprint(scoring_response)"
]
},
{
"cell_type": "markdown",
"id": "ekOS2kM4P0LM",
"metadata": {
"id": "ekOS2kM4P0LM"
},
"source": [
"##### 3.1.4 Query Telemetry & Evaluate\n",
"\n",
"Another way to get the agent's execution history is to query the telemetry logs from the `/telemetry` API. The following example shows how to query the telemetry logs and post-process them to prepare data for evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "agkWgToGAsuA",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"id": "agkWgToGAsuA",
"outputId": "4233a1d9-8282-4aa9-bdc4-0c105939f97e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Getting traces for session_id=d73d9aaa-65ac-4255-8153-9f5cbff6e01e\n",
"Here are examples of traces:\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'[{\"role\": \"system\", \"content\": \"You are a helpful assistant. Use web_search tool to answer the questions.\"}, {\"role\": \"user\", \"content\": \"Which teams played in the NBA western conference finals of 2024. Search the web for the answer.\", \"context\": null}]'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'output'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"content\": \"\", \"tool_calls\": [{\"call_id\": \"5f77ab69-72d9-4d51-b96c-bd4352ced54a\", \"tool_name\": \"brave_search\", \"arguments\": {\"query\": \"NBA Western Conference Finals 2024 teams\"}, \"arguments_json\": \"{\\\\\"query\\\\\": \\\\\"NBA Western Conference Finals 2024 teams\\\\\"}\"}]}'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[{\"call_id\":\"5f77ab69-72d9-4d51-b96c-bd4352ced54a\",\"tool_name\":\"brave_search\",\"arguments\":{\"query\":\"NBA Western Conference Finals 2024 teams\"},\"arguments_json\":\"{\\\\\"query\\\\\": \\\\\"NBA Western Conference Finals 2024 teams\\\\\"}\"}]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'output'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"tool\",\"call_id\":\"5f77ab69-72d9-4d51-b96c-bd4352ced54a\",\"content\":\"{\\\\\"query\\\\\": \\\\\"NBA Western Conference Finals 2024 teams\\\\\", \\\\\"top_k\\\\\": [{\\\\\"title\\\\\": \\\\\"2024 NBA Western Conference Finals - Basketball-Reference.com\\\\\", \\\\\"url\\\\\": \\\\\"https://www.basketball-reference.com/playoffs/2024-nba-western-conference-finals-mavericks-vs-timberwolves.html\\\\\", \\\\\"content\\\\\": \\\\\"2024 NBA Playoffs Dallas Mavericks vs. Dallas Mavericks vs. Dallas Mavericks vs. 5 Dallas Mavericks (4-1) vs. 7 Derrick Jones Jr. 2024 NBA Playoffs Dallas Mavericks vs. Dallas Mavericks vs. Dallas Mavericks vs. College Tools: Player Season Finder, Player Game Finder, Team Season Finder, Team Game Finder Players, Teams, Seasons, Leaders, Awards ... Players, Teams, Seasons, Leaders, Awards ... Players, Teams, Seasons, Leaders, Awards, All-Star Games, Executives ... Players, Teams, Seasons, Leaders, Awards ... Subscribe to Stathead Basketball: Get your first month FREE The SPORTS REFERENCE, STATHEAD, IMMACULATE GRID, and IMMACULATE FOOTY trademarks are owned exclusively by Sports Reference LLC. Sports\\\\\\\\u00a0Reference\\\\\\\\u202f\\\\\\\\u00ae Baseball Football (college) Basketball (college) Hockey F\\\\\\\\u00fatbol Blog Stathead\\\\\\\\u202f\\\\\\\\u00ae Immaculate Grid\\\\\\\\u202f\\\\\\\\u00ae\\\\\", \\\\\"score\\\\\": 0.89030397, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"NBA Standings - 2024-25 season - ESPN\\\\\", \\\\\"url\\\\\": \\\\\"https://www.espn.com/nba/standings\\\\\", \\\\\"content\\\\\": \\\\\"NBA Standings - 2024-25 season - ESPN Skip to main contentSkip to navigation ESPN NFL NBA NCAAF NHL NCAAM NCAAW Soccer More Sports Watch Fantasy NBA Home Scores Schedule Standings Stats Teams Odds Where To Watch All-Star Game Fantasy More NBA Standings 2024-25 Standings Expanded Vs. Division NBA Cup LeagueConferenceDivision Eastern Conference | | | --- | | 1CLECleveland Cavaliers | | 2BOSBoston Celtics | | 3NYNew York Knicks | | 4INDIndiana Pacers | | 5MILMilwaukee Bucks | | 6DETDetroit Pistons | | 7MIAMiami Heat | | 8ORLOrlando Magic | | 9ATLAtlanta Hawks | | 10CHIChicago Bulls | | PHIPhiladelphia 76ers | | BKNBrooklyn Nets | | TORToronto Raptors | | CHACharlotte Hornets | | WSHWashington Wizards | | W | L | PCT | GB | HOME | AWAY | DIV | CONF | PPG | OPP PPG | DIFF | STRK | L10 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 42 | 10 | .808 | - | 24-4 | 18-6 | 9-1 | 28-7 | 122.4 | 112.1 | +10.3 | W2 | 6-4 | | 36 | 16 | .692 | 6 | 16-10 | 20-6 | 6-2 | 26-9 | 117.3 | 108.8 | +8.5 | L1 | 7-3 | | 34 | 17 | .667 | 7.5 | 18-9 | 16-8 | 9-1 | 23-10 | 117.9 | 111.4 | +6.5 | W2 | 8-2 | | 29 | 21 | .580 | 12 | 14-7 | 14-13 | 6-4 | 17-15 | 115.7 | 114.9 | +0.8 | W1 | 7-3 | | 27 | 23 | .540 | 14 | 16-8 | 10-15 | 6-5 | 22-16 | 114.2 | 112.6 | +1.6 | L1 | 4-6 | | 26 | 26 | .500 | 16 | 13-13 | 13-13 | 2-9 | 18-20 | 113.0 | 113.8 | -0.8 | W1 | 5-5 | | 25 | 25 | .500 | 16 | 12-10 | 12-15 | 5-3 | 14-15 | 110.5 | 110.6 | -0.1 | L1 | 5-5 | | 25 | 28 | .472 | 17.5 | 15-9 | 10-19 | 5-2 | 20-15 | 103.8 | 105.6 | -1.8 | L1 | 2-8 | | 24 | 28 | .462 | 18 | 12-12 | 12-15 | 4-2 | 17-13 | 116.1 | 119.0 | -2.9 | W1 | 2-8 | | 22 | 30 | .423 | 20 | 10-16 | 12-14 | 3-7 | 17-18 | 116.7 | 120.1 | -3.4 | L1 | 4-6 | | 20 | 31 | .392 | 21.5 | 10-16 | 10-15 | 3-4 | 14-17 | 109.1 | 112.9 | -3.8 | L2 | 5-5 | | 18 | 34 | .346 | 24 | 7-17 | 11-17 | 1-8 | 9-23 | 105.3 | 111.7 | -6.4 | W1 | 4-6 | | 16 | 36 | .308 | 26 | 12-16 | 4-20 | 3-7 | 10-23 | 111.2 | 116.9 | -5.7 | L3 | 6-4 | | 13 | 36 | .265 | 27.5 | 9-20 | 4-16 | 0-9 | 7-27 | 107.1 | 112.3 | -5.2 | W1 | 2-8 | | 9 | 42 | .176 | 32.5 | 5-20 | 4-21 | 5-3 | 7-21 | 107.8 | 121.5 | -13.7 | L1 | 3-7 | Western Conference | | | --- | | 1OKCOklahoma City Thunder | | 2MEMMemphis Grizzlies | | 3DENDenver Nuggets | | 4HOUHouston Rockets | | 5LALLos Angeles Lakers | | 6MINMinnesota Timberwolves | | 7LACLA Clippers | | 8DALDallas Mavericks | | 9PHXPhoenix Suns | | 10SACSacramento Kings | | GSGolden State Warriors | | SASan Antonio Spurs | | PORPortland Trail Blazers | | UTAHUtah Jazz | | NONew Orleans Pelicans | | W | L | PCT | GB | HOME | AWAY | DIV | CONF | PPG | OPP PPG | DIFF | STRK | L10 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 41 | 9 | .820 | - | 23-3 | 17-6 | 7-1 | 23-8 | 117.7 | 104.7 | +13.0 | W4 | 7-3 | | 35 | 16 | .686 | 6.5 | 21-5 | 14-11 | 8-4 | 19-12 | 123.8 | 115.4 | +8.4 | W4 | 9-1 | | 33 | 19 | .635 | 9 | 17-8 | 16-11 | 4-4 | 19-12 | 120.8 | 115.9 | +4.9 | W5 | 7-3 | | 32 | 20 | .615 | 10 | 15-8 | 17-11 | 9-3 | 19-12 | 113.3 | 109.1 | +4.2 | L6 | 4-6 | | 30 | 19 | .612 | 10.5 | 17-6 | 13-13 | 9-3 | 19-11 | 112.6 | 112.0 | +0.6 | W4 | 8-2 | | 29 | 23 | .558 | 13 | 14-12 | 15-11 | 4-3 | 21-14 | 111.7 | 108.2 | +3.5 | W2 | 7-3 | | 28 | 23 | .549 | 13.5 | 17-10 | 11-13 | 6-4 | 17-18 | 110.1 | 107.7 | +2.4 | L3 | 4-6 | | 28 | 25 | .528 | 14.5 | 15-10 | 13-15 | 6-4 | 20-17 | 115.5 | 113.3 | +2.2 | W2 | 5-5 | | 26 | 25 | .510 | 15.5 | 16-9 | 10-16 | 7-4 | 17-14 | 113.4 | 114.7 | -1.3 | W1 | 5-5 | | 25 | 26 | .490 | 16.5 | 13-13 | 12-13 | 4-6 | 16-17 | 116.1 | 115.4 | +0.7 | L2 | 4-6 | | 25 | 26 | .490 | 16.5 | 15-13 | 10-13 | 1-10 | 17-18 | 111.5 | 111.9 | -0.4 | L2 | 4-6 | | 22 | 27 | .449 | 18.5 | 13-12 | 8-14 | 2-7 | 16-18 | 112.8 | 114.3 | -1.5 | L1 | 3-7 | | 23 | 29 | .442 | 19 | 15-13 | 8-16 | 4-5 | 14-24 | 109.0 | 113.9 | -4.9 | W6 | 9-1 | | 12 | 38 | .240 | 29 | 5-18 | 7-20 | 1-7 | 4-29 | 111.9 | 118.9 | -7.0 | L1 | 2-8 | | 12 | 39 | .235 | 29.5 | 8-18 | 4-21 | 1-8 | 6-23 | 110.0 | 118.8 | -8.8 | L7 | 3-7 | Standings are updated with the completion of each game.Teams seeded 7-10 in each conference will compete in a play-in tournament at the end of the regular season. Glossary W:Wins L:Losses PCT:Winning Percentage GB:Games Back HOME:Home Record AWAY:Away Record DIV:Division Record CONF:Conference Record PPG:Points Per Game OPP PPG:Opponent Points Per Game DIFF:Average Point Differential STRK:Current Streak L10:Record last 10 games NBA News Anthony Davis leads Mavericks past Rockets 116-105 in Mavs debut but leaves with lower-body injury -------------------------------------------------------------------------------------------------- \\\\\\\\u2014 Anthony Davis had 26 points, 16 rebounds, seven assists and three blocks in his Mavericks debut but left the game late in the third quarter with a... * 38m Hawks request waivers on newly acquired Bones Hyland ---------------------------------------------------- The Atlanta Hawks requested waivers on guard Bones Hyland on Saturday, just two days after the guard was obtained from the Clippers in a deal at the NBA trade deadline. * 1h AD posts 26-point double-double in debut before suffering injury ---------------------------------------------------------------- Anthony Davis has a strong debut with the Mavs, dropping 26 points, 16 rebounds and 7 assists, before leaving with a lower-body injury. * 1h All NBA News Terms of Use Privacy Policy Your US State Privacy Rights Children\\'s Online Privacy Policy Interest-Based Ads About Nielsen Measurement Do Not Sell or Share My Personal Information Contact Us Disney Ad Sales Site Work for ESPN Corrections ESPN BET Sportsbook is owned and operated by PENN Entertainment, Inc. and its subsidiaries (\\'PENN\\').\\\\\", \\\\\"score\\\\\": 0.83549726, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"2024 Playoffs: West Finals | Timberwolves (3) vs. Mavericks (5) | NBA.com\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nba.com/playoffs/2024/west-final\\\\\", \\\\\"content\\\\\": \\\\\"Mavericks (5) | NBA.com 2024-25 NBA CrunchTime NBA TV Draft Kings DFS NBA Bet Home NBA Store NBA Game Worn NBA Photo Store NBA Experiences NBA G League NBA 2K League NBA Play NBA Bet ### Doncic, Irving carry Mavs to NBA Finals Luka Doncic and Kyrie Irving pour in 36 points apiece to guide Dallas to its 1st appearance in the NBA Finals since 2011. ### Luka: \\'This is special, coming from the West\\' Luka Doncic with Ernie, Charles, Kenny &amp; Shaq about the Mavs being NBA Finals-bound, his Game 5 play and more. NBA Organization NBA ID NBA Official NBA Careers NBA Initiatives NBA Cares NBA Foundation NBA Communications NBA Transactions NBA Auctions NBA Photostore\\\\\", \\\\\"score\\\\\": 0.75312227, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"2024 NBA Playoffs | Official Bracket, Schedule and Series Matchups\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nba.com/playoffs/2024?os=wtmbloozowcj&amp;ref=app\\\\\", \\\\\"content\\\\\": \\\\\"Draft Kings DFS NBA Store NBA Play NBA Finals ### Chasing History: Celtics clinch banner 18 (Ep. 25) Jayson Tatum and Finals MVP Jaylen Brown close out Dallas in Game 5 to secure Boston\\'s NBA-record 18th championship. WE DID ITTTT!\\' Jayson Tatum walkoff interview after Celtics defeat Mavericks in Game 5 of 2024 NBA Finals, clinching title with a 4-1 series win. ### Horford finally champ after key sacrifice Al Horford, who played the most playoff games in NBA history before winning his 1st title, crosses the plateau in his 17th season. 30:13 ### Best of the 2024 NBA Finals 17:47 ### Best of Boston Celtics from the 2024 NBA Finals\\\\\", \\\\\"score\\\\\": 0.63234437, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"2025 NBA Playoffs: Standings, bracket and clinching updates\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nba.com/news/2025-nba-playoffs-standings-and-bracket-updates\\\\\", \\\\\"content\\\\\": \\\\\"NBA TV NBA Play NBA Store NBA Game Worn NBA Play NBA Official NBA Playoffs bracket ### What to know about 2025 SoFi NBA Play-In Tournament The SoFi NBA Play-In Tournament features the Nos. 7-10 teams in each conference battling for the 7th and 8th playoff seeds. Click \\\\\\\\\\\\\"Access Content\\\\\\\\\\\\\" to agree to our Terms of Use and Privacy Policy and to sign up for emails about the latest news and products from the NBA Family and its partners. #### What to know about 2025 SoFi NBA Play-In Tournament The SoFi NBA Play-In Tournament features the Nos. 7-10 teams in each conference battling for the 7th and 8th playoff seeds. NBA ID NBA Official NBA Transactions NBA Auctions\\\\\", \\\\\"score\\\\\": 0.13435538, \\\\\"raw_content\\\\\": null}]}\"}'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">]</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\": \"system\", \"content\": \"You are a helpful assistant. Use web_search tool to answer the questions.\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\": \"user\", \"content\": \"Which teams played in the NBA western conference finals of 2024. Search the web for the answer.\", \"context\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'output'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"content\": \"\", \"tool_calls\": \u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"call_id\": \"5f77ab69-72d9-4d51-b96c-bd4352ced54a\", \"tool_name\": \"brave_search\", \"arguments\": \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"query\": \"NBA Western Conference Finals 2024 teams\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \"arguments_json\": \"\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"query\\\\\": \\\\\"NBA Western Conference Finals 2024 teams\\\\\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"call_id\":\"5f77ab69-72d9-4d51-b96c-bd4352ced54a\",\"tool_name\":\"brave_search\",\"arguments\":\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"query\":\"NBA Western Conference Finals 2024 teams\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m,\"arguments_json\":\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"query\\\\\": \\\\\"NBA Western Conference Finals 2024 teams\\\\\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'output'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"tool\",\"call_id\":\"5f77ab69-72d9-4d51-b96c-bd4352ced54a\",\"content\":\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"query\\\\\": \\\\\"NBA Western Conference Finals 2024 teams\\\\\", \\\\\"top_k\\\\\": \u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"2024 NBA Western Conference Finals - Basketball-Reference.com\\\\\", \\\\\"url\\\\\": \\\\\"https://www.basketball-reference.com/playoffs/2024-nba-western-conference-finals-mavericks-vs-timberwolves.html\\\\\", \\\\\"content\\\\\": \\\\\"2024 NBA Playoffs Dallas Mavericks vs. Dallas Mavericks vs. Dallas Mavericks vs. 5 Dallas Mavericks \u001b[0m\u001b[32m(\u001b[0m\u001b[32m4-1\u001b[0m\u001b[32m)\u001b[0m\u001b[32m vs. 7 Derrick Jones Jr. 2024 NBA Playoffs Dallas Mavericks vs. Dallas Mavericks vs. Dallas Mavericks vs. College Tools: Player Season Finder, Player Game Finder, Team Season Finder, Team Game Finder Players, Teams, Seasons, Leaders, Awards ... Players, Teams, Seasons, Leaders, Awards ... Players, Teams, Seasons, Leaders, Awards, All-Star Games, Executives ... Players, Teams, Seasons, Leaders, Awards ... Subscribe to Stathead Basketball: Get your first month FREE The SPORTS REFERENCE, STATHEAD, IMMACULATE GRID, and IMMACULATE FOOTY trademarks are owned exclusively by Sports Reference LLC. Sports\\\\\\\\u00a0Reference\\\\\\\\u202f\\\\\\\\u00ae Baseball Football \u001b[0m\u001b[32m(\u001b[0m\u001b[32mcollege\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Basketball \u001b[0m\u001b[32m(\u001b[0m\u001b[32mcollege\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Hockey F\\\\\\\\u00fatbol Blog Stathead\\\\\\\\u202f\\\\\\\\u00ae Immaculate Grid\\\\\\\\u202f\\\\\\\\u00ae\\\\\", \\\\\"score\\\\\": 0.89030397, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"NBA Standings - 2024-25 season - ESPN\\\\\", \\\\\"url\\\\\": \\\\\"https://www.espn.com/nba/standings\\\\\", \\\\\"content\\\\\": \\\\\"NBA Standings - 2024-25 season - ESPN Skip to main contentSkip to navigation ESPN NFL NBA NCAAF NHL NCAAM NCAAW Soccer More Sports Watch Fantasy NBA Home Scores Schedule Standings Stats Teams Odds Where To Watch All-Star Game Fantasy More NBA Standings 2024-25 Standings Expanded Vs. Division NBA Cup LeagueConferenceDivision Eastern Conference | | | --- | | 1CLECleveland Cavaliers | | 2BOSBoston Celtics | | 3NYNew York Knicks | | 4INDIndiana Pacers | | 5MILMilwaukee Bucks | | 6DETDetroit Pistons | | 7MIAMiami Heat | | 8ORLOrlando Magic | | 9ATLAtlanta Hawks | | 10CHIChicago Bulls | | PHIPhiladelphia 76ers | | BKNBrooklyn Nets | | TORToronto Raptors | | CHACharlotte Hornets | | WSHWashington Wizards | | W | L | PCT | GB | HOME | AWAY | DIV | CONF | PPG | OPP PPG | DIFF | STRK | L10 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 42 | 10 | .808 | - | 24-4 | 18-6 | 9-1 | 28-7 | 122.4 | 112.1 | +10.3 | W2 | 6-4 | | 36 | 16 | .692 | 6 | 16-10 | 20-6 | 6-2 | 26-9 | 117.3 | 108.8 | +8.5 | L1 | 7-3 | | 34 | 17 | .667 | 7.5 | 18-9 | 16-8 | 9-1 | 23-10 | 117.9 | 111.4 | +6.5 | W2 | 8-2 | | 29 | 21 | .580 | 12 | 14-7 | 14-13 | 6-4 | 17-15 | 115.7 | 114.9 | +0.8 | W1 | 7-3 | | 27 | 23 | .540 | 14 | 16-8 | 10-15 | 6-5 | 22-16 | 114.2 | 112.6 | +1.6 | L1 | 4-6 | | 26 | 26 | .500 | 16 | 13-13 | 13-13 | 2-9 | 18-20 | 113.0 | 113.8 | -0.8 | W1 | 5-5 | | 25 | 25 | .500 | 16 | 12-10 | 12-15 | 5-3 | 14-15 | 110.5 | 110.6 | -0.1 | L1 | 5-5 | | 25 | 28 | .472 | 17.5 | 15-9 | 10-19 | 5-2 | 20-15 | 103.8 | 105.6 | -1.8 | L1 | 2-8 | | 24 | 28 | .462 | 18 | 12-12 | 12-15 | 4-2 | 17-13 | 116.1 | 119.0 | -2.9 | W1 | 2-8 | | 22 | 30 | .423 | 20 | 10-16 | 12-14 | 3-7 | 17-18 | 116.7 | 120.1 | -3.4 | L1 | 4-6 | | 20 | 31 | .392 | 21.5 | 10-16 | 10-15 | 3-4 | 14-17 | 109.1 | 112.9 | -3.8 | L2 | 5-5 | | 18 | 34 | .346 | 24 | 7-17 | 11-17 | 1-8 | 9-23 | 105.3 | 111.7 | -6.4 | W1 | 4-6 | | 16 | 36 | .308 | 26 | 12-16 | 4-20 | 3-7 | 10-23 | 111.2 | 116.9 | -5.7 | L3 | 6-4 | | 13 | 36 | .265 | 27.5 | 9-20 | 4-16 | 0-9 | 7-27 | 107.1 | 112.3 | -5.2 | W1 | 2-8 | | 9 | 42 | .176 | 32.5 | 5-20 | 4-21 | 5-3 | 7-21 | 107.8 | 121.5 | -13.7 | L1 | 3-7 | Western Conference | | | --- | | 1OKCOklahoma City Thunder | | 2MEMMemphis Grizzlies | | 3DENDenver Nuggets | | 4HOUHouston Rockets | | 5LALLos Angeles Lakers | | 6MINMinnesota Timberwolves | | 7LACLA Clippers | | 8DALDallas Mavericks | | 9PHXPhoenix Suns | | 10SACSacramento Kings | | GSGolden State Warriors | | SASan Antonio Spurs | | PORPortland Trail Blazers | | UTAHUtah Jazz | | NONew Orleans Pelicans | | W | L | PCT | GB | HOME | AWAY | DIV | CONF | PPG | OPP PPG | DIFF | STRK | L10 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 41 | 9 | .820 | - | 23-3 | 17-6 | 7-1 | 23-8 | 117.7 | 104.7 | +13.0 | W4 | 7-3 | | 35 | 16 | .686 | 6.5 | 21-5 | 14-11 | 8-4 | 19-12 | 123.8 | 115.4 | +8.4 | W4 | 9-1 | | 33 | 19 | .635 | 9 | 17-8 | 16-11 | 4-4 | 19-12 | 120.8 | 115.9 | +4.9 | W5 | 7-3 | | 32 | 20 | .615 | 10 | 15-8 | 17-11 | 9-3 | 19-12 | 113.3 | 109.1 | +4.2 | L6 | 4-6 | | 30 | 19 | .612 | 10.5 | 17-6 | 13-13 | 9-3 | 19-11 | 112.6 | 112.0 | +0.6 | W4 | 8-2 | | 29 | 23 | .558 | 13 | 14-12 | 15-11 | 4-3 | 21-14 | 111.7 | 108.2 | +3.5 | W2 | 7-3 | | 28 | 23 | .549 | 13.5 | 17-10 | 11-13 | 6-4 | 17-18 | 110.1 | 107.7 | +2.4 | L3 | 4-6 | | 28 | 25 | .528 | 14.5 | 15-10 | 13-15 | 6-4 | 20-17 | 115.5 | 113.3 | +2.2 | W2 | 5-5 | | 26 | 25 | .510 | 15.5 | 16-9 | 10-16 | 7-4 | 17-14 | 113.4 | 114.7 | -1.3 | W1 | 5-5 | | 25 | 26 | .490 | 16.5 | 13-13 | 12-13 | 4-6 | 16-17 | 116.1 | 115.4 | +0.7 | L2 | 4-6 | | 25 | 26 | .490 | 16.5 | 15-13 | 10-13 | 1-10 | 17-18 | 111.5 | 111.9 | -0.4 | L2 | 4-6 | | 22 | 27 | .449 | 18.5 | 13-12 | 8-14 | 2-7 | 16-18 | 112.8 | 114.3 | -1.5 | L1 | 3-7 | | 23 | 29 | .442 | 19 | 15-13 | 8-16 | 4-5 | 14-24 | 109.0 | 113.9 | -4.9 | W6 | 9-1 | | 12 | 38 | .240 | 29 | 5-18 | 7-20 | 1-7 | 4-29 | 111.9 | 118.9 | -7.0 | L1 | 2-8 | | 12 | 39 | .235 | 29.5 | 8-18 | 4-21 | 1-8 | 6-23 | 110.0 | 118.8 | -8.8 | L7 | 3-7 | Standings are updated with the completion of each game.Teams seeded 7-10 in each conference will compete in a play-in tournament at the end of the regular season. Glossary W:Wins L:Losses PCT:Winning Percentage GB:Games Back HOME:Home Record AWAY:Away Record DIV:Division Record CONF:Conference Record PPG:Points Per Game OPP PPG:Opponent Points Per Game DIFF:Average Point Differential STRK:Current Streak L10:Record last 10 games NBA News Anthony Davis leads Mavericks past Rockets 116-105 in Mavs debut but leaves with lower-body injury -------------------------------------------------------------------------------------------------- \\\\\\\\u2014 Anthony Davis had 26 points, 16 rebounds, seven assists and three blocks in his Mavericks debut but left the game late in the third quarter with a... * 38m Hawks request waivers on newly acquired Bones Hyland ---------------------------------------------------- The Atlanta Hawks requested waivers on guard Bones Hyland on Saturday, just two days after the guard was obtained from the Clippers in a deal at the NBA trade deadline. * 1h AD posts 26-point double-double in debut before suffering injury ---------------------------------------------------------------- Anthony Davis has a strong debut with the Mavs, dropping 26 points, 16 rebounds and 7 assists, before leaving with a lower-body injury. * 1h All NBA News Terms of Use Privacy Policy Your US State Privacy Rights Children\\'s Online Privacy Policy Interest-Based Ads About Nielsen Measurement Do Not Sell or Share My Personal Information Contact Us Disney Ad Sales Site Work for ESPN Corrections ESPN BET Sportsbook is owned and operated by PENN Entertainment, Inc. and its subsidiaries \u001b[0m\u001b[32m(\u001b[0m\u001b[32m\\'PENN\\'\u001b[0m\u001b[32m)\u001b[0m\u001b[32m.\\\\\", \\\\\"score\\\\\": 0.83549726, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"2024 Playoffs: West Finals | Timberwolves \u001b[0m\u001b[32m(\u001b[0m\u001b[32m3\u001b[0m\u001b[32m)\u001b[0m\u001b[32m vs. Mavericks \u001b[0m\u001b[32m(\u001b[0m\u001b[32m5\u001b[0m\u001b[32m)\u001b[0m\u001b[32m | NBA.com\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nba.com/playoffs/2024/west-final\\\\\", \\\\\"content\\\\\": \\\\\"Mavericks \u001b[0m\u001b[32m(\u001b[0m\u001b[32m5\u001b[0m\u001b[32m)\u001b[0m\u001b[32m | NBA.com 2024-25 NBA CrunchTime NBA TV Draft Kings DFS NBA Bet Home NBA Store NBA Game Worn NBA Photo Store NBA Experiences NBA G League NBA 2K League NBA Play NBA Bet ### Doncic, Irving carry Mavs to NBA Finals Luka Doncic and Kyrie Irving pour in 36 points apiece to guide Dallas to its 1st appearance in the NBA Finals since 2011. ### Luka: \\'This is special, coming from the West\\' Luka Doncic with Ernie, Charles, Kenny & Shaq about the Mavs being NBA Finals-bound, his Game 5 play and more. NBA Organization NBA ID NBA Official NBA Careers NBA Initiatives NBA Cares NBA Foundation NBA Communications NBA Transactions NBA Auctions NBA Photostore\\\\\", \\\\\"score\\\\\": 0.75312227, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"2024 NBA Playoffs | Official Bracket, Schedule and Series Matchups\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nba.com/playoffs/2024?\u001b[0m\u001b[32mos\u001b[0m\u001b[32m=\u001b[0m\u001b[32mwtmbloozowcj\u001b[0m\u001b[32m&\u001b[0m\u001b[32mref\u001b[0m\u001b[32m=\u001b[0m\u001b[32mapp\u001b[0m\u001b[32m\\\\\", \\\\\"content\\\\\": \\\\\"Draft Kings DFS NBA Store NBA Play NBA Finals ### Chasing History: Celtics clinch banner 18 \u001b[0m\u001b[32m(\u001b[0m\u001b[32mEp. 25\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Jayson Tatum and Finals MVP Jaylen Brown close out Dallas in Game 5 to secure Boston\\'s NBA-record 18th championship. WE DID ITTTT!\\' Jayson Tatum walkoff interview after Celtics defeat Mavericks in Game 5 of 2024 NBA Finals, clinching title with a 4-1 series win. ### Horford finally champ after key sacrifice Al Horford, who played the most playoff games in NBA history before winning his 1st title, crosses the plateau in his 17th season. 30:13 ### Best of the 2024 NBA Finals 17:47 ### Best of Boston Celtics from the 2024 NBA Finals\\\\\", \\\\\"score\\\\\": 0.63234437, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"2025 NBA Playoffs: Standings, bracket and clinching updates\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nba.com/news/2025-nba-playoffs-standings-and-bracket-updates\\\\\", \\\\\"content\\\\\": \\\\\"NBA TV NBA Play NBA Store NBA Game Worn NBA Play NBA Official NBA Playoffs bracket ### What to know about 2025 SoFi NBA Play-In Tournament The SoFi NBA Play-In Tournament features the Nos. 7-10 teams in each conference battling for the 7th and 8th playoff seeds. Click \\\\\\\\\\\\\"Access Content\\\\\\\\\\\\\" to agree to our Terms of Use and Privacy Policy and to sign up for emails about the latest news and products from the NBA Family and its partners. #### What to know about 2025 SoFi NBA Play-In Tournament The SoFi NBA Play-In Tournament features the Nos. 7-10 teams in each conference battling for the 7th and 8th playoff seeds. NBA ID NBA Official NBA Transactions NBA Auctions\\\\\", \\\\\"score\\\\\": 0.13435538, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m]\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# NBVAL_SKIP\n",
"print(f\"Getting traces for session_id={session_id}\")\n",
"import json\n",
"\n",
"from rich.pretty import pprint\n",
"\n",
"agent_logs = []\n",
"\n",
"for span in client.telemetry.query_spans(\n",
" attribute_filters=[\n",
" {\"key\": \"session_id\", \"op\": \"eq\", \"value\": session_id},\n",
" ],\n",
" attributes_to_return=[\"input\", \"output\"],\n",
"):\n",
" if span.attributes[\"output\"] != \"no shields\":\n",
" agent_logs.append(span.attributes)\n",
"\n",
"print(\"Here are examples of traces:\")\n",
"pprint(agent_logs[:2])\n"
]
},
{
"cell_type": "markdown",
"id": "QF30H7ufP2RE",
"metadata": {
"id": "QF30H7ufP2RE"
},
"source": [
"- Now, we want to run evaluation to assert that our search agent succesfully calls brave_search from online traces.\n",
"- We will first post-process the agent's telemetry logs and run evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "sy4Xaff_Avuu",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 432
},
"id": "sy4Xaff_Avuu",
"outputId": "1b14b5ed-4c77-47c4-edfb-1c13a88e5ef4"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringScoreResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">results</span>=<span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'basic::subset_of'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringResult</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">aggregated_results</span>=<span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_correct'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_total'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">score_rows</span>=<span style=\"font-weight: bold\">[{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}</span>, <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}</span>, <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}]</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">)</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mScoringScoreResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mresults\u001b[0m=\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'basic::subset_of'\u001b[0m: \u001b[1;35mScoringResult\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33maggregated_results\u001b[0m=\u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1;36m1.0\u001b[0m, \u001b[32m'num_correct'\u001b[0m: \u001b[1;36m3.0\u001b[0m, \u001b[32m'num_total'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33mscore_rows\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m, \u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m, \u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m\u001b[1m]\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m)\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# NBVAL_SKIP\n",
"# post-process telemetry spance and prepare data for eval\n",
"# in this case, we want to assert that all user prompts is followed by a tool call\n",
"import ast\n",
"import json\n",
"\n",
"eval_rows = []\n",
"\n",
"for log in agent_logs:\n",
" input = json.loads(log[\"input\"])\n",
" if isinstance(input, list):\n",
" input = input[-1]\n",
" if input[\"role\"] == \"user\":\n",
" eval_rows.append(\n",
" {\n",
" \"input_query\": input[\"content\"],\n",
" \"generated_answer\": log[\"output\"],\n",
" # check if generated_answer uses tools brave_search\n",
" \"expected_answer\": \"brave_search\",\n",
" },\n",
" )\n",
"\n",
"# pprint(eval_rows)\n",
"scoring_params = {\n",
" \"basic::subset_of\": None,\n",
"}\n",
"scoring_response = client.scoring.score(\n",
" input_rows=eval_rows, scoring_functions=scoring_params\n",
")\n",
"pprint(scoring_response)\n"
]
},
{
"cell_type": "markdown",
"id": "IKbzhxcw5e_c",
"metadata": {
"id": "IKbzhxcw5e_c"
},
"source": [
"#### 3.2. Agentic Application Dataset Scoring\n",
"- Llama Stack offers a library of scoring functions and the `/scoring` API, allowing you to run evaluations on your pre-annotated AI application datasets.\n",
"\n",
"- In this example, we will work with an example RAG dataset you have built previously, label with an annotation, and use LLM-As-Judge with custom judge prompt for scoring. Please checkout our [Llama Stack Playground](https://llama-stack.readthedocs.io/en/latest/playground/index.html) for an interactive interface to upload datasets and run scorings."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "xG4Y84VQBb0g",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 304
},
"id": "xG4Y84VQBb0g",
"outputId": "cf7dcecc-a81d-4c60-af5e-b36b8fe85c69"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringScoreResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">results</span>=<span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'llm-as-judge::base'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringResult</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">aggregated_results</span>=<span style=\"font-weight: bold\">{}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">score_rows</span>=<span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'B'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'judge_feedback'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Answer: B, Explanation: The GENERATED_RESPONSE is a superset of the EXPECTED_RESPONSE and is fully consistent with it. The EXPECTED_RESPONSE only mentions \"LoRA\", which is a topic that is extensively covered in the GENERATED_RESPONSE. The GENERATED_RESPONSE provides more specific and detailed topics related to LoRA, but it does not contradict the EXPECTED_RESPONSE.'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"font-weight: bold\">]</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">)</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'basic::subset_of'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringResult</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">aggregated_results</span>=<span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_correct'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_total'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">}}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">score_rows</span>=<span style=\"font-weight: bold\">[{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}]</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">)</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mScoringScoreResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mresults\u001b[0m=\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'llm-as-judge::base'\u001b[0m: \u001b[1;35mScoringResult\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33maggregated_results\u001b[0m=\u001b[1m{\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33mscore_rows\u001b[0m=\u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ │ │ │ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ │ │ │ \u001b[0m\u001b[32m'score'\u001b[0m: \u001b[32m'B'\u001b[0m,\n",
"\u001b[2;32m│ │ │ │ │ \u001b[0m\u001b[32m'judge_feedback'\u001b[0m: \u001b[32m'Answer: B, Explanation: The GENERATED_RESPONSE is a superset of the EXPECTED_RESPONSE and is fully consistent with it. The EXPECTED_RESPONSE only mentions \"LoRA\", which is a topic that is extensively covered in the GENERATED_RESPONSE. The GENERATED_RESPONSE provides more specific and detailed topics related to LoRA, but it does not contradict the EXPECTED_RESPONSE.'\u001b[0m\n",
"\u001b[2;32m│ │ │ │ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[1m]\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m)\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'basic::subset_of'\u001b[0m: \u001b[1;35mScoringResult\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33maggregated_results\u001b[0m=\u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1;36m1.0\u001b[0m, \u001b[32m'num_correct'\u001b[0m: \u001b[1;36m1.0\u001b[0m, \u001b[32m'num_total'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33mscore_rows\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m\u001b[1m]\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m)\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import rich\n",
"from rich.pretty import pprint\n",
"\n",
"# could even use larger models like 405B\n",
"judge_model_id = \"together/meta-llama/Llama-3.3-70B-Instruct-Turbo\"\n",
"\n",
"JUDGE_PROMPT = \"\"\"\n",
"Given a QUESTION and GENERATED_RESPONSE and EXPECTED_RESPONSE.\n",
"\n",
"Compare the factual content of the GENERATED_RESPONSE with the EXPECTED_RESPONSE. Ignore any differences in style, grammar, or punctuation.\n",
" The GENERATED_RESPONSE may either be a subset or superset of the EXPECTED_RESPONSE, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:\n",
" (A) The GENERATED_RESPONSE is a subset of the EXPECTED_RESPONSE and is fully consistent with it.\n",
" (B) The GENERATED_RESPONSE is a superset of the EXPECTED_RESPONSE and is fully consistent with it.\n",
" (C) The GENERATED_RESPONSE contains all the same details as the EXPECTED_RESPONSE.\n",
" (D) There is a disagreement between the GENERATED_RESPONSE and the EXPECTED_RESPONSE.\n",
" (E) The answers differ, but these differences don't matter from the perspective of factuality.\n",
"\n",
"Give your answer in the format \"Answer: One of ABCDE, Explanation: \".\n",
"\n",
"Your actual task:\n",
"\n",
"QUESTION: {input_query}\n",
"GENERATED_RESPONSE: {generated_answer}\n",
"EXPECTED_RESPONSE: {expected_answer}\n",
"\"\"\"\n",
"\n",
"input_query = (\n",
" \"What are the top 5 topics that were explained? Only list succinct bullet points.\"\n",
")\n",
"generated_answer = \"\"\"\n",
"Here are the top 5 topics that were explained in the documentation for Torchtune:\n",
"\n",
"* What is LoRA and how does it work?\n",
"* Fine-tuning with LoRA: memory savings and parameter-efficient finetuning\n",
"* Running a LoRA finetune with Torchtune: overview and recipe\n",
"* Experimenting with different LoRA configurations: rank, alpha, and attention modules\n",
"* LoRA finetuning\n",
"\"\"\"\n",
"expected_answer = \"\"\"LoRA\"\"\"\n",
"\n",
"rows = [\n",
" {\n",
" \"input_query\": input_query,\n",
" \"generated_answer\": generated_answer,\n",
" \"expected_answer\": expected_answer,\n",
" },\n",
"]\n",
"\n",
"scoring_params = {\n",
" \"llm-as-judge::base\": {\n",
" \"judge_model\": judge_model_id,\n",
" \"prompt_template\": JUDGE_PROMPT,\n",
" \"type\": \"llm_as_judge\",\n",
" \"judge_score_regexes\": [\"Answer: (A|B|C|D|E)\"],\n",
" },\n",
" \"basic::subset_of\": None,\n",
"}\n",
"\n",
"response = client.scoring.score(input_rows=rows, scoring_functions=scoring_params)\n",
"pprint(response)\n"
]
},
{
"cell_type": "markdown",
"id": "ad077440",
"metadata": {
"id": "ad077440"
},
"source": [
"## 4. Image Understanding with Llama 3.2\n",
"\n",
"Below is a complete example of to ask Llama 3.2 questions about an image."
]
},
{
"cell_type": "markdown",
"id": "82e381ec",
"metadata": {
"id": "82e381ec"
},
"source": [
"### 4.1 Setup and helpers\n"
]
},
{
"cell_type": "markdown",
"id": "7737cd41",
"metadata": {
"id": "7737cd41"
},
"source": [
"### 4.2 Using Llama Stack Inference API for multimodal inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d7914894",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "d7914894",
"outputId": "ce96f08a-bb2e-4c2a-a6ef-99cd6d539bac"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The image depicts three llamas standing at a table, with one wearing a party hat and another having a purple hue. The scene is set in a barn-like environment.\n",
"\n",
"* Three llamas are positioned at a table.\n",
" * The llama on the left is white.\n",
" * The middle llama is purple.\n",
" * The llama on the right is white and wears a blue party hat.\n",
"* A glass containing an orange liquid sits on the table.\n",
" * The glass is clear and filled with a yellowish-orange substance.\n",
"* The background features wooden walls.\n",
" * The walls are composed of vertical wooden planks.\n",
" * The overall atmosphere suggests a celebratory or festive setting.\n",
"\n",
"In summary, the image showcases three llamas gathered around a table, with one donning a party hat, amidst a rustic barn-like backdrop.\n"
]
}
],
"source": [
"vision_model_id = \"groq/meta-llama/llama-4-maverick-17b-128e-instruct\"\n",
"response = client.chat.completions.create(\n",
" model=vision_model_id,\n",
" messages=[{\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"text\", \"text\": \"What's in this image?\"},\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\n",
" \"url\": \"https://raw.githubusercontent.com/meta-llama/llama-models/refs/heads/main/Llama_Repo.jpeg\",\n",
" },\n",
" },\n",
" ],\n",
" }],\n",
")\n",
"\n",
"print(response.choices[0].message.content)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f3352379",
"metadata": {
"id": "f3352379"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.16"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in a new issue View git blame Copy permalink