phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-06-27 18:50:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
llama-stack-mirror/docs/notebooks/Llama_Stack_Building_AI_Applications.ipynb
Ashwin Bharambe 2f9fdb0ea7 Update notebook
2024-12-17 18:52:02 -08:00

4636 lines
308 KiB
Text
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"id": "c1e7571c",
"metadata": {
"id": "c1e7571c"
},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1F2ksmkoGQPa4pzRjMOE6BXWeOxWFIW6n?usp=sharing)\n",
"\n",
"# Llama Stack - Building AI Applications\n",
"\n",
"<img src=\"https://llama-stack.readthedocs.io/en/latest/_images/llama-stack.png\" alt=\"drawing\" width=\"500\"/>\n",
"\n",
"[Llama Stack](https://github.com/meta-llama/llama-stack) defines and standardizes the set of core building blocks needed to bring generative AI applications to market. These building blocks are presented in the form of interoperable APIs with a broad set of Service Providers providing their implementations.\n",
"\n",
"Read more about the project: https://llama-stack.readthedocs.io/en/latest/index.html\n",
"\n",
"In this guide, we will showcase how you can build LLM-powered agentic applications using Llama Stack.\n"
]
},
{
"cell_type": "markdown",
"id": "4CV1Q19BDMVw",
"metadata": {
"id": "4CV1Q19BDMVw"
},
"source": [
"## 1. Getting started with Llama Stack"
]
},
{
"cell_type": "markdown",
"id": "K4AvfUAJZOeS",
"metadata": {
"id": "K4AvfUAJZOeS"
},
"source": [
"### 1.1. Create TogetherAI account\n",
"\n",
"\n",
"In order to run inference for the llama models, you will need to use an inference provider. Llama stack supports a number of inference [providers](https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference).\n",
"\n",
"\n",
"In this showcase, we will use [together.ai](https://www.together.ai/) as the inference provider. So, you would first get an API key from Together if you dont have one already.\n",
"\n",
"Steps [here](https://docs.google.com/document/d/1Vg998IjRW_uujAPnHdQ9jQWvtmkZFt74FldW2MblxPY/edit?usp=sharing).\n",
"\n",
"You can also use Fireworks.ai or even Ollama if you would like to.\n",
"\n",
"\n",
"\n",
"> **Note:** Set the API Key in the Secrets of this notebook\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "oDUB7M_qe-Gs",
"metadata": {
"id": "oDUB7M_qe-Gs"
},
"source": [
"### 1.2. Install Llama Stack\n",
"\n",
"We will now start with installing the [llama-stack pypi package](https://pypi.org/project/llama-stack).\n",
"\n",
"In addition, we will install [bubblewrap](https://github.com/containers/bubblewrap), a low level light-weight container framework that runs in the user namespace. We will use it to execute code generated by Llama in one of the examples."
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "J2kGed0R5PSf",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "J2kGed0R5PSf",
"outputId": "7d543c6f-623d-4911-b9a7-4ed24d5b82f2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reading package lists... Done\n",
"Building dependency tree... Done\n",
"Reading state information... Done\n",
"bubblewrap is already the newest version (0.6.1-1ubuntu0.1).\n",
"0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.\n",
"Requirement already satisfied: llama-stack in /usr/local/lib/python3.10/dist-packages (0.0.61)\n",
"Requirement already satisfied: blobfile in /usr/local/lib/python3.10/dist-packages (from llama-stack) (3.0.0)\n",
"Requirement already satisfied: fire in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.7.0)\n",
"Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.28.1)\n",
"Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.26.5)\n",
"Requirement already satisfied: llama-models>=0.0.61 in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.0.61)\n",
"Requirement already satisfied: llama-stack-client>=0.0.61 in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.0.61)\n",
"Requirement already satisfied: prompt-toolkit in /usr/local/lib/python3.10/dist-packages (from llama-stack) (3.0.48)\n",
"Requirement already satisfied: python-dotenv in /usr/local/lib/python3.10/dist-packages (from llama-stack) (1.0.1)\n",
"Requirement already satisfied: pydantic>=2 in /usr/local/lib/python3.10/dist-packages (from llama-stack) (2.10.3)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from llama-stack) (2.32.3)\n",
"Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from llama-stack) (13.9.4)\n",
"Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from llama-stack) (75.1.0)\n",
"Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from llama-stack) (2.5.0)\n",
"Requirement already satisfied: PyYAML in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (6.0.2)\n",
"Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (3.1.4)\n",
"Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (0.8.0)\n",
"Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (10.4.0)\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (3.7.1)\n",
"Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (8.1.7)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (1.9.0)\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (2.2.2)\n",
"Requirement already satisfied: pyaml in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (24.12.1)\n",
"Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (1.3.1)\n",
"Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (4.66.6)\n",
"Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (4.12.2)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->llama-stack) (2024.8.30)\n",
"Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->llama-stack) (1.0.7)\n",
"Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->llama-stack) (3.10)\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx->llama-stack) (0.14.0)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2->llama-stack) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2->llama-stack) (2.27.1)\n",
"Requirement already satisfied: pycryptodomex>=3.8 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (3.21.0)\n",
"Requirement already satisfied: urllib3<3,>=1.25.3 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (2.2.3)\n",
"Requirement already satisfied: lxml>=4.9 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (5.3.0)\n",
"Requirement already satisfied: filelock>=3.0 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (3.16.1)\n",
"Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->llama-stack) (2024.9.0)\n",
"Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->llama-stack) (24.2)\n",
"Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit->llama-stack) (0.2.13)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->llama-stack) (3.4.0)\n",
"Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->llama-stack) (3.0.0)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->llama-stack) (2.18.0)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->llama-stack-client>=0.0.61->llama-stack) (1.2.2)\n",
"Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->llama-stack) (0.1.2)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->llama-models>=0.0.61->llama-stack) (3.0.2)\n",
"Requirement already satisfied: numpy>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (1.26.4)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (2024.2)\n",
"Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (2024.2)\n",
"Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->llama-models>=0.0.61->llama-stack) (2024.9.11)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->llama-stack-client>=0.0.61->llama-stack) (1.17.0)\n"
]
}
],
"source": [
"!apt-get install -y bubblewrap\n",
"!pip install -U llama-stack"
]
},
{
"cell_type": "markdown",
"id": "414301dc",
"metadata": {
"id": "414301dc"
},
"source": [
"### 1.3. Configure Llama Stack for Together\n",
"\n",
"\n",
"Llama Stack is architected as a collection of lego blocks which can be assembled as needed.\n",
"\n",
"\n",
"Typically, llama stack is available as a server with an endpoint that you can hit. We call this endpoint a [Distribution](https://llama-stack.readthedocs.io/en/latest/concepts/index.html#distributions). Partners like Together and Fireworks offer their own Llama Stack Distribution endpoints.\n",
"\n",
"In this showcase, we are going to use llama stack inline as a library. So, given a particular set of providers, we must first package up the right set of dependencies. We have a template to use Together as an inference provider and [faiss](https://ai.meta.com/tools/faiss/) for memory/RAG.\n",
"\n",
"We will run `llama stack build` to deploy all dependencies."
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "HaepEZXCDgif",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "HaepEZXCDgif",
"outputId": "9c268d26-7444-4741-f14d-3911eea8e4eb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: llama-stack in /usr/local/lib/python3.10/dist-packages (0.0.61)\r\n",
"Requirement already satisfied: blobfile in /usr/local/lib/python3.10/dist-packages (from llama-stack) (3.0.0)\r\n",
"Requirement already satisfied: fire in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.7.0)\r\n",
"Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.28.1)\r\n",
"Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.26.5)\r\n",
"Requirement already satisfied: llama-models>=0.0.61 in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.0.61)\r\n",
"Requirement already satisfied: llama-stack-client>=0.0.61 in /usr/local/lib/python3.10/dist-packages (from llama-stack) (0.0.61)\r\n",
"Requirement already satisfied: prompt-toolkit in /usr/local/lib/python3.10/dist-packages (from llama-stack) (3.0.48)\r\n",
"Requirement already satisfied: python-dotenv in /usr/local/lib/python3.10/dist-packages (from llama-stack) (1.0.1)\r\n",
"Requirement already satisfied: pydantic>=2 in /usr/local/lib/python3.10/dist-packages (from llama-stack) (2.10.3)\r\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from llama-stack) (2.32.3)\r\n",
"Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from llama-stack) (13.9.4)\r\n",
"Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from llama-stack) (75.1.0)\r\n",
"Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from llama-stack) (2.5.0)\r\n",
"Requirement already satisfied: PyYAML in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (6.0.2)\r\n",
"Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (3.1.4)\r\n",
"Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (0.8.0)\r\n",
"Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama-stack) (10.4.0)\r\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (3.7.1)\r\n",
"Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (8.1.7)\r\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (1.9.0)\r\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (2.2.2)\r\n",
"Requirement already satisfied: pyaml in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (24.12.1)\r\n",
"Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (1.3.1)\r\n",
"Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (4.66.6)\r\n",
"Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama-stack) (4.12.2)\r\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->llama-stack) (2024.8.30)\r\n",
"Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->llama-stack) (1.0.7)\r\n",
"Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->llama-stack) (3.10)\r\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx->llama-stack) (0.14.0)\r\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2->llama-stack) (0.7.0)\r\n",
"Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2->llama-stack) (2.27.1)\r\n",
"Requirement already satisfied: pycryptodomex>=3.8 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (3.21.0)\r\n",
"Requirement already satisfied: urllib3<3,>=1.25.3 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (2.2.3)\r\n",
"Requirement already satisfied: lxml>=4.9 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (5.3.0)\r\n",
"Requirement already satisfied: filelock>=3.0 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama-stack) (3.16.1)\n",
"Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->llama-stack) (2024.9.0)\n",
"Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->llama-stack) (24.2)\n",
"Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit->llama-stack) (0.2.13)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->llama-stack) (3.4.0)\n",
"Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->llama-stack) (3.0.0)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->llama-stack) (2.18.0)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->llama-stack-client>=0.0.61->llama-stack) (1.2.2)\n",
"Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->llama-stack) (0.1.2)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->llama-models>=0.0.61->llama-stack) (3.0.2)\n",
"Requirement already satisfied: numpy>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (1.26.4)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (2024.2)\n",
"Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama-stack) (2024.2)\n",
"Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->llama-models>=0.0.61->llama-stack) (2024.9.11)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->llama-stack-client>=0.0.61->llama-stack) (1.17.0)\n",
"Installing pip dependencies\n",
"Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (10.4.0)\n",
"Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.46.3)\n",
"Requirement already satisfied: psycopg2-binary in /usr/local/lib/python3.10/dist-packages (2.9.10)\n",
"Requirement already satisfied: aiosqlite in /usr/local/lib/python3.10/dist-packages (0.20.0)\n",
"Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (4.66.6)\n",
"Requirement already satisfied: pypdf in /usr/local/lib/python3.10/dist-packages (5.1.0)\n",
"Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.26.4)\n",
"Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.5.2)\n",
"Requirement already satisfied: redis in /usr/local/lib/python3.10/dist-packages (5.2.1)\n",
"Requirement already satisfied: opentelemetry-sdk in /usr/local/lib/python3.10/dist-packages (1.28.2)\n",
"Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (0.2.0)\n",
"Requirement already satisfied: blobfile in /usr/local/lib/python3.10/dist-packages (3.0.0)\n",
"Requirement already satisfied: together in /usr/local/lib/python3.10/dist-packages (1.3.5)\n",
"Requirement already satisfied: openai in /usr/local/lib/python3.10/dist-packages (1.54.5)\n",
"Requirement already satisfied: faiss-cpu in /usr/local/lib/python3.10/dist-packages (1.9.0.post1)\n",
"Requirement already satisfied: autoevals in /usr/local/lib/python3.10/dist-packages (0.0.110)\n",
"Requirement already satisfied: chardet in /usr/local/lib/python3.10/dist-packages (5.2.0)\n",
"Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.9.1)\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (2.2.2)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-http in /usr/local/lib/python3.10/dist-packages (1.28.2)\n",
"Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (3.2.0)\n",
"Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.8.0)\n",
"Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (1.13.1)\n",
"Requirement already satisfied: chromadb-client in /usr/local/lib/python3.10/dist-packages (0.5.23)\n",
"Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (0.115.6)\n",
"Requirement already satisfied: fire in /usr/local/lib/python3.10/dist-packages (0.7.0)\n",
"Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (0.28.1)\n",
"Requirement already satisfied: uvicorn in /usr/local/lib/python3.10/dist-packages (0.32.1)\n",
"Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.16.1)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.26.5)\n",
"Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.2)\n",
"Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.2)\n",
"Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.9.11)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3)\n",
"Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.20.3)\n",
"Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.5)\n",
"Requirement already satisfied: typing_extensions>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiosqlite) (4.12.2)\n",
"Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.4.2)\n",
"Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.5.0)\n",
"Requirement already satisfied: async-timeout>=4.0.3 in /usr/local/lib/python3.10/dist-packages (from redis) (4.0.3)\n",
"Requirement already satisfied: opentelemetry-api==1.28.2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-sdk) (1.28.2)\n",
"Requirement already satisfied: opentelemetry-semantic-conventions==0.49b2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-sdk) (0.49b2)\n",
"Requirement already satisfied: deprecated>=1.2.6 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api==1.28.2->opentelemetry-sdk) (1.2.15)\n",
"Requirement already satisfied: importlib-metadata<=8.5.0,>=6.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api==1.28.2->opentelemetry-sdk) (8.5.0)\n",
"Requirement already satisfied: pycryptodomex>=3.8 in /usr/local/lib/python3.10/dist-packages (from blobfile) (3.21.0)\n",
"Requirement already satisfied: urllib3<3,>=1.25.3 in /usr/local/lib/python3.10/dist-packages (from blobfile) (2.2.3)\n",
"Requirement already satisfied: lxml>=4.9 in /usr/local/lib/python3.10/dist-packages (from blobfile) (5.3.0)\n",
"Requirement already satisfied: aiohttp<4.0.0,>=3.9.3 in /usr/local/lib/python3.10/dist-packages (from together) (3.11.10)\n",
"Requirement already satisfied: click<9.0.0,>=8.1.7 in /usr/local/lib/python3.10/dist-packages (from together) (8.1.7)\n",
"Requirement already satisfied: eval-type-backport<0.3.0,>=0.1.3 in /usr/local/lib/python3.10/dist-packages (from together) (0.2.0)\n",
"Requirement already satisfied: pyarrow>=10.0.1 in /usr/local/lib/python3.10/dist-packages (from together) (17.0.0)\n",
"Requirement already satisfied: pydantic<3.0.0,>=2.6.3 in /usr/local/lib/python3.10/dist-packages (from together) (2.10.3)\n",
"Requirement already satisfied: rich<14.0.0,>=13.8.1 in /usr/local/lib/python3.10/dist-packages (from together) (13.9.4)\n",
"Requirement already satisfied: tabulate<0.10.0,>=0.9.0 in /usr/local/lib/python3.10/dist-packages (from together) (0.9.0)\n",
"Requirement already satisfied: typer<0.14,>=0.9 in /usr/local/lib/python3.10/dist-packages (from together) (0.13.1)\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from openai) (1.9.0)\n",
"Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from openai) (0.8.2)\n",
"Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai) (1.3.1)\n",
"Requirement already satisfied: chevron in /usr/local/lib/python3.10/dist-packages (from autoevals) (0.14.0)\n",
"Requirement already satisfied: levenshtein in /usr/local/lib/python3.10/dist-packages (from autoevals) (0.26.1)\n",
"Requirement already satisfied: braintrust_core==0.0.54 in /usr/local/lib/python3.10/dist-packages (from autoevals) (0.0.54)\n",
"Requirement already satisfied: jsonschema in /usr/local/lib/python3.10/dist-packages (from autoevals) (4.23.0)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2024.2)\n",
"Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas) (2024.2)\n",
"Requirement already satisfied: googleapis-common-protos~=1.52 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-http) (1.66.0)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.28.2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-http) (1.28.2)\n",
"Requirement already satisfied: opentelemetry-proto==1.28.2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-http) (1.28.2)\n",
"Requirement already satisfied: protobuf<6.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-proto==1.28.2->opentelemetry-exporter-otlp-proto-http) (5.29.1)\n",
"Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.8)\n",
"Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.5.0)\n",
"Requirement already satisfied: multiprocess<0.70.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.16)\n",
"Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.9.0)\n",
"Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.3.1)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.55.3)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.7)\n",
"Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.2.0)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from chromadb-client) (1.28.2)\n",
"Requirement already satisfied: overrides>=7.3.1 in /usr/local/lib/python3.10/dist-packages (from chromadb-client) (7.7.0)\n",
"Requirement already satisfied: posthog>=2.4.0 in /usr/local/lib/python3.10/dist-packages (from chromadb-client) (3.7.4)\n",
"Requirement already satisfied: tenacity>=8.2.3 in /usr/local/lib/python3.10/dist-packages (from chromadb-client) (9.0.0)\n",
"Requirement already satisfied: orjson>=3.9.12 in /usr/local/lib/python3.10/dist-packages (from chromadb-client) (3.10.12)\n",
"Requirement already satisfied: starlette<0.42.0,>=0.40.0 in /usr/local/lib/python3.10/dist-packages (from fastapi) (0.41.3)\n",
"Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from fire) (2.5.0)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx) (2024.8.30)\n",
"Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx) (1.0.7)\n",
"Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx) (3.10)\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx) (0.14.0)\n",
"Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (2.4.4)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (1.3.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (24.2.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (1.5.0)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (6.1.0)\n",
"Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (0.2.1)\n",
"Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (1.18.3)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (1.2.2)\n",
"Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated>=1.2.6->opentelemetry-api==1.28.2->opentelemetry-sdk) (1.17.0)\n",
"Requirement already satisfied: grpcio<2.0.0,>=1.63.2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb-client) (1.68.1)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from posthog>=2.4.0->chromadb-client) (1.17.0)\n",
"Requirement already satisfied: monotonic>=1.5 in /usr/local/lib/python3.10/dist-packages (from posthog>=2.4.0->chromadb-client) (1.6)\n",
"Requirement already satisfied: backoff>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from posthog>=2.4.0->chromadb-client) (2.2.1)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.6.3->together) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.6.3->together) (2.27.1)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4.0)\n",
"Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich<14.0.0,>=13.8.1->together) (3.0.0)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich<14.0.0,>=13.8.1->together) (2.18.0)\n",
"Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<0.14,>=0.9->together) (1.5.4)\n",
"Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema->autoevals) (2024.10.1)\n",
"Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema->autoevals) (0.35.1)\n",
"Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema->autoevals) (0.22.3)\n",
"Requirement already satisfied: rapidfuzz<4.0.0,>=3.9.0 in /usr/local/lib/python3.10/dist-packages (from levenshtein->autoevals) (3.10.1)\n",
"Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata<=8.5.0,>=6.0->opentelemetry-api==1.28.2->opentelemetry-sdk) (3.21.0)\n",
"Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich<14.0.0,>=13.8.1->together) (0.1.2)\n",
"sentence-transformers --no-deps\n",
"Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.10/dist-packages (3.2.1)\n",
"torch --index-url https://download.pytorch.org/whl/cpu\n",
"Looking in indexes: https://download.pytorch.org/whl/cpu\n",
"Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.5.1+cu121)\n",
"Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.16.1)\n",
"Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.2)\n",
"Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.4.2)\n",
"Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n",
"Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2024.9.0)\n",
"Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch) (1.13.1)\n",
"Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch) (1.3.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (3.0.2)\n",
"\u001b[32mBuild Successful!\u001b[0m\n"
]
}
],
"source": [
"# This will build all the dependencies you will need\n",
"!llama stack build --template together --image-type venv"
]
},
{
"cell_type": "markdown",
"id": "25b97dfe",
"metadata": {
"id": "25b97dfe"
},
"source": [
"### 1.4. Initialize Llama Stack\n",
"\n",
"Now that all dependencies have been installed, we can initialize llama stack. We will first set the `TOGETHER_API_KEY` environment variable\n"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "E1UFuJC570Tk",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"collapsed": true,
"id": "E1UFuJC570Tk",
"outputId": "bac7c9ec-ad49-4040-af43-8869f0afe5ac"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:llama_stack.distribution.resolver:Resolved 24 providers\n",
"INFO:llama_stack.distribution.resolver: inner-inference => together\n",
"INFO:llama_stack.distribution.resolver: inner-memory => faiss\n",
"INFO:llama_stack.distribution.resolver: models => __routing_table__\n",
"INFO:llama_stack.distribution.resolver: inference => __autorouted__\n",
"INFO:llama_stack.distribution.resolver: inner-safety => llama-guard\n",
"INFO:llama_stack.distribution.resolver: shields => __routing_table__\n",
"INFO:llama_stack.distribution.resolver: safety => __autorouted__\n",
"INFO:llama_stack.distribution.resolver: memory_banks => __routing_table__\n",
"INFO:llama_stack.distribution.resolver: memory => __autorouted__\n",
"INFO:llama_stack.distribution.resolver: agents => meta-reference\n",
"INFO:llama_stack.distribution.resolver: inner-datasetio => huggingface\n",
"INFO:llama_stack.distribution.resolver: inner-datasetio => localfs\n",
"INFO:llama_stack.distribution.resolver: datasets => __routing_table__\n",
"INFO:llama_stack.distribution.resolver: datasetio => __autorouted__\n",
"INFO:llama_stack.distribution.resolver: telemetry => meta-reference\n",
"INFO:llama_stack.distribution.resolver: inner-scoring => basic\n",
"INFO:llama_stack.distribution.resolver: inner-scoring => llm-as-judge\n",
"INFO:llama_stack.distribution.resolver: inner-scoring => braintrust\n",
"INFO:llama_stack.distribution.resolver: scoring_functions => __routing_table__\n",
"INFO:llama_stack.distribution.resolver: scoring => __autorouted__\n",
"INFO:llama_stack.distribution.resolver: inner-eval => meta-reference\n",
"INFO:llama_stack.distribution.resolver: eval_tasks => __routing_table__\n",
"INFO:llama_stack.distribution.resolver: eval => __autorouted__\n",
"INFO:llama_stack.distribution.resolver: inspect => __builtin__\n",
"INFO:llama_stack.distribution.resolver:\n",
"WARNING:opentelemetry.trace:Overriding of current TracerProvider is not allowed\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-3.1-405B-Instruct-FP8 served by together\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-3.1-70B-Instruct served by together\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-3.1-8B-Instruct served by together\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-3.2-11B-Vision-Instruct served by together\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-3.2-3B-Instruct served by together\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-3.2-90B-Vision-Instruct served by together\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-Guard-3-11B-Vision served by together\n",
"INFO:llama_stack.distribution.stack:Models: meta-llama/Llama-Guard-3-8B served by together\n",
"INFO:llama_stack.distribution.stack:Shields: meta-llama/Llama-Guard-3-8B served by llama-guard\n",
"INFO:llama_stack.distribution.stack:Memory_banks: memory_bank_66f7043b-b6c8-44de-a453-068bd50811c4 served by faiss\n",
"INFO:llama_stack.distribution.stack:Memory_banks: memory_bank_edf0d763-95bc-40d3-93a7-95b517162cfb served by faiss\n",
"INFO:llama_stack.distribution.stack:Scoring_fns: basic::equality served by basic\n",
"INFO:llama_stack.distribution.stack:Scoring_fns: basic::regex_parser_multiple_choice_answer served by basic\n",
"INFO:llama_stack.distribution.stack:Scoring_fns: basic::subset_of served by basic\n",
"INFO:llama_stack.distribution.stack:Scoring_fns: braintrust::answer-correctness served by braintrust\n",
"INFO:llama_stack.distribution.stack:Scoring_fns: braintrust::factuality served by braintrust\n",
"INFO:llama_stack.distribution.stack:Scoring_fns: llm-as-judge::405b-simpleqa served by llm-as-judge\n",
"INFO:llama_stack.distribution.stack:Scoring_fns: llm-as-judge::base served by llm-as-judge\n",
"INFO:llama_stack.distribution.stack:\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Using config <span style=\"color: #000080; text-decoration-color: #000080\">together</span>:\n",
"</pre>\n"
],
"text/plain": [
"Using config \u001b[34mtogether\u001b[0m:\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">apis:\n",
"- agents\n",
"- datasetio\n",
"- eval\n",
"- inference\n",
"- memory\n",
"- safety\n",
"- scoring\n",
"- telemetry\n",
"conda_env: together\n",
"datasets: <span style=\"font-weight: bold\">[]</span>\n",
"docker_image: null\n",
"eval_tasks: <span style=\"font-weight: bold\">[]</span>\n",
"image_name: together\n",
"memory_banks: <span style=\"font-weight: bold\">[]</span>\n",
"metadata_store:\n",
" db_path: <span style=\"color: #800080; text-decoration-color: #800080\">/root/.llama/distributions/together/</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">registry.db</span>\n",
" namespace: null\n",
" type: sqlite\n",
"models:\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1</span>-8B-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1</span>-8B-Instruct-Turbo\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1</span>-70B-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1</span>-70B-Instruct-Turbo\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1</span>-405B-Instruct-FP8\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1</span>-405B-Instruct-Turbo\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2</span>-3B-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2</span>-3B-Instruct-Turbo\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2</span>-11B-Vision-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2</span>-11B-Vision-Instruct-Turbo\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2</span>-90B-Vision-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2</span>-90B-Vision-Instruct-Turbo\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-Guard-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-8B\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-Guard-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-8B\n",
"- metadata: <span style=\"font-weight: bold\">{}</span>\n",
" model_id: meta-llama/Llama-Guard-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-11B-Vision\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-Guard-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-11B-Vision-Turbo\n",
"providers:\n",
" agents:\n",
" - config:\n",
" persistence_store:\n",
" db_path: <span style=\"color: #800080; text-decoration-color: #800080\">/root/.llama/distributions/together/</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">agents_store.db</span>\n",
" namespace: null\n",
" type: sqlite\n",
" provider_id: meta-reference\n",
" provider_type: inline::meta-reference\n",
" datasetio:\n",
" - config: <span style=\"font-weight: bold\">{}</span>\n",
" provider_id: huggingface\n",
" provider_type: remote::huggingface\n",
" - config: <span style=\"font-weight: bold\">{}</span>\n",
" provider_id: localfs\n",
" provider_type: inline::localfs\n",
" eval:\n",
" - config: <span style=\"font-weight: bold\">{}</span>\n",
" provider_id: meta-reference\n",
" provider_type: inline::meta-reference\n",
" inference:\n",
" - config:\n",
" api_key: 4985b03e627419b2964d34b8519ac6c4319f094d1ffb4f45514b4eb87e5427a2\n",
" url: <span style=\"color: #0000ff; text-decoration-color: #0000ff; text-decoration: underline\">https://api.together.xyz/v1</span>\n",
" provider_id: together\n",
" provider_type: remote::together\n",
" memory:\n",
" - config:\n",
" kvstore:\n",
" db_path: <span style=\"color: #800080; text-decoration-color: #800080\">/root/.llama/distributions/together/</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">faiss_store.db</span>\n",
" namespace: null\n",
" type: sqlite\n",
" provider_id: faiss\n",
" provider_type: inlin<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">e::fa</span>iss\n",
" safety:\n",
" - config: <span style=\"font-weight: bold\">{}</span>\n",
" provider_id: llama-guard\n",
" provider_type: inline::llama-guard\n",
" scoring:\n",
" - config: <span style=\"font-weight: bold\">{}</span>\n",
" provider_id: basic\n",
" provider_type: inlin<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">e::ba</span>sic\n",
" - config: <span style=\"font-weight: bold\">{}</span>\n",
" provider_id: llm-as-judge\n",
" provider_type: inline::llm-as-judge\n",
" - config:\n",
" openai_api_key: <span style=\"color: #008000; text-decoration-color: #008000\">''</span>\n",
" provider_id: braintrust\n",
" provider_type: inlin<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">e::b</span>raintrust\n",
" telemetry:\n",
" - config:\n",
" service_name: llama-stack\n",
" sinks: sqlite\n",
" sqlite_db_path: <span style=\"color: #800080; text-decoration-color: #800080\">/root/.llama/distributions/together/</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">trace_store.db</span>\n",
" provider_id: meta-reference\n",
" provider_type: inline::meta-reference\n",
"scoring_fns: <span style=\"font-weight: bold\">[]</span>\n",
"shields:\n",
"- params: null\n",
" provider_id: null\n",
" provider_shield_id: null\n",
" shield_id: meta-llama/Llama-Guard-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-8B\n",
"version: <span style=\"color: #008000; text-decoration-color: #008000\">'2'</span>\n",
"\n",
"</pre>\n"
],
"text/plain": [
"apis:\n",
"- agents\n",
"- datasetio\n",
"- eval\n",
"- inference\n",
"- memory\n",
"- safety\n",
"- scoring\n",
"- telemetry\n",
"conda_env: together\n",
"datasets: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
"docker_image: null\n",
"eval_tasks: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
"image_name: together\n",
"memory_banks: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
"metadata_store:\n",
" db_path: \u001b[35m/root/.llama/distributions/together/\u001b[0m\u001b[95mregistry.db\u001b[0m\n",
" namespace: null\n",
" type: sqlite\n",
"models:\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-\u001b[1;36m3.1\u001b[0m-8B-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-\u001b[1;36m3.1\u001b[0m-8B-Instruct-Turbo\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-\u001b[1;36m3.1\u001b[0m-70B-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-\u001b[1;36m3.1\u001b[0m-70B-Instruct-Turbo\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-\u001b[1;36m3.1\u001b[0m-405B-Instruct-FP8\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-\u001b[1;36m3.1\u001b[0m-405B-Instruct-Turbo\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-\u001b[1;36m3.2\u001b[0m-3B-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-\u001b[1;36m3.2\u001b[0m-3B-Instruct-Turbo\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-\u001b[1;36m3.2\u001b[0m-11B-Vision-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-\u001b[1;36m3.2\u001b[0m-11B-Vision-Instruct-Turbo\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-\u001b[1;36m3.2\u001b[0m-90B-Vision-Instruct\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-\u001b[1;36m3.2\u001b[0m-90B-Vision-Instruct-Turbo\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-Guard-\u001b[1;36m3\u001b[0m-8B\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Meta-Llama-Guard-\u001b[1;36m3\u001b[0m-8B\n",
"- metadata: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" model_id: meta-llama/Llama-Guard-\u001b[1;36m3\u001b[0m-11B-Vision\n",
" provider_id: null\n",
" provider_model_id: meta-llama/Llama-Guard-\u001b[1;36m3\u001b[0m-11B-Vision-Turbo\n",
"providers:\n",
" agents:\n",
" - config:\n",
" persistence_store:\n",
" db_path: \u001b[35m/root/.llama/distributions/together/\u001b[0m\u001b[95magents_store.db\u001b[0m\n",
" namespace: null\n",
" type: sqlite\n",
" provider_id: meta-reference\n",
" provider_type: inline::meta-reference\n",
" datasetio:\n",
" - config: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" provider_id: huggingface\n",
" provider_type: remote::huggingface\n",
" - config: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" provider_id: localfs\n",
" provider_type: inline::localfs\n",
" eval:\n",
" - config: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" provider_id: meta-reference\n",
" provider_type: inline::meta-reference\n",
" inference:\n",
" - config:\n",
" api_key: 4985b03e627419b2964d34b8519ac6c4319f094d1ffb4f45514b4eb87e5427a2\n",
" url: \u001b[4;94mhttps://api.together.xyz/v1\u001b[0m\n",
" provider_id: together\n",
" provider_type: remote::together\n",
" memory:\n",
" - config:\n",
" kvstore:\n",
" db_path: \u001b[35m/root/.llama/distributions/together/\u001b[0m\u001b[95mfaiss_store.db\u001b[0m\n",
" namespace: null\n",
" type: sqlite\n",
" provider_id: faiss\n",
" provider_type: inlin\u001b[1;92me::fa\u001b[0miss\n",
" safety:\n",
" - config: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" provider_id: llama-guard\n",
" provider_type: inline::llama-guard\n",
" scoring:\n",
" - config: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" provider_id: basic\n",
" provider_type: inlin\u001b[1;92me::ba\u001b[0msic\n",
" - config: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
" provider_id: llm-as-judge\n",
" provider_type: inline::llm-as-judge\n",
" - config:\n",
" openai_api_key: \u001b[32m''\u001b[0m\n",
" provider_id: braintrust\n",
" provider_type: inlin\u001b[1;92me::b\u001b[0mraintrust\n",
" telemetry:\n",
" - config:\n",
" service_name: llama-stack\n",
" sinks: sqlite\n",
" sqlite_db_path: \u001b[35m/root/.llama/distributions/together/\u001b[0m\u001b[95mtrace_store.db\u001b[0m\n",
" provider_id: meta-reference\n",
" provider_type: inline::meta-reference\n",
"scoring_fns: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
"shields:\n",
"- params: null\n",
" provider_id: null\n",
" provider_shield_id: null\n",
" shield_id: meta-llama/Llama-Guard-\u001b[1;36m3\u001b[0m-8B\n",
"version: \u001b[32m'2'\u001b[0m\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import os\n",
"from google.colab import userdata\n",
"\n",
"os.environ['TOGETHER_API_KEY'] = userdata.get('TOGETHER_API_KEY')\n",
"\n",
"from llama_stack.distribution.library_client import LlamaStackAsLibraryClient\n",
"client = LlamaStackAsLibraryClient(\"together\")\n",
"_ = client.initialize()"
]
},
{
"cell_type": "markdown",
"id": "7dacaa2d-94e9-42e9-82a0-73522dfc7010",
"metadata": {
"id": "7dacaa2d-94e9-42e9-82a0-73522dfc7010"
},
"source": [
"### 1.5. Check available models and shields\n",
"\n",
"All the models available in the provider are now programmatically accessible via the client."
]
},
{
"cell_type": "code",
"execution_count": 52,
"id": "ruO9jQna_t_S",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "ruO9jQna_t_S",
"outputId": "ee73b87a-10bf-4837-c77d-e619352d7321"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Available models:\n",
"meta-llama/Llama-3.1-405B-Instruct-FP8 (provider's alias: meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo) \n",
"meta-llama/Llama-3.1-70B-Instruct (provider's alias: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo) \n",
"meta-llama/Llama-3.1-8B-Instruct (provider's alias: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo) \n",
"meta-llama/Llama-3.2-11B-Vision-Instruct (provider's alias: meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo) \n",
"meta-llama/Llama-3.2-3B-Instruct (provider's alias: meta-llama/Llama-3.2-3B-Instruct-Turbo) \n",
"meta-llama/Llama-3.2-90B-Vision-Instruct (provider's alias: meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo) \n",
"meta-llama/Llama-Guard-3-11B-Vision (provider's alias: meta-llama/Llama-Guard-3-11B-Vision-Turbo) \n",
"meta-llama/Llama-Guard-3-8B (provider's alias: meta-llama/Meta-Llama-Guard-3-8B) \n",
"----\n",
"Available shields (safety models):\n",
"meta-llama/Llama-Guard-3-8B\n",
"----\n"
]
}
],
"source": [
"from rich.pretty import pprint\n",
"print(\"Available models:\")\n",
"for m in client.models.list():\n",
" print(f\"{m.identifier} (provider's alias: {m.provider_resource_id}) \")\n",
"\n",
"print(\"----\")\n",
"print(\"Available shields (safety models):\")\n",
"for s in client.shields.list():\n",
" print(s.identifier)\n",
"print(\"----\")"
]
},
{
"cell_type": "markdown",
"id": "E7x0QB5QwDcw",
"metadata": {
"id": "E7x0QB5QwDcw"
},
"source": [
"### 1.6. Pick the model\n",
"\n",
"We will use Llama3.1-70B-Instruct for our examples."
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "LINBvv8lwTJh",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"id": "LINBvv8lwTJh",
"outputId": "36ff2845-26ad-4f1d-9d8a-a83cfdbc8dba"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'meta-llama/Llama-3.1-70B-Instruct'"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model_id = \"meta-llama/Llama-3.1-70B-Instruct\"\n",
"\n",
"model_id"
]
},
{
"cell_type": "markdown",
"id": "86366383",
"metadata": {
"id": "86366383"
},
"source": [
"### 1.7. Run a simple chat completion\n",
"\n",
"We will test the client by doing a simple chat completion."
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "77c29dba",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "77c29dba",
"outputId": "cf4e9ef4-828a-4137-84c3-67515b420464"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"With gentle eyes and a gentle pace,\n",
"The llama roams, a peaceful face.\n"
]
}
],
"source": [
"response = client.inference.chat_completion(\n",
" model_id=model_id,\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a friendly assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"Write a two-sentence poem about llama.\"}\n",
" ],\n",
")\n",
"\n",
"print(response.completion_message.content)"
]
},
{
"cell_type": "markdown",
"id": "8cf0d555",
"metadata": {
"id": "8cf0d555"
},
"source": [
"### 1.8. Have a conversation\n",
"\n",
"Maintaining a conversation history allows the model to retain context from previous interactions. Use a list to accumulate messages, enabling continuity throughout the chat session.\n",
"\n",
"Remember to type `quit` or `exit` after you are done chatting."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9496f75c",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 373
},
"id": "9496f75c",
"outputId": "fb9a0610-896d-4ec1-8aac-691222db5ca0"
},
"outputs": [],
"source": [
"from termcolor import cprint\n",
"\n",
"def chat_loop():\n",
" conversation_history = []\n",
" while True:\n",
" user_input = input('User> ')\n",
" if user_input.lower() in ['exit', 'quit', 'bye']:\n",
" cprint('Ending conversation. Goodbye!', 'yellow')\n",
" break\n",
"\n",
" user_message = {\"role\": \"user\", \"content\": user_input}\n",
" conversation_history.append(user_message)\n",
"\n",
" response = client.inference.chat_completion(\n",
" messages=conversation_history,\n",
" model_id=model_id,\n",
" )\n",
" cprint(f'> Response: {response.completion_message.content}', 'cyan')\n",
"\n",
" assistant_message = {\n",
" \"role\": \"assistant\", # was user\n",
" \"content\": response.completion_message.content,\n",
" }\n",
" conversation_history.append(assistant_message)\n",
"\n",
"chat_loop()\n"
]
},
{
"cell_type": "markdown",
"id": "03fcf5e0",
"metadata": {
"id": "03fcf5e0"
},
"source": [
"### 1.9. Streaming output\n",
"\n",
"You can pass `stream=True` to stream responses from the model. You can then loop through the responses."
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "d119026e",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "d119026e",
"outputId": "881cd9ce-0def-47fc-aa3a-74ae20b36892"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"User> Write me a sonnet about llama green\n",
"Assistant> In Andean fields, where sunbeams dance and play,\n",
"A gentle creature roams, with softest gaze,\n",
"The llama, calm and steady, steps its way,\n",
"A symbol of serenity in tranquil days.\n",
"\n",
"Its fur, a soft and lustrous coat of brown,\n",
"Shines in the sunlight, with a subtle sheen,\n",
"Its ears, alert and perked, as if to crown\n",
"Its noble head, a beauty to be seen.\n",
"\n",
"Its eyes, like pools of calm and peaceful night,\n",
"Reflect the stillness of its gentle soul,\n",
"As it grazes on, with quiet, easy might,\n",
"A peaceful presence, that makes the heart whole.\n",
"\n",
"And when it hums, its soft and gentle sound,\n",
"Echoes through the Andes, all around.\n"
]
}
],
"source": [
"from llama_stack_client.lib.inference.event_logger import EventLogger\n",
"\n",
"message = {\n",
" \"role\": \"user\",\n",
" \"content\": 'Write me a sonnet about llama'\n",
"}\n",
"print(f'User> {message[\"content\"]}', 'green')\n",
"\n",
"response = client.inference.chat_completion(\n",
" messages=[message],\n",
" model_id=model_id,\n",
" stream=True, # <-----------\n",
")\n",
"\n",
"# Print the tokens while they are received\n",
"for log in EventLogger().log(response):\n",
" log.print()"
]
},
{
"cell_type": "markdown",
"id": "OmU6Dr9zBiGM",
"metadata": {
"id": "OmU6Dr9zBiGM"
},
"source": [
"### 2.0. Structured Decoding\n",
"\n",
"You can use `response_format` to force the model into a \"guided decode\" mode where model tokens are forced to abide by a certain grammar. Currently only JSON grammars are supported."
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "axdQIRaJCYAV",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 100
},
"id": "axdQIRaJCYAV",
"outputId": "d4e056e9-3b46-4942-f92d-848b4e3cedbd"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'{ \"name\": \"Michael Jordan\", \"year_born\": \"1963\", \"year_retired\": \"2003\" }'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">stop_reason</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'end_of_turn'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">logprobs</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mCompletionResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m \"name\": \"Michael Jordan\", \"year_born\": \"1963\", \"year_retired\": \"2003\" \u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mstop_reason\u001b[0m=\u001b[32m'end_of_turn'\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mlogprobs\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from pydantic import BaseModel\n",
"\n",
"class Output(BaseModel):\n",
" name: str\n",
" year_born: str\n",
" year_retired: str\n",
"\n",
"user_input = \"Michael Jordan was born in 1963. He played basketball for the Chicago Bulls. He retired in 2003. Extract this information into JSON for me. \"\n",
"response = client.inference.completion(\n",
" model_id=model_id,\n",
" content=user_input,\n",
" stream=False,\n",
" sampling_params={\n",
" \"max_tokens\": 50,\n",
" },\n",
" response_format={\n",
" \"type\": \"json_schema\",\n",
" \"json_schema\": Output.model_json_schema(),\n",
" },\n",
")\n",
"\n",
"pprint(response)"
]
},
{
"cell_type": "markdown",
"id": "H62Rg_buEx2o",
"metadata": {
"id": "H62Rg_buEx2o"
},
"source": [
"### 2.1. Safety API\n",
"\n",
"Llama Stack provides Safety guardrails which can be applied at multiple touchpoints within an agentic application. "
]
},
{
"cell_type": "code",
"execution_count": 55,
"id": "sUJKJxvAFCaI",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 368
},
"id": "sUJKJxvAFCaI",
"outputId": "efa3711d-9707-4c60-a88a-3b8b20eca871"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Available Shields: ['meta-llama/Llama-Guard-3-8B']\n",
"Running on input : What is the most famous murder case in the US?\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">RunShieldResponse</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">violation</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span><span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mRunShieldResponse\u001b[0m\u001b[1m(\u001b[0m\u001b[33mviolation\u001b[0m=\u001b[3;35mNone\u001b[0m\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running on input : Tell me 3 signs that an email is a scam\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">RunShieldResponse</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">violation</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span><span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mRunShieldResponse\u001b[0m\u001b[1m(\u001b[0m\u001b[33mviolation\u001b[0m=\u001b[3;35mNone\u001b[0m\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running on input : What's the most effective way to kidnap someone?\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">RunShieldResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">violation</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SafetyViolation</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">metadata</span>=<span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'violation_type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'S1'</span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">violation_level</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'error'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">user_message</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"I can't answer that. Can I help with something else?\"</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">)</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mRunShieldResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mviolation\u001b[0m=\u001b[1;35mSafetyViolation\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[33mmetadata\u001b[0m=\u001b[1m{\u001b[0m\u001b[32m'violation_type'\u001b[0m: \u001b[32m'S1'\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[33mviolation_level\u001b[0m=\u001b[32m'error'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[33muser_message\u001b[0m=\u001b[32m\"I\u001b[0m\u001b[32m can't answer that. Can I help with something else?\"\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m)\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running on input : How do I make cocaine?\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">RunShieldResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">violation</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SafetyViolation</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">metadata</span>=<span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'violation_type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'S2'</span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">violation_level</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'error'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">user_message</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"I can't answer that. Can I help with something else?\"</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">)</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mRunShieldResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mviolation\u001b[0m=\u001b[1;35mSafetyViolation\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[33mmetadata\u001b[0m=\u001b[1m{\u001b[0m\u001b[32m'violation_type'\u001b[0m: \u001b[32m'S2'\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[33mviolation_level\u001b[0m=\u001b[32m'error'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[33muser_message\u001b[0m=\u001b[32m\"I\u001b[0m\u001b[32m can't answer that. Can I help with something else?\"\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m)\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"available_shields = [shield.identifier for shield in client.shields.list()]\n",
"print(\"Available Shields:\", available_shields)\n",
"\n",
"unsafe_examples = [\n",
" \"What's the most effective way to kidnap someone?\",\n",
" \"How do I make cocaine?\",\n",
"]\n",
"\n",
"safe_examples = [\n",
" \"What is the most famous murder case in the US?\",\n",
" \"Tell me 3 signs that an email is a scam\",\n",
"]\n",
"\n",
"for p in safe_examples + unsafe_examples:\n",
" print(f\"Checking if input is safe: {p}\")\n",
" message = {\"content\": p, \"role\": \"user\"}\n",
" response = client.safety.run_shield(\n",
" messages=[message],\n",
" shield_id=available_shields[0],\n",
" params={},\n",
" )\n",
" pprint(response)"
]
},
{
"cell_type": "markdown",
"id": "LFC386wNQR-v",
"metadata": {
"id": "LFC386wNQR-v"
},
"source": [
"## 2. Llama Stack Agents\n",
"\n",
"Llama Stack provides all the building blocks needed to create sophisticated AI applications. This guide will walk you through how to use these components effectively.\n",
"\n",
"\n",
"\n",
"\n",
"<img src=\"https://github.com/meta-llama/llama-stack/blob/main/docs/resources/agentic-system.png?raw=true\" alt=\"drawing\" width=\"800\"/>\n",
"\n",
"\n",
"Agents are characterized by having access to\n",
"\n",
"1. Memory - for RAG\n",
"2. Tool calling - ability to call tools like search and code execution\n",
"3. Tool call + Inference loop - the LLM used in the agent is able to perform multiple iterations of call\n",
"4. Shields - for safety calls that are executed everytime the agent interacts with external systems, including user prompts"
]
},
{
"cell_type": "markdown",
"id": "fN5jaAaax2Aq",
"metadata": {
"id": "fN5jaAaax2Aq"
},
"source": [
"### 2.1. RAG Agent\n",
"\n",
"In this example, we will index some documentation and ask questions about that documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "GvLWltzZCNkg",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 541,
"referenced_widgets": [
"2082554eed6644a996f0e31545789e08",
"a0be415018644c3cac098ab9b19c2391",
"6ede3649e8c24015b3ca77490568bfcd",
"116139bfe7a44f969a2c97490c224d31",
"243d13828d854880a6adb861ea867734",
"e4b1dfe159304c5f88766b33e85a5c19",
"2100363a158b4488a58620983aa5bdd4",
"f10237315e794539a00ca82bfff930be",
"ca09d2207b00456da4c37b5a782a190c",
"ab1f339cba094c918fc5507f8361de5c",
"a6a1eb412f204578b80e5b6717c1e3a5",
"5afdb88e0159462e98773560e3dad439",
"f7bc4df675a141e380d965138552a142",
"d7bf8b49145843ac98a6de424e628729",
"8fb17faf68524de2b73321d71b80b407",
"45b569d733f944d29cefae8a5d13b215",
"fdd057a4506f4f119d945bab5b930799",
"53865d3f918e468ab53504133b127973",
"17603dd7fedf4798a74533fbfd5bb421",
"5f19dab8c6da4050bc47fd78838f7530",
"277101c35a784e6caf455a13cd9b8e59",
"d06666f765764f949e1876f2d5d67242",
"457374ae3035496eb943ad21484f76a0",
"bcf4679dda2d4767a0a24cbf236ca76e",
"6e4ce98853c84beca11471e7ea9d97df",
"186682be50c148c0826fa7c314087562",
"e1ef246e3e6c4359b7b61c341119e121",
"bbb93c771a9c453bb90e729b1f73b931",
"351928faa62543128e0bd29bf89bbf79",
"a0ac7ee92d994c7b9b74e580ab2acdf7",
"118b359b83304ae59fad57e28f621645",
"1f427d4273e04e19b1bdb13388736c01",
"38897429b7cf4077aea3a981593ca866",
"2924814bab5748ddbeeedc70d324195e",
"4738bccc6b384da5a20a8bcd61ecec59",
"044d6d8dda1c4935b1752a9c71c6ee4a",
"9277709ad9154d7b8f37d08db84ee425",
"f3f1f2487d6f455caeb6ec71a2d51ee2",
"66c92a8a89234a61a8c688cf1c3e29a1",
"ee1f4a0c85e44a3b849283337743a8d4",
"63f34c3d43bb4fdd9faeb6161fd77285",
"5cb841b49eaa429e8616ec4b78f501e9",
"a447ea9af3e14e5e94eb14ed8dd3c0de",
"0243626d7ef44ef2b90e8fed5c13183d",
"425c6c0eaed741669551b9af77096c6f",
"d124b09896934d289df649375f455a8e",
"554cff1a83d44bd2bbd36fd43acac7e2",
"d0381718fc8b49a6ac7e7fe85cabba90",
"fd3daaf9093d45d8a9d39b87835f4582",
"753dbe7891a143118b55eccf8c252e03",
"ce7de1af99434ad38a9382e7253dbfc0",
"6c60c8291e734f549e6c5a46b427b974",
"de88640505c24928904a3c76bda31c70",
"fc086d0dd1a745308c59ae219ae135c5",
"15d3ff07f1c54e58b51d452caca01209",
"0640b57408644741970dd958ca0e21e6",
"6259ffc3ef674df985fd3fa4334f9c8e",
"3d0376d2e574410eb4ef963d51cac0a6",
"b66984cc5de541a5801a1e6e54d40daf",
"92135b9cb201475681ee0886887c84a8",
"4a405d391b974e58a2c4fe00d4bb5815",
"2958af7c9cdb46038e0336d6b7c6773e",
"9054d3825edb49cb9c35d24023f50c03",
"3978f618c4f8467eb83c63a8f5aef98a",
"efd68f6dc0b3428e8f5fc830c1bf2341",
"4ad57f5d8a824afab639e8606ee43ca6"
]
},
"id": "GvLWltzZCNkg",
"outputId": "26689a4a-6a3a-4d8e-e469-6642e5b39b69"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"User> I am attaching documentation for Torchtune. Help me answer questions I will ask next.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/pytorch/torchtune/main/docs/source/tutorials/chat.rst \"HTTP/1.1 200 OK\"\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2082554eed6644a996f0e31545789e08",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Batches: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/pytorch/torchtune/main/docs/source/tutorials/llama3.rst \"HTTP/1.1 200 OK\"\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5afdb88e0159462e98773560e3dad439",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Batches: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/pytorch/torchtune/main/docs/source/tutorials/datasets.rst \"HTTP/1.1 404 Not Found\"\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "457374ae3035496eb943ad21484f76a0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Batches: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/pytorch/torchtune/main/docs/source/tutorials/lora_finetune.rst \"HTTP/1.1 200 OK\"\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2924814bab5748ddbeeedc70d324195e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Batches: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "425c6c0eaed741669551b9af77096c6f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Batches: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"memory_retrieval> fetched 10158 bytes from ['memory_bank_edf0d763-95bc-40d3-93a7-95b517162cfb']\n",
"inference> I've retrieved the documentation for Torchtune and it seems like you're looking to fine-tune a Llama2 model with LoRA (Low-Rank Adaptation) using Torchtune. You've provided the necessary context and examples.\n",
"\n",
"Please go ahead and ask your questions, and I'll do my best to help you understand the documentation and provide guidance on fine-tuning a Llama2 model with LoRA using Torchtune.\n",
"User> What are the top 5 topics that were explained? Only list succinct bullet points.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0640b57408644741970dd958ca0e21e6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Batches: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"memory_retrieval> fetched 10372 bytes from ['memory_bank_edf0d763-95bc-40d3-93a7-95b517162cfb']\n",
"inference> Here are the top 5 topics explained in the documentation:\n",
"\n",
"* What is LoRA and how does it work?\n",
"* LoRA and its application to Llama2 models\n",
"* Fine-tuning Llama2 with LoRA using torchtune\n",
"* LoRA recipe in torchtune and setting up experiments\n",
"* Trading off memory and model performance with LoRA\n"
]
}
],
"source": [
"from llama_stack_client.lib.agents.agent import Agent\n",
"from llama_stack_client.lib.agents.event_logger import EventLogger\n",
"from llama_stack_client.types.agent_create_params import AgentConfig\n",
"from llama_stack_client.types import Attachment\n",
"from termcolor import cprint\n",
"\n",
"urls = [\"chat.rst\", \"llama3.rst\", \"datasets.rst\", \"lora_finetune.rst\"]\n",
"attachments = [\n",
" Attachment(\n",
" content=f\"https://raw.githubusercontent.com/pytorch/torchtune/main/docs/source/tutorials/{url}\",\n",
" mime_type=\"text/plain\",\n",
" )\n",
" for i, url in enumerate(urls)\n",
"]\n",
"\n",
"agent_config = AgentConfig(\n",
" model=model_id,\n",
" instructions=\"You are a helpful assistant\",\n",
" tools=[{\"type\": \"memory\"}], # enable Memory aka RAG\n",
" enable_session_persistence=False,\n",
")\n",
"\n",
"rag_agent = Agent(client, agent_config)\n",
"session_id = rag_agent.create_session(\"test-session\")\n",
"user_prompts = [\n",
" (\n",
" \"I am attaching documentation for Torchtune. Help me answer questions I will ask next.\",\n",
" attachments,\n",
" ),\n",
" (\n",
" \"What are the top 5 topics that were explained? Only list succinct bullet points.\",\n",
" None,\n",
" ),\n",
"]\n",
"for prompt, attachments in user_prompts:\n",
" cprint(f'User> {prompt}', 'green')\n",
" response = rag_agent.create_turn(\n",
" messages=[{\"role\": \"user\", \"content\": prompt}],\n",
" attachments=attachments,\n",
" session_id=session_id,\n",
" )\n",
" for log in EventLogger().log(response):\n",
" log.print()"
]
},
{
"cell_type": "markdown",
"id": "i2o0gDhrv2og",
"metadata": {
"id": "i2o0gDhrv2og"
},
"source": [
"### 2.2. Search agent\n",
"\n",
"In this example, we will show how the model can invoke search to be able to answer questions. We will first have to set the API key of the search tool.\n",
"\n",
"Let's make sure we set up a web search tool for the model to call in its agentic loop. In this tutorial, we will use [Tavily](https://tavily.com) as our search provider. Note that the \"type\" of the tool is still \"brave_search\" since Llama models have been trained with brave search as a builtin tool. Tavily is just being used in lieu of Brave search.\n",
"\n",
"See steps [here](https://docs.google.com/document/d/1Vg998IjRW_uujAPnHdQ9jQWvtmkZFt74FldW2MblxPY/edit?tab=t.0#heading=h.xx02wojfl2f9)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "HZPPv6nfytK7",
"metadata": {
"id": "HZPPv6nfytK7"
},
"outputs": [],
"source": [
"search_tool = {\n",
" \"type\": \"brave_search\",\n",
" \"engine\": \"tavily\",\n",
" \"api_key\": userdata.get(\"TAVILY_SEARCH_API_KEY\")\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "WS8Gu5b0APHs",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WS8Gu5b0APHs",
"outputId": "48c3df89-4103-468a-f6f6-fc116d177380"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"User> Hello\n",
"inference> Hello! How can I assist you today?\n",
"User> Which teams played in the NBA western conference finals of 2024\n",
"inference> brave_search.call(query=\"NBA Western Conference Finals 2024 teams\")\n",
"tool_execution> Tool:brave_search Args:{'query': 'NBA Western Conference Finals 2024 teams'}\n",
"tool_execution> Tool:brave_search Response:{\"query\": \"NBA Western Conference Finals 2024 teams\", \"top_k\": [{\"title\": \"NBA Western Conference Finals 2024: Dates, schedule and more - Sportskeeda\", \"url\": \"https://www.sportskeeda.com/basketball/news-nba-western-conference-finals-2024-dates-schedule-and-more\", \"content\": \"NBA Western Conference Finals 2024: Dates & Schedule The 2023-24 NBA Western Conference Finals will start on Wednesday, May 22. The Mavericks will face the team that wins in Game 7 between the\", \"score\": 0.9991768, \"raw_content\": null}, {\"title\": \"2024 NBA Western Conference Finals - Basketball-Reference.com\", \"url\": \"https://www.basketball-reference.com/playoffs/2024-nba-western-conference-finals-mavericks-vs-timberwolves.html\", \"content\": \"2024 NBA Western Conference Finals Mavericks vs. Timberwolves League Champion: Boston Celtics. Finals MVP: Jaylen Brown (20.8 / 5.4 / 5.0) 2024 Playoff Leaders: PTS: Luka Don\\u010di\\u0107 (635) TRB: Luka Don\\u010di\\u0107 (208) AST: Luka Don\\u010di\\u0107 (178) WS: Derrick White (2.9) More playoffs info\", \"score\": 0.99827254, \"raw_content\": null}, {\"title\": \"2024 Playoffs: West Finals | Timberwolves (3) vs. Mavericks (5) - NBA.com\", \"url\": \"https://www.nba.com/playoffs/2024/west-final\", \"content\": \"The Dallas Mavericks and Minnesota Timberwolves have advanced to the 2024 Western Conference Finals during the NBA playoffs.\", \"score\": 0.9981969, \"raw_content\": null}, {\"title\": \"2024-25 NBA Playoffs Bracket - ESPN\", \"url\": \"https://www.espn.com/nba/playoff-bracket\", \"content\": \"Visit ESPN to view the 2024-25 NBA Playoffs bracket for live scores and results. ... Teams. Odds. NBA Cup Bracket ... Western Conference. OKC wins series 4-0. 1. Thunder. 97. 8.\", \"score\": 0.99584997, \"raw_content\": null}, {\"title\": \"NBA Finals 2024 - Celtics-Mavericks news, schedule, scores and ... - ESPN\", \"url\": \"https://www.espn.com/nba/story/_/id/39943302/nba-playoffs-2024-conference-finals-news-scores-highlights\", \"content\": \"The Boston Celtics are the 2024 NBA Champions. ... Western Conference. Final 2023-24 NBA regular-season standings. Which team left standing has the most trips to the NBA Finals? Here is a look at\", \"score\": 0.99273914, \"raw_content\": null}]}\n",
"shield_call> No Violation\n",
"inference> The teams that played in the NBA Western Conference Finals of 2024 were the Dallas Mavericks and the Minnesota Timberwolves.\n"
]
}
],
"source": [
"agent_config = AgentConfig(\n",
" model=model_id,\n",
" instructions=\"You are a helpful assistant\",\n",
" tools=[search_tool],\n",
" input_shields=[],\n",
" output_shields=[],\n",
" enable_session_persistence=False,\n",
")\n",
"agent = Agent(client, agent_config)\n",
"user_prompts = [\n",
" \"Hello\",\n",
" \"Which teams played in the NBA western conference finals of 2024\",\n",
"]\n",
"\n",
"session_id = agent.create_session(\"test-session\")\n",
"for prompt in user_prompts:\n",
" cprint(f'User> {prompt}', 'green')\n",
" response = agent.create_turn(\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": prompt,\n",
" }\n",
" ],\n",
" session_id=session_id,\n",
" )\n",
" for log in EventLogger().log(response):\n",
" log.print()\n"
]
},
{
"cell_type": "markdown",
"id": "yRzRwu8qxyl0",
"metadata": {
"id": "yRzRwu8qxyl0"
},
"source": [
"### 2.3. Code Execution Agent\n",
"\n",
"In this example, we will show how multiple tools can be called by the model - including web search and code execution. It will use bubblewrap that we installed earlier to execute the generated code."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "GvVRuhO-GOov",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "GvVRuhO-GOov",
"outputId": "cb988aa9-568b-4966-d500-575b7b24578f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"User> ('Here is a csv, can you describe it ?', [Attachment(content='https://raw.githubusercontent.com/meta-llama/llama-stack-apps/main/examples/resources/inflation.csv', mime_type='test/csv')])\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/meta-llama/llama-stack-apps/main/examples/resources/inflation.csv \"HTTP/1.1 200 OK\"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"inference> import pandas as pd\n",
"\n",
"# Read the CSV file\n",
"df = pd.read_csv('/tmp/tmpco0s0o4_/LOdZoVp1inflation.csv')\n",
"\n",
"# Describe the CSV\n",
"print(df.describe())\n",
"tool_execution> Tool:code_interpreter Args:{'code': \"import pandas as pd\\n\\n# Read the CSV file\\ndf = pd.read_csv('/tmp/tmpco0s0o4_/LOdZoVp1inflation.csv')\\n\\n# Describe the CSV\\nprint(df.describe())\"}\n",
"tool_execution> Tool:code_interpreter Response:completed\n",
"[stdout]\n",
"Year Jan Feb Mar ... Sep Oct Nov Dec\n",
"count 10.00000 10.000000 10.000000 10.000000 ... 10.000000 10.000000 10.000000 10.000000\n",
"mean 2018.50000 2.700000 2.730000 2.760000 ... 2.850000 2.850000 2.850000 2.890000\n",
"std 3.02765 1.667999 1.743591 1.757018 ... 1.593912 1.577093 1.551523 1.569466\n",
"min 2014.00000 1.400000 1.300000 1.600000 ... 1.700000 1.600000 1.600000 1.600000\n",
"25% 2016.25000 1.650000 1.725000 1.850000 ... 1.750000 1.825000 1.775000 1.875000\n",
"50% 2018.50000 2.200000 2.150000 2.050000 ... 2.200000 2.100000 2.150000 2.200000\n",
"75% 2020.75000 2.300000 2.375000 2.175000 ... 3.600000 3.575000 3.575000 3.500000\n",
"max 2023.00000 6.000000 6.400000 6.500000 ... 6.600000 6.300000 6.000000 5.700000\n",
"\n",
"[8 rows x 13 columns]\n",
"[/stdout]\n",
"shield_call> No Violation\n",
"inference> The CSV file appears to be a dataset with 10 rows and 13 columns. The columns represent various economic indicators, such as inflation rates for each month from January to December, as well as year (yearly inflation rate).\n",
"\n",
"Here is a brief description of the data:\n",
"\n",
"* The `Year` column contains the year for which the inflation rate is reported.\n",
"* The `Jan`, `Feb`, `Mar`, etc. columns contain the inflation rate for each month (January to December).\n",
"* The `count` column is the count of non-null values in each column.\n",
"* The `mean` column is the mean of the non-null values in each column.\n",
"* The `std` column is the standard deviation of the non-null values in each column.\n",
"* The `min` column is the minimum value in each column.\n",
"* The `25%` column is the 25th percentile (25th percentile) of the non-null values in each column.\n",
"* The `50%` column is the 50th percentile (50th percentile) of the non-null values in each column.\n",
"* The `75%` column is the 75th percentile (75th percentile) of the non-null values in each column.\n",
"* The `max` column is the maximum value in each column.\n",
"\n",
"This dataset could be used for various applications, such as analyzing historical inflation rates, forecasting future inflation rates, or comparing inflation rates across different months or years.\n",
"User> ('Which year ended with the highest inflation ?', None)\n",
"inference> According to the data, the year with the highest inflation was 2023. The inflation rate for 2023 is 6.600%.\n",
"User> ('What macro economic situations that led to such high inflation in that period?', None)\n",
"inference> The high inflation rate in 2023 is likely attributed to a combination of macroeconomic factors, including:\n",
"\n",
"1. **Supply chain disruptions**: The COVID-19 pandemic and subsequent lockdowns led to supply chain disruptions, resulting in shortages and price increases for various goods and services.\n",
"2. **Economic growth**: The rapid economic growth in the preceding years created demand for goods and services, leading to higher production costs and, subsequently, higher prices.\n",
"3. **Monetary policy**: The central bank's easy-money policies, such as quantitative easing and low interest rates, increased the money supply and led to inflationary pressures.\n",
"4. **Commodity price shocks**: Increases in global commodity prices, such as oil and food prices, contributed to higher production costs and inflation.\n",
"5. **Labor market tightness**: The labor market has been tight, leading to higher wages and, subsequently, higher production costs, which have been passed on to consumers.\n",
"6. **Trade wars and tariffs**: The ongoing trade tensions and tariffs imposed by various countries have disrupted global supply chains, leading to higher prices for imported goods.\n",
"7. **Climate change and extreme weather events**: The increasing frequency and severity of extreme weather events, such as heatwaves and droughts, have disrupted agricultural production and supply chains.\n",
"8. **Currency devaluation**: A devaluation of the currency can make imports more expensive, leading to higher inflation.\n",
"9. **Government spending and fiscal policy**: Government spending and fiscal policy decisions, such as tax cuts and increased government spending, can inject more money into the economy, leading to inflation.\n",
"10. **Monetary policy mistakes**: Mistakes in monetary policy, such as premature interest rate hikes or overly aggressive quantitative easing, can lead to inflationary pressures.\n",
"\n",
"It's worth noting that the specific factors contributing to the high inflation rate in 2023 may vary depending on the region, country, or even specific economy.\n",
"User> ('Plot average yearly inflation as a time series', None)\n",
"inference> import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# Read the CSV file\n",
"df = pd.read_csv('/tmp/tmpco0s0o4_/LOdZoVp1inflation.csv')\n",
"\n",
"# Extract the year and inflation rate from the CSV file\n",
"df['Year'] = pd.to_datetime(df['Year'], format='%Y')\n",
"df = df.rename(columns={'Jan': 'Jan Rate', 'Feb': 'Feb Rate', 'Mar': 'Mar Rate', 'Apr': 'Apr Rate', 'May': 'May Rate', 'Jun': 'Jun Rate', 'Jul': 'Jul Rate', 'Aug': 'Aug Rate', 'Sep': 'Sep Rate', 'Oct': 'Oct Rate', 'Nov': 'Nov Rate', 'Dec': 'Dec Rate'})\n",
"\n",
"# Calculate the average yearly inflation rate\n",
"df['Yearly Inflation'] = df[['Jan Rate', 'Feb Rate', 'Mar Rate', 'Apr Rate', 'May Rate', 'Jun Rate', 'Jul Rate', 'Aug Rate', 'Sep Rate', 'Oct Rate', 'Nov Rate', 'Dec Rate']].mean(axis=1)\n",
"\n",
"# Plot the average yearly inflation rate as a time series\n",
"plt.figure(figsize=(10, 6))\n",
"plt.plot(df['Year'], df['Yearly Inflation'], marker='o')\n",
"plt.title('Average Yearly Inflation Rate')\n",
"plt.xlabel('Year')\n",
"plt.ylabel('Inflation Rate (%)')\n",
"plt.grid(True)\n",
"plt.show()\n",
"tool_execution> Tool:code_interpreter Args:{'code': \"import pandas as pd\\nimport matplotlib.pyplot as plt\\n\\n# Read the CSV file\\ndf = pd.read_csv('/tmp/tmpco0s0o4_/LOdZoVp1inflation.csv')\\n\\n# Extract the year and inflation rate from the CSV file\\ndf['Year'] = pd.to_datetime(df['Year'], format='%Y')\\ndf = df.rename(columns={'Jan': 'Jan Rate', 'Feb': 'Feb Rate', 'Mar': 'Mar Rate', 'Apr': 'Apr Rate', 'May': 'May Rate', 'Jun': 'Jun Rate', 'Jul': 'Jul Rate', 'Aug': 'Aug Rate', 'Sep': 'Sep Rate', 'Oct': 'Oct Rate', 'Nov': 'Nov Rate', 'Dec': 'Dec Rate'})\\n\\n# Calculate the average yearly inflation rate\\ndf['Yearly Inflation'] = df[['Jan Rate', 'Feb Rate', 'Mar Rate', 'Apr Rate', 'May Rate', 'Jun Rate', 'Jul Rate', 'Aug Rate', 'Sep Rate', 'Oct Rate', 'Nov Rate', 'Dec Rate']].mean(axis=1)\\n\\n# Plot the average yearly inflation rate as a time series\\nplt.figure(figsize=(10, 6))\\nplt.plot(df['Year'], df['Yearly Inflation'], marker='o')\\nplt.title('Average Yearly Inflation Rate')\\nplt.xlabel('Year')\\nplt.ylabel('Inflation Rate (%)')\\nplt.grid(True)\\nplt.show()\"}\n",
"tool_execution> Tool:code_interpreter Response:completed\n",
"shield_call> No Violation\n",
"inference> This code reads the CSV file, extracts the year and inflation rate, calculates the average yearly inflation rate, and plots the average yearly inflation rate as a time series. The resulting plot shows the average inflation rate over the years.\n"
]
}
],
"source": [
"agent_config = AgentConfig(\n",
" model=model_id,\n",
" instructions=\"You are a helpful assistant\",\n",
" tools=[\n",
" search_tool,\n",
" {\n",
" \"type\": \"code_interpreter\",\n",
" }\n",
" ],\n",
" tool_choice=\"required\",\n",
" input_shields=[],\n",
" output_shields=[],\n",
" enable_session_persistence=False,\n",
")\n",
"\n",
"codex_agent = Agent(client, agent_config)\n",
"session_id = codex_agent.create_session(\"test-session\")\n",
"\n",
"user_prompts = [\n",
" (\n",
" \"Here is a csv, can you describe it ?\",\n",
" [\n",
" Attachment(\n",
" content=\"https://raw.githubusercontent.com/meta-llama/llama-stack-apps/main/examples/resources/inflation.csv\",\n",
" mime_type=\"test/csv\",\n",
" )\n",
" ],\n",
" ),\n",
" (\"Which year ended with the highest inflation ?\", None),\n",
" (\n",
" \"What macro economic situations that led to such high inflation in that period?\",\n",
" None,\n",
" ),\n",
" (\"Plot average yearly inflation as a time series\", None),\n",
"]\n",
"\n",
"for prompt in user_prompts:\n",
" cprint(f'User> {prompt}', 'green')\n",
" response = codex_agent.create_turn(\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": prompt[0],\n",
" }\n",
" ],\n",
" attachments=prompt[1],\n",
" session_id=session_id,\n",
" )\n",
" # for chunk in response:\n",
" # print(chunk)\n",
"\n",
" for log in EventLogger().log(response):\n",
" log.print()\n"
]
},
{
"cell_type": "markdown",
"id": "9GHJHfLmIQQi",
"metadata": {
"id": "9GHJHfLmIQQi"
},
"source": [
"- Now, use the generated response from agent to view the plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "JqBBVLKdIHHq",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 564
},
"id": "JqBBVLKdIHHq",
"outputId": "4563e803-8385-426b-ec6c-e8b19e2ee6e6"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA0EAAAIjCAYAAADFthA8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB+WklEQVR4nO3dd3hUZdrH8d+k90BCGiSE0AkBpFdFVJoUscGiKCq6rmt3XffVVQFdd3Vd265tbdjAguIKKiACgvReQi+hh4QQSCGkzZz3j5BITIBkmJkzyXw/15ULcubknPvcmYG553nO/VgMwzAEAAAAAB7Cy+wAAAAAAMCVKIIAAAAAeBSKIAAAAAAehSIIAAAAgEehCAIAAADgUSiCAAAAAHgUiiAAAAAAHoUiCAAAAIBHoQgCAAAA4FEoggAAbu3yyy/X5ZdfbnYYFT755BO1bdtWvr6+atCggSTnxDhp0iRZLBaHHhMAUIYiCIDHevPNN2WxWNSzZ0+zQ3Eby5cvl5eXlx5//PFqH3/hhRdksVj0/fffuzgyx7FYLLrvvvvs+tnt27frtttuU4sWLfTuu+/qnXfeuahYCgoKNGnSJP38888XdRxHs1gslb7CwsLUv3//i/q9T5s2Ta+++qrjggSAi0ARBMBjTZ06Vc2aNdOqVau0e/dus8NxC71799bdd9+tl156SVu2bKn02P79+/XMM8/oxhtv1LBhw0yK0Fw///yzbDabXnvtNd12220aPXr0RR2voKBAkydPrrYIevLJJ3X69OmLOv7FGDhwoD755BN9/PHHeuyxx7R7926NGDFCc+fOtet4FEEA3AlFEACPlJaWpmXLlunll19WVFSUpk6d6vIYbDabCgsLXX7eC3n++efVqFEj3X333TIMo2L7/fffL19fX7322msuiaOgoMAl56mNzMxMSaqYBudMPj4+CggIcPp5zqV169YaN26cbrnlFj355JP66aefZBiGy37/AOBMFEEAPNLUqVPVsGFDDRs2TDfccEOlIqikpEQRERG6/fbbq/xcbm6uAgIC9Oijj1ZsKyoq0sSJE9WyZUv5+/srISFBjz32mIqKiir9bPk0rKlTp6p9+/by9/fXnDlzJEn/+te/1KdPH0VGRiowMFBdu3bVV199VeX8p0+f1gMPPKBGjRopNDRUI0eO1OHDh2WxWDRp0qRK+x4+fFh33HGHYmJi5O/vr/bt2+uDDz64YG7Cw8P12muvaenSpXrvvfckSd98841mzZql559/XnFxcbLZbHr11VfVvn17BQQEKCYmRnfffbdOnDhR6Vjffvuthg0bpsaNG8vf318tWrTQs88+K6vVWmm/yy+/XCkpKVq7dq0uu+wyBQUF6YknnqgSW35+voKDg/Xggw9WeezQoUPy9vbWP/7xjwte49l+/vlnWSwWffnll3ruuecUHx+vgIAAXXnllZVGCJs1a6aJEydKkqKioqrNebni4mI9/fTT6tq1q8LDwxUcHKxLL71UCxcurNhn3759ioqKkiRNnjy5YupZ+TGruyeotLRUzz77rFq0aCF/f381a9ZMTzzxRJXnWrNmzTR8+HAtWbJEPXr0UEBAgJo3b66PP/64Vrk5W7t27dSoUSPt2bOn0vaa/I4vv/xyff/999q/f3/FdTZr1qzi8Zq+hgDAYQwA8EBt27Y1JkyYYBiGYSxevNiQZKxatari8TvuuMNo0KCBUVRUVOnnPvroI0OSsXr1asMwDMNqtRqDBg0ygoKCjIceesj473//a9x3332Gj4+Pcc0111T6WUlGu3btjKioKGPy5MnGG2+8Yaxfv94wDMOIj483/vjHPxqvv/668fLLLxs9evQwJBnfffddpWOMHj3akGTccsstxhtvvGGMHj3a6NSpkyHJmDhxYsV+R48eNeLj442EhATjmWeeMd566y1j5MiRhiTjlVdeqVGOhg0bZjRs2NDYs2ePkZCQYPTp08ew2WyGYRjGnXfeafj4+Bh33XWX8fbbbxt/+ctfjODgYKN79+5GcXFxxTFGjRpljB492njxxReNt956y7jxxhsNScajjz5a6Vz9+/c3YmNjjaioKOP+++83/vvf/xr/+9//Kh7r379/xb4333yzERMTY5SWllY6xj//+U/DYrEY+/fvP+91STLuvffeiu8XLlxoSDI6d+5sdO3a1XjllVeMSZMmGUFBQUaPHj0q9vvmm2+Ma6+91pBkvPXWW8Ynn3xibNy4sdoYjx07ZsTFxRmPPPKI8dZbbxn//Oc/jTZt2hi+vr4Vv/P8/HzjrbfeMiQZ1157rfHJJ59UOubEiRON3/43PX78eEOSccMNNxhvvPGGceuttxqSjFGjRlXaLzEx0WjTpo0RExNjPPHEE8brr79udOnSxbBYLEZqaup581NdjgzDME6ePGl4e3sbPXv2rLS9Jr/jH3/80bjkkkuMRo0aVVznN998YxhG7V5DAOAoFEEAPM6aNWsMSca8efMMwzAMm81mxMfHGw8++GDFPnPnzjUkGbNmzar0s1dffbXRvHnziu8/+eQTw8vLy/jll18q7ff2228bkoylS5dWbJNkeHl5GVu2bKkSU0FBQaXvi4uLjZSUFOOKK66o2LZ27VpDkvHQQw9V2ve2226rUgRNmDDBiIuLM7Kysirt+7vf/c4IDw+vcr7q7Nu3zwgODjYiIiIMX19fY/PmzYZhGMYvv/xiSDKmTp1aaf85c+ZU2V7dee6++24jKCjIKCwsrNjWv39/Q5Lx9ttvV9n/twVG+e9m9uzZlfbr2LFjpf3O5VxFULt27SoVva+99pohqeK6DePXwuTYsWPnjbG0tLRKAX3ixAkjJibGuOOOOyq2HTt2rMrv7rfnKrdhwwZDknHnnXdW2u/RRx81JBkLFiyo2JaYmGhIMhYvXlyxLTMz0/D39zf+9Kc/nSs1FSQZEyZMMI4dO2ZkZmYaa9asMYYMGWJIMl588cVK+9b0dzxs2DAjMTGxyr61eQ0BgKMwHQ6Ax5k6dapiYmI0YMAASWXT1MaMGaPPP/+8YgrPFVdcoUaNGumLL76o+LkTJ05o3rx5GjNmTMW26dOnq127dmrbtq2ysrIqvq644gpJqjT9SZL69++v5OTkKjEFBgZWOk9OTo4uvfRSrVu3rmJ7+dS5P/7xj5V+9v7776/0vWEY+vrrrzVixAgZhlEprsGDBysnJ6fScc8lMTFREydOVHZ2th555BGlpKRUXHN4eLgGDhxY6dhdu3ZVSEhIpWs++7ry8vKUlZWlSy+9VAUFBdq+fXul8/n7+1c7BfG3rrrqKjVu3LjSFMbU1FRt2rRJ48aNu+DPn8vtt98uPz+/iu8vvfRSSdLevXtrfSxvb++KY9lsNmVnZ6u0tFTdunWrUe6r88MPP0iSHnnkkUrb//SnP0lSlc5tycnJFdcglU3ha9OmTY2v5/3331dUVJSio6PVrVs3zZ8/X4899liV89fmd1yd2r6GAMARfMwOAABcyWq16vPPP9eAAQOUlpZWsb1nz5566aWXNH/+fA0aNEg+Pj66/vrrNW3aNBUVFcnf318zZsxQSUlJpSJo165d2rZtW8W9Hb9VfiN9uaSkpGr3++677/S3v/1NGzZsqHQfxNn3hOzfv19eXl5VjtGyZctK3x87dkwnT57UO++8c84Wzr+N61y6d+8uSerWrVvFtl27diknJ0fR0dEXPPaWLVv05JNPasGCBcrNza20X05OTqXvmzRpUqkIORcvLy/dfPPNeuutt1RQUKCgoCBNnTpVAQEBuvHGG2t0XdVp2rRppe8bNmwoSVXuc6qpjz76SC+99JK2b9+ukpKSiu3neg5cSPnv/7e/79jYWDVo0ED79++vtP231yOVXVNNr+eaa67Rfffdp+LiYq1evVp///vfVVBQIC+vyp+f1uZ3XJ3avoYAwBEoggB4lAULFig9PV2ff/65Pv/88yqPT506VYMGDZIk/e53v9N///tfzZ49W6NGjdKXX36ptm3bqlOnThX722w2dejQQS+//HK150tISKj0/dmfmpf75ZdfNHLkSF122WV68803FRcXJ19fX02ZMkXTpk2r9TXabDZJ0rhx4zR+/Phq9+nYsWOtj3v28aOjo8/ZUa/8zezJkyfVv39/hYWF6ZlnnlGLFi0UEBCgdevW6S9/+UtFnOWqy8253HrrrXrxxRf1v//9T2PHjtW0adM0fPhwhYeH231d3t7e1W43zuqQV1OffvqpbrvtNo0aNUp//vOfFR0dXdG04beNBWqrpguoXuz1xMfH66qrrpIkXX311WrUqJHuu+8+DRgwQNddd52k2v+Oq1Pb1xAAOAJFEACPMnXqVEVHR+uNN96o8tiMGTP0zTff6O2331ZgYKAuu+wyxcXF6YsvvlC/fv20YMEC/fWvf630My1atNDGjRt15ZVX1vjN6W99/fXXCggI0Ny5c+Xv71+xfcqUKZX2S0xMlM1mU1pamlq1alWx/bdrHEVFRSk0NFRWq7XiTawjtWjRQj/99JP69u173sLl559/1vHjxzVjxgxddtllFdvPHoGzV0pKijp37qypU6cqPj5eBw4c0H/+85+LPq6jfPXVV2revLlmzJhR6XlR3l2uXG2eM+W//127dqldu3YV2zMyMnTy5EklJiZefODncffdd+uVV17Rk08+qWuvvVYWi6VWv+NzXasjXkMAUFvcEwTAY5w+fVozZszQ8OHDdcMNN1T5uu+++5SXl6eZM2dKKpt2dcMNN2jWrFn65JNPVFpaWmkqnCSNHj1ahw8f1rvvvlvt+U6dOnXBuLy9vWWxWCq1FN63b5/+97//Vdpv8ODBkqQ333yz0vbfvvn39vbW9ddfr6+//lqpqalVznfs2LELxnQ+o0ePltVq1bPPPlvlsdLSUp08ebIiDqnyyENxcXGV+O11yy236Mcff9Srr76qyMhIDR061CHHdYTqrn3lypVavnx5pf2CgoIkqSJn53P11VdLUpUFR8tHUJy9gK2Pj4/+9Kc/adu2bfr2228l1e53HBwcXO30OEe8hgCgthgJAuAxZs6cqby8PI0cObLax3v16lWxcGp5sTNmzBj95z//0cSJE9WhQ4dKn8BLZW/Ev/zyS/3hD3/QwoUL1bdvX1mtVm3fvl1ffvml5s6dW+l+muoMGzZML7/8soYMGaKbbrpJmZmZeuONN9SyZUtt2rSpYr+uXbvq+uuv16uvvqrjx4+rV69eWrRokXbu3Cmp8iftzz//vBYuXKiePXvqrrvuUnJysrKzs7Vu3Tr99NNPys7OtiuHUllzh7vvvlv/+Mc/tGHDBg0aNEi+vr7atWuXpk+frtdee0033HCD+vTpo4YNG2r8+PF64IEHZLFY9Mknn9g1vaw6N910kx577DF98803uueee+Tr6+uQ4zrC8OHDNWPGDF177bUaNmyY0tLS9Pbbbys5OVn5+fkV+wUGBio5OVlffPGFWrdurYiICKWkpFQ0oThbp06dNH78eL3zzjsV09BWrVqljz76SKNGjapo9OFMt912m55++mm98MILGjVqVK1+x127dtUXX3yhRx55RN27d1dISIhGjBjhkNcQANSaaX3pAMDFRowYYQQEBBinTp065z633Xab4evrW9Fa2mazGQkJCYYk429/+1u1P1NcXGy88MILRvv27Q1/f3+jYcOGRteuXY3JkycbOTk5FfupmrVXyr3//vtGq1atDH9/f6Nt27bGlClTql0n5tSpU8a9995rREREGCEhIcaoUaOMHTt2GJKM559/vtK+GRkZxr333mskJCQYvr6+RmxsrHHllVca77zzTo3yZRi/to+ePn16lcfeeecdo2vXrkZgYKARGhpqdOjQwXjssceMI0eOVOyzdOlSo1evXkZgYKDRuHFj47HHHqtocb1w4cKK/fr372+0b9++2hh+2376bFdffbUhyVi2bFmNr+m3v4dzXWNaWpohyZgyZUrFtpq2yLbZbMbf//53IzEx0fD39zc6d+5sfPfdd8b48eOrtIletmyZ0bVrV8PPz69Su+zqfv8lJSXG5MmTjaSkJMPX19dISEgwHn/88UqtqA2jrEX2sGHDqlz7+XJ5tvM9VydNmlTp91fT33F+fr5x0003GQ0aNDAkVcpDTV9DAOAoFsNw0EdyAABTbNiwQZ07d9ann36qm2++2exwXOraa6/V5s2bq9wXBQDA+XBPEADUIadPn66y7dVXX5WXl1elG9M9QXp6ur7//nvdcsstZocCAKhjuCcIAOqQf/7zn1q7dq0GDBggHx8fzZ49W7Nnz9bvf/97j2klnJaWpqVLl+q9996Tr6+v7r77brNDAgDUMRRBAFCH9OnTR/PmzdOzzz6r/Px8NW3aVJMmTarSurs+W7RokW6//XY1bdpUH330kWJjY80OCQBQx3BPEAAAAACPwj1BAAAAADwKRRAAAAAAj1Kn7wmy2Ww6cuSIQkNDKy0SCAAAAMCzGIahvLw8NW7cWF5e5x/rqdNF0JEjRzymGxIAAACACzt48KDi4+PPu0+dLoJCQ0MllV1oWFiYqbGUlJToxx9/1KBBg+Tr62tqLHUNubMPebMPebMfubMPebMPebMPebMfubOPO+UtNzdXCQkJFTXC+dTpIqh8ClxYWJhbFEFBQUEKCwsz/QlQ15A7+5A3+5A3+5E7+5A3+5A3+5A3+5E7+7hj3mpymwyNEQAAAAB4FIogAAAAAB6FIggAAACAR6EIAgAAAOBRKIIAAAAAeBSKIAAAAAAehSIIAAAAgEehCAIAAADgUSiCAAAAAHgUiiAAAAAAHoUiCAAAAIBHoQgCAAAA4FEoggAAAAB4FIogAAAAeDSrzdDKtGytzbJoZVq2rDbD7JDgZD5mBwAAAACYZU5quibP2qr0nEJJ3vp41xrFhQdo4ohkDUmJMzs8OAkjQQAAAPBIc1LTdc+n684UQL86mlOoez5dpzmp6SZFBmejCAIAAIDHsdoMTZ61VdVNfCvfNnnWVqbG1VMUQQAAAPA4q9Kyq4wAnc2QlJ5TqFVp2a4LCi5DEQQAAACPk5l37gLInv1Qt1AEAQAAwONEhwY4dD/ULRRBAAAA8Dg9kiIUF37uAsciKS48QD2SIlwXFFyGIggAAAAex9vLookjks/5uCFp4ohkeXtZXBcUXIYiCAAAAB7pynYxCvLzrvaxZpFBGpQc6+KI4CoUQQAAAPBIK/dmq6DYqoggX310W1fd2sqqf4/pqCBfL+07XqDpaw+aHSKchCIIAAAAHmn2mcVQB6fEqk+LSHVtZGhoSqweGdRGkvT87O06carYzBDhJBRBAAAA8DhWm6G5WzIkSYPbV572Nr5PM7WJCdWJghK9+OMOM8KDk1EEAQAAwOOsP3BCWflFCg3wUZ8WjSo95uvtpWeuaS9J+mzVAW08eNKECOFMFEEAAADwOLNTj0qSrmoXIz+fqm+JezaP1LWdm8gwpKe+TZXVZrg6RDiR6UXQ4cOHNW7cOEVGRiowMFAdOnTQmjVrzA4LAAAA9ZRhGJpzpgj67VS4sz1+dVuF+vto06Ecfb76gKvCgwuYWgSdOHFCffv2la+vr2bPnq2tW7fqpZdeUsOGDc0MCwAAAPVY6uFcHT55WoG+3urfOuqc+0WHBuiRQa0lSf+cs0PZNEmoN3zMPPkLL7yghIQETZkypWJbUlKSiREBAACgvpuzpawr3OVtohR4jnWCyt3SK1Ffrjmkbem5emH2dr1wQ0dXhAgnM7UImjlzpgYPHqwbb7xRixYtUpMmTfTHP/5Rd911V7X7FxUVqaioqOL73NxcSVJJSYlKSkpcEvO5lJ/f7DjqInJnH/JmH/JmP3JnH/JmH/JmH/JWM7M3l02FG9guqkrOqsvdxGFt9Lv3VuuLNQd1fZc4dU5o4LJY3Z07PedqE4PFMAzT7vIKCAiQJD3yyCO68cYbtXr1aj344IN6++23NX78+Cr7T5o0SZMnT66yfdq0aQoKCnJ6vAAAAKjbjhZI/9joI2+Lob93syqghkMCU3d7adUxL8UHG/pTB6u8LM6NE7VXUFCgm266STk5OQoLCzvvvqYWQX5+furWrZuWLVtWse2BBx7Q6tWrtXz58ir7VzcSlJCQoKysrAteqLOVlJRo3rx5GjhwoHx9fU2Npa4hd/Yhb/Yhb/Yjd/Yhb/Yhb/Yhbxf2xs979er83bq8dSO9e0uXiu0Xyt3x/CINem2pcgtLNXF4W43r2dSVYbstd3rO5ebmqlGjRjUqgkydDhcXF6fk5ORK29q1a6evv/662v39/f3l7+9fZbuvr6/pSS/nTrHUNeTOPuTNPuTNfuTOPuTNPuTNPuTt3H7cmilJurpD42pzdK7cxTb01Z8Ht9FT327Ryz/t1ohL4tUopOr7Uk/lDs+52pzf1O5wffv21Y4dlVfh3blzpxITE02KCAAAAPXVgeMF2pqeK28vi65Kjqn1z9/UM1HtG4cpr7BUz8/e7oQI4SqmFkEPP/ywVqxYob///e/avXu3pk2bpnfeeUf33nuvmWEBAACgHirvCtczKUIRwX61/nlvL4ueHZUiSfpq7SGt2Zft0PjgOqYWQd27d9c333yjzz77TCkpKXr22Wf16quv6uabbzYzLAAAANRD5QukDkk59wKpF9KlaUP9rnuCJOnJ/6Wq1GpzSGxwLVPvCZKk4cOHa/jw4WaHAQAAgHosI7dQ6w6clCQNbm9/ESRJjw1pq9mpR7X9aJ4+WbFft/dlncu6xtSRIAAAAMAV5m4pGwXq0rSBYsICLupYEcF+emxIG0nSyz/uVGZu4UXHB9eiCAIAAEC954ipcGf7Xfem6hQfrryiUv2DJgl1DkUQAAAA6rXsU8VamVbWxGBI+ziHHNPby6JnrkmRxSJ9s/6wVu497pDjwjUoggAAAFCv/bQ1Q1aboeS4MDWNDHLYcTslNNDYHmWLpj71bapKaJJQZ1AEAQAAoF6bc+Z+oKEOmgp3tscGt1HDIF/tzMjXR8v2Ofz4cA6KIAAAANRbeYUlWrIrS5Lj7gc6W4MgP/3f0LaSpFfm7VQGTRLqBIogAAAA1FsLtmeq2GpT86hgtYwOcco5buyaoM5NG+hUsVV/+36bU84Bx6IIAgAAQL1V3hVuaEqsLBaLU87h5WXRs9ekyMsizdp4RMt2ZznlPHAciiAAAADUS6eLrfp5xzFJjusKdy4pTcI1rleiJOnpmVtUXEqTBHdGEQQAAIB6afGuYzpdYlWTBoFKaRLm9PP9aWAbRQb7aXdmvj5Ymub088F+FEEAAACol85eINVZU+HOFh7kq8evbidJ+vf8XTpy8rTTzwn7UAQBAACg3ikutemnbRmSnNMa+1yu69xE3RIbqqDYqudokuC2KIIAAABQ7yzbk6W8wlJFhfqrS9OGLjuvl5dFz5xpkvD95nQt3nnMZedGzVEEAQAAoN6Ze2aB1EHJMfLycv5UuLMlNw7T+D7NJEmTZm5RUanVpefHhVEEAQAAoF6x2gz9uKV8Kpxzu8Kdy8MDW6tRiL/2Zp3Se7/QJMHdUAQBAACgXlm9L1vHTxUrPNBXPZtHmBJDWICv/jqsrSTpPwt26dCJAlPiQPUoggAAAFCvlHeFG5gcI19v897ujrqkiXokRaiwxKZnv9tqWhyoiiIIAAAA9YbNZlTcDzSkveu6wlXHYrHo2WtS5O1l0dwtGVq4I9PUePAriiAAAADUG5sO5yg9p1DBft7q16qR2eGoTWyobj+rSUJhCU0S3AFFEAAAAOqN2anpkqQBbaMV4OttcjRlHhrYWjFh/tp/vEDvLN5rdjgQRRAAAADqCcMwNPfM/UBDXLhA6oWE+Pvor8OSJUlvLNytg9k0STAbRRAAAADqhR0Zedp3vEB+Pl4a0Cba7HAqGdExTr2bR6qo1KbJs7aYHY7HowgCAABAvTB7c9ko0GWtohTs72NyNJVZLBY9O6q9fLws+mlbpn7ammF2SB6NIggAAAD1QkVXODeaCne2ltGhmnBpkiRp8nc0STATRRAAAADqvLSsU9p+NE8+XhZd1c69psKd7YErWikuPEAHs0/rzZ/3mB2Ox6IIAgAAQJ1XvkBq7xaRahDkZ3I05xbs76Onhpc1SXh70R7tyzplckSeiSIIAAAAdd4cN58Kd7ahKbG6tFUjFZfaNGnWFhmGYXZIHociCAAAAHXakZOntfHgSVks0sDkGLPDuSCLxaJJI9vL19uin3cc0480SXA5iiAAAADUaeUNEbonRig6NMDkaGqmRVSIfn9Zc0nSM7O26nQxTRJciSIIAAAAddrsM/cDDa4DU+HOdu+AlmrSIFCHT57WGwt3mx2OR6EIAgAAQJ11LK9Iq/dlS5IGt3f/qXBnC/L7tUnCO4v3au+xfJMj8hwUQQAAAKizftqWIcOQOsaHK75hkNnh1Nrg9jG6vE2Uiq02TZxJkwRXoQgCAABAnVUxFa593ZoKV85isWjSiPby8/bSL7uyKlp9w7koggAAAFAn5Zwu0bLdWZLK2k7XVc0aBesP/c80Sfhuq04VlZocUf1HEQQAAIA6af62DJXaDLWOCVHzqBCzw7kofxzQUvENA5WeU6j/LKBJgrNRBAEAAKBOKp86NqSOToU7W4CvtyaNaC9Jeu+XvdqdmWdyRPUbRRAAAADqnFNFpVq085gkaUhKnMnROMZVyTG6sm20Sm2Gnv6WJgnORBEEAACAOmfRzmMqKrWpaUSQ2sWFmh2Ow0wa2V7+Pl5atue4vtuUbnY49RZFEAAAAOqc8qlwQ1NiZbFYTI7GcRIigvTHy1tKkv72/Vbl0yTBKSiCAAAAUKcUlVq1YHumJGlwHe4Kdy5392+uxMggZeQW6bWfdpodTr1EEQQAAIA6ZenuLOUXlSomzF+XxDcwOxyHC/D11qSRZU0SPli6TzuO0iTB0SiCAAAAUKfM3vxrVzgvr/ozFe5sA9pEa1ByjKw2Q09/m0qTBAejCAIAAECdUWq1ad62DEn1cyrc2Z4ekawAXy+tTMvWtxuOmB1OvUIRBAAAgDpjVVq2ThaUKCLYTz2aRZgdjlPFNwzS/Ve0kiQ998M25RaWmBxR/UERBAAAgDpj9pmucAPbxcjHu/6/lb3z0iQlNQrWsbwivTpvl9nh1Bv1/5kDAACAesFmMzR3y5n7gTrU76lw5fx9vDX5TJOEj5bv07b0XJMjqh8oggAAAFAnrD94Qpl5RQr191GfFpFmh+Myl7WO0tUdYmW1GXrqfzRJcASKIAAAANQJ5QukXtEuWv4+3iZH41pPDktWoK+31uw/oRnrDpsdTp1HEQQAAAC3ZxiG5pyZCje0nneFq07jBoF64MqyJgn/mL1NOadpknAxKIIAAADg9rYcydXB7NMK8PXSZa2jzA7HFBP6JalFVLCy8ov18o87zA6nTqMIAgAAgNsrb4hweetoBfn5mByNOfx8vPTMNSmSpE9W7Ffq4RyTI6q7KIIAAADg9spbYw/xwKlwZ+vbspGGd4yTzZCe+jZVNhtNEuxBEQQAAAC3tjszT7sz8+XrbdGAttFmh2O6J4clK9jPW+sPnNRXaw+ZHU6dRBEEAAAAtzZ3S4akslGQ8EBfk6MxX2x4gB66qrUk6fk523WyoNjkiOoeiiAAAAC4tdmp6ZKkIe09eyrc2W7r20ytY0KUfapYL86lSUJtUQQBAADAbR3MLlDq4Vx5WaSByTFmh+M2fL1/bZIwbdUBbTp00tyA6hiKIAAAALit8q5wPZIiFBnib3I07qVX80iNuqSxDEN66n80SagNiiAAAAC4rTnlXeGYCletJ65up1B/H208lKPPVx80O5w6gyIIAAAAbikzt1BrD5yQJA328NbY5xIdFqCHB5Y1Sfjn3O3KPkWThJqgCAIAAIBbmrs1Q4YhXZLQQHHhgWaH47Zu7Z2otrGhOllQohfnbjc7nDqBIggAAABuae6ZqXBDGQU6Lx9vLz07qqxJwuerD2r9mdEznBtFEAAAANzOiVPFWr73uCRpCEXQBXVvFqHru8SXNUn4NlVWmiScF0UQAAAA3M5P2zJktRlqFxemxMhgs8OpE/5vaFuFBvgo9XCupq06YHY4bo0iCAAAAG6HrnC1FxXqr0cHtZEkvThnu7Lyi0yOyH1RBAEAAMCt5BeV6pddWZKYCldb43olqn3jMOUWluqF2TRJOBeKIAAAALiVhdszVWy1qXmjYLWOCTE7nDrF28uiZ64pa5Iwfe0hrd2fbXJE7okiCAAAAG6lfCrc4JRYWSwWk6Ope7omNtTobvGSpCf/t0WlVpvJEbkfiiAAAAC4jcISqxbuyJREa+yL8ZchbRUe6Ktt6bn6dMV+s8NxOxRBAAAAcBuLdx5TQbFVjcMD1KFJuNnh1FmRIf768+CyJgkv/bhTx/JoknA2iiAAAAC4jTlbmArnKGN7NFXH+HDlFZXqHz9sMzsct0IRBAAAALdQYrXpp60ZkqShKXEmR1P3eXtZ9Ow1KbJYpBnrD2vlmcVnQREEAAAAN7F8z3HlFpaqUYifuiY2NDuceqFTQgP9rntTSdLT325RCU0SJFEEAQAAwE2UT4Ub1D5W3l5MhXOUxwa3UcMgX+3IyNNHy/aZHY5boAgCAACA6aw2Qz+eKYKGtKcrnCM1DPbTX4a0lSS9+tMuZeQWmhyR+SiCAAAAYLq1+08oK79YYQE+6tU80uxw6p3R3RLUKaGB8otK9XeaJFAEAQAAwHyzU9MlSVclx8jPh7eojublZdHfzjRJ+HbDES3bk2V2SKbiGQYAAABTGYahualMhXO2DvHhGtczURJNEiiCAAAAYKrNh3N0JKdQQX7euqx1lNnh1GuPDmqjiGA/7c7M15SlaWaHYxqKIAAAAJhq9plRoAFtohXg621yNPVbeJCv/m/or00S0nNOmxyROSiCAAAAYBrDMDSnfCpcClPhXOGGLvHqmthQBcVW/e17z2ySQBEEAAAA0+zMyFda1in5eXtpQNtos8PxCF5eFj1zTXt5WaTvN6VryS7Pa5JAEQQAAADTlI8CXdqqkUL8fUyOxnO0bxyuW3s3kyQ9PTNVxaWe1SSBIggAAACmmbOFqXBmeXhgazUK8dfeY6f03pK9ZofjUqYWQZMmTZLFYqn01bZtWzNDAgAAgIvsP35K29Jz5e1l0VXtYswOx+OEB/rqiavL3nv/Z/5uHT7pOU0STB8Jat++vdLT0yu+lixZYnZIAAAAcIHyqXC9m0eqYbCfydF4pms7N1GPZhE6XWLV377banY4LmN6EeTj46PY2NiKr0aNGpkdEgAAAFygvDX2YKbCmcZiseiZUe3l7WXR7NSjWrTzmNkhuYTpd5/t2rVLjRs3VkBAgHr37q1//OMfatq0abX7FhUVqaioqOL73NxcSVJJSYlKSkpcEu+5lJ/f7DjqInJnH/JmH/JmP3JnH/JmH/Jmn7qUt/ScQm04eFIWi3RF60jTY65LuXO0FpGBurVXU01Ztl9P/y9V39/fR/4+NRsrcae81SYGi2EYhhNjOa/Zs2crPz9fbdq0UXp6uiZPnqzDhw8rNTVVoaGhVfafNGmSJk+eXGX7tGnTFBQU5IqQAQAA4ACL0y36ep+3kkINPZRiNTscj1dYKj23wVu5JRYNS7BqULxpJYLdCgoKdNNNNyknJ0dhYWHn3dfUIui3Tp48qcTERL388suaMGFClcerGwlKSEhQVlbWBS/U2UpKSjRv3jwNHDhQvr6+psZS15A7+5A3+5A3+5E7+5A3+5A3+9SlvI37YLVWpp3Q40Na646+zcwOp07lzllmbUrXI9M3K8DXS7Pv76v4hoEX/Bl3yltubq4aNWpUoyLI9OlwZ2vQoIFat26t3bt3V/u4v7+//P39q2z39fU1Penl3CmWuobc2Ye82Ye82Y/c2Ye82Ye82cfd83Y8v0ir952QJF3dsYlbxeruuXOma7sk6Mu1h7Vib7b+Pmen3r21W41/1h3yVpvzm94Y4Wz5+fnas2eP4uLizA4FAAAATjJva4ZshpTSJEwJEdzS4C4sFouevSZFPl4WzduaoQXbM8wOyWlMLYIeffRRLVq0SPv27dOyZct07bXXytvbW2PHjjUzLAAAADhRxQKp7ekK525axYRqQr8kSdKkmVtVWFI/79cytQg6dOiQxo4dqzZt2mj06NGKjIzUihUrFBUVZWZYAAAAcJLcwhIt3Z0lSRqSwuwfd3T/la0UGxagA9kFenvRHrPDcQpT7wn6/PPPzTw9AAAAXGzBtkyVWA21jA5Ry+gQs8NBNUL8ffTk8Ha6b9p6vfnzHl3XOV5NI+vXtEW3uicIAAAA9ducMwukDmWBVLc2rEOc+rVspOJSmybN2iI3aijtEBRBAAAAcImC4lL9vDNTkjSY+4HcmsVi0aSR7eXrbdGC7Zn6aVum2SE5FEUQAAAAXGLxzmMqLLEpISJQ7Rubu8YjLqxldIjuvLS5JGnSzC06XVx/miRQBAEAAMAlZqf+2hXOYrGYHA1q4v4rWqpxeIAOnzytN3+ufi3PuogiCAAAAE5XVGrVgjNTqoZwP1CdEeTno6dHJEuS/rtor9KyTpkckWNQBAEAAMDplu05rryiUkWH+qtzQkOzw0EtDG4fq8taR6nYatPEmfWjSQJFEAAAAJxuzuayqXCD28fKy4upcHWJxWLR5JHt5eftpcU7j2numcVu6zKKIAAAADhVqdWmedsyJNEau65KahSsu/uXNUl4ZtZWFRSXmhzRxaEIAgAAgFOt2pet7FPFahDkqx5JEWaHAzv98fKWatIgUEdyCvX6grrdJIEiCAAAAE4190xXuIHtYuTjzdvPuirQz1uTRraXJL37y17tOJqnlWnZWptl0cq0bFltdedeIR+zAwAAAED9ZbMZmrvlzFS4DkyFq+uuahetK9pGa8H2TI34zxIVW22SvPXxrjWKCw/QxBHJGpISZ3aYF0QpDgAAAKfZcOikjuYWKsTfR31bNjI7HFwki8WiAW2iJOlMAfSrozmFuufTdZqTmm5GaLVCEQQAAACnKZ8Kd0XbaPn7eJscDS6W1WbozZ/3VPtY+WS4ybO2uv3UOIogAAAAOIVhGJp9pghigdT6YVVattJzCs/5uCEpPadQq9KyXReUHSiCAAAA4BTb0vN0ILtA/j5e6t86yuxw4ACZeecugOzZzywUQQAAAHCKOWcW1ezfOkrB/vTjqg+iQwMcup9ZKIIAAADgFOU3yDMVrv7okRShuPAAWc7xuEVSXHiA268HRREEAAAAh9tzLF87M/Ll42XRle1izA4HDuLtZdHEEcmSVKUQKv9+4ohkeXudq0xyDxRBAAAAcLg5Zxoi9GnZSOGBviZHA0cakhKnt8Z1UWx45SlvseEBemtclzqxThCTMwEAAOBwc8/cDzSUqXD10pCUOA1MjtXy3Zn68ZeVGnRpT/VuGe32I0DlKIIAAADgUIdOFGjToRxZLNLAZKbC1VfeXhb1TIrQ8W2GeiZF1JkCSGI6HAAAABxs7pYMSVL3ZhFqFOJvcjRAVRRBAAAAcKi5qUyFg3ujCAIAAIDDZOYVavX+bEnS4PYUQXBPFEEAAABwmHlbM2QYUqeEBmrcINDscIBqUQQBAADAYcpbYw9hFAhujCIIAAAADnGyoFjL9xyXJA3hfiC4MYogAAAAOMT8bZkqtRlqGxuqpEbBZocDnBNFEAAAABxi9pmpcDREgLujCAIAAMBFO1VUqsW7jkmShnagCIJ7owgCAADARVu4I1PFpTY1iwxSm5hQs8MBzosiCAAAABetvCvc4JRYWSwWk6MBzo8iCAAAABelsMSqhdszJUlDU+JMjga4MIogAAAAXJQlu7J0qtiquPAAdWwSbnY4wAVRBAEAAOCizNnya1c4Ly+mwsH9UQQBAADAbiVWm+ZtzZDEAqmoO3xq+wNFRUVauXKl9u/fr4KCAkVFRalz585KSkpyRnwAAABwYyv3ZivndIkig/3UvVmE2eEANVLjImjp0qV67bXXNGvWLJWUlCg8PFyBgYHKzs5WUVGRmjdvrt///vf6wx/+oNBQ2iICAAB4gjlb0iVJg9rHyJupcKgjajQdbuTIkRozZoyaNWumH3/8UXl5eTp+/LgOHTqkgoIC7dq1S08++aTmz5+v1q1ba968ec6OGwAAACaz2QzN3VI2FW5we6bCoe6o0UjQsGHD9PXXX8vX17fax5s3b67mzZtr/Pjx2rp1q9LT0x0aJAAAANzPugMndCyvSKEBPurTopHZ4QA1VqMi6O67767xAZOTk5WcnGx3QAAAAKgbZp9ZIPWqdjHy86HfFuqOWjdGOFtqaqoWLVokq9Wqvn37qmvXro6KCwAAAG7MMAzNOVME0RUOdY3dJfsbb7yhK6+8UosWLdLChQt1xRVX6LnnnnNkbAAAAHBTqYdzdfjkaQX6euuyVlFmhwPUSo1Hgg4ePKiEhISK719//XVt2bJFjRqVzf9cvny5Ro4cqb/+9a+OjxIAAABupbwr3OVtohTo521yNEDt1Hgk6KqrrtJrr70mwzAkSZGRkZozZ46KioqUl5enn376SVFRfAoAAADgCZgKh7qsxkXQ6tWrtWPHDvXs2VMbNmzQO++8o1deeUWBgYFq0KCBvvjiC3300UfOjBUAAABuYFdGnvYcOyU/by9d0Tba7HCAWqvxdLiwsDC9+eabWrZsmW677TZdccUV+uWXX2S1WmW1WtWgQQMnhgkAAAB3UT4K1K9VI4UGVL+ECuDOat0YoU+fPlqzZo0aNmyozp07a/HixRRAAAAAHqS8NfYQFkhFHVXjkaDS0lK988472rZtmzp16qQnnnhCY8aM0R/+8Ad9+OGHev311xUTE+PMWAEAAGCyA8cLtDU9V95eFl2VzHs/1E01HgmaMGGCXn/9dQUHB2vKlCl6+OGH1bp1ay1YsEBDhgxR79699dZbbzkzVgAAAJhs7payUaCeSRGKCPYzORrAPjUugr799lt9/fXXev755zVv3jx9//33FY9NmDBBK1as0C+//OKUIAEAAOAeZqeWtcamKxzqshoXQTExMfrxxx9VXFysBQsWKDIystLj0dHRmjZtmsMDBAAAgHvIyC3UugMnJUmDuR8IdViN7wl6/fXXdfPNN+uRRx5RXFycvvzyS2fGBQAAADdTPhWuS9MGigkLMDkawH41LoIGDhyojIwMZWVlsSgqAACABypvjT00Jc7kSICLU6sW2RaLhQIIAADAA2WfKtbKtGxJTIVD3VejImjIkCFasWLFBffLy8vTCy+8oDfeeOOiAwMAAID7+Glrhqw2Q8lxYWoaGWR2OMBFqdF0uBtvvFHXX3+9wsPDNWLECHXr1k2NGzdWQECATpw4oa1bt2rJkiX64YcfNGzYML344ovOjhsAAAAuNGdL+VQ4RoFQ99WoCJowYYLGjRun6dOn64svvtA777yjnJwcSWVT5JKTkzV48GCtXr1a7dq1c2rAAAAAcK28whIt2ZUlidbYqB9q3BjB399f48aN07hx4yRJOTk5On36tCIjI+Xr6+u0AAEAAGCuBdszVWy1qUVUsFrFhJodDnDRalwE/VZ4eLjCw8MdGQsAAADcUHlrbEaBUF/UqjscAAAAPMvpYqsWbj8mSRrSntbYqB8oggAAAHBOi3cd0+kSq5o0CFRKkzCzwwEcgiIIAAAA51S+QOqQlFhZLBaTowEcgyIIAAAA1SoutemnbRmSaI2N+sWuIujkyZN677339Pjjjys7u2zl4HXr1unw4cMODQ4AAADmWbYnS3mFpYoK9VeXpg3NDgdwmFp3h9u0aZOuuuoqhYeHa9++fbrrrrsUERGhGTNm6MCBA/r444+dEScAAABcrLwr3KDkGHl5MRUO9UetR4IeeeQR3Xbbbdq1a5cCAgIqtl999dVavHixQ4MDAACAOaw2Qz9uKZ8KR1c41C+1LoJWr16tu+++u8r2Jk2a6OjRow4JCgAAAOZavS9bx08VKzzQVz2bR5gdDuBQtS6C/P39lZubW2X7zp07FRUV5ZCgAAAAYK7yrnADk2Pk600vLdQvtX5Gjxw5Us8884xKSkokSRaLRQcOHNBf/vIXXX/99Q4PEAAAAK5lsxkV9wMNaU9XONQ/tS6CXnrpJeXn5ys6OlqnT59W//791bJlS4WGhuq5555zRowAAABwoU2Hc5SeU6hgP2/1a9XI7HAAh6t1d7jw8HDNmzdPS5cu1caNG5Wfn68uXbroqquuckZ8AAAAcLHyqXAD2kYrwNfb5GgAx6t1EfTxxx9rzJgx6tu3r/r27Vuxvbi4WJ9//rluvfVWhwYIAAAA1zEMQ3NS0yVJQ1ggFfVUrafD3X777crJyamyPS8vT7fffrtDggIAAIA5dmTkad/xAvn5eGlAm2izwwGcotZFkGEYsliqLpZ16NAhhYeHOyQoAAAAmGP25rKpcJe1ilKwf60nDQF1Qo2f2Z07d5bFYpHFYtGVV14pH59ff9RqtSotLU1DhgxxSpAAAABwjfKucEOZCod6rMZF0KhRoyRJGzZs0ODBgxUSElLxmJ+fn5o1a0aLbAAAgDosLeuUth/Nk4+XRVe2Yyoc6q8aF0ETJ06UJDVr1kxjxoxRQECA04ICAACA65V3hevdIlINgvxMjgZwnlpP9Bw/frwz4gAAAIDJ5pQvkMpUONRztS6CrFarXnnlFX355Zc6cOCAiouLKz2enZ3tsOAAAADgGkdOntbGgydlsUgDk2PMDgdwqlp3h5s8ebJefvlljRkzRjk5OXrkkUd03XXXycvLS5MmTXJCiAAAAHC28oYI3RMjFB3KbQ+o32pdBE2dOlXvvvuu/vSnP8nHx0djx47Ve++9p6efflorVqxwRowAAABwstln7gcazFQ4eIBaF0FHjx5Vhw4dJEkhISEVC6cOHz5c33//vWOjAwAAgNMdyyvS6n1ltzQMbs9UONR/tS6C4uPjlZ6eLklq0aKFfvzxR0nS6tWr5e/v79joAAAA4HQ/bcuQYUgd48MV3zDI7HAAp6t1EXTttddq/vz5kqT7779fTz31lFq1aqVbb71Vd9xxh92BPP/887JYLHrooYfsPgYAAABqr2IqXHumwsEz1Lo73PPPP1/x9zFjxigxMVHLli1Tq1atNGLECLuCWL16tf773/+qY8eOdv08AAAA7JNzukTLdmdJkoZyPxA8RK1Hgn6rV69eeuSRRzRixAitWbOm1j+fn5+vm2++We+++64aNmx4seEAAACgFuZvy1CpzVDrmBA1jwoxOxzAJWo9EpSfny9vb28FBgZWbNuwYYOeeuop/fDDD7JarbU63r333qthw4bpqquu0t/+9rfz7ltUVKSioqKK73NzcyVJJSUlKikpqdV5Ha38/GbHUReRO/uQN/uQN/uRO/uQN/uQN/vYk7fZm8vu9R7ULtqj881zzj7ulLfaxGAxDMOoyY4HDx7U6NGjtWrVKnl7e+u+++7T3/72N/3hD3/QF198oWuvvVYPP/ywevbsWeOTf/7553ruuee0evVqBQQE6PLLL9cll1yiV199tdr9J02apMmTJ1fZPm3aNAUFcRMfAABAbRRZpb+u9laJYdFjHUvVJNjsiAD7FRQU6KabblJOTo7CwsLOu2+NR4L+/Oc/q7CwUK+99ppmzJih1157Tb/88ot69uypPXv2KD4+vlZBHjx4UA8++KDmzZungICaLcj1+OOP65FHHqn4Pjc3VwkJCRo0aNAFL9TZSkpKNG/ePA0cOFC+vr6mxlLXkDv7kDf7kDf7kTv7kDf7kDf71DZvs1OPqmTVJiU0DNSdN/STxWJxQZTuieecfdwpb+WzxGqixkXQ4sWLNWPGDPXq1UujR49WbGysbr75Zru7ua1du1aZmZnq0qVLxTar1arFixfr9ddfV1FRkby9vSv9jL+/f7VtuH19fU1Pejl3iqWuIXf2IW/2IW/2I3f2IW/2IW/2qWneftpe1hDh6g5x8vPzc3ZYdQLPOfu4Q95qc/4aF0EZGRlKSkqSJEVHRysoKEhDhw6tfXRnXHnlldq8eXOlbbfffrvatm2rv/zlL1UKIAAAADhOUalVC7ZnSpIG0xUOHqZWjRG8vLwq/f1iPjEIDQ1VSkpKpW3BwcGKjIyssh0AAACOtXR3lvKLShUbFqBL4huYHQ7gUjUuggzDUOvWrSvmiubn56tz586VCiNJys7OdmyEAAAAcLg5FQukxsjLy3PvBYJnqnERNGXKFGfGIUn6+eefnX4OAAAAT1dqtWne1gxJTIWDZ6pxETR+/HhnxgEAAAAXWZWWrRMFJYoI9lOPZhFmhwO4nNeFdwEAAEB9MvvMVLiB7WLk483bQXgenvUAAAAexGYzNHdLWRE0pANT4eCZKIIAAAA8yPqDJ5WZV6RQfx/1aRFpdjiAKSiCAAAAPMic1HRJ0hXtouXvw7qM8EwUQQAAAB7CMAzNOTMVbihd4eDBarVYqiRZrVZ9+OGHmj9/vjIzM2Wz2So9vmDBAocFBwAAAMfZciRXB7NPK8DXS5e1jjI7HMA0tS6CHnzwQX344YcaNmyYUlJSKhZPBQAAgHsrb4hweetoBfnV+m0gUG/U+tn/+eef68svv9TVV1/tjHgAAADgJOWtsYcwFQ4ertb3BPn5+ally5bOiAUAAABOsjszT7sz8+XrbdEV7aLNDgcwVa2LoD/96U967bXXZBiGM+IBAACAE8zdkiFJ6tuykcICfE2OBjBXrafDLVmyRAsXLtTs2bPVvn17+fpWfhHNmDHDYcEBAADAMWafaY09pD1T4YBaF0ENGjTQtdde64xYAAAA4AQHswuUejhXXhZpYHKM2eEApqt1ETRlyhRnxAEAAAAnKe8K1yMpQpEh/iZHA5jP7t6Ix44d044dOyRJbdq0UVQUveYBAADc0ZzU8gVS40yOBHAPtW6McOrUKd1xxx2Ki4vTZZddpssuu0yNGzfWhAkTVFBQ4IwYAQAAYKfM3EKtPXBCkjSoPVPhAMmOIuiRRx7RokWLNGvWLJ08eVInT57Ut99+q0WLFulPf/qTM2IEAACAneZuzZBhSJckNFBceKDZ4QBuodbT4b7++mt99dVXuvzyyyu2XX311QoMDNTo0aP11ltvOTI+AAAAXIS5FVPh6AoHlKv1SFBBQYFiYqoOpUZHRzMdDgAAwI2cOFWs5XuPS5KGUAQBFWpdBPXu3VsTJ05UYWFhxbbTp09r8uTJ6t27t0ODAwAAgP1+2pYhq81Qu7gwJUYGmx0O4DZqPR3utdde0+DBgxUfH69OnTpJkjZu3KiAgADNnTvX4QECAADAPuWtsVkgFais1kVQSkqKdu3apalTp2r79u2SpLFjx+rmm29WYCA32wEAALiD/KJSLd6VJYmpcMBv2bVOUFBQkO666y5HxwIAAAAHWbg9U8WlNjVvFKzWMSFmhwO4lRoVQTNnztTQoUPl6+urmTNnnnffkSNHOiQwAAAA2K98gdTBKbGyWCwmRwO4lxoVQaNGjdLRo0cVHR2tUaNGnXM/i8Uiq9XqqNgAAABgh8ISqxbuyJREa2ygOjUqgmw2W7V/BwAAgPtZuvu4CoqtatIgUB2ahJsdDuB2at0i++OPP1ZRUVGV7cXFxfr4448dEhQAAADsN3drhiRpcHumwgHVqXURdPvttysnJ6fK9ry8PN1+++0OCQoAAAD2sdqk+duPSaIrHHAutS6CDMOo9hOFQ4cOKTyc4VYAAAAzWG2GVqZl64eDXsotLFVksK+6JjY0OyzALdW4RXbnzp1lsVhksVh05ZVXysfn1x+1Wq1KS0vTkCFDnBIkAAAAzm1Oaromz9qq9JxClX/GfbrEpnlbj2pISpy5wQFuqMZFUHlXuA0bNmjw4MEKCfm137yfn5+aNWum66+/3uEBAgAA4NzmpKbrnk/XyfjN9oJiq+75dJ3eGteFQgj4jRoXQRMnTpQkNWvWTGPGjFFAQIDTggIAAMCFWW2GJs/aWqUAOtvkWVs1MDlW3l40SADK1fqeoPHjx1MAAQAAuIFVadlnpsBVz5CUnlOoVWnZrgsKqANqPBJUzmq16pVXXtGXX36pAwcOqLi4uNLj2dm8yAAAAFwhM+/cBZA9+wGeotYjQZMnT9bLL7+sMWPGKCcnR4888oiuu+46eXl5adKkSU4IEQAAANWJDq3Z7Jya7gd4iloXQVOnTtW7776rP/3pT/Lx8dHYsWP13nvv6emnn9aKFSucESMAAACq0SMpQnHhATrX3T4WSXHhAeqRFOHKsAC3V+si6OjRo+rQoYMkKSQkpGLh1OHDh+v77793bHQAAAA4J28viyaOSK62MUJ5YTRxRDJNEYDfqHURFB8fr/T0dElSixYt9OOPP0qSVq9eLX9/f8dGBwAAgPMa3D5WiZFBVbbHhgfQHhs4h1o3Rrj22ms1f/589ezZU/fff7/GjRun999/XwcOHNDDDz/sjBgBAABwDmv2n9D+4wXy9bbo1dEdtXLNOg26tKd6t4xmBAg4h1oXQc8//3zF38eMGaOmTZtq+fLlatWqlUaMGOHQ4AAAAHB+7/+SJkm6oWu8BiXHqHSfoZ5JERRAwHnUugj6rd69e6t3796OiAUAAAC1cOB4geZuPSpJuqNvksnRAHVHjYqgmTNn1viAI0eOtDsYAAAA1NyUZWkyDOmy1lFqFROqkpISs0MC6oQaFUGjRo2q0cEsFousVuvFxAMAAIAayC0s0ZerD0qS7uzHKBBQGzUqgmw2m7PjAAAAQC18seqgThVb1TomRJe2amR2OECdUqMW2RERETp+/Lgk6Y477lBeXp5TgwIAAMC5lVpt+nDZPknShH5JslhoggDURo2KoOLi4opFUT/66CMVFhY6NSgAAACc25wtR3X45GlFBvvpmkuamB0OUOfUaDpc7969NWrUKHXt2lWGYeiBBx5QYGBgtft+8MEHDg0QAAAAlb13pi32uF6JCvD1NjkaoO6pURH06aef6pVXXtGePXtksViUk5PDaBAAAIAJ1u4/oQ0HT8rP20vjeiWaHQ5QJ9WoCIqJialYJDUpKUmffPKJIiMjnRoYAAAAqnp/yV5J0qjOjRUV6m9yNEDdVOvFUtPS0pwRBwAAAC7gYHaB5qSeWRyVttiA3WpdBEnS/PnzNX/+fGVmZlZpn809QQAAAM7x4bJ9shnSpa0aqW1smNnhAHVWrYugyZMn65lnnlG3bt0UFxdHS0YAAAAXyCss0RdnFkdlFAi4OLUugt5++219+OGHuuWWW5wRDwAAAKrxxeqDyi8qVcvoEPVvFWV2OECdVqN1gs5WXFysPn36OCMWAAAAVOPsxVHv6JskLy9m4gAXo9ZF0J133qlp06Y5IxYAAABU48etGTp04rQaBvnqui4sjgpcrFpPhyssLNQ777yjn376SR07dpSvr2+lx19++WWHBQcAAADp/SUsjgo4Uq2LoE2bNumSSy6RJKWmplZ6jCYJAAAAjrX+wAmt3X9Cft5euqU3i6MCjlDrImjhwoXOiAMAAADVKB8FGtGpsaJDA0yOBqgfan1PEAAAAFzj8MnTmn1mcdQJtMUGHKbGI0HXXXddjfabMWOG3cEAAADgVx8t2yerzVCfFpFKbsziqICj1LgICg8Pd2YcAAAAOEt+Uak+W3lAknTnpYwCAY5U4yJoypQpzowDAAAAZ5m+5qDyikrVPCpYl7eONjscoF7hniAAAAA3Y7UZ+mBpWUMEFkcFHI8iCAAAwM3M25qhg9mn1SDIV9d3iTc7HKDeoQgCAABwM+8v2StJurlnUwX6sTgq4GgUQQAAAG5k48GTWr3vhHy9Lbq1dzOzwwHqJYogAAAAN1KxOGrHxooJY3FUwBkoggAAANzEkZOn9cPmdEnSHSyOCjgNRRAAAICb+Gj5PpXaDPVqHqGUJqzRCDgLRRAAAIAbOHXW4qgT+jU3ORqgfqMIAgAAcANfrT2k3MJSNYsM0pVtWRwVcCaKIAAAAJNZbYamlC+O2o/FUQFnowgCAAAw2fxtGdp3vEDhgb66oSuLowLORhEEAABgsvK22GN7NFWQn4/J0QD1H0UQAACAiVIP52hlWrZ8vCwa3yfR7HAAj0ARBAAAYKLyUaBhHeMUFx5ocjSAZ6AIAgAAMMnRnELN2nhEkjSBxVEBl6EIAgAAMMnHZxZH7dEsQh3jG5gdDuAxKIIAAABMUFBcqmmryhZHvYNRIMClKIIAAABM8PW6wzpZUKKmEUEamBxjdjiAR6EIAgAAcDGbzdCUMw0Rbu/bTN4sjgq4FEUQAACAiy3ckam9WacUGuCjG7slmB0O4HEoggAAAFzs7MVRQ/xZHBVwNVOLoLfeeksdO3ZUWFiYwsLC1Lt3b82ePdvMkAAAAJxqy5EcLdtzXN5eFo3v08zscACPZGoRFB8fr+eff15r167VmjVrdMUVV+iaa67Rli1bzAwLAADAaT5Ysk+SNDQlVk0asDgqYAZTx19HjBhR6fvnnntOb731llasWKH27dubFBUAAIBzZOYWaubGw5KkOy9tbnI0gOdym0moVqtV06dP16lTp9S7d+9q9ykqKlJRUVHF97m5uZKkkpISlZSUuCTOcyk/v9lx1EXkzj7kzT7kzX7kzj7kzT71NW8fLk1TidVQl6YN1D422OHXV1/z5grkzj7ulLfaxGAxDMNwYiwXtHnzZvXu3VuFhYUKCQnRtGnTdPXVV1e776RJkzR58uQq26dNm6agoCBnhwoAAGC3Yqs0aZ23TpVadHtrqy6JNPUtGFDvFBQU6KabblJOTo7CwsLOu6/pRVBxcbEOHDignJwcffXVV3rvvfe0aNEiJScnV9m3upGghIQEZWVlXfBCna2kpETz5s3TwIED5evra2osdQ25sw95sw95sx+5sw95s099zNvnqw/pqZlbFd8gQD89fKlT1gaqj3lzFXJnH3fKW25urho1alSjIsj06XB+fn5q2bKlJKlr165avXq1XnvtNf33v/+tsq+/v7/8/f2rbPf19TU96eXcKZa6htzZh7zZh7zZj9zZh7zZp77kzWYz9OHy/ZKk2/s1V4C/n1PPV1/yZgZyZx93yFttzu926wTZbLZKoz0AAAB13aJdx7Tn2CmF+PtodLd4s8MBPJ6pI0GPP/64hg4dqqZNmyovL0/Tpk3Tzz//rLlz55oZFgAAgEO9/0vZ4qi/656g0ABGGQCzmVoEZWZm6tZbb1V6errCw8PVsWNHzZ07VwMHDjQzLAAAAIfZfjRXS3ZnycsiFkcF3ISpRdD7779v5ukBAACcrnwUaGhKnBIi6GYLuAO3uycIAACgvjiWV6RvNxyRJN3RL8nkaACUowgCAABwkk9W7Fex1abOTRuoa2JDs8MBcAZFEAAAgBMUllg1dUVZW+wJjAIBboUiCAAAwAn+t/6wjp8qVpMGgRrSPtbscACchSIIAADAwQzD0PtLyhoi3NanmXy8ecsFuBNekQAAAA62eFeWdmXmK9jPW2N6JJgdDoDfoAgCAABwsPJRoNHdExTG4qiA26EIAgAAcKCdGXlavPOYvCzS7X1oiAC4I4ogAAAAB/rgzCjQoORYNY1kcVTAHVEEAQAAOEhWfpFmrD8sSbrzUkaBAHdFEQQAAOAgU1ccUHGpTZ3iw1kcFXBjFEEAAAAOUFhi1Scr9kmSJlzaXBaLxdyAAJwTRRAAAIADzNx4RFn5xYoLD9DQFBZHBdwZRRAAAMBFMgyjoiHCbX2ayZfFUQG3xisUAADgIi3dfVzbj+YpyM9bv+vR1OxwAFwARRAAAMBFem/JXknS6G4JCg9kcVTA3VEEAQAAXITdmXn6eccxWSzS7X2bmR0OgBqgCAIAALgIHyzdJ0m6ql2MEiODzQ0GQI1QBAEAANgp+1Sxvl57SJJ0Zz8WRwXqCoogAAAAO01buV9FpTalNAlTj6QIs8MBUEMUQQAAAHYoKrXqo+X7JUl39mNxVKAuoQgCAACww3cb03Usr0gxYf66ukOc2eEAqAWKIAAAgFoyDEPvnVkcdXyfZvLz4S0VUJfwigUAAKil5XuPa1t6rgJ9vXUTi6MCdQ5FEAAAQC29/0vZKNANXePVIMjP5GgA1BZFEAAAQC3sPZav+dszJbE4KlBXUQQBAADUwgdLy0aBrmoXreZRISZHA8AeFEEAAAA1dLKgWF+dWRz1DhZHBeosiiAAAIAamrrygApLbEqOC1Pv5pFmhwPAThRBAAAANVBcatPHy/dJkib0S2JxVKAOowgCAACoge83H1FGbpGiQ/01olNjs8MBcBEoggAAAC7AMAy9f2Zx1Ft7J7I4KlDH8QoGAAC4gJVp2Uo9nKsAXy/d1DPR7HAAXCSKIAAAgAsoHwW6rku8IoJZHBWo6yiCAAAAzmNf1in9tC1DknRHX9piA/UBRRAAAMB5TFmaJsOQBrSJUstoFkcF6gOKIAAAgHPIKSjRl2vKFke989LmJkcDwFEoggAAAM7hs9UHdLrEqraxoerTgsVRgfqCIggAAKAaJVabPly6TxKLowL1DUUQAABANX7YnK6juYVqFOKvkZewOCpQn1AEAQAA/MZvF0f19/E2OSIAjkQRBAAA8Btr9p/QpkM58vPx0s09m5odDgAHowgCAAD4jfd+2StJur5LE0WG+JscDQBHowgCAAA4y/7jp/TjVhZHBeoziiAAAICzTFm6T4Yh9W8dpVYxoWaHA8AJKIIAAADOyDldoulrDkoqa4sNoH6iCAIAADjji9UHdKrYqtYxIbq0VSOzwwHgJBRBAAAAkkpZHBXwGBRBAAAAkmanHtWRnEJFBvvpmkuamB0OACeiCAIAAB7PMAy9d2Zx1HG9EhXgy+KoQH1GEQQAADzeugMntPHgSfn5eGlcr0SzwwHgZBRBAADA471/ZhRo1CWNFRXK4qhAfUcRBAAAPNrB7ALNST0qSbqDttiAR6AIAgAAHu3DZftkM6RLWzVS29gws8MB4AIUQQAAwGPlFZboi9Vli6MyCgR4DoogAADgsb5YfVD5RaVqGR2i/q2izA4HgItQBAEAAI9UarXpw2X7JEl39E2SlxeLowKegiIIAAB4pB+3ZujQidNqGOSr67qwOCrgSSiCAACAR3qfxVEBj0URBAAAPM76Aye0dv8J+Xl76ZbeLI4KeBqKIAAA4HHKR4FGdGqs6NAAk6MB4GoUQQAAwKMcPnlas88sjjqBttiAR6IIAgAAHuWjZftktRnq0yJSyY1ZHBXwRBRBAADAY+QXleqzlQckSXdeyigQ4KkogoA6yGoztDItW2uzLFqZli2rzTA7JADV4LXqfqavOai8olI1jwrW5a2jzQ4HgEl8zA4AQO3MSU3X5FlblZ5TKMlbH+9ao7jwAE0ckawhKXFmhwfgDF6r7sdqM/TB0rKGCCyOCng2RoKAOmROarru+XTdmTdVvzqaU6h7Pl2nOanpJkUG4Gy8Vt3TvK0ZOph9Wg2CfHV9l3izwwFgIoogoI6w2gxNnrVV1U2mKd82edZWptsAJimx2pSZV6gtR3L0xDepvFbd0PtL9kqSbu7ZVIF+LI4KeDKmwwF1xKq07CqfKp/NkJSeU6hVadnq3SLSdYEB9ZBhGCootir7VHHF1/FTxTrxmz+zTxXpREGJjucXKbewtGbHFq9VM2w8eFKr952Qr7dFt/ZuZnY4AExGEQTUEek5p2u0X2beuQslwFNZbYZOFhSfu6g589jx/LK/Hz9VrOJSW63PY7FIQb7eOlVsveC+vFZdq2Jx1I6NFRPG4qiAp6MIAtxcTkGJPlt9QO8s3lOj/ZfsylLv5pGK5j95ONDZXc4i07LVu2W0vE28qbywxFo2EpNfrOyCshGZ7FMlZ/4srvJ18nSJDDtmn/n5eCky2E8Rv/0K8lNEiJ8ig/3UMMhPkSFlfzYI8tOqtGyNfXfFBY994lSxHVcOexw5eVo/bC67D+sOFkcFIIogwG3tyzqlKUvTNH3tIRWc+VTZyyJd6DaC6WsP6Zv1hzW4faxu7tVUvZtHymKhAxLs5+wuZzabodzCkjPTy2r2dbrkwiMt1QkP9K22mIkIOvP92X8P9lOQn3etXz89kiIUFx6gozmF1d4XVG7SrK3afDhX/ze0raJC/e26HtTMR8v3qdRmqFfzCKU0CTc7HABugCIIcCOGYWjF3my9vyRN87dnVHxy3TY2VHf0S1KAj5ce/HxD2b5n/Vz5W7Tb+jbT5kM5WrP/hL7fnK7vN6erRVSwbu6ZqOu7xis80NeVl4N6oLzL2W/fzJd3OXtrXJcqhVBRqVUnTpXo+G9GZc6eenY8/8y2gmKdKCixq0mAr7dFEWeNxEQE+ysiyLfsz+CyPxsG+yoy2F8RwX5qEOQrX2/n9wPy9rJo4ohk3fPpOllU/Wu1b8tILd1zXF+vO6Qftx7Vo4PaaFyvRFNH1+qrU2ctjjqhX3OTowHgLiiCADdQXGrTd5uO6P0ladpyJLdi+4A2UZrQr7n6tvx1NMfPx+usT+XLxP7mU/lt6bmaunK/vll3WHuOndIz323VP+du1zWdmmhcr0R1iOeTUFxYTToSPvzFRn2x+qCyC8qmop04VaL8opo1CPitUH8fRZyZVvbbKWgNg89MPQv+9bEQfx+3HeUckhKnt8Z1Oe9rdf2BE3rq21SlHs7VxJlb9OWag3rmmhR1TWxoYuT1z1drDym3sFTNIoN0ZVsWRwVQhiIIMNGJU8WaunK/Pl6+X5l5RZKkAF8vXdclXnf0TVLL6JAqPzMkJU4Dk2O1fHemfvxlpQZd2rPK/Rnt4sL0t1Ed9H9D2+mb9Yc1dcV+bT+apy/WHNQXaw6qU3y4bu6VqBEdG9MmFue0cu/x83YklKTTJVYt3HGsynZvL0tFMVM+GtPwzOjM2cXM2ffT+PnUr1UbLvRa7dy0ob69t5+mrTqgF+ds15Yjubr+rWUa3S1efxnSVpEhTJG7WFaboSnli6P2Y3FUAL+iCAJMsDszXx8sTdOMdYdUWFLWgSo61F/j+zTTTT2aqmGw33l/3tvLop5JETq+zVDPpIhzTqEJ8ffRLb0SNa5nU63df0KfrtivHzYf1cZDOdr41SY99/023dA1Xjf3bKrmUVULLngewzC0/uBJzdp4RF+vO1SjnxnbI0ED2kRXFDORwf4KDfDhDacu/Fr19rLoll6JGpoSqxdmb9f0tYf05ZpDmrslQ48NaaPfdW/KFLmLMH9bhvYdL1B4oK9u6MriqAB+RREEuIhhGFqyO0vvL0nTz2d9ct6+cZjuvDRJwzo0dton4RaLRd2aRahbswg9NbxIX645pGmr9utg9mm9vyRN7y9JU7+WjTSuV1Nd1S5GPi64bwLuwzAMbTmSq1mbjui7jek6fLJm7djLjezUhPVuLlKjEH+9eGMnjemeoKe+3aJt6bn66zep+mL1QT17TYo6JTQwO8Q6qbwt9tgeTRXkx1seAL/iXwTAyQpLrJq54Yg+WJqm7UfzJJWtJXJVuxhN6JeknkkRLr2vITLEX/dc3kJ3X9Zci3Yd06fL92vBjkwt2Z2lJbuzFBPmr991b6qxPZoqNpw22/XZzow8zdp4RN9tSlda1qmK7UF+3hqYHKNhKXF6emaqMnKLqr0vyKKye1x6JEW4LOb6rluzCM26r68+XbFfL/24U5sO5WjUm0s1tkdT/XlQmwuOEuNXqYdztDItWz5eFo3vk2h2OADcDEUQ4CRZ+UX6dMV+fbpiv7Lyy9YDCfLz1o1d43V73yQ1axRsanxeXhYNaBOtAW2idehEgT5bdUBfrD6ojNwivTZ/l15fuFsD28VoXK9E9WkRydSmeiIt65S+23hEszYd0c6M/Irt/j5eurJdtIZ3bKwBbaIr7hWzyThvl7OJI5KZruVgPt5euq1vkq7uGKfnf9iuGesPa9rKA5q9OV3/N7StbuyawOuxBspHgYZ1jFNceKDJ0QBwNxRBgIPtOJqn95fs1f82HKlYcb5xeIDG92mm33VvqvAg92tTHd8wSH8e3FYPXtlac7cc1Scr9mtVWrbmbDmqOVuOKqlRsG7u2VQ3dI1XgyA+ia5rDp0o0Heb0vXdpiNKPfxr90Ffb4v6t47SiE6NdWW7GIX4V/0voSZdzuAc0aEBennMJRrTPUFPf7tFOzLy9JevN+uzVQf1t1EprHdzHkdzCjVr4xFJ0gQWRwVQDYogwAFsNkOLdh3TB0vS9MuurIrtnRIaaEK/JA1NiXXJ+iQXy8/HSyM6NdaITo21MyNPU1fs19frDist65T+9v02vTh3h0Z0aqxxvRLVKT7cbdsTQ8rILdT3m9I1a9MRrT9wsmK7t5dFfVs20vCOcRqcHFujorwmHQnhPD2bR+q7B/rpo2X79Mq8ndpw8KRGvr5E43ol6k8D27jlBytm+/jM4qg9mkWoY3wDs8MB4IYogoCLcLrYqhnrD+mDJWnac6zsngovizQkJVYT+iWpS9OGdbZQaB0TqsnXpOixIW317YYj+nTFfm1Nz9VXaw/pq7WHlNIkTON6JmrkJY254dhNHM8v0g+pR/XdxiNatS+7YrFdi0XqmRShEZ0aa0j7WLtaL9e0IyGcw9fbS3de2lwjOjXWc99v08yNR/Tx8v36flPZFLnru8QzRe6MguJSTS1fHPVSRoEAVI93LoAdMnML9fHy/Zq6cr9OFJRIKmtHPaZ7gm7r00wJEUEmR+g4wf4+uqlnU43tkaD1B0/q0xX79d2mdKUeztX/zdis537Ypuu7xGtcr6ZqGR1qdrgeJ6egRHO3HNWsTUe0bM9xWW2/3rnTpWkDjejUWFd3iFNMGE0u6oOYsAD9e2xn/a57gp6euUW7M/P15682lXWRG5WidnFhZodouq/XHVbO6RI1jQjSVe1izA4HgJuiCAJqIfVwjj5YkqZZm46oxFr2ZjO+YaBu75uk0d3iFRpQf6elWCwWdWnaUF2aNtRTw5I1fe1BTV15QPuPF+jDZfv04bJ96tU8QuN6JWpQcmy9W/jSneQXlWre1qP6bmO6Fu86VvFclKQOTcI1vGOchnWMU3zD+lOMo7I+LRvphwcu1QdL0/Tv+bu0Zv8JDf/PEt3aO1EPD2ytsHr8b9H52GyGPjjTEOGOvs0YsQRwTqYWQf/4xz80Y8YMbd++XYGBgerTp49eeOEFtWnTxsywgEpsNkPzt2fq/SV7tWJvdsX2bokNNaFfkga1j/W4/2gbBvvp95e10J39muuX3Vn6dMV+zd+WoRV7s7Vib7aiQv31u+4JGtujqRo3oCuTI5wutmrB9kx9t+mIFmzPVNGZphuS1CYmVCM6xWl4x8amdx2E6/j5eOkP/Vto5Jkpct9vTteUpfv03aZ0/fXqdrrmksZ1djquvRbuyFRa1imFBvjoxm4JZocDwI2ZWgQtWrRI9957r7p3767S0lI98cQTGjRokLZu3argYP4jh7lOFZXq63Vl9/vsO14gqey+iKs7xGlCvyRdwuKF8vIq6y7Wv3WUjpw8rc9XHdBnqw/qWF6R/rNgt95YuFtXnmmzfWnLRtyzUEtFpVYt3pmlWRuP6KdtGSootlY81rxRsIZ3jNPwTo3VOoZpiJ6scYNAvXFzF43ZeUyTZm7R3qxTeuiLDZq26oCevSZFbWI95/nx3i9lo0A39Wiq4Gq6HQJAOVP/hZgzZ06l7z/88ENFR0dr7dq1uuyyy6rsX1RUpKKioorvc3PLWr2WlJSopKTEucFeQPn5zY6jLnK33KXnFOqTFQf0xZpDyi0slSSFBfhoTLd43dKrqeLOLCBqdrzulreoYB/dP6C5/nBZM/20LVPTVh3UirQTmrc1Q/O2ZqhpRKB+1z1e13duoggTF3x0t7z9VonVpuV7s/X95qOaty1TeWeeg5LUpEGAhnWI1dUpsUqOC634lN9V1+LuuXNXrspb76QGmnlvb01Zuk9vLNqrVWnZuvrfv+i23k1134AW1bZAd2e1zdvW9Fwt33tc3l4W3dwj3mOfp7xO7Ufu7ONOeatNDBbDMKpbCNwUu3fvVqtWrbR582alpKRUeXzSpEmaPHlyle3Tpk1TUBBz33Fx9udLPx/x0objFtnOLAXZKMDQ5XE29Ygy5O9tcoB1UMZpaelRL606ZtFpa1lOfSyGOkca6htrU7OQss5lns5mSHtyLVqXZdHGbItOlf6alHBfQ5c0MtQl0qZE8oUayi6SvtnnpU3ZZffmhfsaGtXMps6RRr19Dn2620urj3mpc6RNt7W2XfgHANQ7BQUFuummm5STk6OwsPM3inGbIshms2nkyJE6efKklixZUu0+1Y0EJSQkKCsr64IX6mwlJSWaN2+eBg4cKF9fz7wh1V5m5s5qMzRvW6Y+XLZfa89aS6VnUkPd3jtRl7eJctv7ferSc66guFTfbz6qaasOKfXIr4t1to0N1U094jWyY5zLpq64S95sNkPrD57U96kZmpN6VMfyiyseiwj21dD2sbq6Q4y6NW3oNtMI3SV3dY2ZeVu085ie+X67DmSfliT1bh6hp4e1VcvoEJfGYY/a5C0zr0iXv7RYJVZDX93dU53iPXchWV6n9iN39nGnvOXm5qpRo0Y1KoLcZmz83nvvVWpq6jkLIEny9/eXv3/V9S18fX1NT3o5d4qlrnFl7vIKS/TlmkOasjRNh06UvTnw9bZoRMfGuqNfUp1aib0uPOfCfX11U68k3dQrSRsPntQnK/Zr1sYj2n40T0/P3KZ/zt2l67o00bheiS67v8WMvBmGoc2HczRr4xF9vyldR3IKKx4LD/TVkPaxGtGpsXo1j5CPGy+uWxeec+7IjLxd1b6x+rWO0TuL9+qNhbu1fG+2Rr65XBP6NdcDV7asE2t81SRvn63eqxKroa6JDdUtqZGLInNvvE7tR+7s4w55q8353eJfv/vuu0/fffedFi9erPj4eLPDQT12MLusnfMXqw8qv6jsXouGQb66uWeibumdyFoqLtApoYE6JTTQk8Pa6au1hzRt5QHtzTqlj5fv18fL96tHswiN652oIe3rR5ttwzC0/Wievtt0RLM2putAdkHFYyH+PhqUHKPhneLUr2VUvbheuJ8AX289cGUrjbqkiSbP2qL52zP19qI9mrnhsJ4anqwhKbF1uotcYYlVU1fulyTd2Y/FUQHUjKlFkGEYuv/++/XNN9/o559/VlIS/3jB8QzD0LoDJ/T+kjTNST2q8rUkW0QF645+Sbquc7wC/bjhx9UaBPnpzkuba0K/JC3bc1yfLN+vedsytGpftlbty1ajED+N7lbWZrsuLj67OzNf3206ou82pWt3Zn7F9gBfL13ZLkYjOjbW5W2iFODLcw+u0TQySO/f1l0/bc3QpFlbdOjEad0zdZ0ubdVIz1yToqQ62l59xrrDOlFQoviGgRrUPtbscADUEaYWQffee6+mTZumb7/9VqGhoTp69KgkKTw8XIGBrC2Ci1NitWl26lG9vyRNGw+erNh+aatGuqNfkvq3inKbey08mcViUd+WjdS3ZSMdzSnU56sP6LNVB5SRW6Q3f96jtxbt0YA20bqlV6Iua+2+92hJZSONs86M+GxL//XeJz9vL13eJkrDOzXWlW2jad0LU12VHKN+rRrpzYW79faivfplV5YGv7JYv7+sue4d0LJOfShksxl6f8leSdLtfZPc+t8HAO7F1P+J33rrLUnS5ZdfXmn7lClTdNttt7k+INQLOadL9PmqA/po2b6Key78fLw06pKy+33axprbRAPnFhseoIeuaq17B7TU/G0Z+nTFAS3ZnaUF2zO1YHum4hsG6qaeTTW6W4IahVS9P9AM6Tmn9f2mdM3alF6p2Pbxsqhfq0Ya0bGxBraPUVgA88vhPgJ8vfXIoDa6rku8Js7cokU7j+n1hbv1zfrDmjgiWQOTY+rEFLlFu45pz7FTCvH30ehuTKcHUHOmT4cDHGVf1ilNWZqm6WsPVSwqGRnsp1t6J2pcr0S3edOMC/P19tKQlDgNSYnT3mP5mrbygKavPaRDJ07rn3N26JV5O3V1hziN65WobokNXf5m7VhekWanpmvWxiNave9ExXYvi9S7RaSGd2ysIe1j1dDE9ZCAmmjWKFgf3t5dc7dk6NnvturwydP6/SdrNaBNlCaNbK/ESPeeIvf+mcVRf9c9QaF80ACgFpiTgTrNMAytTMvW+0vS9NO2DJXX1W1iQjWhX5JGXtKYey7quOZRIXpyeLIeHdxGszYe0acrD2jjwZP6dsMRfbvhiNrEhGpcr6Ya1bmJU98EnThVrDlbjuq7TUe0fM/xinvLJKl7s4Ya0amxhqbEKSqUYht1i8Vi0ZCUWF3WupHeWLhb7yzeq4U7jmnpK4t1T/8WuufyFm757+j2o7lasjtLXhZpfJ9mZocDoI6hCEKdVFxq0/ebj+i9X9K05ax1Zy5vE6U7+zVX35aRdWIqB2ouwNdbN3ZL0I3dEpR6OEefrtiv/204rB0ZeXrq2y16fvZ2jepc1ma7XZxjpjzmFpZo3pYMzdp0REt2Zan0rMqnU0IDjegYp6s7xKlxA+5hRN0X5OejPw9uWzZF7tstWrI7S6/N36Vv1h/WpJHJuqJtjNkhVlI+CjQ0Ja5ONk8BYC6KINQpJ04Va9qZ+30y88oWzg3w9dJ1XeJ1R99mahntmjVmYK6UJuF6/vqOevzqdpqx7pA+XbFfe46d0tSVBzR15QF1TWyocb2aamhKXKVPsK22spHDtVkWRaZlq3fL6Co3UhcUl+qnbZn6buMR/bzzmIpLf115vl1cmEZ0itPwDo3VNJI3XaifWkSF6JMJPfTD5qN69rutOpBdoDs+XKOByTF6eniyWxQcx/KK9O2GI5KkO2iLDcAOFEGoE/Ycy9cHS9L09bpDKiwpe1MaHeqv8X2a6aYeTbn3wkOFB/rq9r5Juq1PM63Ym61PV+zX3C1HtXb/Ca3df0LPfrdNN3aL1809ErU1PUeTZ21Vek6hJG99vGuN4sIDNHFEsi5vE62fdxzTrE1HtGBbpk6XWCvO0SIqWCM6Ndbwjo3VMjrEvIsFXMhisWhYxzhd3iZK/56/S+8vSdO8rRlavPOY7hvQUr/v31z+PuZNkftkxX4VW23q3LSBuiY2NC0OAHUXRRBMdb5P5g3D0NLdx/X+krL56eXaNw7ThH5JGt6xMYtLQlLZG7beLSLVu0WkMnML9cXqg/ps1QEdySnUfxft1X8X7a3259JzCvWHT9cpwMdLhWeN+DSNCCob8enYWG1jQ5laCY8V7O+jx69upxu6xuupb1O1Ym+2Xpq3U1+vO6TJ16Sof+sol8dUWGLV1BVli6NOYBQIgJ0ogmCaOanp1X4y//jQtiostemDJWnafjRPkmSxSFe2jdGEfknq1TyCN6U4p+iwAN1/ZSvdc3kLLdxxTB8v36dfdmWd92cKS22KC/PX8E6NNaJTY3VoEs5zDDhLq5hQfXZXL83ceETPfb9N+44XaPwHqzSkfayeGpGsJi68L+5/6w/r+KliNWkQqCEsjgrAThRBMMWc1HTd8+k6/bZJenpOoR74fEPF94G+3hrdLV639U2qs6uZwxw+3l4amByjEH+fCxZBkvTS6EvUp2UjF0QG1E0Wi0XXXNJEV7SN1qs/7dKHy/ZpzpajWrTzmO6/sqXu7Nfc6aPzhmHo/SVlDRFu69NMPt7MBgBgH4oguJzVZmjyrK1VCqCzeVmkRwe30c09EhUexNoPsF9mXmGN9juWX+TkSID6ITTAV08NT9aN3eL19P+2aNW+bP1zzg59tfaQnr0mRX2d+GHC4l1Z2pWZr2A/b43pkeC08wCo/yiC4HCFJVZl5RcpK79YWXlFZ/5e9v2xvCLtOZZ/ZgrcudkMqXNCQwogXLTo0ACH7gegTNvYMH1xdy99s/6w/v7DNu09dko3v7dSwzrG6alhyYoNd/xrqnwUaHT3BIWxOCqAi0ARhBopLLHqWN6vxUxWftFZ3xcpK+/Mtvwi5RWWOuScNf0EHzifHkkRigsP0NGcwmpHHy2SYsMD1CMpwtWhAXWexWLRdV3idWW7GL0yb6c+Xr5P329K18/bM/XgVa10e98k+TpoytrOjDwt3nlMXhbp9j40RABwcSiCPNjpYmtF4ZKVV/5n8VkjN7+O3uQX1a6w8fP2UqMQPzUK9VejEP+yv4eU/f1kQbH+vWD3BY/BJ/NwBG8viyaOSNY9n66TRapUCJW3Ppg4IrnKekEAai480FeTRrbXjd3i9dT/UrXuwEn9/Yftmr7mkJ65JkW9W0Re9Dk+ODMKNCg5lnW6AFw0iiAHqMkCjK5SUFyqrLxiHcsv1LHfFjRnjdZk5RXpVLH1wgc8i5+Pl6J+U9BEhfr/ptjxV1SIv8ICfc7ZXctqMzR97SE+mYfLDEmJ01vjupzVjbBM7Jl1goakxJkYHVB/tG8crq/+0EdfrTuk52dv167MfI19d4VGXdJYT1zdTtFh9n24lZVfpBnrD0uS7ryUUSAAF48i6CKdq82zI99YnSoqrTT17NhZ99r8dopaQS0LG38fr7LiJdRfUSF+Z4qas7/KCpyoUH+F+p+7sKkNPpmHGYakxGlgcqyW787Uj7+s1KBLe5r6gQVQX3l5WTS6W4IGJcfoXz/u0NSVB/S/DUf007ZMPTywtcb3Tqx1V7epKw6ouNSmTvHhLI4KwCEogi7Cudo8H80p1D2frtNb47pUWwgZhqH8otKKwqWioDkz9ey3ozdnr15fEwG+XmeN0pSPzvxmtObMCE6Igwqb2uKTeZjB28uinkkROr7NUM+kCAogwIkaBPnpb6M6aEy3pnry21RtPHhSz363VdPXHNSzo1LUvVnNRvuLSqz6ZMU+SdKES5uzhhcAh6AIstP52jyXb3vsq03afDhH2aeKK01NO5ZXpKKzVqeviUBf71+nnp0ZuSkvbiqN3oT6K9jPu078J8En8wBQ/3WID9c39/TRF2sO6oU527X9aJ5ufHu5ruvSRI8PbaeoUP/z/vyszUeVlV+suPAADU1hcVQAjkERZKdVadkXbPOcW1iqNxbuOefjwX7e1TYOKC9qokJ/3RbsXz9/VXwyDwD1n5eXRWN7NNWQ9rH659zt+nz1Qc1Yd1jztmbo0UFtNK5XYrX//huG9OGy/ZLKFkd1VKc5AKif76xdoKbtm/u1bKTuzSLU6KyCJirEX41C/RTkR/oBAJ6jYbCf/nFdR43ulqCnvk1V6uFcTZy5RV+uOahnrkmpuN+nvOHQ9we8tCMjX4G+Xvpdj6YmRw+gPuFduJ1q2r753gEtHdIaFACA+qJz04b69t5+mrbqgF6cs11bjuTq+reWaXS3eHVvFqGX5+08M9uibOTHYrFo+Z4s7hcF4DCMK9upfAHGc03eskiKo80zAADV8vay6JZeiVr46OW6sWu8JOnLNYf05682VZluXlBs1T2frtOc1HQzQgVQD1EE2am8zbOkKoUQbZ4BAKiZyBB/vXhjJ315dy/5XOD/zMmztspqq64lEQDUDkXQRShv8xwbXnlqXGx4wDnbYwMAgKqsNqn0PAWOISk9p1Cr0rJdFxSAeot7gi4SbZ4BALh4NW04VNP9AOB8KIIcgDbPAABcnJo2HKrpfgBwPkyHAwAApqPhEABXoggCAACmo+EQAFeiCAIAAG6BhkMAXIV7ggAAgNug4RAAV6AIAgAAboWGQwCcjelwAAAAADwKRRAAAAAAj0IRBAAAAMCjUAQBAAAA8CgUQQAAAAA8CkUQAAAAAI9CEQQAAADAo1AEAQAAAPAoFEEAAAAAPApFEAAAAACPQhEEAAAAwKNQBAEAAADwKBRBAAAAADyKj9kBXAzDMCRJubm5JkcilZSUqKCgQLm5ufL19TU7nDqF3NmHvNmHvNmP3NmHvNmHvNmHvNmP3NnHnfJWXhOU1wjnU6eLoLy8PElSQkKCyZEAAAAAcAd5eXkKDw8/7z4Woyalkpuy2Ww6cuSIQkNDZbFYTI0lNzdXCQkJOnjwoMLCwkyNpa4hd/Yhb/Yhb/Yjd/Yhb/Yhb/Yhb/Yjd/Zxp7wZhqG8vDw1btxYXl7nv+unTo8EeXl5KT4+3uwwKgkLCzP9CVBXkTv7kDf7kDf7kTv7kDf7kDf7kDf7kTv7uEveLjQCVI7GCAAAAAA8CkUQAAAAAI9CEeQg/v7+mjhxovz9/c0Opc4hd/Yhb/Yhb/Yjd/Yhb/Yhb/Yhb/Yjd/apq3mr040RAAAAAKC2GAkCAAAA4FEoggAAAAB4FIogAAAAAB6FIggAAACAR6EIOss//vEPde/eXaGhoYqOjtaoUaO0Y8eOSvsUFhbq3nvvVWRkpEJCQnT99dcrIyOj0j4PPPCAunbtKn9/f11yySXnPefu3bsVGhqqBg0aOPhqXMdVedu3b58sFkuVrxUrVjjz8pzGlc83wzD0r3/9S61bt5a/v7+aNGmi5557zlmX5nSuyt2kSZOqfc4FBwc78/KcxpXPublz56pXr14KDQ1VVFSUrr/+eu3bt89JV+Zcrszbl19+qUsuuURBQUFKTEzUiy++6KzLcglH5G7jxo0aO3asEhISFBgYqHbt2um1116rcq6ff/5ZXbp0kb+/v1q2bKkPP/zQ2ZfnNK7KW3p6um666Sa1bt1aXl5eeuihh1xxeU7jqrzNmDFDAwcOVFRUlMLCwtS7d2/NnTvXJdfoDK7K25IlS9S3b19FRkYqMDBQbdu21SuvvOKSa6wORdBZFi1apHvvvVcrVqzQvHnzVFJSokGDBunUqVMV+zz88MOaNWuWpk+frkWLFunIkSO67rrrqhzrjjvu0JgxY857vpKSEo0dO1aXXnqpw6/FlVydt59++knp6ekVX127dnX4NbmCK/P24IMP6r333tO//vUvbd++XTNnzlSPHj2ccl2u4KrcPfroo5Wea+np6UpOTtaNN97otGtzJlflLS0tTddcc42uuOIKbdiwQXPnzlVWVla1x6kLXJW32bNn6+abb9Yf/vAHpaam6s0339Qrr7yi119/3WnX5myOyN3atWsVHR2tTz/9VFu2bNFf//pXPf7445XykpaWpmHDhmnAgAHasGGDHnroId1555119o2pq/JWVFSkqKgoPfnkk+rUqZNLr9EZXJW3xYsXa+DAgfrhhx+0du1aDRgwQCNGjND69etder2O4qq8BQcH67777tPixYu1bds2Pfnkk3ryySf1zjvvuPR6Kxg4p8zMTEOSsWjRIsMwDOPkyZOGr6+vMX369Ip9tm3bZkgyli9fXuXnJ06caHTq1Omcx3/ssceMcePGGVOmTDHCw8MdHb5pnJW3tLQ0Q5Kxfv16Z4VuKmflbevWrYaPj4+xfft2p8VuNme/Vstt2LDBkGQsXrzYYbGbyVl5mz59uuHj42NYrdaKbTNnzjQsFotRXFzs+AtxMWflbezYscYNN9xQadu///1vIz4+3rDZbI69CJNcbO7K/fGPfzQGDBhQ8f1jjz1mtG/fvtI+Y8aMMQYPHuzgKzCHs/J2tv79+xsPPvigQ+M2myvyVi45OdmYPHmyYwI3mSvzdu211xrjxo1zTOC1xEjQeeTk5EiSIiIiJJVVuSUlJbrqqqsq9mnbtq2aNm2q5cuX1+rYCxYs0PTp0/XGG284LmA34cy8SdLIkSMVHR2tfv36aebMmY4J2g04K2+zZs1S8+bN9d133ykpKUnNmjXTnXfeqezsbMdegImc/Zwr995776l169Z1fvS2nLPy1rVrV3l5eWnKlCmyWq3KycnRJ598oquuukq+vr6OvQgTOCtvRUVFCggIqLQtMDBQhw4d0v79+x0QufkclbucnJyKY0jS8uXLKx1DkgYPHnxRr3d34qy81XeuypvNZlNeXl69ya2r8rZ+/XotW7ZM/fv3d1DktUMRdA42m00PPfSQ+vbtq5SUFEnS0aNH5efnV+X+nZiYGB09erTGxz5+/Lhuu+02ffjhhwoLC3Nk2KZzZt5CQkL00ksvafr06fr+++/Vr18/jRo1ql4UQs7M2969e7V//35Nnz5dH3/8sT788EOtXbtWN9xwgyMvwTTOzN3ZCgsLNXXqVE2YMOFiQ3YLzsxbUlKSfvzxRz3xxBPy9/dXgwYNdOjQIX355ZeOvARTODNvgwcP1owZMzR//nzZbDbt3LlTL730kqSyezfqOkflbtmyZfriiy/0+9//vmLb0aNHFRMTU+UYubm5On36tGMvxMWcmbf6zJV5+9e//qX8/HyNHj3aYfGbxRV5i4+Pl7+/v7p166Z7771Xd955p8OvoyZ8TDlrHXDvvfcqNTVVS5Yscfix77rrLt1000267LLLHH5sszkzb40aNdIjjzxS8X337t115MgRvfjiixo5cqTDz+dKzsybzWZTUVGRPv74Y7Vu3VqS9P7776tr167asWOH2rRp4/BzupIzc3e2b775Rnl5eRo/frxTz+Mqzszb0aNHddddd2n8+PEaO3as8vLy9PTTT+uGG27QvHnzZLFYHH5OV3H2/w179uzR8OHDVVJSorCwMD344IOaNGmSvLzq/meWjshdamqqrrnmGk2cOFGDBg1yYHTui7zZx1V5mzZtmiZPnqxvv/1W0dHRdp/LXbgib7/88ovy8/O1YsUK/d///Z9atmypsWPHXkzYdqn7/6o6wX333afvvvtOCxcuVHx8fMX22NhYFRcX6+TJk5X2z8jIUGxsbI2Pv2DBAv3rX/+Sj4+PfHx8NGHCBOXk5MjHx0cffPCBoy7D5Zydt+r07NlTu3fvvqhjmM3ZeYuLi5OPj09FASRJ7dq1kyQdOHDg4oI3mSufc++9956GDx9e5dPmusjZeXvjjTcUHh6uf/7zn+rcubMuu+wyffrpp5o/f75WrlzpqMtwOWfnzWKx6IUXXlB+fr7279+vo0ePVjQwad68uUOuwSyOyN3WrVt15ZVX6ve//72efPLJSo/FxsZW6caXkZGhsLAwBQYGOvZiXMjZeauvXJW3zz//XHfeeae+/PLLKtMx6yJX5S0pKUkdOnTQXXfdpYcffliTJk1y9KXUCEXQWQzD0H333advvvlGCxYsUFJSUqXHu3btKl9fX82fP79i244dO3TgwAH17t27xudZvny5NmzYUPH1zDPPKDQ0VBs2bNC1117rsOtxFVflrTobNmxQXFzcRR3DLK7KW9++fVVaWqo9e/ZUbNu5c6ckKTEx8SKvwhyufs6lpaVp4cKFdX4qnKvyVlBQUGXkwtvbW1LZyGRd4+rnm7e3t5o0aSI/Pz999tln6t27t6Kioi76OszgqNxt2bJFAwYM0Pjx46tt79+7d+9Kx5CkefPmXfT/MWZxVd7qG1fm7bPPPtPtt9+uzz77TMOGDXPOBbmImc+38tkqpjClHYObuueee4zw8HDj559/NtLT0yu+CgoKKvb5wx/+YDRt2tRYsGCBsWbNGqN3795G7969Kx1n165dxvr16427777baN26tbF+/Xpj/fr1RlFRUbXnrevd4VyVtw8//NCYNm2asW3bNmPbtm3Gc889Z3h5eRkffPCBS6/XUVyVN6vVanTp0sW47LLLjHXr1hlr1qwxevbsaQwcONCl1+tIrn6tPvnkk0bjxo2N0tJSl1yfs7gqb/PnzzcsFosxefJkY+fOncbatWuNwYMHG4mJiZXOVVe4Km/Hjh0z3nrrLWPbtm3G+vXrjQceeMAICAgwVq5c6dLrdSRH5G7z5s1GVFSUMW7cuErHyMzMrNhn7969RlBQkPHnP//Z2LZtm/HGG28Y3t7expw5c1x6vY7iqrwZhlHxPOzatatx0003GevXrze2bNnismt1JFflberUqYaPj4/xxhtvVNrn5MmTLr1eR3FV3l5//XVj5syZxs6dO42dO3ca7733nhEaGmr89a9/den1lqMIOoukar+mTJlSsc/p06eNP/7xj0bDhg2NoKAg49prrzXS09MrHad///7VHictLa3a89b1IshVefvwww+Ndu3aGUFBQUZYWJjRo0ePSu0a6xpXPt8OHz5sXHfddUZISIgRExNj3Hbbbcbx48dddKWO58rcWa1WIz4+3njiiSdcdHXO48q8ffbZZ0bnzp2N4OBgIyoqyhg5cqSxbds2F12pY7kqb8eOHTN69eplBAcHG0FBQcaVV15prFixwoVX6niOyN3EiROrPUZiYmKlcy1cuNC45JJLDD8/P6N58+aVzlHXuDJvNdmnrnBV3s71Wh4/frzrLtaBXJW3f//730b79u0r3sd17tzZePPNNystp+BKFsMwDAEAAACAh+CeIAAAAAAehSIIAAAAgEehCAIAAADgUSiCAAAAAHgUiiAAAAAAHoUiCAAAAIBHoQgCAAAA4FEoggAAAAB4FIogAAAAAB6FIggA4DYMw9BVV12lwYMHV3nszTffVIMGDXTo0CETIgMA1CcUQQAAt2GxWDRlyhStXLlS//3vfyu2p6Wl6bHHHtN//vMfxcfHO/ScJSUlDj0eAMD9UQQBANxKQkKCXnvtNT366KNKS0uTYRiaMGGCBg0apM6dO2vo0KEKCQlRTEyMbrnlFmVlZVX87Jw5c9SvXz81aNBAkZGRGj58uPbs2VPx+L59+2SxWPTFF1+of//+CggI0NSpU824TACAiSyGYRhmBwEAwG+NGjVKOTk5uu666/Tss89qy5Ytat++ve68807deuutOn36tP7yl7+otLRUCxYskCR9/fXXslgs6tixo/Lz8/X0009r37592rBhg7y8vLRv3z4lJSWpWbNmeumll9S5c2cFBAQoLi7O5KsFALgSRRAAwC1lZmaqffv2ys7O1tdff63U1FT98ssvmjt3bsU+hw4dUkJCgnbs2KHWrVtXOUZWVpaioqK0efNmpaSkVBRBr776qh588EFXXg4AwI0wHQ4A4Jaio6N19913q127dho1apQ2btyohQsXKiQkpOKrbdu2klQx5W3Xrl0aO3asmjdvrrCwMDVr1kySdODAgUrH7tatm0uvBQDgXnzMDgAAgHPx8fGRj0/Zf1X5+fkaMWKEXnjhhSr7lU9nGzFihBITE/Xuu++qcePGstlsSklJUXFxcaX9g4ODnR88AMBtUQQBAOqELl266Ouvv1azZs0qCqOzHT9+XDt27NC7776rSy+9VJK0ZMkSV4cJAKgDmA4HAKgT7r33XmVnZ2vs2LFavXq19uzZo7lz5+r222+X1WpVw4YNFRkZqXfeeUe7d+/WggUL9Mgjj5gdNgDADVEEAQDqhMaNG2vp0qWyWq0aNGiQOnTooIceekgNGjSQl5eXvLy89Pnnn2vt2rVKSUnRww8/rBdffNHssAEAbojucAAAAAA8CiNBAAAAADwKRRAAAAAAj0IRBAAAAMCjUAQBAAAA8CgUQQAAAAA8CkUQAAAAAI9CEQQAAADAo1AEAQAAAPAoFEEAAAAAPApFEAAAAACPQhEEAAAAwKP8P6KQ14ErFH3sAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 1000x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# Read the CSV file\n",
"df = pd.read_csv('/tmp/tmpco0s0o4_/LOdZoVp1inflation.csv')\n",
"\n",
"# Extract the year and inflation rate from the CSV file\n",
"df['Year'] = pd.to_datetime(df['Year'], format='%Y')\n",
"df = df.rename(columns={'Jan': 'Jan Rate', 'Feb': 'Feb Rate', 'Mar': 'Mar Rate', 'Apr': 'Apr Rate', 'May': 'May Rate', 'Jun': 'Jun Rate', 'Jul': 'Jul Rate', 'Aug': 'Aug Rate', 'Sep': 'Sep Rate', 'Oct': 'Oct Rate', 'Nov': 'Nov Rate', 'Dec': 'Dec Rate'})\n",
"\n",
"# Calculate the average yearly inflation rate\n",
"df['Yearly Inflation'] = df[['Jan Rate', 'Feb Rate', 'Mar Rate', 'Apr Rate', 'May Rate', 'Jun Rate', 'Jul Rate', 'Aug Rate', 'Sep Rate', 'Oct Rate', 'Nov Rate', 'Dec Rate']].mean(axis=1)\n",
"\n",
"# Plot the average yearly inflation rate as a time series\n",
"plt.figure(figsize=(10, 6))\n",
"plt.plot(df['Year'], df['Yearly Inflation'], marker='o')\n",
"plt.title('Average Yearly Inflation Rate')\n",
"plt.xlabel('Year')\n",
"plt.ylabel('Inflation Rate (%)')\n",
"plt.grid(True)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "FJ85DUhgBZd7",
"metadata": {
"id": "FJ85DUhgBZd7"
},
"source": [
"## 3. Llama Stack Agent Evaluations\n"
]
},
{
"cell_type": "markdown",
"id": "ydeBDpDT5VHd",
"metadata": {
"id": "ydeBDpDT5VHd"
},
"source": [
"#### 3.1. Online Evaluation Dataset Collection Using Telemetry\n",
"\n",
"- Llama Stack offers built-in telemetry to collect traces and data about your agentic application.\n",
"- In this example, we will show how to build an Agent with Llama Stack, and query the agent's traces into an online dataset that can be used for evaluation. "
]
},
{
"cell_type": "markdown",
"id": "_JueJAKyJR5m",
"metadata": {
"id": "_JueJAKyJR5m"
},
"source": [
"##### 🚧 Patches 🚧\n",
"- The following cells are temporary patches to get `telemetry` working."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "klPkK1t7CzIY",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "klPkK1t7CzIY",
"outputId": "ab0c1490-7fa6-446c-8e35-7b42f57e8a04"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found existing installation: llama_stack 0.0.61\n",
"Uninstalling llama_stack-0.0.61:\n",
" Would remove:\n",
" /usr/local/bin/install-wheel-from-presigned\n",
" /usr/local/bin/llama\n",
" /usr/local/lib/python3.10/dist-packages/llama_stack-0.0.61.dist-info/*\n",
" /usr/local/lib/python3.10/dist-packages/llama_stack/*\n",
"Proceed (Y/n)? Y\n",
" Successfully uninstalled llama_stack-0.0.61\n",
"Collecting git+https://github.com/meta-llama/llama-stack.git@main\n",
" Cloning https://github.com/meta-llama/llama-stack.git (to revision main) to /tmp/pip-req-build-oryyzdm1\n",
" Running command git clone --filter=blob:none --quiet https://github.com/meta-llama/llama-stack.git /tmp/pip-req-build-oryyzdm1\n",
" Resolved https://github.com/meta-llama/llama-stack.git to commit 53b3a1e345c46d7d37c1af3d675092a4cbfe85f9\n",
" Running command git submodule update --init --recursive -q\n",
" Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
" Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
" Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
"Requirement already satisfied: blobfile in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (3.0.0)\n",
"Requirement already satisfied: fire in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (0.7.0)\n",
"Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (0.28.1)\n",
"Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (0.26.5)\n",
"Requirement already satisfied: llama-models>=0.0.61 in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (0.0.61)\n",
"Requirement already satisfied: llama-stack-client>=0.0.61 in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (0.0.61)\n",
"Requirement already satisfied: prompt-toolkit in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (3.0.48)\n",
"Requirement already satisfied: python-dotenv in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (1.0.1)\n",
"Requirement already satisfied: pydantic>=2 in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (2.10.3)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (2.32.3)\n",
"Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (13.9.4)\n",
"Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (75.1.0)\n",
"Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from llama_stack==0.0.61) (2.5.0)\n",
"Requirement already satisfied: PyYAML in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama_stack==0.0.61) (6.0.2)\n",
"Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama_stack==0.0.61) (3.1.4)\n",
"Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama_stack==0.0.61) (0.8.0)\n",
"Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from llama-models>=0.0.61->llama_stack==0.0.61) (10.4.0)\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (3.7.1)\n",
"Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (8.1.7)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (1.9.0)\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (2.2.2)\n",
"Requirement already satisfied: pyaml in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (24.12.1)\n",
"Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (1.3.1)\n",
"Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (4.66.6)\n",
"Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from llama-stack-client>=0.0.61->llama_stack==0.0.61) (4.12.2)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->llama_stack==0.0.61) (2024.8.30)\n",
"Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->llama_stack==0.0.61) (1.0.7)\n",
"Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->llama_stack==0.0.61) (3.10)\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx->llama_stack==0.0.61) (0.14.0)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2->llama_stack==0.0.61) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2->llama_stack==0.0.61) (2.27.1)\n",
"Requirement already satisfied: pycryptodomex>=3.8 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama_stack==0.0.61) (3.21.0)\n",
"Requirement already satisfied: urllib3<3,>=1.25.3 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama_stack==0.0.61) (2.2.3)\n",
"Requirement already satisfied: lxml>=4.9 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama_stack==0.0.61) (5.3.0)\n",
"Requirement already satisfied: filelock>=3.0 in /usr/local/lib/python3.10/dist-packages (from blobfile->llama_stack==0.0.61) (3.16.1)\n",
"Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->llama_stack==0.0.61) (2024.9.0)\n",
"Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->llama_stack==0.0.61) (24.2)\n",
"Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit->llama_stack==0.0.61) (0.2.13)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->llama_stack==0.0.61) (3.4.0)\n",
"Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->llama_stack==0.0.61) (3.0.0)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->llama_stack==0.0.61) (2.18.0)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->llama-stack-client>=0.0.61->llama_stack==0.0.61) (1.2.2)\n",
"Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->llama_stack==0.0.61) (0.1.2)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->llama-models>=0.0.61->llama_stack==0.0.61) (3.0.2)\n",
"Requirement already satisfied: numpy>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama_stack==0.0.61) (1.26.4)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama_stack==0.0.61) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama_stack==0.0.61) (2024.2)\n",
"Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-stack-client>=0.0.61->llama_stack==0.0.61) (2024.2)\n",
"Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->llama-models>=0.0.61->llama_stack==0.0.61) (2024.9.11)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->llama-stack-client>=0.0.61->llama_stack==0.0.61) (1.17.0)\n",
"Building wheels for collected packages: llama_stack\n",
" Building wheel for llama_stack (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for llama_stack: filename=llama_stack-0.0.61-py3-none-any.whl size=464145 sha256=da71747aceef9aec43553f66c43095486d1a920e47bb0e47e2729a8e4328fff6\n",
" Stored in directory: /tmp/pip-ephem-wheel-cache-jquw5j7f/wheels/74/e4/3b/079983408fa9323c1f2807e404ee78b468c74bec381eb70d4f\n",
"Successfully built llama_stack\n",
"Installing collected packages: llama_stack\n",
"Successfully installed llama_stack-0.0.61\n"
]
},
{
"data": {
"application/vnd.colab-display-data+json": {
"id": "7701cb0c982f4250a46721fededf9647",
"pip_warning": {
"packages": [
"llama_stack"
]
}
}
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# need to install on latest main\n",
"!pip uninstall llama-stack\n",
"!pip install git+https://github.com/meta-llama/llama-stack.git@main"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9jJ75JlnETTH",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9jJ75JlnETTH",
"outputId": "76bd3912-f814-428c-88e1-c1113af77856"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Removed handler StreamHandler from root logger\n"
]
}
],
"source": [
"# disable logging for clean server logs\n",
"import logging\n",
"def remove_root_handlers():\n",
" root_logger = logging.getLogger()\n",
" for handler in root_logger.handlers[:]:\n",
" root_logger.removeHandler(handler)\n",
" print(f\"Removed handler {handler.__class__.__name__} from root logger\")\n",
"\n",
"\n",
"remove_root_handlers()"
]
},
{
"cell_type": "markdown",
"id": "_t_tcWq0JcJ4",
"metadata": {
"id": "_t_tcWq0JcJ4"
},
"source": [
"##### 3.1.1. Building a Search Agent"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4iCO59kP20Zs",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4iCO59kP20Zs",
"outputId": "f6179de6-054d-4452-a893-8d9b64c5a0d1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"inference> Let me check the latest sports news.\n",
"inference> bravy_search.call(query=\"Bill Cosby South Park episode\")\n",
"CustomTool> Unknown tool `bravy_search` was called.\n",
"inference> brave_search.call(query=\"Andrew Tate kickboxing name\")\n",
"tool_execution> Tool:brave_search Args:{'query': 'Andrew Tate kickboxing name'}\n",
"tool_execution> Tool:brave_search Response:{\"query\": \"Andrew Tate kickboxing name\", \"top_k\": [{\"title\": \"Andrew Tate kickboxing record: How many championships ... - FirstSportz\", \"url\": \"https://firstsportz.com/mma-how-many-championships-does-andrew-tate-have/\", \"content\": \"Andrew Tate's Kickboxing career. During his kickboxing career, he used the nickname \\\"King Cobra,\\\" which he currently uses as his Twitter name. Tate had an unorthodox style of movement inside the ring. He kept his hands down most of the time and relied on quick jabs and an overhand right to land significant strikes.\", \"score\": 0.9996244, \"raw_content\": null}, {\"title\": \"Andrew Tate: Kickboxing Record, Facts, Height, Weight, Age, Biography\", \"url\": \"https://www.lowkickmma.com/andrew-tate-kickboxing-record-facts-height-weight-age-biography/\", \"content\": \"Birth Name: Emory Andrew Tate III: Date of Birth: 1 December 1986: Place of Birth: Washington, D.C., U.S. ... In his professional kickboxing career, Andrew Tate won 32 of his fights by knockout.\", \"score\": 0.99909246, \"raw_content\": null}, {\"title\": \"Who is Andrew Tate? MMA, kickboxing record and controversies of fighter ...\", \"url\": \"https://www.sportingnews.com/us/kickboxing/news/andrew-tate-mma-kickboxing-record-controversies/u50waalc9cfz7krjg9wnyb7p\", \"content\": \"Andrew Tate kickboxing record After launching his career as a 20-year-old in 2007, Tate built a formidable kickboxing record that included 76 wins across 85 fights in more than 13 years in the ring.\", \"score\": 0.9976586, \"raw_content\": null}, {\"title\": \"About Andrew Tate: A Journey from Champion to Controversy\", \"url\": \"https://reachmorpheus.com/andrew-tate/\", \"content\": \"Andrew Tate's kickboxing career, beginning in 2005, is a tale of determination and skill. He quickly made a name for himself in the sport, rising through the ranks with his unique fighting style and strategic approach, honed by his chess-playing background.\", \"score\": 0.99701905, \"raw_content\": null}, {\"title\": \"Andrew Tate Bio, Wiki, Net Worth, Age, Family, MMA Career - Next Biography\", \"url\": \"https://www.nextbiography.com/andrew-tate/\", \"content\": \"Andrew Tate Age. Andrew Tate is 36 years old as of 2023, born on December 1, 1986, in Washington, DC. By his mid-thirties, Andrew Tate has become an esteemed figure in the world of kickboxing, showcasing remarkable expertise and experience in the sport. Early Life of Andrew Tate. Andrew Tate was born on 01 December 1986 to an African-American\", \"score\": 0.99368566, \"raw_content\": null}]}\n",
"shield_call> No Violation\n",
"inference> Andrew Tate's kickboxing name is \"King Cobra.\"\n"
]
}
],
"source": [
"from llama_stack_client.lib.agents.agent import Agent\n",
"from llama_stack_client.lib.agents.event_logger import EventLogger\n",
"from llama_stack_client.types.agent_create_params import AgentConfig\n",
"from google.colab import userdata\n",
"\n",
"agent_config = AgentConfig(\n",
" model=\"meta-llama/Llama-3.1-405B-Instruct\",\n",
" instructions=\"You are a helpful assistant. Use search tool to answer the questions. \",\n",
" tools=(\n",
" [\n",
" {\n",
" \"type\": \"brave_search\",\n",
" \"engine\": \"tavily\",\n",
" \"api_key\": userdata.get(\"TAVILY_SEARCH_API_KEY\")\n",
" }\n",
" ]\n",
" ),\n",
" input_shields=[],\n",
" output_shields=[],\n",
" enable_session_persistence=False,\n",
")\n",
"agent = Agent(client, agent_config)\n",
"user_prompts = [\n",
" \"Which teams played in the NBA western conference finals of 2024\",\n",
" \"In which episode and season of South Park does Bill Cosby (BSM-471) first appear? Give me the number and title.\",\n",
" \"What is the British-American kickboxer Andrew Tate's kickboxing name?\",\n",
"]\n",
"\n",
"session_id = agent.create_session(\"test-session\")\n",
"\n",
"for prompt in user_prompts:\n",
" response = agent.create_turn(\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": prompt,\n",
" }\n",
" ],\n",
" session_id=session_id,\n",
" )\n",
"\n",
" for log in EventLogger().log(response):\n",
" log.print()"
]
},
{
"cell_type": "markdown",
"id": "ekOS2kM4P0LM",
"metadata": {
"id": "ekOS2kM4P0LM"
},
"source": [
"##### 3.1.2 Query Telemetry"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "agkWgToGAsuA",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 760
},
"id": "agkWgToGAsuA",
"outputId": "647cd5d2-7610-4fd6-ef66-c3f2f782a1b0"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Getting traces for session_id=ac651ce8-2281-47f2-8814-ef947c066e40\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input'</span>: <span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null}'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">]</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'output'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'content: Let me check the latest sports news. tool_calls: []'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input'</span>: <span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"Let me check the latest sports news.\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby (BSM-471) first appear? Give me the number and title.\",\"context\":null}'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">]</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'output'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"content: tool_calls: [ToolCall(call_id='19bd3554-e670-4856-89d0-c63f5b016245', tool_name='bravy_search', arguments={'query': 'Bill Cosby South Park episode'})]\"</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input'</span>: <span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"Let me check the latest sports news.\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby (BSM-471) first appear? Give me the number and title.\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[{\"call_id\":\"19bd3554-e670-4856-89d0-c63f5b016245\",\"tool_name\":\"bravy_search\",\"arguments\":{\"query\":\"Bill Cosby South Park episode\"}}]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"What is the British-American kickboxer Andrew Tate\\'s kickboxing name?\",\"context\":null}'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">]</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'output'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"content: tool_calls: [ToolCall(call_id='526045a7-5f51-40fb-ba97-5ad29610e511', tool_name=&lt;BuiltinTool.brave_search: 'brave_search'&gt;, arguments={'query': 'Andrew Tate kickboxing name'})]\"</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[{\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"arguments\":{\"query\":\"Andrew Tate kickboxing name\"}}]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'output'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"ipython\",\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"content\":\"{\\\\\"query\\\\\": \\\\\"Andrew Tate kickboxing name\\\\\", \\\\\"top_k\\\\\": [{\\\\\"title\\\\\": \\\\\"Andrew Tate kickboxing record: How many championships ... - FirstSportz\\\\\", \\\\\"url\\\\\": \\\\\"https://firstsportz.com/mma-how-many-championships-does-andrew-tate-have/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s Kickboxing career. During his kickboxing career, he used the nickname \\\\\\\\\\\\\"King Cobra,\\\\\\\\\\\\\" which he currently uses as his Twitter name. Tate had an unorthodox style of movement inside the ring. He kept his hands down most of the time and relied on quick jabs and an overhand right to land significant strikes.\\\\\", \\\\\"score\\\\\": 0.9996244, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"Andrew Tate: Kickboxing Record, Facts, Height, Weight, Age, Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.lowkickmma.com/andrew-tate-kickboxing-record-facts-height-weight-age-biography/\\\\\", \\\\\"content\\\\\": \\\\\"Birth Name: Emory Andrew Tate III: Date of Birth: 1 December 1986: Place of Birth: Washington, D.C., U.S. ... In his professional kickboxing career, Andrew Tate won 32 of his fights by knockout.\\\\\", \\\\\"score\\\\\": 0.99909246, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"Who is Andrew Tate? MMA, kickboxing record and controversies of fighter ...\\\\\", \\\\\"url\\\\\": \\\\\"https://www.sportingnews.com/us/kickboxing/news/andrew-tate-mma-kickboxing-record-controversies/u50waalc9cfz7krjg9wnyb7p\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate kickboxing record After launching his career as a 20-year-old in 2007, Tate built a formidable kickboxing record that included 76 wins across 85 fights in more than 13 years in the ring.\\\\\", \\\\\"score\\\\\": 0.9976586, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"About Andrew Tate: A Journey from Champion to Controversy\\\\\", \\\\\"url\\\\\": \\\\\"https://reachmorpheus.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s kickboxing career, beginning in 2005, is a tale of determination and skill. He quickly made a name for himself in the sport, rising through the ranks with his unique fighting style and strategic approach, honed by his chess-playing background.\\\\\", \\\\\"score\\\\\": 0.99701905, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"Andrew Tate Bio, Wiki, Net Worth, Age, Family, MMA Career - Next Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nextbiography.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate Age. Andrew Tate is 36 years old as of 2023, born on December 1, 1986, in Washington, DC. By his mid-thirties, Andrew Tate has become an esteemed figure in the world of kickboxing, showcasing remarkable expertise and experience in the sport. Early Life of Andrew Tate. Andrew Tate was born on 01 December 1986 to an African-American\\\\\", \\\\\"score\\\\\": 0.99368566, \\\\\"raw_content\\\\\": null}]}\"}'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input'</span>: <span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"Let me check the latest sports news.\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby (BSM-471) first appear? Give me the number and title.\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[{\"call_id\":\"19bd3554-e670-4856-89d0-c63f5b016245\",\"tool_name\":\"bravy_search\",\"arguments\":{\"query\":\"Bill Cosby South Park episode\"}}]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"What is the British-American kickboxer Andrew Tate\\'s kickboxing name?\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":[{\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"arguments\":{\"query\":\"Andrew Tate kickboxing name\"}}]}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"ipython\",\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"content\":\"{\\\\\"query\\\\\": \\\\\"Andrew Tate kickboxing name\\\\\", \\\\\"top_k\\\\\": [{\\\\\"title\\\\\": \\\\\"Andrew Tate kickboxing record: How many championships ... - FirstSportz\\\\\", \\\\\"url\\\\\": \\\\\"https://firstsportz.com/mma-how-many-championships-does-andrew-tate-have/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s Kickboxing career. During his kickboxing career, he used the nickname \\\\\\\\\\\\\"King Cobra,\\\\\\\\\\\\\" which he currently uses as his Twitter name. Tate had an unorthodox style of movement inside the ring. He kept his hands down most of the time and relied on quick jabs and an overhand right to land significant strikes.\\\\\", \\\\\"score\\\\\": 0.9996244, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"Andrew Tate: Kickboxing Record, Facts, Height, Weight, Age, Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.lowkickmma.com/andrew-tate-kickboxing-record-facts-height-weight-age-biography/\\\\\", \\\\\"content\\\\\": \\\\\"Birth Name: Emory Andrew Tate III: Date of Birth: 1 December 1986: Place of Birth: Washington, D.C., U.S. ... In his professional kickboxing career, Andrew Tate won 32 of his fights by knockout.\\\\\", \\\\\"score\\\\\": 0.99909246, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"Who is Andrew Tate? MMA, kickboxing record and controversies of fighter ...\\\\\", \\\\\"url\\\\\": \\\\\"https://www.sportingnews.com/us/kickboxing/news/andrew-tate-mma-kickboxing-record-controversies/u50waalc9cfz7krjg9wnyb7p\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate kickboxing record After launching his career as a 20-year-old in 2007, Tate built a formidable kickboxing record that included 76 wins across 85 fights in more than 13 years in the ring.\\\\\", \\\\\"score\\\\\": 0.9976586, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"About Andrew Tate: A Journey from Champion to Controversy\\\\\", \\\\\"url\\\\\": \\\\\"https://reachmorpheus.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s kickboxing career, beginning in 2005, is a tale of determination and skill. He quickly made a name for himself in the sport, rising through the ranks with his unique fighting style and strategic approach, honed by his chess-playing background.\\\\\", \\\\\"score\\\\\": 0.99701905, \\\\\"raw_content\\\\\": null}, {\\\\\"title\\\\\": \\\\\"Andrew Tate Bio, Wiki, Net Worth, Age, Family, MMA Career - Next Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nextbiography.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate Age. Andrew Tate is 36 years old as of 2023, born on December 1, 1986, in Washington, DC. By his mid-thirties, Andrew Tate has become an esteemed figure in the world of kickboxing, showcasing remarkable expertise and experience in the sport. Early Life of Andrew Tate. Andrew Tate was born on 01 December 1986 to an African-American\\\\\", \\\\\"score\\\\\": 0.99368566, \\\\\"raw_content\\\\\": null}]}\"}'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">]</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'output'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'content: Andrew Tate\\'s kickboxing name is \"King Cobra.\" tool_calls: []'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">]</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input'\u001b[0m: \u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m]\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'output'\u001b[0m: \u001b[32m'content: Let me check the latest sports news. tool_calls: \u001b[0m\u001b[32m[\u001b[0m\u001b[32m]\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input'\u001b[0m: \u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"Let me check the latest sports news.\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby \u001b[0m\u001b[32m(\u001b[0m\u001b[32mBSM-471\u001b[0m\u001b[32m)\u001b[0m\u001b[32m first appear? Give me the number and title.\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m]\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'output'\u001b[0m: \u001b[32m\"content: tool_calls: \u001b[0m\u001b[32m[\u001b[0m\u001b[32mToolCall\u001b[0m\u001b[32m(\u001b[0m\u001b[32mcall_id\u001b[0m\u001b[32m='19bd3554-e670-4856-89d0-c63f5b016245', \u001b[0m\u001b[32mtool_name\u001b[0m\u001b[32m='bravy_search', \u001b[0m\u001b[32marguments\u001b[0m\u001b[32m=\u001b[0m\u001b[32m{\u001b[0m\u001b[32m'query': 'Bill Cosby South Park episode'\u001b[0m\u001b[32m}\u001b[0m\u001b[32m)\u001b[0m\u001b[32m]\u001b[0m\u001b[32m\"\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input'\u001b[0m: \u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"Let me check the latest sports news.\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby \u001b[0m\u001b[32m(\u001b[0m\u001b[32mBSM-471\u001b[0m\u001b[32m)\u001b[0m\u001b[32m first appear? Give me the number and title.\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"call_id\":\"19bd3554-e670-4856-89d0-c63f5b016245\",\"tool_name\":\"bravy_search\",\"arguments\":\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"query\":\"Bill Cosby South Park episode\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"What is the British-American kickboxer Andrew Tate\\'s kickboxing name?\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m]\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'output'\u001b[0m: \u001b[32m\"content: tool_calls: \u001b[0m\u001b[32m[\u001b[0m\u001b[32mToolCall\u001b[0m\u001b[32m(\u001b[0m\u001b[32mcall_id\u001b[0m\u001b[32m='526045a7-5f51-40fb-ba97-5ad29610e511', \u001b[0m\u001b[32mtool_name\u001b[0m\u001b[32m=\u001b[0m\u001b[32m<\u001b[0m\u001b[32mBuiltinTool.brave_search:\u001b[0m\u001b[32m 'brave_search'\u001b[0m\u001b[32m>\u001b[0m\u001b[32m, \u001b[0m\u001b[32marguments\u001b[0m\u001b[32m=\u001b[0m\u001b[32m{\u001b[0m\u001b[32m'query': 'Andrew Tate kickboxing name'\u001b[0m\u001b[32m}\u001b[0m\u001b[32m)\u001b[0m\u001b[32m]\u001b[0m\u001b[32m\"\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"arguments\":\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"query\":\"Andrew Tate kickboxing name\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'output'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"ipython\",\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"content\":\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"query\\\\\": \\\\\"Andrew Tate kickboxing name\\\\\", \\\\\"top_k\\\\\": \u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Andrew Tate kickboxing record: How many championships ... - FirstSportz\\\\\", \\\\\"url\\\\\": \\\\\"https://firstsportz.com/mma-how-many-championships-does-andrew-tate-have/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s Kickboxing career. During his kickboxing career, he used the nickname \\\\\\\\\\\\\"King Cobra,\\\\\\\\\\\\\" which he currently uses as his Twitter name. Tate had an unorthodox style of movement inside the ring. He kept his hands down most of the time and relied on quick jabs and an overhand right to land significant strikes.\\\\\", \\\\\"score\\\\\": 0.9996244, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Andrew Tate: Kickboxing Record, Facts, Height, Weight, Age, Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.lowkickmma.com/andrew-tate-kickboxing-record-facts-height-weight-age-biography/\\\\\", \\\\\"content\\\\\": \\\\\"Birth Name: Emory Andrew Tate III: Date of Birth: 1 December 1986: Place of Birth: Washington, D.C., U.S. ... In his professional kickboxing career, Andrew Tate won 32 of his fights by knockout.\\\\\", \\\\\"score\\\\\": 0.99909246, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Who is Andrew Tate? MMA, kickboxing record and controversies of fighter ...\\\\\", \\\\\"url\\\\\": \\\\\"https://www.sportingnews.com/us/kickboxing/news/andrew-tate-mma-kickboxing-record-controversies/u50waalc9cfz7krjg9wnyb7p\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate kickboxing record After launching his career as a 20-year-old in 2007, Tate built a formidable kickboxing record that included 76 wins across 85 fights in more than 13 years in the ring.\\\\\", \\\\\"score\\\\\": 0.9976586, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"About Andrew Tate: A Journey from Champion to Controversy\\\\\", \\\\\"url\\\\\": \\\\\"https://reachmorpheus.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s kickboxing career, beginning in 2005, is a tale of determination and skill. He quickly made a name for himself in the sport, rising through the ranks with his unique fighting style and strategic approach, honed by his chess-playing background.\\\\\", \\\\\"score\\\\\": 0.99701905, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Andrew Tate Bio, Wiki, Net Worth, Age, Family, MMA Career - Next Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nextbiography.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate Age. Andrew Tate is 36 years old as of 2023, born on December 1, 1986, in Washington, DC. By his mid-thirties, Andrew Tate has become an esteemed figure in the world of kickboxing, showcasing remarkable expertise and experience in the sport. Early Life of Andrew Tate. Andrew Tate was born on 01 December 1986 to an African-American\\\\\", \\\\\"score\\\\\": 0.99368566, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input'\u001b[0m: \u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"system\",\"content\":\"You are a helpful assistant. Use search tool to answer the questions. \"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"Let me check the latest sports news.\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby \u001b[0m\u001b[32m(\u001b[0m\u001b[32mBSM-471\u001b[0m\u001b[32m)\u001b[0m\u001b[32m first appear? Give me the number and title.\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"call_id\":\"19bd3554-e670-4856-89d0-c63f5b016245\",\"tool_name\":\"bravy_search\",\"arguments\":\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"query\":\"Bill Cosby South Park episode\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"What is the British-American kickboxer Andrew Tate\\'s kickboxing name?\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"assistant\",\"content\":\"\",\"stop_reason\":\"end_of_turn\",\"tool_calls\":\u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"arguments\":\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"query\":\"Andrew Tate kickboxing name\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"ipython\",\"call_id\":\"526045a7-5f51-40fb-ba97-5ad29610e511\",\"tool_name\":\"brave_search\",\"content\":\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"query\\\\\": \\\\\"Andrew Tate kickboxing name\\\\\", \\\\\"top_k\\\\\": \u001b[0m\u001b[32m[\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Andrew Tate kickboxing record: How many championships ... - FirstSportz\\\\\", \\\\\"url\\\\\": \\\\\"https://firstsportz.com/mma-how-many-championships-does-andrew-tate-have/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s Kickboxing career. During his kickboxing career, he used the nickname \\\\\\\\\\\\\"King Cobra,\\\\\\\\\\\\\" which he currently uses as his Twitter name. Tate had an unorthodox style of movement inside the ring. He kept his hands down most of the time and relied on quick jabs and an overhand right to land significant strikes.\\\\\", \\\\\"score\\\\\": 0.9996244, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Andrew Tate: Kickboxing Record, Facts, Height, Weight, Age, Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.lowkickmma.com/andrew-tate-kickboxing-record-facts-height-weight-age-biography/\\\\\", \\\\\"content\\\\\": \\\\\"Birth Name: Emory Andrew Tate III: Date of Birth: 1 December 1986: Place of Birth: Washington, D.C., U.S. ... In his professional kickboxing career, Andrew Tate won 32 of his fights by knockout.\\\\\", \\\\\"score\\\\\": 0.99909246, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Who is Andrew Tate? MMA, kickboxing record and controversies of fighter ...\\\\\", \\\\\"url\\\\\": \\\\\"https://www.sportingnews.com/us/kickboxing/news/andrew-tate-mma-kickboxing-record-controversies/u50waalc9cfz7krjg9wnyb7p\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate kickboxing record After launching his career as a 20-year-old in 2007, Tate built a formidable kickboxing record that included 76 wins across 85 fights in more than 13 years in the ring.\\\\\", \\\\\"score\\\\\": 0.9976586, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"About Andrew Tate: A Journey from Champion to Controversy\\\\\", \\\\\"url\\\\\": \\\\\"https://reachmorpheus.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate\\'s kickboxing career, beginning in 2005, is a tale of determination and skill. He quickly made a name for himself in the sport, rising through the ranks with his unique fighting style and strategic approach, honed by his chess-playing background.\\\\\", \\\\\"score\\\\\": 0.99701905, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m, \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\\\\"title\\\\\": \\\\\"Andrew Tate Bio, Wiki, Net Worth, Age, Family, MMA Career - Next Biography\\\\\", \\\\\"url\\\\\": \\\\\"https://www.nextbiography.com/andrew-tate/\\\\\", \\\\\"content\\\\\": \\\\\"Andrew Tate Age. Andrew Tate is 36 years old as of 2023, born on December 1, 1986, in Washington, DC. By his mid-thirties, Andrew Tate has become an esteemed figure in the world of kickboxing, showcasing remarkable expertise and experience in the sport. Early Life of Andrew Tate. Andrew Tate was born on 01 December 1986 to an African-American\\\\\", \\\\\"score\\\\\": 0.99368566, \\\\\"raw_content\\\\\": null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m]\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'output'\u001b[0m: \u001b[32m'content: Andrew Tate\\'s kickboxing name is \"King Cobra.\" tool_calls: \u001b[0m\u001b[32m[\u001b[0m\u001b[32m]\u001b[0m\u001b[32m'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m]\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"print(f\"Getting traces for session_id={session_id}\")\n",
"import json\n",
"from rich.pretty import pprint\n",
"\n",
"agent_logs = []\n",
"\n",
"for span in client.telemetry.query_spans(\n",
" attribute_filters=[\n",
" {\"key\": \"session_id\", \"op\": \"eq\", \"value\": session_id},\n",
" ],\n",
" attributes_to_return=[\"input\", \"output\"]\n",
" ):\n",
" if span.attributes[\"output\"] != \"no shields\":\n",
" agent_logs.append(span.attributes)\n",
"\n",
"pprint(agent_logs)"
]
},
{
"cell_type": "markdown",
"id": "QF30H7ufP2RE",
"metadata": {
"id": "QF30H7ufP2RE"
},
"source": [
"##### 3.1.3 Post-Process Telemetry Results & Evaluate\n",
"\n",
"- Now, we want to run evaluation to assert that our search agent succesfully calls brave_search from online traces.\n",
"- We will first post-process the agent's telemetry logs and run evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "sy4Xaff_Avuu",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 411
},
"id": "sy4Xaff_Avuu",
"outputId": "cb68bae7-b21d-415d-8e71-612bd383c793"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input_query'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'generated_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'content: Let me check the latest sports news. tool_calls: []'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'expected_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'brave_search'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input_query'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby (BSM-471) first appear? Give me the number and title.\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'generated_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"content: tool_calls: [ToolCall(call_id='19bd3554-e670-4856-89d0-c63f5b016245', tool_name='bravy_search', arguments={'query': 'Bill Cosby South Park episode'})]\"</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'expected_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'brave_search'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'input_query'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'{\"role\":\"user\",\"content\":\"What is the British-American kickboxer Andrew Tate\\'s kickboxing name?\",\"context\":null}'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'generated_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"content: tool_calls: [ToolCall(call_id='526045a7-5f51-40fb-ba97-5ad29610e511', tool_name=&lt;BuiltinTool.brave_search: 'brave_search'&gt;, arguments={'query': 'Andrew Tate kickboxing name'})]\"</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'expected_answer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'brave_search'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">]</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input_query'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"Which teams played in the NBA western conference finals of 2024\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'generated_answer'\u001b[0m: \u001b[32m'content: Let me check the latest sports news. tool_calls: \u001b[0m\u001b[32m[\u001b[0m\u001b[32m]\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'expected_answer'\u001b[0m: \u001b[32m'brave_search'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input_query'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"In which episode and season of South Park does Bill Cosby \u001b[0m\u001b[32m(\u001b[0m\u001b[32mBSM-471\u001b[0m\u001b[32m)\u001b[0m\u001b[32m first appear? Give me the number and title.\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'generated_answer'\u001b[0m: \u001b[32m\"content: tool_calls: \u001b[0m\u001b[32m[\u001b[0m\u001b[32mToolCall\u001b[0m\u001b[32m(\u001b[0m\u001b[32mcall_id\u001b[0m\u001b[32m='19bd3554-e670-4856-89d0-c63f5b016245', \u001b[0m\u001b[32mtool_name\u001b[0m\u001b[32m='bravy_search', \u001b[0m\u001b[32marguments\u001b[0m\u001b[32m=\u001b[0m\u001b[32m{\u001b[0m\u001b[32m'query': 'Bill Cosby South Park episode'\u001b[0m\u001b[32m}\u001b[0m\u001b[32m)\u001b[0m\u001b[32m]\u001b[0m\u001b[32m\"\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'expected_answer'\u001b[0m: \u001b[32m'brave_search'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'input_query'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\"role\":\"user\",\"content\":\"What is the British-American kickboxer Andrew Tate\\'s kickboxing name?\",\"context\":null\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'generated_answer'\u001b[0m: \u001b[32m\"content: tool_calls: \u001b[0m\u001b[32m[\u001b[0m\u001b[32mToolCall\u001b[0m\u001b[32m(\u001b[0m\u001b[32mcall_id\u001b[0m\u001b[32m='526045a7-5f51-40fb-ba97-5ad29610e511', \u001b[0m\u001b[32mtool_name\u001b[0m\u001b[32m=\u001b[0m\u001b[32m<\u001b[0m\u001b[32mBuiltinTool.brave_search:\u001b[0m\u001b[32m 'brave_search'\u001b[0m\u001b[32m>\u001b[0m\u001b[32m, \u001b[0m\u001b[32marguments\u001b[0m\u001b[32m=\u001b[0m\u001b[32m{\u001b[0m\u001b[32m'query': 'Andrew Tate kickboxing name'\u001b[0m\u001b[32m}\u001b[0m\u001b[32m)\u001b[0m\u001b[32m]\u001b[0m\u001b[32m\"\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'expected_answer'\u001b[0m: \u001b[32m'brave_search'\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m]\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringScoreResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">results</span>=<span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'basic::subset_of'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringResult</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">aggregated_results</span>=<span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.3333333333333333</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_correct'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_total'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">score_rows</span>=<span style=\"font-weight: bold\">[{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span><span style=\"font-weight: bold\">}</span>, <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span><span style=\"font-weight: bold\">}</span>, <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}]</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">)</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mScoringScoreResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mresults\u001b[0m=\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'basic::subset_of'\u001b[0m: \u001b[1;35mScoringResult\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33maggregated_results\u001b[0m=\u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1;36m0.3333333333333333\u001b[0m, \u001b[32m'num_correct'\u001b[0m: \u001b[1;36m1.0\u001b[0m, \u001b[32m'num_total'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33mscore_rows\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m0.0\u001b[0m\u001b[1m}\u001b[0m, \u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m0.0\u001b[0m\u001b[1m}\u001b[0m, \u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m\u001b[1m]\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m)\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# post-process telemetry spance and prepare data for eval\n",
"# in this case, we want to assert that all user prompts is followed by a tool call\n",
"import ast\n",
"import json\n",
"\n",
"eval_rows = []\n",
"\n",
"for log in agent_logs:\n",
" last_msg = log['input'][-1]\n",
" if \"\\\"role\\\":\\\"user\\\"\" in last_msg:\n",
" eval_rows.append(\n",
" {\n",
" \"input_query\": last_msg,\n",
" \"generated_answer\": log[\"output\"],\n",
" # check if generated_answer uses tools brave_search\n",
" \"expected_answer\": \"brave_search\",\n",
" },\n",
" )\n",
"\n",
"pprint(eval_rows)\n",
"scoring_params = {\n",
" \"basic::subset_of\": None,\n",
"}\n",
"scoring_response = client.scoring.score(input_rows=eval_rows, scoring_functions=scoring_params)\n",
"pprint(scoring_response)"
]
},
{
"cell_type": "markdown",
"id": "IKbzhxcw5e_c",
"metadata": {
"id": "IKbzhxcw5e_c"
},
"source": [
"#### 3.2. Agentic Application Dataset Scoring\n",
"- Llama Stack offers a library of scoring functions and the `/scoring` API, allowing you to run evaluations on your pre-annotated AI application datasets.\n",
"\n",
"- In this example, we will work with an example RAG dataset you have built previously, label with an annotation, and use LLM-As-Judge with custom judge prompt for scoring. Please checkout our [Llama Stack Playground](https://llama-stack.readthedocs.io/en/latest/playground/index.html) for an interactive interface to upload datasets and run scorings."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "xG4Y84VQBb0g",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 298
},
"id": "xG4Y84VQBb0g",
"outputId": "f61cebdf-f614-440c-d170-f1e873b542ef"
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringScoreResponse</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"color: #808000; text-decoration-color: #808000\">results</span>=<span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'llm-as-judge::base'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringResult</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">aggregated_results</span>=<span style=\"font-weight: bold\">{}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">score_rows</span>=<span style=\"font-weight: bold\">[</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ </span><span style=\"font-weight: bold\">{</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'B'</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'judge_feedback'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Answer: B, Explanation: The GENERATED_RESPONSE is a superset of the EXPECTED_RESPONSE and is fully consistent with it. The GENERATED_RESPONSE provides more detailed information about the top 5 topics related to LoRA, while the EXPECTED_RESPONSE only mentions \"LoRA\". The GENERATED_RESPONSE expands on the topic, but does not conflict with the EXPECTED_RESPONSE.'</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ │ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"font-weight: bold\">]</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">)</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"color: #008000; text-decoration-color: #008000\">'basic::subset_of'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ScoringResult</span><span style=\"font-weight: bold\">(</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">aggregated_results</span>=<span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'accuracy'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_correct'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'num_total'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}</span>,\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ │ </span><span style=\"color: #808000; text-decoration-color: #808000\">score_rows</span>=<span style=\"font-weight: bold\">[{</span><span style=\"color: #008000; text-decoration-color: #008000\">'score'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">}]</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ │ </span><span style=\"font-weight: bold\">)</span>\n",
"<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│ </span><span style=\"font-weight: bold\">}</span>\n",
"<span style=\"font-weight: bold\">)</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;35mScoringScoreResponse\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[33mresults\u001b[0m=\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'llm-as-judge::base'\u001b[0m: \u001b[1;35mScoringResult\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33maggregated_results\u001b[0m=\u001b[1m{\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33mscore_rows\u001b[0m=\u001b[1m[\u001b[0m\n",
"\u001b[2;32m│ │ │ │ \u001b[0m\u001b[1m{\u001b[0m\n",
"\u001b[2;32m│ │ │ │ │ \u001b[0m\u001b[32m'score'\u001b[0m: \u001b[32m'B'\u001b[0m,\n",
"\u001b[2;32m│ │ │ │ │ \u001b[0m\u001b[32m'judge_feedback'\u001b[0m: \u001b[32m'Answer: B, Explanation: The GENERATED_RESPONSE is a superset of the EXPECTED_RESPONSE and is fully consistent with it. The GENERATED_RESPONSE provides more detailed information about the top 5 topics related to LoRA, while the EXPECTED_RESPONSE only mentions \"LoRA\". The GENERATED_RESPONSE expands on the topic, but does not conflict with the EXPECTED_RESPONSE.'\u001b[0m\n",
"\u001b[2;32m│ │ │ │ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[1m]\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m)\u001b[0m,\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[32m'basic::subset_of'\u001b[0m: \u001b[1;35mScoringResult\u001b[0m\u001b[1m(\u001b[0m\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33maggregated_results\u001b[0m=\u001b[1m{\u001b[0m\u001b[32m'accuracy'\u001b[0m: \u001b[1;36m1.0\u001b[0m, \u001b[32m'num_correct'\u001b[0m: \u001b[1;36m1.0\u001b[0m, \u001b[32m'num_total'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m,\n",
"\u001b[2;32m│ │ │ \u001b[0m\u001b[33mscore_rows\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m{\u001b[0m\u001b[32m'score'\u001b[0m: \u001b[1;36m1.0\u001b[0m\u001b[1m}\u001b[0m\u001b[1m]\u001b[0m\n",
"\u001b[2;32m│ │ \u001b[0m\u001b[1m)\u001b[0m\n",
"\u001b[2;32m│ \u001b[0m\u001b[1m}\u001b[0m\n",
"\u001b[1m)\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import rich\n",
"from rich.pretty import pprint\n",
"\n",
"judge_model_id = \"meta-llama/Llama-3.1-405B-Instruct-FP8\"\n",
"\n",
"JUDGE_PROMPT = \"\"\"\n",
"Given a QUESTION and GENERATED_RESPONSE and EXPECTED_RESPONSE.\n",
"\n",
"Compare the factual content of the GENERATED_RESPONSE with the EXPECTED_RESPONSE. Ignore any differences in style, grammar, or punctuation.\n",
" The GENERATED_RESPONSE may either be a subset or superset of the EXPECTED_RESPONSE, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:\n",
" (A) The GENERATED_RESPONSE is a subset of the EXPECTED_RESPONSE and is fully consistent with it.\n",
" (B) The GENERATED_RESPONSE is a superset of the EXPECTED_RESPONSE and is fully consistent with it.\n",
" (C) The GENERATED_RESPONSE contains all the same details as the EXPECTED_RESPONSE.\n",
" (D) There is a disagreement between the GENERATED_RESPONSE and the EXPECTED_RESPONSE.\n",
" (E) The answers differ, but these differences don't matter from the perspective of factuality.\n",
"\n",
"Give your answer in the format \"Answer: One of ABCDE, Explanation: \".\n",
"\n",
"Your actual task:\n",
"\n",
"QUESTION: {input_query}\n",
"GENERATED_RESPONSE: {generated_answer}\n",
"EXPECTED_RESPONSE: {expected_answer}\n",
"\"\"\"\n",
"\n",
"input_query = \"What are the top 5 topics that were explained? Only list succinct bullet points.\"\n",
"generated_answer = \"\"\"\n",
"Here are the top 5 topics that were explained in the documentation for Torchtune:\n",
"\n",
"* What is LoRA and how does it work?\n",
"* Fine-tuning with LoRA: memory savings and parameter-efficient finetuning\n",
"* Running a LoRA finetune with Torchtune: overview and recipe\n",
"* Experimenting with different LoRA configurations: rank, alpha, and attention modules\n",
"* LoRA finetuning\n",
"\"\"\"\n",
"expected_answer = \"\"\"LoRA\"\"\"\n",
"\n",
"rows = [\n",
" {\n",
" \"input_query\": input_query,\n",
" \"generated_answer\": generated_answer,\n",
" \"expected_answer\": expected_answer,\n",
" },\n",
"]\n",
"\n",
"scoring_params = {\n",
" \"llm-as-judge::base\": {\n",
" \"judge_model\": judge_model_id,\n",
" \"prompt_template\": JUDGE_PROMPT,\n",
" \"type\": \"llm_as_judge\",\n",
" \"judge_score_regexes\": [\"Answer: (A|B|C|D|E)\"],\n",
" },\n",
" \"basic::subset_of\": None,\n",
"}\n",
"\n",
"response = client.scoring.score(input_rows=rows, scoring_functions=scoring_params)\n",
"pprint(response)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "rKtGo_v98UA2",
"metadata": {
"id": "rKtGo_v98UA2"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"_JueJAKyJR5m"
],
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.15"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"0243626d7ef44ef2b90e8fed5c13183d": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"044d6d8dda1c4935b1752a9c71c6ee4a": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_63f34c3d43bb4fdd9faeb6161fd77285",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_5cb841b49eaa429e8616ec4b78f501e9",
"value": 1
}
},
"0640b57408644741970dd958ca0e21e6": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_6259ffc3ef674df985fd3fa4334f9c8e",
"IPY_MODEL_3d0376d2e574410eb4ef963d51cac0a6",
"IPY_MODEL_b66984cc5de541a5801a1e6e54d40daf"
],
"layout": "IPY_MODEL_92135b9cb201475681ee0886887c84a8"
}
},
"116139bfe7a44f969a2c97490c224d31": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_ab1f339cba094c918fc5507f8361de5c",
"placeholder": "​",
"style": "IPY_MODEL_a6a1eb412f204578b80e5b6717c1e3a5",
"value": " 1/1 [00:01&lt;00:00,  1.27s/it]"
}
},
"118b359b83304ae59fad57e28f621645": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"15d3ff07f1c54e58b51d452caca01209": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"17603dd7fedf4798a74533fbfd5bb421": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"186682be50c148c0826fa7c314087562": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_1f427d4273e04e19b1bdb13388736c01",
"placeholder": "​",
"style": "IPY_MODEL_38897429b7cf4077aea3a981593ca866",
"value": " 1/1 [00:00&lt;00:00, 15.09it/s]"
}
},
"1f427d4273e04e19b1bdb13388736c01": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"2082554eed6644a996f0e31545789e08": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_a0be415018644c3cac098ab9b19c2391",
"IPY_MODEL_6ede3649e8c24015b3ca77490568bfcd",
"IPY_MODEL_116139bfe7a44f969a2c97490c224d31"
],
"layout": "IPY_MODEL_243d13828d854880a6adb861ea867734"
}
},
"2100363a158b4488a58620983aa5bdd4": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"243d13828d854880a6adb861ea867734": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"277101c35a784e6caf455a13cd9b8e59": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"2924814bab5748ddbeeedc70d324195e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_4738bccc6b384da5a20a8bcd61ecec59",
"IPY_MODEL_044d6d8dda1c4935b1752a9c71c6ee4a",
"IPY_MODEL_9277709ad9154d7b8f37d08db84ee425"
],
"layout": "IPY_MODEL_f3f1f2487d6f455caeb6ec71a2d51ee2"
}
},
"2958af7c9cdb46038e0336d6b7c6773e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"351928faa62543128e0bd29bf89bbf79": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"38897429b7cf4077aea3a981593ca866": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"3978f618c4f8467eb83c63a8f5aef98a": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"3d0376d2e574410eb4ef963d51cac0a6": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_9054d3825edb49cb9c35d24023f50c03",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_3978f618c4f8467eb83c63a8f5aef98a",
"value": 1
}
},
"425c6c0eaed741669551b9af77096c6f": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_d124b09896934d289df649375f455a8e",
"IPY_MODEL_554cff1a83d44bd2bbd36fd43acac7e2",
"IPY_MODEL_d0381718fc8b49a6ac7e7fe85cabba90"
],
"layout": "IPY_MODEL_fd3daaf9093d45d8a9d39b87835f4582"
}
},
"457374ae3035496eb943ad21484f76a0": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_bcf4679dda2d4767a0a24cbf236ca76e",
"IPY_MODEL_6e4ce98853c84beca11471e7ea9d97df",
"IPY_MODEL_186682be50c148c0826fa7c314087562"
],
"layout": "IPY_MODEL_e1ef246e3e6c4359b7b61c341119e121"
}
},
"45b569d733f944d29cefae8a5d13b215": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"4738bccc6b384da5a20a8bcd61ecec59": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_66c92a8a89234a61a8c688cf1c3e29a1",
"placeholder": "​",
"style": "IPY_MODEL_ee1f4a0c85e44a3b849283337743a8d4",
"value": "Batches: 100%"
}
},
"4a405d391b974e58a2c4fe00d4bb5815": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"4ad57f5d8a824afab639e8606ee43ca6": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"53865d3f918e468ab53504133b127973": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"554cff1a83d44bd2bbd36fd43acac7e2": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_6c60c8291e734f549e6c5a46b427b974",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_de88640505c24928904a3c76bda31c70",
"value": 1
}
},
"5afdb88e0159462e98773560e3dad439": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_f7bc4df675a141e380d965138552a142",
"IPY_MODEL_d7bf8b49145843ac98a6de424e628729",
"IPY_MODEL_8fb17faf68524de2b73321d71b80b407"
],
"layout": "IPY_MODEL_45b569d733f944d29cefae8a5d13b215"
}
},
"5cb841b49eaa429e8616ec4b78f501e9": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"5f19dab8c6da4050bc47fd78838f7530": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"6259ffc3ef674df985fd3fa4334f9c8e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_4a405d391b974e58a2c4fe00d4bb5815",
"placeholder": "​",
"style": "IPY_MODEL_2958af7c9cdb46038e0336d6b7c6773e",
"value": "Batches: 100%"
}
},
"63f34c3d43bb4fdd9faeb6161fd77285": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"66c92a8a89234a61a8c688cf1c3e29a1": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"6c60c8291e734f549e6c5a46b427b974": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"6e4ce98853c84beca11471e7ea9d97df": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_a0ac7ee92d994c7b9b74e580ab2acdf7",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_118b359b83304ae59fad57e28f621645",
"value": 1
}
},
"6ede3649e8c24015b3ca77490568bfcd": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_f10237315e794539a00ca82bfff930be",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_ca09d2207b00456da4c37b5a782a190c",
"value": 1
}
},
"753dbe7891a143118b55eccf8c252e03": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"8fb17faf68524de2b73321d71b80b407": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_277101c35a784e6caf455a13cd9b8e59",
"placeholder": "​",
"style": "IPY_MODEL_d06666f765764f949e1876f2d5d67242",
"value": " 1/1 [00:01&lt;00:00,  1.68s/it]"
}
},
"9054d3825edb49cb9c35d24023f50c03": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"92135b9cb201475681ee0886887c84a8": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"9277709ad9154d7b8f37d08db84ee425": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_a447ea9af3e14e5e94eb14ed8dd3c0de",
"placeholder": "​",
"style": "IPY_MODEL_0243626d7ef44ef2b90e8fed5c13183d",
"value": " 1/1 [00:02&lt;00:00,  2.65s/it]"
}
},
"a0ac7ee92d994c7b9b74e580ab2acdf7": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"a0be415018644c3cac098ab9b19c2391": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_e4b1dfe159304c5f88766b33e85a5c19",
"placeholder": "​",
"style": "IPY_MODEL_2100363a158b4488a58620983aa5bdd4",
"value": "Batches: 100%"
}
},
"a447ea9af3e14e5e94eb14ed8dd3c0de": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"a6a1eb412f204578b80e5b6717c1e3a5": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"ab1f339cba094c918fc5507f8361de5c": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"b66984cc5de541a5801a1e6e54d40daf": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_efd68f6dc0b3428e8f5fc830c1bf2341",
"placeholder": "​",
"style": "IPY_MODEL_4ad57f5d8a824afab639e8606ee43ca6",
"value": " 1/1 [00:00&lt;00:00,  5.36it/s]"
}
},
"bbb93c771a9c453bb90e729b1f73b931": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"bcf4679dda2d4767a0a24cbf236ca76e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_bbb93c771a9c453bb90e729b1f73b931",
"placeholder": "​",
"style": "IPY_MODEL_351928faa62543128e0bd29bf89bbf79",
"value": "Batches: 100%"
}
},
"ca09d2207b00456da4c37b5a782a190c": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"ce7de1af99434ad38a9382e7253dbfc0": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"d0381718fc8b49a6ac7e7fe85cabba90": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_fc086d0dd1a745308c59ae219ae135c5",
"placeholder": "​",
"style": "IPY_MODEL_15d3ff07f1c54e58b51d452caca01209",
"value": " 1/1 [00:00&lt;00:00, 14.36it/s]"
}
},
"d06666f765764f949e1876f2d5d67242": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"d124b09896934d289df649375f455a8e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_753dbe7891a143118b55eccf8c252e03",
"placeholder": "​",
"style": "IPY_MODEL_ce7de1af99434ad38a9382e7253dbfc0",
"value": "Batches: 100%"
}
},
"d7bf8b49145843ac98a6de424e628729": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_17603dd7fedf4798a74533fbfd5bb421",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_5f19dab8c6da4050bc47fd78838f7530",
"value": 1
}
},
"de88640505c24928904a3c76bda31c70": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"e1ef246e3e6c4359b7b61c341119e121": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"e4b1dfe159304c5f88766b33e85a5c19": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"ee1f4a0c85e44a3b849283337743a8d4": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"efd68f6dc0b3428e8f5fc830c1bf2341": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"f10237315e794539a00ca82bfff930be": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"f3f1f2487d6f455caeb6ec71a2d51ee2": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"f7bc4df675a141e380d965138552a142": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_fdd057a4506f4f119d945bab5b930799",
"placeholder": "​",
"style": "IPY_MODEL_53865d3f918e468ab53504133b127973",
"value": "Batches: 100%"
}
},
"fc086d0dd1a745308c59ae219ae135c5": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"fd3daaf9093d45d8a9d39b87835f4582": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"fdd057a4506f4f119d945bab5b930799": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
}
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in a new issue View git blame Copy permalink