From cb27cbd4b5935e2d5b760be68c75f4e7ba662452 Mon Sep 17 00:00:00 2001 From: Dinesh Yeduguru Date: Wed, 22 Jan 2025 16:17:50 -0800 Subject: [PATCH 1/6] update docs for tools and telemetry --- .../source/building_applications/telemetry.md | 164 +--------------- docs/source/building_applications/tools.md | 181 ++++++++++++++++++ 2 files changed, 182 insertions(+), 163 deletions(-) create mode 100644 docs/source/building_applications/tools.md diff --git a/docs/source/building_applications/telemetry.md b/docs/source/building_applications/telemetry.md index 70c54ac98..9a71b9fc8 100644 --- a/docs/source/building_applications/telemetry.md +++ b/docs/source/building_applications/telemetry.md @@ -1,8 +1,4 @@ # Telemetry -```{note} -The telemetry system is currently experimental and subject to change. We welcome feedback and contributions to help improve it. -``` - The Llama Stack telemetry system provides comprehensive tracing, metrics, and logging capabilities. It supports multiple sink types including OpenTelemetry, SQLite, and Console output. @@ -44,58 +40,6 @@ structured_log_event = SpanStartPayload( - **SQLite**: Store events in a local SQLite database. This is needed if you want to query the events later through the Llama Stack API. - **Console**: Print events to the console. -## APIs - -The telemetry API is designed to be flexible for different user flows like debugging/visualization in UI, monitoring, and saving traces to datasets. -The telemetry system exposes the following HTTP endpoints: - -### Log Event -```http -POST /telemetry/log-event -``` -Logs a telemetry event (unstructured log, metric, or structured log) with optional TTL. - -### Query Traces -```http -POST /telemetry/query-traces -``` -Retrieves traces based on filters with pagination support. Parameters: -- `attribute_filters`: List of conditions to filter traces -- `limit`: Maximum number of traces to return (default: 100) -- `offset`: Number of traces to skip (default: 0) -- `order_by`: List of fields to sort by - -### Get Span Tree -```http -POST /telemetry/get-span-tree -``` -Retrieves a hierarchical view of spans starting from a specific span. Parameters: -- `span_id`: ID of the root span to retrieve -- `attributes_to_return`: Optional list of specific attributes to include -- `max_depth`: Optional maximum depth of the span tree to return - -### Query Spans -```http -POST /telemetry/query-spans -``` -Retrieves spans matching specified filters and returns selected attributes. Parameters: -- `attribute_filters`: List of conditions to filter traces -- `attributes_to_return`: List of specific attributes to include in results -- `max_depth`: Optional maximum depth of spans to traverse (default: no limit) - -Returns a flattened list of spans with requested attributes. - -### Save Spans to Dataset -This is useful for saving traces to a dataset for running evaluations. For example, you can save the input/output of each span that is part of an agent session/turn to a dataset and then run an eval task on it. See example in [Example: Save Spans to Dataset](#example-save-spans-to-dataset). -```http -POST /telemetry/save-spans-to-dataset -``` -Queries spans and saves their attributes to a dataset. Parameters: -- `attribute_filters`: List of conditions to filter traces -- `attributes_to_save`: List of span attributes to save to the dataset -- `dataset_id`: ID of the dataset to save to -- `max_depth`: Optional maximum depth of spans to traverse (default: no limit) - ## Providers ### Meta-Reference Provider @@ -133,110 +77,4 @@ Once the Jaeger instance is running, you can visualize traces by navigating to h ## Querying Traces Stored in SQLIte -The `sqlite` sink allows you to query traces without an external system. Here are some example queries: - -Querying Traces for a agent session -The client SDK is not updated to support the new telemetry API. It will be updated soon. You can manually query traces using the following curl command: - -``` bash - curl -X POST 'http://localhost:8321/alpha/telemetry/query-traces' \ --H 'Content-Type: application/json' \ --d '{ - "attribute_filters": [ - { - "key": "session_id", - "op": "eq", - "value": "dd667b87-ca4b-4d30-9265-5a0de318fc65" }], - "limit": 100, - "offset": 0, - "order_by": ["start_time"] - - [ - { - "trace_id": "6902f54b83b4b48be18a6f422b13e16f", - "root_span_id": "5f37b85543afc15a", - "start_time": "2024-12-04T08:08:30.501587", - "end_time": "2024-12-04T08:08:36.026463" - }, - ........ -] -}' - -``` - -Querying spans for a specifc root span id - -``` bash -curl -X POST 'http://localhost:8321/alpha/telemetry/get-span-tree' \ --H 'Content-Type: application/json' \ --d '{ "span_id" : "6cceb4b48a156913", "max_depth": 2 }' - -{ - "span_id": "6cceb4b48a156913", - "trace_id": "dafa796f6aaf925f511c04cd7c67fdda", - "parent_span_id": "892a66d726c7f990", - "name": "retrieve_rag_context", - "start_time": "2024-12-04T09:28:21.781995", - "end_time": "2024-12-04T09:28:21.913352", - "attributes": { - "input": [ - "{\"role\":\"system\",\"content\":\"You are a helpful assistant\"}", - "{\"role\":\"user\",\"content\":\"What are the top 5 topics that were explained in the documentation? Only list succinct bullet points.\",\"context\":null}" - ] - }, - "children": [ - { - "span_id": "1a2df181854064a8", - "trace_id": "dafa796f6aaf925f511c04cd7c67fdda", - "parent_span_id": "6cceb4b48a156913", - "name": "MemoryRouter.query_documents", - "start_time": "2024-12-04T09:28:21.787620", - "end_time": "2024-12-04T09:28:21.906512", - "attributes": { - "input": null - }, - "children": [], - "status": "ok" - } - ], - "status": "ok" -} - -``` - -## Example: Save Spans to Dataset -Save all spans for a specific agent session to a dataset. -``` bash -curl -X POST 'http://localhost:8321/alpha/telemetry/save-spans-to-dataset' \ --H 'Content-Type: application/json' \ --d '{ - "attribute_filters": [ - { - "key": "session_id", - "op": "eq", - "value": "dd667b87-ca4b-4d30-9265-5a0de318fc65" - } - ], - "attributes_to_save": ["input", "output"], - "dataset_id": "my_dataset", - "max_depth": 10 -}' -``` - -Save all spans for a specific agent turn to a dataset. -```bash -curl -X POST 'http://localhost:8321/alpha/telemetry/save-spans-to-dataset' \ --H 'Content-Type: application/json' \ --d '{ - "attribute_filters": [ - { - "key": "turn_id", - "op": "eq", - "value": "123e4567-e89b-12d3-a456-426614174000" - } - ], - "attributes_to_save": ["input", "output"], - "dataset_id": "my_dataset", - "max_depth": 10 -}' -``` +The `sqlite` sink allows you to query traces without an external system. Here are some example queries. Refer to the notebook at [Llama Stack Building AI Applications](../notebooks/Llama_Stack_Building_AI_Applications.ipynb) for more examples on how to query traces and spaces. diff --git a/docs/source/building_applications/tools.md b/docs/source/building_applications/tools.md new file mode 100644 index 000000000..b4319e3a2 --- /dev/null +++ b/docs/source/building_applications/tools.md @@ -0,0 +1,181 @@ +# Tools + +Tools are functions that can be invoked by an agent to perform tasks. They are organized into tool groups and registered with specific providers. Each tool group represents a collection of related tools from a single provider. + +Tools are treated as any other resource in llama stack like models. you can register them, have providers for them etc. + +When instatiating an agent, you can provide it a list of tool groups that it has access to. Agent gets the corresponding tool definitions for the specified tool gropus and pass along to the model + +Refer to the [Building AI Applications](../notebooks/Llama_Stack_Building_AI_Applications.ipynb) notebook for more examples on how to use tools. + +## Types of Tool Group providers + +There are three types of providers for tool groups that are supported by llama stack. + +1. Built-in providers +2. Model Context Protocol (MCP) providers +3. Tools provided by the client + +### Built-in providers + +Built-in providers come packaged with LlamaStack. These providers provide common functionalities like web search, code interpretation, and computational capabilities. + +#### Web Search providers +There are three web search providers that are supported by llama stack. + +1. Brave Search +2. Bing Search +3. Tavily Search + +Example client SDK call to register a "websearch" toolgroup that is provided by brave-search. + +```python +# Register Brave Search tool group +client.toolgroups.register( + toolgroup_id="builtin::websearch", + provider_id="brave-search", + args={"max_results": 5} +) +``` + +The tool requires an API key which can be provided either in the configuration or through the request header `X-LlamaStack-Provider-Data`. +The api key is required to be passed in the header as `X-LlamaStack-Provider-Data` as `{"brave_search_api_key": }` for brave search. + + + +#### Code Interpreter + +The Code Interpreter tool allows execution of Python code within a controlled environment. It includes safety measures to prevent potentially dangerous operations. + +```python +# Register Code Interpreter tool group +client.toolgroups.register( + toolgroup_id="builtin::code_interpreter", + provider_id="code_interpreter" +) +``` + +Features: +- Secure execution environment using `bwrap` sandboxing +- Matplotlib support for generating plots +- Disabled dangerous system operations +- Configurable execution timeouts + +#### WolframAlpha + +The WolframAlpha tool provides access to computational knowledge through the WolframAlpha API. + +```python +# Register WolframAlpha tool group +client.toolgroups.register( + toolgroup_id="builtin::wolfram_alpha", + provider_id="wolfram-alpha" +) +``` + +Example usage: +```python +result = client.tools.invoke_tool( + tool_name="wolfram_alpha", + args={"query": "solve x^2 + 2x + 1 = 0"} +) +``` + +### Memory Tool + +The Memory tool enables retrieval of context from various types of memory banks (vector, key-value, keyword, and graph). + +```python +# Register Memory tool group +client.toolgroups.register( + toolgroup_id="builtin::memory", + provider_id="memory", + args={ + "max_chunks": 5, + "max_tokens_in_context": 4096 + } +) +``` + +Features: +- Support for multiple memory bank types +- Configurable query generation +- Context retrieval with token limits + + +> **Note:** By default, llama stack run.yaml defines toolgroups for web search, code interpreter and memory, that are provided by tavily-search, code-interpreter and memory providers. + +## Model Context Protocol (MCP) Tools + +MCP tools are special tools that can interact with llama stack over model context protocol. These tools are dynamically discovered from an MCP endpoint and can be used to extend the agent's capabilities. + +```python +# Register MCP tools +client.toolgroups.register( + toolgroup_id="builtin::filesystem", + provider_id="model-context-protocol", + mcp_endpoint=URL(uri="http://localhost:8000/sse"), +) +``` + +MCP tools require: +- A valid MCP endpoint URL +- The endpoint must implement the Model Context Protocol +- Tools are discovered dynamically from the endpoint + +## Tool Structure + +Each tool has the following components: + +- `name`: Unique identifier for the tool +- `description`: Human-readable description of the tool's functionality +- `parameters`: List of parameters the tool accepts + - `name`: Parameter name + - `parameter_type`: Data type (string, number, etc.) + - `description`: Parameter description + - `required`: Whether the parameter is required (default: true) + - `default`: Default value if any + +Example tool definition: +```python +{ + "name": "web_search", + "description": "Search the web for information", + "parameters": [ + { + "name": "query", + "parameter_type": "string", + "description": "The query to search for", + "required": True + } + ] +} +``` + +## Tool Invocation + +Tools can be invoked using the `invoke_tool` method: + +```python +result = client.tools.invoke_tool( + tool_name="web_search", + kwargs={"query": "What is the capital of France?"} +) +``` + +The result contains: +- `content`: The tool's output +- `error_message`: Optional error message if the tool failed +- `error_code`: Optional error code if the tool failed + +## Listing Available Tools + +You can list all available tools or filter by tool group: + +```python +# List all tools +all_tools = client.tools.list_tools() + +# List tools in a specific group +group_tools = client.tools.list_tools(toolgroup_id="search_tools") +``` From 73243f1348d7b211b21e1ad9dda1238282c91678 Mon Sep 17 00:00:00 2001 From: Dinesh Yeduguru Date: Wed, 22 Jan 2025 16:21:30 -0800 Subject: [PATCH 2/6] update index.md --- docs/source/building_applications/index.md | 55 +++++++++++++++------- docs/source/building_applications/tools.md | 4 +- 2 files changed, 40 insertions(+), 19 deletions(-) diff --git a/docs/source/building_applications/index.md b/docs/source/building_applications/index.md index 61b7038cd..b9170e092 100644 --- a/docs/source/building_applications/index.md +++ b/docs/source/building_applications/index.md @@ -262,37 +262,58 @@ response = agent.create_turn( ``` ### Adding Tools to Agents +```{toctree} +:hidden: +:maxdepth: 3 -Agents can be enhanced with various tools: +tools +``` -1. **Search**: Web search capabilities through providers like Brave -2. **Code Interpreter**: Execute code snippets -3. **RAG**: Memory and document retrieval -4. **Function Calling**: Custom function execution -5. **WolframAlpha**: Mathematical computations -6. **Photogen**: Image generation +Agents can be enhanced with various tools. For detailed information about available tools, their configuration, and providers, see the [Tools](tools.md) documentation. -Example of configuring an agent with tools: +Tools are configured through the `toolgroups` parameter in the agent configuration. Each tool group can be specified either as a string or with additional arguments: ```python +from llama_stack_client.lib.agents.agent import Agent +from llama_stack_client.types.agent_create_params import AgentConfig + agent_config = AgentConfig( model="Llama3.2-3B-Instruct", - tools=[ + instructions="You are a helpful assistant", + # Configure tool groups + toolgroups=[ + # Simple string format + "builtin::code_interpreter", + # With arguments format { - "type": "brave_search", - "api_key": "YOUR_API_KEY", - "engine": "brave" - }, - { - "type": "code_interpreter", - "enable_inline_code_execution": True + "name": "builtin::websearch", + "args": { + "max_results": 5 + } } ], tool_choice="auto", - tool_prompt_format="json" + tool_prompt_format="json", + # Optional safety configuration + input_shields=["content_safety"], + output_shields=["content_safety"], + # Control the inference loop + max_infer_iters=10, + sampling_params={ + "strategy": { + "type": "top_p", + "temperature": 0.7, + "top_p": 0.95 + }, + "max_tokens": 2048 + } ) + +agent = Agent(client, agent_config) ``` +For details on available tool groups, providers, and their configuration options, refer to the [Tools](tools.md) documentation. + ## Building RAG-Enhanced Agents One of the most powerful patterns is combining agents with RAG capabilities. Here's a complete example: diff --git a/docs/source/building_applications/tools.md b/docs/source/building_applications/tools.md index b4319e3a2..519f3c699 100644 --- a/docs/source/building_applications/tools.md +++ b/docs/source/building_applications/tools.md @@ -6,7 +6,7 @@ Tools are treated as any other resource in llama stack like models. you can regi When instatiating an agent, you can provide it a list of tool groups that it has access to. Agent gets the corresponding tool definitions for the specified tool gropus and pass along to the model -Refer to the [Building AI Applications](../notebooks/Llama_Stack_Building_AI_Applications.ipynb) notebook for more examples on how to use tools. +Refer to the [Building AI Applications](../../notebooks/Llama_Stack_Building_AI_Applications.ipynb) notebook for more examples on how to use tools. ## Types of Tool Group providers @@ -81,7 +81,7 @@ result = client.tools.invoke_tool( ) ``` -### Memory Tool +#### Memory The Memory tool enables retrieval of context from various types of memory banks (vector, key-value, keyword, and graph). From 151174a9dbe7d5fcecf30d24a4fd68cfa2329e2f Mon Sep 17 00:00:00 2001 From: Dinesh Yeduguru Date: Wed, 22 Jan 2025 16:56:30 -0800 Subject: [PATCH 3/6] update link --- docs/source/building_applications/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/building_applications/tools.md b/docs/source/building_applications/tools.md index 519f3c699..7ccea47da 100644 --- a/docs/source/building_applications/tools.md +++ b/docs/source/building_applications/tools.md @@ -6,7 +6,7 @@ Tools are treated as any other resource in llama stack like models. you can regi When instatiating an agent, you can provide it a list of tool groups that it has access to. Agent gets the corresponding tool definitions for the specified tool gropus and pass along to the model -Refer to the [Building AI Applications](../../notebooks/Llama_Stack_Building_AI_Applications.ipynb) notebook for more examples on how to use tools. +Refer to the [Building AI Applications](https://github.com/meta-llama/llama-stack/blob/main/docs/notebooks/Llama_Stack_Building_AI_Applications.ipynb) notebook for more examples on how to use tools. ## Types of Tool Group providers From a5be81dac5fbfe3b3aaed40d171c9be2dc378ae4 Mon Sep 17 00:00:00 2001 From: Dinesh Yeduguru Date: Wed, 22 Jan 2025 17:04:04 -0800 Subject: [PATCH 4/6] update link in telemetry --- docs/source/building_applications/telemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/building_applications/telemetry.md b/docs/source/building_applications/telemetry.md index 9a71b9fc8..ee640398b 100644 --- a/docs/source/building_applications/telemetry.md +++ b/docs/source/building_applications/telemetry.md @@ -77,4 +77,4 @@ Once the Jaeger instance is running, you can visualize traces by navigating to h ## Querying Traces Stored in SQLIte -The `sqlite` sink allows you to query traces without an external system. Here are some example queries. Refer to the notebook at [Llama Stack Building AI Applications](../notebooks/Llama_Stack_Building_AI_Applications.ipynb) for more examples on how to query traces and spaces. +The `sqlite` sink allows you to query traces without an external system. Here are some example queries. Refer to the notebook at [Llama Stack Building AI Applications](https://github.com/meta-llama/llama-stack/blob/main/docs/notebooks/Llama_Stack_Building_AI_Applications.ipynb) for more examples on how to query traces and spaces. From 19dee34ebdc9a8172c09fa17b60033a3d24d5045 Mon Sep 17 00:00:00 2001 From: Dinesh Yeduguru Date: Wed, 22 Jan 2025 17:12:28 -0800 Subject: [PATCH 5/6] Address feedback --- docs/source/building_applications/tools.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/source/building_applications/tools.md b/docs/source/building_applications/tools.md index 7ccea47da..65b6e1f8a 100644 --- a/docs/source/building_applications/tools.md +++ b/docs/source/building_applications/tools.md @@ -1,16 +1,16 @@ # Tools -Tools are functions that can be invoked by an agent to perform tasks. They are organized into tool groups and registered with specific providers. Each tool group represents a collection of related tools from a single provider. +Tools are functions that can be invoked by an agent to perform tasks. They are organized into tool groups and registered with specific providers. Each tool group represents a collection of related tools from a single provider. They are organized into groups so that state can be externalized -- the collection operates on the same state typically. -Tools are treated as any other resource in llama stack like models. you can register them, have providers for them etc. +Tools are treated as any other resource in llama stack like models. You can register them, have providers for them etc. -When instatiating an agent, you can provide it a list of tool groups that it has access to. Agent gets the corresponding tool definitions for the specified tool gropus and pass along to the model +When instatiating an agent, you can provide it a list of tool groups that it has access to. Agent gets the corresponding tool definitions for the specified tool groups and passes them along to the model. Refer to the [Building AI Applications](https://github.com/meta-llama/llama-stack/blob/main/docs/notebooks/Llama_Stack_Building_AI_Applications.ipynb) notebook for more examples on how to use tools. ## Types of Tool Group providers -There are three types of providers for tool groups that are supported by llama stack. +There are three types of providers for tool groups that are supported by Llama Stack. 1. Built-in providers 2. Model Context Protocol (MCP) providers @@ -18,10 +18,10 @@ There are three types of providers for tool groups that are supported by llama s ### Built-in providers -Built-in providers come packaged with LlamaStack. These providers provide common functionalities like web search, code interpretation, and computational capabilities. +Built-in providers come packaged with Llama Stack. These providers provide common functionalities like web search, code interpretation, and computational capabilities. #### Web Search providers -There are three web search providers that are supported by llama stack. +There are three web search providers that are supported by Llama Stack. 1. Brave Search 2. Bing Search @@ -38,14 +38,13 @@ client.toolgroups.register( ) ``` -The tool requires an API key which can be provided either in the configuration or through the request header `X-LlamaStack-Provider-Data`. -The api key is required to be passed in the header as `X-LlamaStack-Provider-Data` as `{"brave_search_api_key": }` for brave search. +The tool requires an API key which can be provided either in the configuration or through the request header `X-LlamaStack-Provider-Data`. The format of the header is `{"_api_key": }`. #### Code Interpreter -The Code Interpreter tool allows execution of Python code within a controlled environment. It includes safety measures to prevent potentially dangerous operations. +The Code Interpreter allows execution of Python code within a controlled environment. ```python # Register Code Interpreter tool group @@ -75,7 +74,7 @@ client.toolgroups.register( Example usage: ```python -result = client.tools.invoke_tool( +result = client.tool_runtime.invoke_tool( tool_name="wolfram_alpha", args={"query": "solve x^2 + 2x + 1 = 0"} ) @@ -157,7 +156,7 @@ Example tool definition: Tools can be invoked using the `invoke_tool` method: ```python -result = client.tools.invoke_tool( +result = client.tool_runtime.invoke_tool( tool_name="web_search", kwargs={"query": "What is the capital of France?"} ) From 09d2e04c5cb4f44df762085a0564db82c6006639 Mon Sep 17 00:00:00 2001 From: Dinesh Yeduguru Date: Wed, 22 Jan 2025 17:38:56 -0800 Subject: [PATCH 6/6] Address feedback: v2 --- docs/source/building_applications/tools.md | 26 ++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/docs/source/building_applications/tools.md b/docs/source/building_applications/tools.md index 65b6e1f8a..1339a14ae 100644 --- a/docs/source/building_applications/tools.md +++ b/docs/source/building_applications/tools.md @@ -1,6 +1,7 @@ # Tools -Tools are functions that can be invoked by an agent to perform tasks. They are organized into tool groups and registered with specific providers. Each tool group represents a collection of related tools from a single provider. They are organized into groups so that state can be externalized -- the collection operates on the same state typically. +Tools are functions that can be invoked by an agent to perform tasks. They are organized into tool groups and registered with specific providers. Each tool group represents a collection of related tools from a single provider. They are organized into groups so that state can be externalized: the collection operates on the same state typically. +An example of this would be a "db_access" tool group that contains tools for interacting with a database. "list_tables", "query_table", "insert_row" could be examples of tools in this group. Tools are treated as any other resource in llama stack like models. You can register them, have providers for them etc. @@ -14,7 +15,7 @@ There are three types of providers for tool groups that are supported by Llama S 1. Built-in providers 2. Model Context Protocol (MCP) providers -3. Tools provided by the client +3. Client provided tools ### Built-in providers @@ -108,6 +109,8 @@ Features: MCP tools are special tools that can interact with llama stack over model context protocol. These tools are dynamically discovered from an MCP endpoint and can be used to extend the agent's capabilities. +Refer to https://github.com/modelcontextprotocol/server for available MCP servers. + ```python # Register MCP tools client.toolgroups.register( @@ -122,6 +125,25 @@ MCP tools require: - The endpoint must implement the Model Context Protocol - Tools are discovered dynamically from the endpoint + +## Tools provided by the client + +These tools are registered along with the agent config and are specific to the agent for which they are registered. The main difference between these tools and the tools provided by the built-in providers is that the execution of these tools is handled by the client and the agent transfers the tool call to the client and waits for the result from the client. + +```python +# Example agent config with client provided tools +config = AgentConfig( + toolgroups=[ + "builtin::websearch", + ], + client_tools=[ + ToolDef(name="client_tool", description="Client provided tool") + ] +) +``` + +Refer to [llama-stack-apps](https://github.com/meta-llama/llama-stack-apps/blob/main/examples/agents/e2e_loop_with_custom_tools.py) for an example of how to use client provided tools. + ## Tool Structure Each tool has the following components: