phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-05 18:22:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix: add tool-calling example with tool response

Browse source
This commit is contained in:
Suraj Subramanian 2025-04-07 10:48:10 -07:00
parent c52ccc4bbd
commit a2734d24e7
3 changed files with 233 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

94
llama_stack/models/llama/llama4/prompt_format.md
View file

@ -136,16 +136,13 @@ We are continuing the format for zero shot function calling used in previous ver
<|begin_of_text|><|header_start|>system<|header_end|> <|begin_of_text|><|header_start|>system<|header_end|>
You are an expert in composing functions. You are given a question and a set of possible functions. You are an expert in composing functions. You are given a question and a set of possible functions.
Based on the question, you will need to make one or more function/tool calls to achieve the purpose. Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the function can be used, point it out. If the given question lacks the parameters required by the function, If none of the function can be used, point it out. If the given question lacks the parameters required by the function,
also point it out. You should only return the function call in tools call sections. also point it out. You should only return the function call in tools call sections.
If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]
If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] You SHOULD NOT include any other text in the response.
You SHOULD NOT include any other text in the response. Here is a list of functions in JSON format that you can invoke.
[
Here is a list of functions in JSON format that you can invoke.
[
{ {
"name": "get_weather", "name": "get_weather",
"description": "Get weather info for places", "description": "Get weather info for places",
@ -167,7 +164,7 @@ Here is a list of functions in JSON format that you can invoke.
} }
} }
} }
<|eot|><|header_start|>user<|header_end|> <|eot|><|header_start|>user<|header_end|>
What is the weather in SF and Seattle?<|eot|><|header_start|>assistant<|header_end|> What is the weather in SF and Seattle?<|eot|><|header_start|>assistant<|header_end|>
@ -197,8 +194,8 @@ Similar to the above example, you can also provide information for all the avail
<|begin_of_text|><|header_start|>user<|header_end|> <|begin_of_text|><|header_start|>user<|header_end|>
Questions: Can you retrieve the details for the user with the ID 7890, who has black as their special request? Questions: Can you retrieve the details for the user with the ID 7890, who has black as their special request?
Here is a list of functions in JSON format that you can invoke: Here is a list of functions in JSON format that you can invoke:
[ [
{ {
"name": "get_user_info", "name": "get_user_info",
"description": "Retrieve details for a specific user by their unique identifier. Note that the provided function is in Python 3 syntax.", "description": "Retrieve details for a specific user by their unique identifier. Note that the provided function is in Python 3 syntax.",
@ -220,11 +217,9 @@ Here is a list of functions in JSON format that you can invoke:
} }
} }
} }
] ]
Should you decide to return the function call(s), put them in the format of [func1(params_name=params_value, params_name2=params_value2...), func2(params)]
Should you decide to return the function call(s), put them in the format of [func1(params_name=params_value, params_name2=params_value2...), func2(params)] You SHOULD NOT include any other text in the response.<|eot|><|header_start|>assistant<|header_end|>
You SHOULD NOT include any other text in the response.<|eot|><|header_start|>assistant<|header_end|>
``` ```
@ -259,12 +254,12 @@ Think very carefully before calling functions.
If you choose to call a function ONLY reply in the following format with no prefix or suffix: If you choose to call a function ONLY reply in the following format with no prefix or suffix:
<function=example_function_name>{"example_name": "example_value"}</function> <function=example_function_name>{"example_name": "example_value"}</function>
Reminder: Reminder:
- If looking for real time information use relevant functions before falling back to brave_search - If looking for real time information use relevant functions before falling back to brave_search
- Function calls MUST follow the specified format, start with <function= and end with </function> - Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified - Required parameters MUST be specified
- Only call one function at a time - Only call one function at a time
- Put the entire function call reply on one line<|eot_id|><|eot|><|header_start|>user<|header_end|> - Put the entire function call reply on one line<|eot|><|header_start|>user<|header_end|>
Use tools to get latest trending songs<|eot|><|header_start|>assistant<|header_end|> Use tools to get latest trending songs<|eot|><|header_start|>assistant<|header_end|>
@ -273,5 +268,54 @@ Use tools to get latest trending songs<|eot|><|header_start|>assistant<|header_e
##### Model Response Format ##### Model Response Format
``` ```
<function=trending_songs>{"n": "10"}</function><|eot|> <function=trending_songs>{"n": 10}</function><|eot|>
``` ```
## Parsing tool outputs
This example shows how the model interprets the output of a tool call and synthesizes it into a response.
##### Input Prompt Format
```
<|begin_of_text|><|header_start|>system<|header_end|>
You are an expert assistant who can answer general questions or invoke tools when necessary. You have access to the following functions:
Use the function 'trending_songs' to 'Returns the trending songs on a Music site':
{"name": "trending_songs", "description": "Returns the trending songs on a Music site", "parameters": {"genre": {"description": "The genre of the songs to return", "param_type": "str", "required": false}, "n": {"description": "The number of songs to return", "param_type": "int", "required": true}}}
Think very carefully before calling functions.
If you choose to call a function ONLY reply in the following format with no prefix or suffix:
<function=example_function_name>{"example_name": "example_value"}</function>
Reminder:
- Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- In addition to tool calls, you should also augment your responses by using the tool outputs.<|eot|><|header_start|>user<|header_end|>
Get the top 2 latest trending songs<|eot|><|header_start|>assistant<|header_end|>
<function=trending_songs>{"n": "2"}</function><|eot|><|header_start|>ipython<|header_end|>
[{"name": "Song 1", "artist": "Artist 1", "genre": "Genre 1"}, {"name": "Song 2", "artist": "Artist 2", "genre": "Genre 2"}]<|eom|><|header_start|>assistant<|header_end|>
```
##### Model Response Format
```
The top 2 latest trending songs are:
1. Song 1 by Artist 1 (Genre 1)
2. Song 2 by Artist 2 (Genre 2)<|eot|>
```
##### Notes
- Tool outputs should be passed back to the model in the `tool` role, which uses the `<|ipython|>` tag.
- The model parses the tool output contents until it encounters the `<|eom|>` tag. It uses this to synthesize an appropriate response to the query.

81
llama_stack/models/llama/llama4/prompts.py
View file

@ -185,16 +185,13 @@ def usecases(base_model: bool = False) -> List[UseCase | str]:
RawMessage( RawMessage(
role="system", role="system",
content="""You are an expert in composing functions. You are given a question and a set of possible functions. content="""You are an expert in composing functions. You are given a question and a set of possible functions.
Based on the question, you will need to make one or more function/tool calls to achieve the purpose. Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the function can be used, point it out. If the given question lacks the parameters required by the function, If none of the function can be used, point it out. If the given question lacks the parameters required by the function,
also point it out. You should only return the function call in tools call sections. also point it out. You should only return the function call in tools call sections.
If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]
If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] You SHOULD NOT include any other text in the response.
You SHOULD NOT include any other text in the response. Here is a list of functions in JSON format that you can invoke.
[
Here is a list of functions in JSON format that you can invoke.
[
{ {
"name": "get_weather", "name": "get_weather",
"description": "Get weather info for places", "description": "Get weather info for places",
@ -216,7 +213,7 @@ Here is a list of functions in JSON format that you can invoke.
} }
} }
} }
""", """,
), ),
RawMessage( RawMessage(
role="user", role="user",
@ -243,8 +240,8 @@ Here is a list of functions in JSON format that you can invoke.
RawMessage( RawMessage(
role="user", role="user",
content="""Questions: Can you retrieve the details for the user with the ID 7890, who has black as their special request? content="""Questions: Can you retrieve the details for the user with the ID 7890, who has black as their special request?
Here is a list of functions in JSON format that you can invoke: Here is a list of functions in JSON format that you can invoke:
[ [
{ {
"name": "get_user_info", "name": "get_user_info",
"description": "Retrieve details for a specific user by their unique identifier. Note that the provided function is in Python 3 syntax.", "description": "Retrieve details for a specific user by their unique identifier. Note that the provided function is in Python 3 syntax.",
@ -266,11 +263,9 @@ Here is a list of functions in JSON format that you can invoke:
} }
} }
} }
] ]
Should you decide to return the function call(s), put them in the format of [func1(params_name=params_value, params_name2=params_value2...), func2(params)]
Should you decide to return the function call(s), put them in the format of [func1(params_name=params_value, params_name2=params_value2...), func2(params)] You SHOULD NOT include any other text in the response.""",
You SHOULD NOT include any other text in the response.""",
), ),
] ]
], ],
@ -293,12 +288,12 @@ You SHOULD NOT include any other text in the response.""",
RawMessage( RawMessage(
role="user", role="user",
content="""You have access to the following functions:\nUse the function 'trending_songs' to 'Returns the trending songs on a Music site':\n{"name": "trending_songs", "description": "Returns the trending songs on a Music site", "parameters": {"genre": {"description": "The genre of the songs to return", "param_type": "str", "required": false}, "n": {"description": "The number of songs to return", "param_type": "int", "required": true}}}\n\nThink very carefully before calling functions.\nIf you choose to call a function ONLY reply in the following format with no prefix or suffix:\n\n<function=example_function_name>{"example_name": "example_value"}</function> content="""You have access to the following functions:\nUse the function 'trending_songs' to 'Returns the trending songs on a Music site':\n{"name": "trending_songs", "description": "Returns the trending songs on a Music site", "parameters": {"genre": {"description": "The genre of the songs to return", "param_type": "str", "required": false}, "n": {"description": "The number of songs to return", "param_type": "int", "required": true}}}\n\nThink very carefully before calling functions.\nIf you choose to call a function ONLY reply in the following format with no prefix or suffix:\n\n<function=example_function_name>{"example_name": "example_value"}</function>
Reminder: Reminder:
- If looking for real time information use relevant functions before falling back to brave_search - If looking for real time information use relevant functions before falling back to brave_search
- Function calls MUST follow the specified format, start with <function= and end with </function> - Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified - Required parameters MUST be specified
- Only call one function at a time - Only call one function at a time
- Put the entire function call reply on one line<|eot_id|>""", - Put the entire function call reply on one line""",
), ),
RawMessage( RawMessage(
role="user", role="user",
@ -307,6 +302,44 @@ Reminder:
] ]
], ],
), ),
Llama4UseCase(
title="Parsing tool outputs",
description=textwrap.dedent(
"""This example shows how the model interprets the output of a tool call and synthesizes it into a response."""
),
dialogs=[
[
RawMessage(
role="system",
content="""You are an expert assistant who can answer general questions or invoke tools when necessary. You have access to the following functions:\nUse the function 'trending_songs' to 'Returns the trending songs on a Music site':\n{"name": "trending_songs", "description": "Returns the trending songs on a Music site", "parameters": {"genre": {"description": "The genre of the songs to return", "param_type": "str", "required": false}, "n": {"description": "The number of songs to return", "param_type": "int", "required": true}}}\n\nThink very carefully before calling functions.\nIf you choose to call a function ONLY reply in the following format with no prefix or suffix:\n\n<function=example_function_name>{"example_name": "example_value"}</function>
Reminder:
- Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- In addition to tool calls, you should also augment your responses by using the tool outputs.""",
),
RawMessage(
role="user",
content="Get the top 2 latest trending songs",
),
RawMessage(
role="assistant",
content='<function=trending_songs>{"n": "2"}</function>',
),
RawMessage(
role="tool",
content='[{"name": "Song 1", "artist": "Artist 1", "genre": "Genre 1"}, {"name": "Song 2", "artist": "Artist 2", "genre": "Genre 2"}]',
),
]
],
notes=textwrap.dedent(
"""
- Tool outputs should be passed back to the model in the `tool` role, which uses the `<|ipython|>` tag.
- The model parses the tool output contents until it encounters the `<|eom|>` tag. It uses this to synthesize an appropriate response to the query.
"""
),
),
] ]
) )

2
scripts/generate_prompt_format.py
View file

@ -13,7 +13,7 @@
# the top-level of this source tree. # the top-level of this source tree.
# Run this script: # Run this script:
# torchrun --nproc_per_node=8 scripts/generate_prompt_format.py meta-llama/Llama-4-17B-Omni-Instruct-BF16-16E ~/.llama/checkpoints/Llama-4-17B-Omni-Instruct-BF16-16E/ llama_stack.models.llama.llama4.prompts llama_stack/models/llama/llama4/prompt_format.md # torchrun --nproc_per_node=8 scripts/generate_prompt_format.py meta-llama/Llama-4-Scout-17B-16E ~/.llama/checkpoints/Llama-4-Scout-17B-16E/ llama_stack.models.llama.llama4.prompts llama_stack/models/llama/llama4/prompt_format.md
import importlib import importlib