mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-06-27 18:50:41 +00:00
fix: loosen tool call checks in inference store (#2420)
# What does this PR do? This loosens up the tool call function name and arguments checks in `tests/integration/inference/test_openai_completion.py::test_inference_store_tool_calls` because the small models we use in CI cannot reliably get the tool call function name or arguments exactly right. Closes #2345 ## Test Plan I ran this flaking test in a loop, let it run many dozens of times, and didn't observe any flakes after the changes. Previously it flaked quite regularly. ``` while uv run pytest -s -v \ 'tests/integration/inference/test_openai_completion.py::test_inference_store_tool_calls[llama_stack_client-txt=3B-False]' \ --stack-config=http://localhost:8321 \ --text-model="meta-llama/Llama-3.2-3B-Instruct" \ --embedding-model=all-MiniLM-L6-v2; do; sleep 0.1; done ``` Signed-off-by: Ben Browning <bbrownin@redhat.com>
This commit is contained in:
parent
5639ad7466
commit
ee57e58f29
1 changed files with 7 additions and 3 deletions
|
@ -372,10 +372,14 @@ def test_inference_store_tool_calls(compat_client, client_with_models, text_mode
|
|||
)
|
||||
assert input_content == message, retrieved_response
|
||||
tool_calls = retrieved_response.choices[0].message.tool_calls
|
||||
# sometimes model doesn't ouptut tool calls, but we still want to test that the tool was called
|
||||
# sometimes model doesn't output tool calls, but we still want to test that the tool was called
|
||||
if tool_calls:
|
||||
# because we test with small models, just check that we retrieved
|
||||
# a tool call with a name and arguments string, but ignore contents
|
||||
assert len(tool_calls) == 1
|
||||
assert tool_calls[0].function.name == "get_weather"
|
||||
assert "tokyo" in tool_calls[0].function.arguments.lower()
|
||||
assert tool_calls[0].function.name
|
||||
assert tool_calls[0].function.arguments
|
||||
else:
|
||||
# failed tool call parses show up as a message with content, so ensure
|
||||
# that the retrieve response content matches the original request
|
||||
assert retrieved_response.choices[0].message.content == content
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue