phoenix-oss/llama-stack
1
0
Fork 0
forked from phoenix-oss/llama-stack-mirror
Code Pull requests Releases Packages 2 Activity Actions

add distro report (#847)

Browse source 
# What does this PR do?

Generate distro reports to cover inference, agents, and vector_io. 


## Test Plan

Report generated through `/opt/miniconda3/envs/stack/bin/pytest -s -v
tests/client-sdk/ --report`


## Sources

Please link relevant resources if necessary.


## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the
other checks if that's the case).
- [ ] Ran pre-commit to handle lint / formatting issues.
- [ ] Read the [contributor
guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md),
      Pull Request section?
- [ ] Updated relevant documentation.
- [ ] Wrote necessary unit or integration tests.
This commit is contained in:
Sixian Yi 2025-01-22 19:20:49 -08:00 • committed by GitHub
parent 23f1980f9c
commit 597869a2aa
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 328 additions and 60 deletions
Download patch file Download diff file Expand all files Collapse all files

44
llama_stack/templates/cerebras/report.md Normal file
View file

@ -0,0 +1,44 @@
# Report for cerebras distribution
## Supported Models:
| Model Descriptor | cerebras |
|:---|:---|
| meta-llama/Llama-3-8B-Instruct | ❌ |
| meta-llama/Llama-3-70B-Instruct | ❌ |
| meta-llama/Llama-3.1-8B-Instruct | ✅ |
| meta-llama/Llama-3.1-70B-Instruct | ❌ |
| meta-llama/Llama-3.1-405B-Instruct-FP8 | ❌ |
| meta-llama/Llama-3.2-1B-Instruct | ❌ |
| meta-llama/Llama-3.2-3B-Instruct | ❌ |
| meta-llama/Llama-3.2-11B-Vision-Instruct | ❌ |
| meta-llama/Llama-3.2-90B-Vision-Instruct | ❌ |
| meta-llama/Llama-3.3-70B-Instruct | ✅ |
| meta-llama/Llama-Guard-3-11B-Vision | ❌ |
| meta-llama/Llama-Guard-3-1B | ❌ |
| meta-llama/Llama-Guard-3-8B | ❌ |
| meta-llama/Llama-Guard-2-8B | ❌ |
## Inference:
| Model | API | Capability | Test | Status |
|:----- |:-----|:-----|:-----|:-----|
| Llama-3.1-8B-Instruct | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | streaming | test_image_chat_completion_streaming | ❌ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ❌ |
| Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | streaming | test_text_completion_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | non_streaming | test_text_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | structured_output | test_text_completion_structured_output | ❌ |
## Vector_io:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /retrieve | | test_vector_db_retrieve | ✅ |
## Agents:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /create_agent_turn | rag | test_rag_agent | ✅ |
| /create_agent_turn | custom_tool | test_custom_tool | ❌ |
| /create_agent_turn | code_execution | test_code_interpreter_for_attachments | ✅ |

57
llama_stack/templates/fireworks/report.md
View file

@ -3,43 +3,42 @@
## Supported Models: ## Supported Models:
| Model Descriptor | fireworks | | Model Descriptor | fireworks |
|:---|:---| |:---|:---|
| meta-llama/Llama-3-8B-Instruct | ❌ | | Llama-3-8B-Instruct | ❌ |
| meta-llama/Llama-3-70B-Instruct | ❌ | | Llama-3-70B-Instruct | ❌ |
| meta-llama/Llama-3.1-8B-Instruct | ✅ | | Llama3.1-8B-Instruct | ✅ |
| meta-llama/Llama-3.1-70B-Instruct | ✅ | | Llama3.1-70B-Instruct | ✅ |
| meta-llama/Llama-3.1-405B-Instruct-FP8 | ✅ | | Llama3.1-405B-Instruct | ✅ |
| meta-llama/Llama-3.2-1B-Instruct | ✅ | | Llama3.2-1B-Instruct | ✅ |
| meta-llama/Llama-3.2-3B-Instruct | ✅ | | Llama3.2-3B-Instruct | ✅ |
| meta-llama/Llama-3.2-11B-Vision-Instruct | ✅ | | Llama3.2-11B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.2-90B-Vision-Instruct | ✅ | | Llama3.2-90B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.3-70B-Instruct | ✅ | | Llama3.3-70B-Instruct | ✅ |
| meta-llama/Llama-Guard-3-11B-Vision | ✅ | | Llama-Guard-3-11B-Vision | ✅ |
| meta-llama/Llama-Guard-3-1B | ❌ | | Llama-Guard-3-1B | ❌ |
| meta-llama/Llama-Guard-3-8B | ✅ | | Llama-Guard-3-8B | ✅ |
| meta-llama/Llama-Guard-2-8B | ❌ | | Llama-Guard-2-8B | ❌ |
## Inference: ## Inference:
| Model | API | Capability | Test | Status | | Model | API | Capability | Test | Status |
|:----- |:-----|:-----|:-----|:-----| |:----- |:-----|:-----|:-----|:-----|
| Text | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ |
| Vision | /chat_completion | streaming | test_image_chat_completion_streaming | ✅ | | Llama-3.2-11B-Vision-Instruct | /chat_completion | streaming | test_image_chat_completion_streaming | ✅ |
| Vision | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ✅ | | Llama-3.2-11B-Vision-Instruct | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ✅ |
| Text | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ |
| Text | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ |
| Text | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ |
| Text | /completion | streaming | test_text_completion_streaming | ✅ | | Llama-3.1-8B-Instruct | /completion | streaming | test_text_completion_streaming | ✅ |
| Text | /completion | non_streaming | test_text_completion_non_streaming | ✅ | | Llama-3.1-8B-Instruct | /completion | non_streaming | test_text_completion_non_streaming | ✅ |
| Text | /completion | structured_output | test_text_completion_structured_output | ✅ | | Llama-3.1-8B-Instruct | /completion | structured_output | test_text_completion_structured_output | ✅ |
## Memory: ## Vector_io:
| API | Capability | Test | Status | | API | Capability | Test | Status |
|:-----|:-----|:-----|:-----| |:-----|:-----|:-----|:-----|
| /insert, /query | inline | test_memory_bank_insert_inline_and_query | ✅ | | /retrieve | | test_vector_db_retrieve | ✅ |
| /insert, /query | url | test_memory_bank_insert_from_url_and_query | ✅ |
## Agents: ## Agents:
| API | Capability | Test | Status | | API | Capability | Test | Status |
|:-----|:-----|:-----|:-----| |:-----|:-----|:-----|:-----|
| create_agent_turn | rag | test_rag_agent | ✅ | | /create_agent_turn | rag | test_rag_agent | ✅ |
| create_agent_turn | custom_tool | test_custom_tool | ✅ | | /create_agent_turn | custom_tool | test_custom_tool | ✅ |
| create_agent_turn | code_execution | test_code_execution | ❌ | | /create_agent_turn | code_execution | test_code_interpreter_for_attachments | ✅ |

44
llama_stack/templates/ollama/report.md Normal file
View file

@ -0,0 +1,44 @@
# Report for ollama distribution
## Supported Models:
| Model Descriptor | ollama |
|:---|:---|
| Llama-3-8B-Instruct | ❌ |
| Llama-3-70B-Instruct | ❌ |
| Llama3.1-8B-Instruct | ✅ |
| Llama3.1-70B-Instruct | ✅ |
| Llama3.1-405B-Instruct | ✅ |
| Llama3.2-1B-Instruct | ✅ |
| Llama3.2-3B-Instruct | ✅ |
| Llama3.2-11B-Vision-Instruct | ✅ |
| Llama3.2-90B-Vision-Instruct | ✅ |
| Llama3.3-70B-Instruct | ✅ |
| Llama-Guard-3-11B-Vision | ❌ |
| Llama-Guard-3-1B | ✅ |
| Llama-Guard-3-8B | ✅ |
| Llama-Guard-2-8B | ❌ |
## Inference:
| Model | API | Capability | Test | Status |
|:----- |:-----|:-----|:-----|:-----|
| Llama-3.1-8B-Instruct | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | streaming | test_image_chat_completion_streaming | ❌ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ❌ |
| Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | streaming | test_text_completion_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | non_streaming | test_text_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | structured_output | test_text_completion_structured_output | ✅ |
## Vector_io:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /retrieve | | test_vector_db_retrieve | ✅ |
## Agents:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /create_agent_turn | rag | test_rag_agent | ✅ |
| /create_agent_turn | custom_tool | test_custom_tool | ✅ |
| /create_agent_turn | code_execution | test_code_interpreter_for_attachments | ✅ |

44
llama_stack/templates/tgi/report.md Normal file
View file

@ -0,0 +1,44 @@
# Report for tgi distribution
## Supported Models:
| Model Descriptor | tgi |
|:---|:---|
| Llama-3-8B-Instruct | ✅ |
| Llama-3-70B-Instruct | ✅ |
| Llama3.1-8B-Instruct | ✅ |
| Llama3.1-70B-Instruct | ✅ |
| Llama3.1-405B-Instruct | ✅ |
| Llama3.2-1B-Instruct | ✅ |
| Llama3.2-3B-Instruct | ✅ |
| Llama3.2-11B-Vision-Instruct | ✅ |
| Llama3.2-90B-Vision-Instruct | ✅ |
| Llama3.3-70B-Instruct | ✅ |
| Llama-Guard-3-11B-Vision | ✅ |
| Llama-Guard-3-1B | ✅ |
| Llama-Guard-3-8B | ✅ |
| Llama-Guard-2-8B | ✅ |
## Inference:
| Model | API | Capability | Test | Status |
|:----- |:-----|:-----|:-----|:-----|
| Llama-3.1-8B-Instruct | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | streaming | test_image_chat_completion_streaming | ❌ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ❌ |
| Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | streaming | test_text_completion_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | non_streaming | test_text_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | structured_output | test_text_completion_structured_output | ✅ |
## Vector_io:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /retrieve | | test_vector_db_retrieve | ✅ |
## Agents:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /create_agent_turn | rag | test_rag_agent | ✅ |
| /create_agent_turn | custom_tool | test_custom_tool | ✅ |
| /create_agent_turn | code_execution | test_code_interpreter_for_attachments | ✅ |

44
llama_stack/templates/together/report.md Normal file
View file

@ -0,0 +1,44 @@
# Report for together distribution
## Supported Models:
| Model Descriptor | together |
|:---|:---|
| Llama-3-8B-Instruct | ❌ |
| Llama-3-70B-Instruct | ❌ |
| Llama3.1-8B-Instruct | ✅ |
| Llama3.1-70B-Instruct | ✅ |
| Llama3.1-405B-Instruct | ✅ |
| Llama3.2-1B-Instruct | ❌ |
| Llama3.2-3B-Instruct | ✅ |
| Llama3.2-11B-Vision-Instruct | ✅ |
| Llama3.2-90B-Vision-Instruct | ✅ |
| Llama3.3-70B-Instruct | ✅ |
| Llama-Guard-3-11B-Vision | ✅ |
| Llama-Guard-3-1B | ❌ |
| Llama-Guard-3-8B | ✅ |
| Llama-Guard-2-8B | ❌ |
## Inference:
| Model | API | Capability | Test | Status |
|:----- |:-----|:-----|:-----|:-----|
| Llama-3.1-8B-Instruct | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | streaming | test_image_chat_completion_streaming | ✅ |
| Llama-3.2-11B-Vision-Instruct | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | streaming | test_text_completion_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | non_streaming | test_text_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /completion | structured_output | test_text_completion_structured_output | ✅ |
## Vector_io:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /retrieve | | test_vector_db_retrieve | ✅ |
## Agents:
| API | Capability | Test | Status |
|:-----|:-----|:-----|:-----|
| /create_agent_turn | rag | test_rag_agent | ✅ |
| /create_agent_turn | custom_tool | test_custom_tool | ✅ |
| /create_agent_turn | code_execution | test_code_interpreter_for_attachments | ✅ |

14
tests/client-sdk/metadata.py
View file

@ -4,6 +4,7 @@
# This source code is licensed under the terms described in the LICENSE file in # This source code is licensed under the terms described in the LICENSE file in
# the root directory of this source tree. # the root directory of this source tree.
from llama_stack.providers.datatypes import Api
INFERENCE_API_CAPA_TEST_MAP = { INFERENCE_API_CAPA_TEST_MAP = {
"chat_completion": { "chat_completion": {
@ -27,10 +28,9 @@ INFERENCE_API_CAPA_TEST_MAP = {
}, },
} }
MEMORY_API_TEST_MAP = { VECTORIO_API_TEST_MAP = {
"/insert, /query": { "retrieve": {
"inline": ["test_memory_bank_insert_inline_and_query"], "": ["test_vector_db_retrieve"],
"url": ["test_memory_bank_insert_from_url_and_query"],
} }
} }
@ -44,7 +44,7 @@ AGENTS_API_TEST_MAP = {
API_MAPS = { API_MAPS = {
"inference": INFERENCE_API_CAPA_TEST_MAP, Api.inference: INFERENCE_API_CAPA_TEST_MAP,
"memory": MEMORY_API_TEST_MAP, Api.vector_io: VECTORIO_API_TEST_MAP,
"agents": AGENTS_API_TEST_MAP, Api.agents: AGENTS_API_TEST_MAP,
} }

103
tests/client-sdk/report.py
View file

@ -12,8 +12,9 @@ from pathlib import Path
from urllib.parse import urlparse from urllib.parse import urlparse
import pytest import pytest
from llama_models.datatypes import CoreModelId
from llama_models.sku_list import ( from llama_models.sku_list import (
all_registered_models,
llama3_1_instruct_models, llama3_1_instruct_models,
llama3_2_instruct_models, llama3_2_instruct_models,
llama3_3_instruct_models, llama3_3_instruct_models,
@ -22,6 +23,7 @@ from llama_models.sku_list import (
) )
from llama_stack.distribution.library_client import LlamaStackAsLibraryClient from llama_stack.distribution.library_client import LlamaStackAsLibraryClient
from llama_stack.providers.datatypes import Api
from llama_stack.providers.tests.env import get_env_or_fail from llama_stack.providers.tests.env import get_env_or_fail
from llama_stack_client import LlamaStackClient from llama_stack_client import LlamaStackClient
@ -42,6 +44,45 @@ def featured_models_repo_names():
return [model.huggingface_repo for model in models if not model.variant] return [model.huggingface_repo for model in models if not model.variant]
SUPPORTED_MODELS = {
"ollama": set(
[
CoreModelId.llama3_1_8b_instruct.value,
CoreModelId.llama3_1_8b_instruct.value,
CoreModelId.llama3_1_70b_instruct.value,
CoreModelId.llama3_1_70b_instruct.value,
CoreModelId.llama3_1_405b_instruct.value,
CoreModelId.llama3_1_405b_instruct.value,
CoreModelId.llama3_2_1b_instruct.value,
CoreModelId.llama3_2_1b_instruct.value,
CoreModelId.llama3_2_3b_instruct.value,
CoreModelId.llama3_2_3b_instruct.value,
CoreModelId.llama3_2_11b_vision_instruct.value,
CoreModelId.llama3_2_11b_vision_instruct.value,
CoreModelId.llama3_2_90b_vision_instruct.value,
CoreModelId.llama3_2_90b_vision_instruct.value,
CoreModelId.llama3_3_70b_instruct.value,
CoreModelId.llama_guard_3_8b.value,
CoreModelId.llama_guard_3_1b.value,
]
),
"tgi": set(
[
model.core_model_id.value
for model in all_registered_models()
if model.huggingface_repo
]
),
"vllm": set(
[
model.core_model_id.value
for model in all_registered_models()
if model.huggingface_repo
]
),
}
class Report: class Report:
def __init__(self): def __init__(self):
@ -90,6 +131,8 @@ class Report:
# test function -> test nodeid # test function -> test nodeid
self.test_data = dict() self.test_data = dict()
self.test_name_to_nodeid = defaultdict(list) self.test_name_to_nodeid = defaultdict(list)
self.vision_model_id = None
self.text_model_id = None
@pytest.hookimpl(tryfirst=True) @pytest.hookimpl(tryfirst=True)
def pytest_runtest_logreport(self, report): def pytest_runtest_logreport(self, report):
@ -113,20 +156,28 @@ class Report:
report.append(dividor) report.append(dividor)
rows = [] rows = []
if self.image_name in SUPPORTED_MODELS:
try: for model in all_registered_models():
if (
"Instruct" not in model.core_model_id.value
and "Guard" not in model.core_model_id.value
) or (model.variant):
continue
row = f"| {model.core_model_id.value} |"
if model.core_model_id.value in SUPPORTED_MODELS[self.image_name]:
row += " ✅ |"
else:
row += " ❌ |"
rows.append(row)
else:
supported_models = {m.identifier for m in self.client.models.list()} supported_models = {m.identifier for m in self.client.models.list()}
except Exception as e: for model in featured_models_repo_names():
cprint(f"Error getting models: {e}", "red") row = f"| {model} |"
supported_models = set() if model in supported_models:
row += " ✅ |"
for m_name in featured_models_repo_names(): else:
row = f"| {m_name} |" row += " ❌ |"
if m_name in supported_models: rows.append(row)
row += " ✅ |"
else:
row += " ❌ |"
rows.append(row)
report.extend(rows) report.extend(rows)
report.append("\n## Inference:") report.append("\n## Inference:")
@ -134,23 +185,28 @@ class Report:
"| Model | API | Capability | Test | Status |", "| Model | API | Capability | Test | Status |",
"|:----- |:-----|:-----|:-----|:-----|", "|:----- |:-----|:-----|:-----|:-----|",
] ]
for api, capa_map in API_MAPS["inference"].items(): for api, capa_map in API_MAPS[Api.inference].items():
for capa, tests in capa_map.items(): for capa, tests in capa_map.items():
for test_name in tests: for test_name in tests:
model_type = "Text" if "text" in test_name else "Vision" model_id = (
self.text_model_id
if "text" in test_name
else self.vision_model_id
)
test_nodeids = self.test_name_to_nodeid[test_name] test_nodeids = self.test_name_to_nodeid[test_name]
assert len(test_nodeids) > 0 assert len(test_nodeids) > 0
# There might be more than one parametrizations for the same test function. We take # There might be more than one parametrizations for the same test function. We take
# the result of the first one for now. Ideally we should mark the test as failed if # the result of the first one for now. Ideally we should mark the test as failed if
# any of the parametrizations failed. # any of the parametrizations failed.
test_table.append( test_table.append(
f"| {model_type} | /{api} | {capa} | {test_name} | {self._print_result_icon(self.test_data[test_nodeids[0]])} |" f"| {model_id} | /{api} | {capa} | {test_name} | {self._print_result_icon(self.test_data[test_nodeids[0]])} |"
) )
report.extend(test_table) report.extend(test_table)
for api_group in ["memory", "agents"]: for api_group in [Api.vector_io, Api.agents]:
api_capitalized = api_group.capitalize() api_capitalized = api_group.name.capitalize()
report.append(f"\n## {api_capitalized}:") report.append(f"\n## {api_capitalized}:")
test_table = [ test_table = [
"| API | Capability | Test | Status |", "| API | Capability | Test | Status |",
@ -162,7 +218,7 @@ class Report:
test_nodeids = self.test_name_to_nodeid[test_name] test_nodeids = self.test_name_to_nodeid[test_name]
assert len(test_nodeids) > 0 assert len(test_nodeids) > 0
test_table.append( test_table.append(
f"| {api} | {capa} | {test_name} | {self._print_result_icon(self.test_data[test_nodeids[0]])} |" f"| /{api} | {capa} | {test_name} | {self._print_result_icon(self.test_data[test_nodeids[0]])} |"
) )
report.extend(test_table) report.extend(test_table)
@ -173,6 +229,13 @@ class Report:
def pytest_runtest_makereport(self, item, call): def pytest_runtest_makereport(self, item, call):
func_name = getattr(item, "originalname", item.name) func_name = getattr(item, "originalname", item.name)
if "text_model_id" in item.funcargs:
text_model = item.funcargs["text_model_id"].split("/")[1]
self.text_model_id = self.text_model_id or text_model
elif "vision_model_id" in item.funcargs:
vision_model = item.funcargs["vision_model_id"].split("/")[1]
self.vision_model_id = self.vision_model_id or vision_model
self.test_name_to_nodeid[func_name].append(item.nodeid) self.test_name_to_nodeid[func_name].append(item.nodeid)
def _print_result_icon(self, result): def _print_result_icon(self, result):

38
tests/client-sdk/vector_io/test_vector_io.py
View file

@ -6,8 +6,36 @@
import random import random
import pytest
def test_vector_db_retrieve(llama_stack_client, embedding_model):
@pytest.fixture(scope="function")
def empty_vector_db_registry(llama_stack_client):
vector_dbs = [
vector_db.identifier for vector_db in llama_stack_client.vector_dbs.list()
]
for vector_db_id in vector_dbs:
llama_stack_client.vector_dbs.unregister(vector_db_id=vector_db_id)
@pytest.fixture(scope="function")
def single_entry_vector_db_registry(llama_stack_client, empty_vector_db_registry):
vector_db_id = f"test_vector_db_{random.randint(1000, 9999)}"
llama_stack_client.vector_dbs.register(
vector_db_id=vector_db_id,
embedding_model="all-MiniLM-L6-v2",
embedding_dimension=384,
provider_id="faiss",
)
vector_dbs = [
vector_db.identifier for vector_db in llama_stack_client.vector_dbs.list()
]
return vector_dbs
def test_vector_db_retrieve(
llama_stack_client, embedding_model, empty_vector_db_registry
):
# Register a memory bank first # Register a memory bank first
vector_db_id = f"test_vector_db_{random.randint(1000, 9999)}" vector_db_id = f"test_vector_db_{random.randint(1000, 9999)}"
llama_stack_client.vector_dbs.register( llama_stack_client.vector_dbs.register(
@ -26,14 +54,16 @@ def test_vector_db_retrieve(llama_stack_client, embedding_model):
assert response.provider_resource_id == vector_db_id assert response.provider_resource_id == vector_db_id
def test_vector_db_list(llama_stack_client): def test_vector_db_list(llama_stack_client, empty_vector_db_registry):
vector_dbs_after_register = [ vector_dbs_after_register = [
vector_db.identifier for vector_db in llama_stack_client.vector_dbs.list() vector_db.identifier for vector_db in llama_stack_client.vector_dbs.list()
] ]
assert len(vector_dbs_after_register) == 0 assert len(vector_dbs_after_register) == 0
def test_vector_db_register(llama_stack_client, embedding_model): def test_vector_db_register(
llama_stack_client, embedding_model, empty_vector_db_registry
):
vector_db_id = f"test_vector_db_{random.randint(1000, 9999)}" vector_db_id = f"test_vector_db_{random.randint(1000, 9999)}"
llama_stack_client.vector_dbs.register( llama_stack_client.vector_dbs.register(
vector_db_id=vector_db_id, vector_db_id=vector_db_id,
@ -48,7 +78,7 @@ def test_vector_db_register(llama_stack_client, embedding_model):
assert vector_dbs_after_register == [vector_db_id] assert vector_dbs_after_register == [vector_db_id]
def test_vector_db_unregister(llama_stack_client): def test_vector_db_unregister(llama_stack_client, single_entry_vector_db_registry):
vector_dbs = [ vector_dbs = [
vector_db.identifier for vector_db in llama_stack_client.vector_dbs.list() vector_db.identifier for vector_db in llama_stack_client.vector_dbs.list()
] ]