phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-06 10:42:39 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix

Browse source
This commit is contained in:
Sixian Yi 2025-01-22 18:27:42 -08:00
parent a55e56ed2b
commit 97f435b4df
4 changed files with 131 additions and 60 deletions
Download patch file Download diff file Expand all files Collapse all files

28
llama_stack/templates/fireworks/report.md
View file

@ -3,20 +3,20 @@
## Supported Models: ## Supported Models:
| Model Descriptor | fireworks | | Model Descriptor | fireworks |
|:---|:---| |:---|:---|
| meta-llama/Llama-3-8B-Instruct | ❌ | | Llama-3-8B-Instruct | ❌ |
| meta-llama/Llama-3-70B-Instruct | ❌ | | Llama-3-70B-Instruct | ❌ |
| meta-llama/Llama-3.1-8B-Instruct | ✅ | | Llama3.1-8B-Instruct | ✅ |
| meta-llama/Llama-3.1-70B-Instruct | ✅ | | Llama3.1-70B-Instruct | ✅ |
| meta-llama/Llama-3.1-405B-Instruct-FP8 | ✅ | | Llama3.1-405B-Instruct | ✅ |
| meta-llama/Llama-3.2-1B-Instruct | ✅ | | Llama3.2-1B-Instruct | ✅ |
| meta-llama/Llama-3.2-3B-Instruct | ✅ | | Llama3.2-3B-Instruct | ✅ |
| meta-llama/Llama-3.2-11B-Vision-Instruct | ✅ | | Llama3.2-11B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.2-90B-Vision-Instruct | ✅ | | Llama3.2-90B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.3-70B-Instruct | ✅ | | Llama3.3-70B-Instruct | ✅ |
| meta-llama/Llama-Guard-3-11B-Vision | ✅ | | Llama-Guard-3-11B-Vision | ✅ |
| meta-llama/Llama-Guard-3-1B | ❌ | | Llama-Guard-3-1B | ❌ |
| meta-llama/Llama-Guard-3-8B | ✅ | | Llama-Guard-3-8B | ✅ |
| meta-llama/Llama-Guard-2-8B | ❌ | | Llama-Guard-2-8B | ❌ |
## Inference: ## Inference:
| Model | API | Capability | Test | Status | | Model | API | Capability | Test | Status |

40
llama_stack/templates/ollama/report.md
View file

@ -3,27 +3,27 @@
## Supported Models: ## Supported Models:
| Model Descriptor | ollama | | Model Descriptor | ollama |
|:---|:---| |:---|:---|
| meta-llama/Llama-3-8B-Instruct | ❌ | | Llama-3-8B-Instruct | ❌ |
| meta-llama/Llama-3-70B-Instruct | ❌ | | Llama-3-70B-Instruct | ❌ |
| meta-llama/Llama-3.1-8B-Instruct | ✅ | | Llama3.1-8B-Instruct | ✅ |
| meta-llama/Llama-3.1-70B-Instruct | ❌ | | Llama3.1-70B-Instruct | ✅ |
| meta-llama/Llama-3.1-405B-Instruct-FP8 | ❌ | | Llama3.1-405B-Instruct | ✅ |
| meta-llama/Llama-3.2-1B-Instruct | ❌ | | Llama3.2-1B-Instruct | ✅ |
| meta-llama/Llama-3.2-3B-Instruct | ❌ | | Llama3.2-3B-Instruct | ✅ |
| meta-llama/Llama-3.2-11B-Vision-Instruct | ❌ | | Llama3.2-11B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.2-90B-Vision-Instruct | ❌ | | Llama3.2-90B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.3-70B-Instruct | ❌ | | Llama3.3-70B-Instruct | ✅ |
| meta-llama/Llama-Guard-3-11B-Vision | ❌ | | Llama-Guard-3-11B-Vision | ❌ |
| meta-llama/Llama-Guard-3-1B | ❌ | | Llama-Guard-3-1B | ✅ |
| meta-llama/Llama-Guard-3-8B | ❌ | | Llama-Guard-3-8B | ✅ |
| meta-llama/Llama-Guard-2-8B | ❌ | | Llama-Guard-2-8B | ❌ |
## Inference: ## Inference:
| Model | API | Capability | Test | Status | | Model | API | Capability | Test | Status |
|:----- |:-----|:-----|:-----|:-----| |:----- |:-----|:-----|:-----|:-----|
| Llama-3.1-8B-Instruct | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | streaming | test_text_chat_completion_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | streaming | test_image_chat_completion_streaming | ❌ | | Llama-3.2-11B-Vision-Instruct | /chat_completion | streaming | test_image_chat_completion_streaming | ❌ |
| Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ❌ | | Llama-3.2-11B-Vision-Instruct | /chat_completion | non_streaming | test_image_chat_completion_non_streaming | ❌ |
| Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | non_streaming | test_text_chat_completion_non_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_streaming | ✅ |
| Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ | | Llama-3.1-8B-Instruct | /chat_completion | tool_calling | test_text_chat_completion_with_tool_calling_and_non_streaming | ✅ |
@ -34,11 +34,11 @@
## Vector_io: ## Vector_io:
| API | Capability | Test | Status | | API | Capability | Test | Status |
|:-----|:-----|:-----|:-----| |:-----|:-----|:-----|:-----|
| retrieve | | test_vector_db_retrieve | ✅ | | /retrieve | | test_vector_db_retrieve | ✅ |
## Agents: ## Agents:
| API | Capability | Test | Status | | API | Capability | Test | Status |
|:-----|:-----|:-----|:-----| |:-----|:-----|:-----|:-----|
| create_agent_turn | rag | test_rag_agent | ✅ | | /create_agent_turn | rag | test_rag_agent | ✅ |
| create_agent_turn | custom_tool | test_custom_tool | ❌ | | /create_agent_turn | custom_tool | test_custom_tool | ✅ |
| create_agent_turn | code_execution | test_code_interpreter_for_attachments | ✅ | | /create_agent_turn | code_execution | test_code_interpreter_for_attachments | ✅ |

28
llama_stack/templates/together/report.md
View file

@ -3,20 +3,20 @@
## Supported Models: ## Supported Models:
| Model Descriptor | together | | Model Descriptor | together |
|:---|:---| |:---|:---|
| meta-llama/Llama-3-8B-Instruct | ❌ | | Llama-3-8B-Instruct | ❌ |
| meta-llama/Llama-3-70B-Instruct | ❌ | | Llama-3-70B-Instruct | ❌ |
| meta-llama/Llama-3.1-8B-Instruct | ✅ | | Llama3.1-8B-Instruct | ✅ |
| meta-llama/Llama-3.1-70B-Instruct | ✅ | | Llama3.1-70B-Instruct | ✅ |
| meta-llama/Llama-3.1-405B-Instruct-FP8 | ✅ | | Llama3.1-405B-Instruct | ✅ |
| meta-llama/Llama-3.2-1B-Instruct | ❌ | | Llama3.2-1B-Instruct | ❌ |
| meta-llama/Llama-3.2-3B-Instruct | ✅ | | Llama3.2-3B-Instruct | ✅ |
| meta-llama/Llama-3.2-11B-Vision-Instruct | ✅ | | Llama3.2-11B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.2-90B-Vision-Instruct | ✅ | | Llama3.2-90B-Vision-Instruct | ✅ |
| meta-llama/Llama-3.3-70B-Instruct | ✅ | | Llama3.3-70B-Instruct | ✅ |
| meta-llama/Llama-Guard-3-11B-Vision | ✅ | | Llama-Guard-3-11B-Vision | ✅ |
| meta-llama/Llama-Guard-3-1B | ❌ | | Llama-Guard-3-1B | ❌ |
| meta-llama/Llama-Guard-3-8B | ✅ | | Llama-Guard-3-8B | ✅ |
| meta-llama/Llama-Guard-2-8B | ❌ | | Llama-Guard-2-8B | ❌ |
## Inference: ## Inference:
| Model | API | Capability | Test | Status | | Model | API | Capability | Test | Status |

95
tests/client-sdk/report.py
View file

@ -12,8 +12,9 @@ from pathlib import Path
from urllib.parse import urlparse from urllib.parse import urlparse
import pytest import pytest
from llama_models.datatypes import CoreModelId
from llama_models.sku_list import ( from llama_models.sku_list import (
all_registered_models,
llama3_1_instruct_models, llama3_1_instruct_models,
llama3_2_instruct_models, llama3_2_instruct_models,
llama3_3_instruct_models, llama3_3_instruct_models,
@ -40,7 +41,79 @@ def featured_models_repo_names():
*llama3_3_instruct_models(), *llama3_3_instruct_models(),
*safety_models(), *safety_models(),
] ]
return [model.huggingface_repo for model in models if not model.variant] return [model for model in models if not model.variant]
SUPPORTED_MODELS = {
"ollama": set(
[
CoreModelId.llama3_1_8b_instruct.value,
CoreModelId.llama3_1_8b_instruct.value,
CoreModelId.llama3_1_70b_instruct.value,
CoreModelId.llama3_1_70b_instruct.value,
CoreModelId.llama3_1_405b_instruct.value,
CoreModelId.llama3_1_405b_instruct.value,
CoreModelId.llama3_2_1b_instruct.value,
CoreModelId.llama3_2_1b_instruct.value,
CoreModelId.llama3_2_3b_instruct.value,
CoreModelId.llama3_2_3b_instruct.value,
CoreModelId.llama3_2_11b_vision_instruct.value,
CoreModelId.llama3_2_11b_vision_instruct.value,
CoreModelId.llama3_2_90b_vision_instruct.value,
CoreModelId.llama3_2_90b_vision_instruct.value,
CoreModelId.llama3_3_70b_instruct.value,
CoreModelId.llama_guard_3_8b.value,
CoreModelId.llama_guard_3_1b.value,
]
),
"fireworks": set(
[
CoreModelId.llama3_1_8b_instruct.value,
CoreModelId.llama3_1_70b_instruct.value,
CoreModelId.llama3_1_405b_instruct.value,
CoreModelId.llama3_2_1b_instruct.value,
CoreModelId.llama3_2_3b_instruct.value,
CoreModelId.llama3_2_11b_vision_instruct.value,
CoreModelId.llama3_2_90b_vision_instruct.value,
CoreModelId.llama3_3_70b_instruct.value,
CoreModelId.llama_guard_3_8b.value,
CoreModelId.llama_guard_3_11b_vision.value,
]
),
"together": set(
[
CoreModelId.llama3_1_8b_instruct.value,
CoreModelId.llama3_1_70b_instruct.value,
CoreModelId.llama3_1_405b_instruct.value,
CoreModelId.llama3_2_3b_instruct.value,
CoreModelId.llama3_2_11b_vision_instruct.value,
CoreModelId.llama3_2_90b_vision_instruct.value,
CoreModelId.llama3_3_70b_instruct.value,
CoreModelId.llama_guard_3_8b.value,
CoreModelId.llama_guard_3_11b_vision.value,
]
),
"tgi": set(
[
model.core_model_id.value
for model in all_registered_models()
if model.huggingface_repo
]
),
"vllm": set(
[
model.core_model_id.value
for model in all_registered_models()
if model.huggingface_repo
]
),
"cerebras": set(
[
CoreModelId.llama3_1_8b_instruct.value,
CoreModelId.llama3_3_70b_instruct.value,
]
),
}
class Report: class Report:
@ -119,16 +192,14 @@ class Report:
report.append(dividor) report.append(dividor)
rows = [] rows = []
for model in all_registered_models():
try: if (
supported_models = {m.identifier for m in self.client.models.list()} "Instruct" not in model.core_model_id.value
except Exception as e: and "Guard" not in model.core_model_id.value
cprint(f"Error getting models: {e}", "red") ) or (model.variant):
supported_models = set() continue
row = f"| {model.core_model_id.value} |"
for m_name in featured_models_repo_names(): if model.core_model_id.value in SUPPORTED_MODELS[self.image_name]:
row = f"| {m_name} |"
if m_name in supported_models:
row += " ✅ |" row += " ✅ |"
else: else:
row += " ❌ |" row += " ❌ |"