phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-06-27 18:50:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
llama-stack-mirror/tests/verifications/REPORT.md
ehhuang bcbc56baa2
feat: adds test suite to verify provider's OAI compat endpoints (#1901) 
# What does this PR do?


## Test Plan
pytest verifications/openai/test_chat_completion.py --provider together
2025-04-08 21:21:38 -07:00

3.2 KiB
Raw Blame History

Test Results Report

Generated on: 2025-04-08 21:14:02

This report was generated by running python tests/verifications/generate_report.py

Legend

  • ✅ - Test passed
  • ❌ - Test failed
  • ⚪ - Test not applicable or not run for this model

Summary

Provider Pass Rate Tests Passed Total Tests
Together 67.7% 21 31
Fireworks 90.3% 28 31
Openai 100.0% 22 22

Together

Tests run on: 2025-04-08 16:19:59

pytest tests/verifications/openai/test_chat_completion.py --provider=together -v
Test Llama-3.3-70B-Instruct Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct
test_chat_non_streaming_basic (case 0) ✅ ✅ ✅
test_chat_non_streaming_basic (case 1) ✅ ✅ ✅
test_chat_non_streaming_image (case 0) ⚪ ✅ ✅
test_chat_non_streaming_structured_output (case 0) ✅ ✅ ✅
test_chat_non_streaming_structured_output (case 1) ✅ ✅ ✅
test_chat_non_streaming_tool_calling (case 0) ✅ ✅ ✅
test_chat_streaming_basic (case 0) ✅ ❌ ❌
test_chat_streaming_basic (case 1) ✅ ❌ ❌
test_chat_streaming_image (case 0) ⚪ ❌ ❌
test_chat_streaming_structured_output (case 0) ✅ ❌ ❌
test_chat_streaming_structured_output (case 1) ✅ ❌ ❌

Fireworks

Tests run on: 2025-04-08 16:18:28

pytest tests/verifications/openai/test_chat_completion.py --provider=fireworks -v
Test Llama-3.3-70B-Instruct Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct
test_chat_non_streaming_basic (case 0) ✅ ✅ ✅
test_chat_non_streaming_basic (case 1) ✅ ✅ ✅
test_chat_non_streaming_image (case 0) ⚪ ✅ ✅
test_chat_non_streaming_structured_output (case 0) ✅ ✅ ✅
test_chat_non_streaming_structured_output (case 1) ✅ ✅ ✅
test_chat_non_streaming_tool_calling (case 0) ✅ ❌ ❌
test_chat_streaming_basic (case 0) ✅ ✅ ✅
test_chat_streaming_basic (case 1) ✅ ✅ ✅
test_chat_streaming_image (case 0) ⚪ ✅ ✅
test_chat_streaming_structured_output (case 0) ✅ ✅ ✅
test_chat_streaming_structured_output (case 1) ❌ ✅ ✅

Openai

Tests run on: 2025-04-08 16:22:02

pytest tests/verifications/openai/test_chat_completion.py --provider=openai -v
Test gpt-4o gpt-4o-mini
test_chat_non_streaming_basic (case 0) ✅ ✅
test_chat_non_streaming_basic (case 1) ✅ ✅
test_chat_non_streaming_image (case 0) ✅ ✅
test_chat_non_streaming_structured_output (case 0) ✅ ✅
test_chat_non_streaming_structured_output (case 1) ✅ ✅
test_chat_non_streaming_tool_calling (case 0) ✅ ✅
test_chat_streaming_basic (case 0) ✅ ✅
test_chat_streaming_basic (case 1) ✅ ✅
test_chat_streaming_image (case 0) ✅ ✅
test_chat_streaming_structured_output (case 0) ✅ ✅
test_chat_streaming_structured_output (case 1) ✅ ✅