phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-02 08:44:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

llama-4-scout-17b-16e-instruct passing tests

Browse source
This commit is contained in:
Matt Clayton 2025-04-25 13:38:40 -04:00
parent 6135bdec22
commit cfc6bdae68
3 changed files with 1150 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

46
tests/verifications/REPORT.md
View file

@ -19,6 +19,7 @@
| Together | 50.0% | 40 | 80 |
| Fireworks | 50.0% | 40 | 80 |
| Openai | 100.0% | 56 | 56 |
| Lmstudio | 100.0% | 24 | 24 |
@ -230,3 +231,48 @@ pytest tests/verifications/openai_api/test_chat_completion.py --provider=openai
| test_chat_streaming_tool_calling | ✅ | ✅ |
| test_chat_streaming_tool_choice_none | ✅ | ✅ |
| test_chat_streaming_tool_choice_required | ✅ | ✅ |
## Lmstudio
```bash
# Run all tests for this provider:
pytest tests/verifications/openai_api/test_chat_completion.py --provider=lmstudio -v
# Example: Run only the 'earth' case of test_chat_non_streaming_basic:
pytest tests/verifications/openai_api/test_chat_completion.py --provider=lmstudio -k "test_chat_non_streaming_basic and earth"
```
**Model Key (Lmstudio)**
| Display Name | Full Model ID |
| --- | --- |
| Llama-4-Scout-Instruct | `llama-4-scout-17b-16e-instruct` |
| Test | Llama-4-Scout-Instruct |
| --- | --- |
| test_chat_non_streaming_basic (earth) | ✅ |
| test_chat_non_streaming_basic (saturn) | ✅ |
| test_chat_non_streaming_multi_turn_tool_calling (add_product_tool) | ✅ |
| test_chat_non_streaming_multi_turn_tool_calling (compare_monthly_expense_tool) | ✅ |
| test_chat_non_streaming_multi_turn_tool_calling (get_then_create_event_tool) | ✅ |
| test_chat_non_streaming_multi_turn_tool_calling (text_then_weather_tool) | ✅ |
| test_chat_non_streaming_multi_turn_tool_calling (weather_tool_then_text) | ✅ |
| test_chat_non_streaming_structured_output (calendar) | ✅ |
| test_chat_non_streaming_structured_output (math) | ✅ |
| test_chat_non_streaming_tool_calling | ✅ |
| test_chat_non_streaming_tool_choice_none | ✅ |
| test_chat_non_streaming_tool_choice_required | ✅ |
| test_chat_streaming_basic (earth) | ✅ |
| test_chat_streaming_basic (saturn) | ✅ |
| test_chat_streaming_multi_turn_tool_calling (add_product_tool) | ✅ |
| test_chat_streaming_multi_turn_tool_calling (compare_monthly_expense_tool) | ✅ |
| test_chat_streaming_multi_turn_tool_calling (get_then_create_event_tool) | ✅ |
| test_chat_streaming_multi_turn_tool_calling (text_then_weather_tool) | ✅ |
| test_chat_streaming_multi_turn_tool_calling (weather_tool_then_text) | ✅ |
| test_chat_streaming_structured_output (calendar) | ✅ |
| test_chat_streaming_structured_output (math) | ✅ |
| test_chat_streaming_tool_calling | ✅ |
| test_chat_streaming_tool_choice_none | ✅ |
| test_chat_streaming_tool_choice_required | ✅ |

12
tests/verifications/conf/lmstudio.yaml
View file

@ -1,16 +1,10 @@
base_url: http://localhost:1234/v1/
models:
- meta-llama-3.1-8b-instruct
- llama-3.2-3b-instruct
- llama-4-scout-17b-16e-instruct
model_display_names:
meta-llama-3.1-8b-instruct: Llama-3.1-8b-Instruct
llama-3.2-3b-instruct: Llama-3.2-3b-Instruct
llama-4-scout-17b-16e-instruct: Llama-4-Scout-Instruct
test_exclusions:
meta-llama-3.1-8b-instruct:
- test_chat_non_streaming_image
- test_chat_streaming_image
- test_chat_multi_turn_multiple_images
llama-3.2-3b-instruct:
llama-4-scout-17b-16e-instruct:
- test_chat_non_streaming_image
- test_chat_streaming_image
- test_chat_multi_turn_multiple_images

1101
tests/verifications/test_results/lmstudio.json Normal file
View file

File diff suppressed because it is too large Load diff