mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-12-23 03:02:25 +00:00
fix: Resolve Llama4 tool calling 500 errors (Issue #2584)
This commit fixes the tool calling failures with Llama4 models that were
returning 500 errors while Together API worked correctly. The root cause
was that the system was using Llama3's JSON format for all models instead
of Llama4's python_list format.
Key changes:
- NEW: llama_stack/models/llama/llama4/interface.py - Complete Llama4 interface
with python_list tool format support
- MODIFIED: prompt_adapter.py - Added model-aware decode_assistant_message()
that uses Llama4ChatFormat for llama4 models and Llama3ChatFormat for others
- MODIFIED: openai_compat.py - Updated to pass model_id parameter to enable
model-specific format detection
- MODIFIED: sku_list.py - Enhanced with provider alias support for better
model resolution
- NEW: tests/unit/models/test_decode_assistant_message.py - Comprehensive unit
tests for the new decode_assistant_message function
The fix ensures that:
- Llama4 models (meta-llama/Llama-4-*) use python_list format: [func(args)]
- Other models continue using JSON format: {"type": "function", ...}
- Backward compatibility is maintained for existing models
- Tool calling works correctly across different model families
- Graceful fallback when Llama4 dependencies are unavailable
Testing:
- All 17 unit tests pass (9 original + 8 new)
- Conditional imports prevent torch dependency issues
- Comprehensive test coverage for different model types and scenarios
Fixes #2584
This commit is contained in:
parent
f731f369a2
commit
857496ea3e
5 changed files with 482 additions and 7 deletions
|
|
@ -22,6 +22,20 @@ def resolve_model(descriptor: str) -> Model | None:
|
|||
for m in all_registered_models():
|
||||
if descriptor in (m.descriptor(), m.huggingface_repo):
|
||||
return m
|
||||
|
||||
# Check provider aliases by attempting to import and check common providers
|
||||
try:
|
||||
from llama_stack.providers.remote.inference.together.models import MODEL_ENTRIES as TOGETHER_ENTRIES
|
||||
|
||||
for entry in TOGETHER_ENTRIES:
|
||||
if descriptor in entry.aliases and entry.llama_model:
|
||||
# Find the model by its descriptor
|
||||
for m in all_registered_models():
|
||||
if m.descriptor() == entry.llama_model:
|
||||
return m
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue