phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-04 04:04:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix: Remove bfcl scoring function as not supported (#3281)
Some checks failed
Integration Tests (Replay) / Integration Tests (, , , client=, vision=) (push) Failing after 1s
Details
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 2s
Details
Pre-commit / pre-commit (push) Failing after 1s
Details
Test Llama Stack Build / build-single-provider (push) Failing after 1s
Details
Vector IO Integration Tests / test-matrix (push) Failing after 2s
Details
Test Llama Stack Build / build-custom-container-distribution (push) Failing after 0s
Details
Test Llama Stack Build / generate-matrix (push) Failing after 2s
Details
Test Llama Stack Build / build (push) Has been skipped
Details
Test Llama Stack Build / build-ubi9-container-distribution (push) Failing after 1s
Details
Python Package Build Test / build (3.12) (push) Failing after 0s
Details
Python Package Build Test / build (3.13) (push) Failing after 1s
Details
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Details
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 5s
Details
Test External API and Providers / test-external (venv) (push) Failing after 1s
Details
UI Tests / ui-tests (22) (push) Failing after 0s
Details
Unit Tests / unit-tests (3.12) (push) Failing after 1s
Details
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 8s
Details
Unit Tests / unit-tests (3.13) (push) Failing after 1s
Details
Update ReadTheDocs / update-readthedocs (push) Failing after 1s
Details

Browse source 
# What does this PR do?
BFCL scoring function is not supported, removing it. 

Also minor fixes as the llama stack run is broken for open-benchmark for
test plan verification
1. Correct the model paths for supported models
2. Fix another issue as there is no `provider_id` for DatasetInput but
logger assumes it exists.
``` 
File "/Users/swapna942/llama-stack/llama_stack/core/stack.py", line 332, in construct_stack
    await register_resources(run_config, impls)
  File "/Users/swapna942/llama-stack/llama_stack/core/stack.py", line 108, in register_resources
    logger.debug(f"registering {rsrc.capitalize()} {obj} for provider {obj.provider_id}")
                                                                       ^^^^^^^^^^^^^^^
  File "/Users/swapna942/llama-stack/.venv/lib/python3.13/site-packages/pydantic/main.py", line 991, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'DatasetInput' object has no attribute 'provider_id'
```

## Test Plan
```llama stack build --distro open-benchmark --image-type venv``` and run the server succeeds


Issue Link: https://github.com/llamastack/llama-stack/issues/3282
This commit is contained in:
slekkala1 2025-08-29 11:03:52 -07:00 • committed by GitHub
parent 3130ca0a78
commit efdb5558b8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 12 additions and 1482 deletions
Download patch file Download diff file Expand all files Collapse all files

16
llama_stack/distributions/open-benchmark/open_benchmark.py
View file

@ -43,7 +43,7 @@ def get_inference_providers() -> tuple[list[Provider], dict[str, list[ProviderMo
"openai",
[
ProviderModelEntry(
provider_model_id="openai/gpt-4o",
provider_model_id="gpt-4o",
model_type=ModelType.llm,
)
],
@ -53,7 +53,7 @@ def get_inference_providers() -> tuple[list[Provider], dict[str, list[ProviderMo
"anthropic",
[
ProviderModelEntry(
provider_model_id="anthropic/claude-3-5-sonnet-latest",
provider_model_id="claude-3-5-sonnet-latest",
model_type=ModelType.llm,
)
],
@ -206,13 +206,6 @@ def get_distribution_template() -> DistributionTemplate:
uri="huggingface://datasets/llamastack/math_500?split=test",
),
),
DatasetInput(
dataset_id="bfcl",
purpose=DatasetPurpose.eval_messages_answer,
source=URIDataSource(
uri="huggingface://datasets/llamastack/bfcl_v3?split=train",
),
),
DatasetInput(
dataset_id="ifeval",
purpose=DatasetPurpose.eval_messages_answer,
@ -250,11 +243,6 @@ def get_distribution_template() -> DistributionTemplate:
dataset_id="math_500",
scoring_functions=["basic::regex_parser_math_response"],
),
BenchmarkInput(
benchmark_id="meta-reference-bfcl",
dataset_id="bfcl",
scoring_functions=["basic::bfcl"],
),
BenchmarkInput(
benchmark_id="meta-reference-ifeval",
dataset_id="ifeval",