Botao Chen f369871083 feat: [New Eval Benchamark] IfEval (#1708) ... # What does this PR do? In this PR, we added a new eval open benchmark IfEval based on paper https://arxiv.org/abs/2311.07911 to measure the model capability of instruction following. ## Test Plan spin up a llama stack server with open-benchmark template run `llama-stack-client --endpoint xxx eval run-benchmark "meta-reference-ifeval" --model-id "meta-llama/Llama-3.3-70B-Instruct" --output-dir "/home/markchen1015/" --num-examples 20` on client side and get the eval aggregate results		2025-03-19 16:39:59 -07:00
..
agents	feat(agent): support multiple tool groups (#1556)	2025-03-17 22:13:09 -07:00
datasetio	fix: Call pandas.read_* in a seperate thread (#1698)	2025-03-19 10:46:37 -07:00
eval	feat(api): (1/n) datasets api clean up (#1573)	2025-03-17 16:55:45 -07:00
inference	fix: Updating ToolCall.arguments to allow for json strings that can be decoded on client side (#1685)	2025-03-19 10:36:19 -07:00
ios/inference	chore: removed executorch submodule (#1265)	2025-02-25 21:57:21 -08:00
post_training	chore: fix mypy violations in post_training modules (#1548)	2025-03-18 14:58:16 -07:00
safety	feat(agent): support multiple tool groups (#1556)	2025-03-17 22:13:09 -07:00
scoring	feat: [New Eval Benchamark] IfEval (#1708)	2025-03-19 16:39:59 -07:00
telemetry	refactor: move all datetime.now() calls to UTC (#1589)	2025-03-13 15:34:53 -07:00
tool_runtime	chore: Make code interpreter async (#1654)	2025-03-18 14:13:46 -07:00
vector_io	feat: Qdrant inline provider (#1273)	2025-03-18 14:04:21 -07:00
__init__.py	impls -> inline, adapters -> remote (#381)	2024-11-06 14:54:05 -08:00