Botao Chen e3edca7739 feat: [new open benchmark] Math 500 (#1538) ... ## What does this PR do? Created a new math_500 open-benchmark based on OpenAI's [Let's Verify Step by Step](https://arxiv.org/abs/2305.20050) paper and hugging face's [HuggingFaceH4/MATH-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500) dataset. The challenge part of this benchmark is to parse the generated and expected answer and verify if they are same. For the parsing part, we refer to [Minerva: Solving Quantitative Reasoning Problems with Language Models](https://research.google/blog/minerva-solving-quantitative-reasoning-problems-with-language-models/). To simply the parse logic, as the next step, we plan to also refer to what [simple-eval](https://github.com/openai/simple-evals) is doing, using llm as judge to check if the generated answer matches the expected answer or not ## Test Plan on sever side, spin up a server with open-benchmark template `llama stack run llama_stack/templates/open-benchamrk/run.yaml` on client side, issue an open benchmark eval request `llama-stack-client --endpoint xxx eval run-benchmark "meta-reference-math-500" --model-id "meta-llama/Llama-3.3-70B-Instruct" --output-dir "/home/markchen1015/" --num-examples 20` and get ther aggregated eval results <img width="238" alt="Screenshot 2025-03-10 at 7 57 04 PM" src="https://github.com/user-attachments/assets/2c9da042-3b70-470e-a7c4-69f4cc24d1fb" /> check the generated answer and the related scoring and they make sense		2025-03-10 20:38:28 -07:00
..
__init__.py	add missing __init__	2024-11-25 09:42:46 -08:00
aggregation_utils.py	chore(lint): update Ruff ignores for project conventions and maintainability (#1184)	2025-02-28 09:36:49 -08:00
base_scoring_fn.py	test: revamp eval related integration tests (#1433)	2025-03-06 10:51:35 -08:00
basic_scoring_utils.py	feat: [new open benchmark] Math 500 (#1538)	2025-03-10 20:38:28 -07:00