4 commits

Author	SHA1	Message	Date
Xi Yan	d1f758abf6	change schema -> dataset_schema	2024-11-13 10:52:16 -05:00
Dinesh Yeduguru	fdff24e77a	Inference to use provider resource id to register and validate (#428) ... This PR changes the way model id gets translated to the final model name that gets passed through the provider. Major changes include: 1) Providers are responsible for registering an object and as part of the registration returning the object with the correct provider specific name of the model provider_resource_id 2) To help with the common look ups different names a new ModelLookup class is created. Tested all inference providers including together, fireworks, vllm, ollama, meta reference and bedrock	2024-11-12 20:02:00 -08:00
Dinesh Yeduguru	3802edfc50	migrate evals to resource (#421) ... * migrate evals to resource * remove listing of providers's evals * change the order of params in register * fix after rebase * linter fix --------- Co-authored-by: Dinesh Yeduguru <dineshyv@fb.com>	2024-11-11 17:24:03 -08:00
Xi Yan	b4416b72fd	Folder restructure for evals/datasets/scoring (#419) ... * rename evals related stuff * fix datasetio * fix scoring test * localfs -> LocalFS * refactor scoring * refactor scoring * remove 8b_correctness scoring_fn from tests * tests w/ eval params * scoring fn braintrust fixture * import	2024-11-11 17:35:40 -05:00

Renamed from llama_stack/providers/inline/meta_reference/eval/eval.py (Browse further)