4 commits

Author	SHA1	Message	Date
Dinesh Yeduguru	fdff24e77a	Inference to use provider resource id to register and validate (#428) ... This PR changes the way model id gets translated to the final model name that gets passed through the provider. Major changes include: 1) Providers are responsible for registering an object and as part of the registration returning the object with the correct provider specific name of the model provider_resource_id 2) To help with the common look ups different names a new ModelLookup class is created. Tested all inference providers including together, fireworks, vllm, ollama, meta reference and bedrock	2024-11-12 20:02:00 -08:00
Xi Yan	84c6fbbd93	fix tests after registration migration & rename meta-reference -> basic / llm_as_judge provider (#424) ... * rename meta-reference -> basic * config rename * impl rename * rename llm_as_judge, fix test * util * rebase * naming fix	2024-11-12 10:35:44 -05:00
Dinesh Yeduguru	0a3b3d5fb6	migrate scoring fns to resource (#422) ... * fix after rebase * remove print --------- Co-authored-by: Dinesh Yeduguru <dineshyv@fb.com>	2024-11-11 17:28:48 -08:00
Xi Yan	b4416b72fd	Folder restructure for evals/datasets/scoring (#419) ... * rename evals related stuff * fix datasetio * fix scoring test * localfs -> LocalFS * refactor scoring * refactor scoring * remove 8b_correctness scoring_fn from tests * tests w/ eval params * scoring fn braintrust fixture * import	2024-11-11 17:35:40 -05:00