18 commits

Author	SHA1	Message	Date
Xi Yan	d1633dc412	huggingface provider	2024-11-07 15:20:22 -08:00
Xi Yan	cc6edf6287	Merge branch 'eval_task_register' into mmlu_benchmark	2024-11-07 14:41:50 -08:00
Xi Yan	f05db9a25c	add eval_id for jobs	2024-11-07 14:30:46 -08:00
Xi Yan	ea80f623fb	add default task_eval_id for routing	2024-11-07 14:19:33 -08:00
Xi Yan	51c20f9c29	api refactor	2024-11-07 13:54:26 -08:00
Xi Yan	97dcd5704c	Merge branch 'main' into eval_task_register	2024-11-07 13:08:58 -08:00
Ashwin Bharambe	694c142b89	Add provider deprecation support; change directory structure (#397) ... * Add provider deprecation support; change directory structure * fix a couple dangling imports * move the meta_reference safety dir also	2024-11-07 13:04:53 -08:00
Xi Yan	93995ecc4c	test wip	2024-11-07 11:11:27 -08:00
Xi Yan	3322aa9ee4	mmlu scoring fn	2024-11-07 10:54:00 -08:00
Xi Yan	b946afddc0	datasetdef files	2024-11-07 10:28:51 -08:00
Xi Yan	d75095033d	huggingface provider	2024-11-07 10:21:25 -08:00
Xi Yan	3f1ac29d57	test eval works	2024-11-06 21:40:38 -08:00
Xi Yan	413a1b6d8b	fix eval	2024-11-06 21:10:54 -08:00
Xi Yan	56239fce90	scoring fix	2024-11-06 18:07:16 -08:00
Xi Yan	0bce74402f	scoring test pass	2024-11-06 17:27:55 -08:00
Xi Yan	def6d5d8ad	scoring resolve	2024-11-06 17:04:25 -08:00
Xi Yan	8fc2d212a2	fix safety signature mismatch (#388) ... * fix safety sig * shield_type->identifier	2024-11-06 16:30:47 -08:00
Ashwin Bharambe	994732e2e0	impls -> inline, adapters -> remote (#381)	2024-11-06 14:54:05 -08:00