4 commits

Author	SHA1	Message	Date
Xi Yan	6192bf43a4	[Evals API][10/n] API updates for EvalTaskDef + new test migration (#379) ... * wip * scoring fn api * eval api * eval task * evaluate api update * pre commit * unwrap context -> config * config field doc * typo * naming fix * separate benchmark / app eval * api name * rename * wip tests * wip * datasetio test * delete unused * fixture * scoring resolve * fix scoring register * scoring test pass * score batch * scoring fix * fix eval * test eval works * remove type ignore * api refactor * add default task_eval_id for routing * add eval_id for jobs * remove type ignore * only keep 1 run_eval * fix optional * register task required * register task required * delete old tests * delete old tests * fixture return impl	2024-11-07 21:24:12 -08:00
Ashwin Bharambe	489f74a70b	Allow simpler initialization of RemoteProviderConfig; fix issue in httpx client	2024-11-06 19:19:26 -08:00
Ashwin Bharambe	7cf4c905f3	add support for remote providers in tests	2024-11-04 20:30:46 -08:00
Ashwin Bharambe	ffedb81c11	Significantly simpler and malleable test setup (#360) ... * Significantly simpler and malleable test setup * convert memory tests * refactor fixtures and add support for composable fixtures * Fix memory to use the newer fixture organization * Get agents tests working * Safety tests work * yet another refactor to make this more general now it accepts --inference-model, --safety-model options also * get multiple providers working for meta-reference (for inference + safety) * Add README.md --------- Co-authored-by: Ashwin Bharambe <ashwin@meta.com>	2024-11-04 17:36:43 -08:00