Xi Yan
6192bf43a4
[Evals API][10/n] API updates for EvalTaskDef + new test migration ( #379 )
...
* wip
* scoring fn api
* eval api
* eval task
* evaluate api update
* pre commit
* unwrap context -> config
* config field doc
* typo
* naming fix
* separate benchmark / app eval
* api name
* rename
* wip tests
* wip
* datasetio test
* delete unused
* fixture
* scoring resolve
* fix scoring register
* scoring test pass
* score batch
* scoring fix
* fix eval
* test eval works
* remove type ignore
* api refactor
* add default task_eval_id for routing
* add eval_id for jobs
* remove type ignore
* only keep 1 run_eval
* fix optional
* register task required
* register task required
* delete old tests
* delete old tests
* fixture return impl
2024-11-07 21:24:12 -08:00
Ashwin Bharambe
694c142b89
Add provider deprecation support; change directory structure ( #397 )
...
* Add provider deprecation support; change directory structure
* fix a couple dangling imports
* move the meta_reference safety dir also
2024-11-07 13:04:53 -08:00
Ashwin Bharambe
994732e2e0
impls -> inline, adapters -> remote (#381 )
2024-11-06 14:54:05 -08:00
Ashwin Bharambe
b10e9f46bb
Enable remote::vllm ( #384 )
...
* Enable remote::vllm
* Kill the giant list of hard coded models
2024-11-06 14:42:44 -08:00
Ashwin Bharambe
7cf4c905f3
add support for remote providers in tests
2024-11-04 20:30:46 -08:00
Ashwin Bharambe
ffedb81c11
Significantly simpler and malleable test setup ( #360 )
...
* Significantly simpler and malleable test setup
* convert memory tests
* refactor fixtures and add support for composable fixtures
* Fix memory to use the newer fixture organization
* Get agents tests working
* Safety tests work
* yet another refactor to make this more general
now it accepts --inference-model, --safety-model options also
* get multiple providers working for meta-reference (for inference + safety)
* Add README.md
---------
Co-authored-by: Ashwin Bharambe <ashwin@meta.com>
2024-11-04 17:36:43 -08:00
