Xi Yan
f429e75b3e
fix tests
2024-11-07 21:31:05 -08:00
Xi Yan
0443b36cc1
merge
2024-11-07 21:27:08 -08:00
Xi Yan
6192bf43a4
[Evals API][10/n] API updates for EvalTaskDef + new test migration ( #379 )
...
* wip
* scoring fn api
* eval api
* eval task
* evaluate api update
* pre commit
* unwrap context -> config
* config field doc
* typo
* naming fix
* separate benchmark / app eval
* api name
* rename
* wip tests
* wip
* datasetio test
* delete unused
* fixture
* scoring resolve
* fix scoring register
* scoring test pass
* score batch
* scoring fix
* fix eval
* test eval works
* remove type ignore
* api refactor
* add default task_eval_id for routing
* add eval_id for jobs
* remove type ignore
* only keep 1 run_eval
* fix optional
* register task required
* register task required
* delete old tests
* delete old tests
* fixture return impl
2024-11-07 21:24:12 -08:00
Xi Yan
edeb6dcf04
mmlu loose
2024-11-07 18:36:41 -08:00
Xi Yan
6ee02ca23b
fix
2024-11-07 18:25:39 -08:00
Xi Yan
33b6d9b7b7
merge
2024-11-07 18:15:13 -08:00
Xi Yan
027ee2335c
delete old tests
2024-11-07 18:06:21 -08:00
Xi Yan
94a56cc3f3
register task required
2024-11-07 16:41:23 -08:00
Xi Yan
fd581c3d88
only keep 1 run_eval
2024-11-07 16:17:49 -08:00
Xi Yan
37d87c585a
wip huggingface register
2024-11-07 15:59:55 -08:00
Xi Yan
d1633dc412
huggingface provider
2024-11-07 15:20:22 -08:00
Xi Yan
cc6edf6287
Merge branch 'eval_task_register' into mmlu_benchmark
2024-11-07 14:41:50 -08:00
Xi Yan
f05db9a25c
add eval_id for jobs
2024-11-07 14:30:46 -08:00
Xi Yan
51c20f9c29
api refactor
2024-11-07 13:54:26 -08:00
Xi Yan
97dcd5704c
Merge branch 'main' into eval_task_register
2024-11-07 13:08:58 -08:00
Ashwin Bharambe
694c142b89
Add provider deprecation support; change directory structure ( #397 )
...
* Add provider deprecation support; change directory structure
* fix a couple dangling imports
* move the meta_reference safety dir also
2024-11-07 13:04:53 -08:00
Xi Yan
93995ecc4c
test wip
2024-11-07 11:11:27 -08:00
Xi Yan
283b5c1def
Merge branch 'main' into eval_task_register
2024-11-06 21:50:09 -08:00
Xi Yan
3f1ac29d57
test eval works
2024-11-06 21:40:38 -08:00
Xi Yan
413a1b6d8b
fix eval
2024-11-06 21:10:54 -08:00
Ashwin Bharambe
489f74a70b
Allow simpler initialization of RemoteProviderConfig; fix issue in httpx client
2024-11-06 19:19:26 -08:00
Xi Yan
56239fce90
scoring fix
2024-11-06 18:07:16 -08:00
Ashwin Bharambe
064d2a5287
Remove the safety adapter for Together; we can just use "meta-reference" ( #387 )
2024-11-06 17:36:57 -08:00
Xi Yan
c5cf9c30be
score batch
2024-11-06 17:30:46 -08:00
Xi Yan
0bce74402f
scoring test pass
2024-11-06 17:27:55 -08:00
Xi Yan
0351072531
fix scoring register
2024-11-06 17:18:16 -08:00
Xi Yan
def6d5d8ad
scoring resolve
2024-11-06 17:04:25 -08:00
Xi Yan
c53733d1a3
fixture
2024-11-06 16:41:17 -08:00
Xi Yan
00869799a1
Merge branch 'main' into eval_task_register
2024-11-06 16:34:22 -08:00
Ashwin Bharambe
7c340f0236
rename test_inference -> test_text_inference
2024-11-06 16:12:50 -08:00
Xi Yan
10eda0af59
delete unused
2024-11-06 16:08:04 -08:00
Ashwin Bharambe
3b54ce3499
remote::vllm now works with vision models
2024-11-06 16:07:17 -08:00
Xi Yan
1fe4099bd0
datasetio test
2024-11-06 16:00:38 -08:00
Xi Yan
1b7e19d5d0
Merge branch 'main' into eval_task_register
2024-11-06 15:05:46 -08:00
Ashwin Bharambe
994732e2e0
impls -> inline, adapters -> remote (#381 )
2024-11-06 14:54:05 -08:00
Ashwin Bharambe
b10e9f46bb
Enable remote::vllm ( #384 )
...
* Enable remote::vllm
* Kill the giant list of hard coded models
2024-11-06 14:42:44 -08:00
Dinesh Yeduguru
6ebd553da5
fix routing tables look up key for memory bank ( #383 )
...
Co-authored-by: Dinesh Yeduguru <dineshyv@fb.com>
2024-11-06 13:32:46 -08:00
Xi Yan
683a370d23
wip tests
2024-11-06 10:03:49 -08:00
Ashwin Bharambe
cde9bc1388
Enable vision models for (Together, Fireworks, Meta-Reference, Ollama) ( #376 )
...
* Enable vision models for Together and Fireworks
* Works with ollama 0.4.0 pre-release with the vision model
* localize media for meta_reference inference
* Fix
2024-11-05 16:22:33 -08:00
Ashwin Bharambe
7cf4c905f3
add support for remote providers in tests
2024-11-04 20:30:46 -08:00
Ashwin Bharambe
ffedb81c11
Significantly simpler and malleable test setup ( #360 )
...
* Significantly simpler and malleable test setup
* convert memory tests
* refactor fixtures and add support for composable fixtures
* Fix memory to use the newer fixture organization
* Get agents tests working
* Safety tests work
* yet another refactor to make this more general
now it accepts --inference-model, --safety-model options also
* get multiple providers working for meta-reference (for inference + safety)
* Add README.md
---------
Co-authored-by: Ashwin Bharambe <ashwin@meta.com>
2024-11-04 17:36:43 -08:00
Ashwin Bharambe
37b330b4ef
add dynamic clients for all APIs ( #348 )
...
* add dynamic clients for all APIs
* fix openapi generator
* inference + memory + agents tests now pass with "remote" providers
* Add docstring which fixes openapi generator :/
2024-10-31 14:46:25 -07:00
Ashwin Bharambe
eccd7dc4a9
Avoid warnings from pydantic for overriding schema
...
Also fix structured output in completions
2024-10-28 21:39:48 -07:00
Xi Yan
ed833bb758
[Evals API][7/n] braintrust scoring provider ( #333 )
...
* wip scoring refactor
* llm as judge, move folders
* test full generation + eval
* extract score regex to llm context
* remove prints, cleanup braintrust in this branch
* braintrust skeleton
* datasetio test fix
* braintrust provider
* remove prints
* dependencies
* change json -> class
* json -> class
* remove initialize
* address nits
* check identifier prefix
* braintrust scoring identifier check, rebase
* udpate MANIFEST
* manifest
* remove braintrust scoring_fn
* remove comments
* tests
* imports fix
2024-10-28 18:59:35 -07:00
Xi Yan
7b8748c53e
[Evals API][6/n] meta-reference llm as judge, registration for ScoringFnDefs ( #330 )
...
* wip scoring refactor
* llm as judge, move folders
* test full generation + eval
* extract score regex to llm context
* remove prints, cleanup braintrust in this branch
* change json -> class
* remove initialize
* address nits
* check identifier prefix
* udpate MANIFEST
2024-10-28 14:08:42 -07:00
Ashwin Bharambe
b7d2b83d55
Allow passing provider_registry to resolve_impls()
2024-10-28 11:58:16 -07:00
Dinesh Yeduguru
9b85d9a841
completion() for fireworks ( #329 )
2024-10-25 16:12:10 -07:00
Dinesh Yeduguru
7ec79f3b9d
completion() for together ( #324 )
...
* completion() for together
* test fixes
* fix client building
2024-10-25 14:21:12 -07:00
Xi Yan
abdf7cddf3
[Evals API][4/n] evals with generation meta-reference impl ( #303 )
...
* wip
* dataset validation
* test_scoring
* cleanup
* clean up test
* comments
* error checking
* dataset client
* test client:
* datasetio client
* clean up
* basic scoring function works
* scorer wip
* equality scorer
* score batch impl
* score batch
* update scoring test
* refactor
* validate scorer input
* address comments
* evals with generation
* add all rows scores to ScoringResult
* minor typing
* bugfix
* scoring function def rename
* rebase name
* refactor
* address comments
* Update iOS inference instructions for new quantization
* Small updates to quantization config
* Fix score threshold in faiss
* Bump version to 0.0.45
* Handle both ipv6 and ipv4 interfaces together
* update manifest for build templates
* Update getting_started.md
* chatcompletion & completion input type validation
* inclusion->subsetof
* error checking
* scoring_function -> scoring_fn rename, scorer -> scoring_fn rename
* address comments
* [Evals API][5/n] fixes to generate openapi spec (#323 )
* generate openapi
* typing comment, dataset -> dataset_id
* remove custom type
* sample eval run.yaml
---------
Co-authored-by: Dalton Flanagan <6599399+dltn@users.noreply.github.com>
Co-authored-by: Ashwin Bharambe <ashwin.bharambe@gmail.com>
2024-10-25 13:12:39 -07:00
Dinesh Yeduguru
3e1c3fdb3f
completion() for tgi ( #295 )
2024-10-24 16:02:41 -07:00
