Xi Yan
1031f1404b
add register model to unit test
2024-11-11 10:35:59 -05:00
Xi Yan
e690eb7ad3
Merge branch 'main' into mmlu_benchmark
2024-11-11 10:22:32 -05:00
Dinesh Yeduguru
ec644d3418
migrate model to Resource and new registration signature ( #410 )
...
* resource oriented object design for models
* add back llama_model field
* working tests
* register singature fix
* address feedback
---------
Co-authored-by: Dinesh Yeduguru <dineshyv@fb.com>
2024-11-08 16:12:57 -08:00
Dalton Flanagan
5625aef48a
Add pip install helper for test and direct scenarios ( #404 )
...
* initial branch commit
* pip install helptext
* remove print
* pre-commit
2024-11-08 15:18:21 -05:00
Dinesh Yeduguru
d800a16acd
Resource oriented design for shields ( #399 )
...
* init
* working bedrock tests
* bedrock test for inference fixes
* use env vars for bedrock guardrail vars
* add register in meta reference
* use correct shield impl in meta ref
* dont add together fixture
* right naming
* minor updates
* improved registration flow
* address feedback
---------
Co-authored-by: Dinesh Yeduguru <dineshyv@fb.com>
2024-11-08 12:16:11 -08:00
Xi Yan
f429e75b3e
fix tests
2024-11-07 21:31:05 -08:00
Xi Yan
0443b36cc1
merge
2024-11-07 21:27:08 -08:00
Xi Yan
6192bf43a4
[Evals API][10/n] API updates for EvalTaskDef + new test migration ( #379 )
...
* wip
* scoring fn api
* eval api
* eval task
* evaluate api update
* pre commit
* unwrap context -> config
* config field doc
* typo
* naming fix
* separate benchmark / app eval
* api name
* rename
* wip tests
* wip
* datasetio test
* delete unused
* fixture
* scoring resolve
* fix scoring register
* scoring test pass
* score batch
* scoring fix
* fix eval
* test eval works
* remove type ignore
* api refactor
* add default task_eval_id for routing
* add eval_id for jobs
* remove type ignore
* only keep 1 run_eval
* fix optional
* register task required
* register task required
* delete old tests
* delete old tests
* fixture return impl
2024-11-07 21:24:12 -08:00
Xi Yan
edeb6dcf04
mmlu loose
2024-11-07 18:36:41 -08:00
Xi Yan
6ee02ca23b
fix
2024-11-07 18:25:39 -08:00
Xi Yan
33b6d9b7b7
merge
2024-11-07 18:15:13 -08:00
Xi Yan
027ee2335c
delete old tests
2024-11-07 18:06:21 -08:00
Xi Yan
94a56cc3f3
register task required
2024-11-07 16:41:23 -08:00
Xi Yan
fd581c3d88
only keep 1 run_eval
2024-11-07 16:17:49 -08:00
Xi Yan
37d87c585a
wip huggingface register
2024-11-07 15:59:55 -08:00
Xi Yan
d1633dc412
huggingface provider
2024-11-07 15:20:22 -08:00
Xi Yan
cc6edf6287
Merge branch 'eval_task_register' into mmlu_benchmark
2024-11-07 14:41:50 -08:00
Xi Yan
f05db9a25c
add eval_id for jobs
2024-11-07 14:30:46 -08:00
Xi Yan
51c20f9c29
api refactor
2024-11-07 13:54:26 -08:00
Xi Yan
97dcd5704c
Merge branch 'main' into eval_task_register
2024-11-07 13:08:58 -08:00
Ashwin Bharambe
694c142b89
Add provider deprecation support; change directory structure ( #397 )
...
* Add provider deprecation support; change directory structure
* fix a couple dangling imports
* move the meta_reference safety dir also
2024-11-07 13:04:53 -08:00
Xi Yan
93995ecc4c
test wip
2024-11-07 11:11:27 -08:00
Xi Yan
283b5c1def
Merge branch 'main' into eval_task_register
2024-11-06 21:50:09 -08:00
Xi Yan
3f1ac29d57
test eval works
2024-11-06 21:40:38 -08:00
Xi Yan
413a1b6d8b
fix eval
2024-11-06 21:10:54 -08:00
Ashwin Bharambe
489f74a70b
Allow simpler initialization of RemoteProviderConfig; fix issue in httpx client
2024-11-06 19:19:26 -08:00
Xi Yan
56239fce90
scoring fix
2024-11-06 18:07:16 -08:00
Ashwin Bharambe
064d2a5287
Remove the safety adapter for Together; we can just use "meta-reference" ( #387 )
2024-11-06 17:36:57 -08:00
Xi Yan
c5cf9c30be
score batch
2024-11-06 17:30:46 -08:00
Xi Yan
0bce74402f
scoring test pass
2024-11-06 17:27:55 -08:00
Xi Yan
0351072531
fix scoring register
2024-11-06 17:18:16 -08:00
Xi Yan
def6d5d8ad
scoring resolve
2024-11-06 17:04:25 -08:00
Xi Yan
c53733d1a3
fixture
2024-11-06 16:41:17 -08:00
Xi Yan
00869799a1
Merge branch 'main' into eval_task_register
2024-11-06 16:34:22 -08:00
Ashwin Bharambe
7c340f0236
rename test_inference -> test_text_inference
2024-11-06 16:12:50 -08:00
Xi Yan
10eda0af59
delete unused
2024-11-06 16:08:04 -08:00
Ashwin Bharambe
3b54ce3499
remote::vllm now works with vision models
2024-11-06 16:07:17 -08:00
Xi Yan
1fe4099bd0
datasetio test
2024-11-06 16:00:38 -08:00
Xi Yan
1b7e19d5d0
Merge branch 'main' into eval_task_register
2024-11-06 15:05:46 -08:00
Ashwin Bharambe
994732e2e0
impls -> inline, adapters -> remote (#381 )
2024-11-06 14:54:05 -08:00
Ashwin Bharambe
b10e9f46bb
Enable remote::vllm ( #384 )
...
* Enable remote::vllm
* Kill the giant list of hard coded models
2024-11-06 14:42:44 -08:00
Dinesh Yeduguru
6ebd553da5
fix routing tables look up key for memory bank ( #383 )
...
Co-authored-by: Dinesh Yeduguru <dineshyv@fb.com>
2024-11-06 13:32:46 -08:00
Xi Yan
683a370d23
wip tests
2024-11-06 10:03:49 -08:00
Ashwin Bharambe
cde9bc1388
Enable vision models for (Together, Fireworks, Meta-Reference, Ollama) ( #376 )
...
* Enable vision models for Together and Fireworks
* Works with ollama 0.4.0 pre-release with the vision model
* localize media for meta_reference inference
* Fix
2024-11-05 16:22:33 -08:00
Ashwin Bharambe
7cf4c905f3
add support for remote providers in tests
2024-11-04 20:30:46 -08:00
Ashwin Bharambe
ffedb81c11
Significantly simpler and malleable test setup ( #360 )
...
* Significantly simpler and malleable test setup
* convert memory tests
* refactor fixtures and add support for composable fixtures
* Fix memory to use the newer fixture organization
* Get agents tests working
* Safety tests work
* yet another refactor to make this more general
now it accepts --inference-model, --safety-model options also
* get multiple providers working for meta-reference (for inference + safety)
* Add README.md
---------
Co-authored-by: Ashwin Bharambe <ashwin@meta.com>
2024-11-04 17:36:43 -08:00
Ashwin Bharambe
37b330b4ef
add dynamic clients for all APIs ( #348 )
...
* add dynamic clients for all APIs
* fix openapi generator
* inference + memory + agents tests now pass with "remote" providers
* Add docstring which fixes openapi generator :/
2024-10-31 14:46:25 -07:00
Ashwin Bharambe
eccd7dc4a9
Avoid warnings from pydantic for overriding schema
...
Also fix structured output in completions
2024-10-28 21:39:48 -07:00
Xi Yan
ed833bb758
[Evals API][7/n] braintrust scoring provider ( #333 )
...
* wip scoring refactor
* llm as judge, move folders
* test full generation + eval
* extract score regex to llm context
* remove prints, cleanup braintrust in this branch
* braintrust skeleton
* datasetio test fix
* braintrust provider
* remove prints
* dependencies
* change json -> class
* json -> class
* remove initialize
* address nits
* check identifier prefix
* braintrust scoring identifier check, rebase
* udpate MANIFEST
* manifest
* remove braintrust scoring_fn
* remove comments
* tests
* imports fix
2024-10-28 18:59:35 -07:00
Xi Yan
7b8748c53e
[Evals API][6/n] meta-reference llm as judge, registration for ScoringFnDefs ( #330 )
...
* wip scoring refactor
* llm as judge, move folders
* test full generation + eval
* extract score regex to llm context
* remove prints, cleanup braintrust in this branch
* change json -> class
* remove initialize
* address nits
* check identifier prefix
* udpate MANIFEST
2024-10-28 14:08:42 -07:00
