Xi Yan
8e1a182b57
imports fix
2024-10-28 18:58:11 -07:00
Xi Yan
a48a16cd53
tests
2024-10-28 14:13:13 -07:00
Xi Yan
e2aa592e26
Merge branch 'main' into evals_9
2024-10-28 14:11:51 -07:00
Xi Yan
7b8748c53e
[Evals API][6/n] meta-reference llm as judge, registration for ScoringFnDefs ( #330 )
...
* wip scoring refactor
* llm as judge, move folders
* test full generation + eval
* extract score regex to llm context
* remove prints, cleanup braintrust in this branch
* change json -> class
* remove initialize
* address nits
* check identifier prefix
* udpate MANIFEST
2024-10-28 14:08:42 -07:00
Xi Yan
04a4784287
Update README.md
2024-10-28 13:25:44 -07:00
Xi Yan
3fa1eaf37d
Update README.md
2024-10-28 13:18:55 -07:00
Xi Yan
0d4215e125
Update README.md
2024-10-28 13:18:34 -07:00
Xi Yan
8f5a850de9
Update README.md
2024-10-28 13:16:23 -07:00
Xi Yan
f3aab94029
Merge branch 'evals_8' into evals_9
2024-10-28 13:08:42 -07:00
Xi Yan
dc79f1c2c6
Merge branch 'main' into evals_8
2024-10-28 13:08:28 -07:00
Xi Yan
ffb3965ade
remove Field for return_type
2024-10-28 13:04:41 -07:00
Xi Yan
ccbe1756ae
remove comments
2024-10-28 12:42:02 -07:00
Xi Yan
f5d41b582e
remove braintrust scoring_fn
2024-10-28 12:41:06 -07:00
Ashwin Bharambe
b7d2b83d55
Allow passing provider_registry to resolve_impls()
2024-10-28 11:58:16 -07:00
Xi Yan
b5ed80ac15
manifest
2024-10-28 11:57:54 -07:00
Xi Yan
8a142ba210
merge
2024-10-28 11:57:40 -07:00
Xi Yan
09ad08afd4
udpate MANIFEST
2024-10-28 11:57:08 -07:00
Xi Yan
93d6c1a4e6
braintrust scoring identifier check, rebase
2024-10-28 11:45:37 -07:00
Xi Yan
488b967d33
Merge branch 'evals_8' into evals_9
2024-10-28 11:43:04 -07:00
Xi Yan
3a4039aea9
check identifier prefix
2024-10-28 11:42:07 -07:00
Xi Yan
e3f80fa4aa
address nits
2024-10-28 11:38:14 -07:00
Xi Yan
8627e27b17
remove initialize
2024-10-28 11:35:08 -07:00
Xi Yan
e2f6c04e1c
json -> class
2024-10-28 11:30:02 -07:00
Xi Yan
0dad0d0d67
Merge branch 'evals_8' into evals_9
2024-10-28 11:26:18 -07:00
Xi Yan
8aee752c19
change json -> class
2024-10-28 11:24:32 -07:00
Ashwin Bharambe
8a3b64d1be
Bump version to 0.0.47
2024-10-27 22:30:38 -07:00
Xi Yan
e567a60007
dependencies
2024-10-27 17:31:00 -07:00
Xi Yan
ae75eb0f95
remove prints
2024-10-27 17:29:41 -07:00
Xi Yan
38186f7903
braintrust provider
2024-10-27 17:24:10 -07:00
Xi Yan
68346fac39
datasetio test fix
2024-10-27 16:43:02 -07:00
Xi Yan
d3d2243dfb
braintrust skeleton
2024-10-27 12:32:07 -07:00
Xi Yan
caf6a266e0
Merge branch 'main' into evals_8
2024-10-27 12:08:23 -07:00
Xi Yan
46bb8884a7
distributions readme typos
2024-10-27 11:57:21 -07:00
Xi Yan
46d3e64c9a
Merge branch 'main' into evals_8
2024-10-27 11:42:00 -07:00
Dalton Flanagan
44c05c6e7d
add vision instruct models for fireworks
2024-10-27 17:54:54 +00:00
Xi Yan
91e5ad18b0
remove prints, cleanup braintrust in this branch
2024-10-25 17:39:05 -07:00
Xi Yan
9b410a87bf
extract score regex to llm context
2024-10-25 17:03:01 -07:00
Xi Yan
247a53d393
test full generation + eval
2024-10-25 16:52:59 -07:00
Xi Yan
16620a8185
llm as judge, move folders
2024-10-25 16:41:36 -07:00
Dinesh Yeduguru
9b85d9a841
completion() for fireworks ( #329 )
2024-10-25 16:12:10 -07:00
Xi Yan
bf8bc7a781
wip scoring refactor
2024-10-25 15:03:03 -07:00
Dinesh Yeduguru
7ec79f3b9d
completion() for together ( #324 )
...
* completion() for together
* test fixes
* fix client building
2024-10-25 14:21:12 -07:00
Xi Yan
8a74e400d6
Update getting_started.md
2024-10-25 13:30:33 -07:00
Xi Yan
f168752bba
Update getting_started.md
2024-10-25 13:27:43 -07:00
Xi Yan
abdf7cddf3
[Evals API][4/n] evals with generation meta-reference impl ( #303 )
...
* wip
* dataset validation
* test_scoring
* cleanup
* clean up test
* comments
* error checking
* dataset client
* test client:
* datasetio client
* clean up
* basic scoring function works
* scorer wip
* equality scorer
* score batch impl
* score batch
* update scoring test
* refactor
* validate scorer input
* address comments
* evals with generation
* add all rows scores to ScoringResult
* minor typing
* bugfix
* scoring function def rename
* rebase name
* refactor
* address comments
* Update iOS inference instructions for new quantization
* Small updates to quantization config
* Fix score threshold in faiss
* Bump version to 0.0.45
* Handle both ipv6 and ipv4 interfaces together
* update manifest for build templates
* Update getting_started.md
* chatcompletion & completion input type validation
* inclusion->subsetof
* error checking
* scoring_function -> scoring_fn rename, scorer -> scoring_fn rename
* address comments
* [Evals API][5/n] fixes to generate openapi spec (#323 )
* generate openapi
* typing comment, dataset -> dataset_id
* remove custom type
* sample eval run.yaml
---------
Co-authored-by: Dalton Flanagan <6599399+dltn@users.noreply.github.com>
Co-authored-by: Ashwin Bharambe <ashwin.bharambe@gmail.com>
2024-10-25 13:12:39 -07:00
Ashwin Bharambe
426d821e7f
Bump version to 0.0.46
2024-10-25 13:10:55 -07:00
Sachin Mehta
c05fbf14b3
Added hadamard transform for spinquant ( #326 )
...
* Added hadamard transform for spinquant
* Changed from config to model_args
* Added an assertion for model args
* Use enum.value to check against str
* pre-commit
---------
Co-authored-by: Sachin Mehta <sacmehta@fb.com>
Co-authored-by: Ashwin Bharambe <ashwin.bharambe@gmail.com>
2024-10-25 12:58:48 -07:00
Xi Yan
07f9bf723f
fix broken --list-templates with adding build.yaml files for packaging ( #327 )
...
* add build files to templates
* fix templates
* manifest
* symlink
* symlink
* precommit
* change everything to docker build.yaml
* remove image_type in templates
* fix build from templates CLI
* fix readmes
2024-10-25 12:51:22 -07:00
Ashwin Bharambe
afae4e3d8e
Update docker build flow a little
2024-10-25 10:06:21 -07:00
Ashwin Bharambe
5bed6c276c
Move function around
2024-10-25 09:18:22 -07:00
