[Evals API][6/n] meta-reference llm as judge, registration for ScoringFnDefs (#330)

* wip scoring refactor

* llm as judge, move folders

* test full generation + eval

* extract score regex to llm context

* remove prints, cleanup braintrust in this branch

* change json -> class

* remove initialize

* address nits

* check identifier prefix

* udpate MANIFEST
This commit is contained in:
Xi Yan 2024-10-28 14:08:42 -07:00 committed by GitHub
parent 04a4784287
commit 7b8748c53e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
20 changed files with 360 additions and 50 deletions

View file

@ -20,6 +20,7 @@ def available_providers() -> List[ProviderSpec]:
api_dependencies=[
Api.datasetio,
Api.datasets,
Api.inference,
],
),
]