phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-06-27 18:50:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

[Evals API][7/n] braintrust scoring provider (#333)

Browse source 
* wip scoring refactor

* llm as judge, move folders

* test full generation + eval

* extract score regex to llm context

* remove prints, cleanup braintrust in this branch

* braintrust skeleton

* datasetio test fix

* braintrust provider

* remove prints

* dependencies

* change json -> class

* json -> class

* remove initialize

* address nits

* check identifier prefix

* braintrust scoring identifier check, rebase

* udpate MANIFEST

* manifest

* remove braintrust scoring_fn

* remove comments

* tests

* imports fix
This commit is contained in:
Xi Yan 2024-10-28 18:59:35 -07:00 • committed by GitHub
parent ae671eaf7a
commit ed833bb758
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 274 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

11
llama_stack/providers/registry/scoring.py
View file

@ -23,4 +23,15 @@ def available_providers() -> List[ProviderSpec]:
Api.inference,
],
),
InlineProviderSpec(
api=Api.scoring,
provider_type="braintrust",
pip_packages=["autoevals", "openai"],
module="llama_stack.providers.impls.braintrust.scoring",
config_class="llama_stack.providers.impls.braintrust.scoring.BraintrustScoringConfig",
api_dependencies=[
Api.datasetio,
Api.datasets,
],
),
]