[Evals API][11/n] huggingface dataset provider + mmlu scoring fn (#392)

Browse source

* wip

* scoring fn api

* eval api

* eval task

* evaluate api update

* pre commit

* unwrap context -> config

* config field doc

* typo

* naming fix

* separate benchmark / app eval

* api name

* rename

* wip tests

* wip

* datasetio test

* delete unused

* fixture

* scoring resolve

* fix scoring register

* scoring test pass

* score batch

* scoring fix

* fix eval

* test eval works

* huggingface provider

* datasetdef files

* mmlu scoring fn

* test wip

* remove type ignore

* api refactor

* add default task_eval_id for routing

* add eval_id for jobs

* remove type ignore

* huggingface provider

* wip huggingface register

* only keep 1 run_eval

* fix optional

* register task required

* register task required

* delete old tests

* fix

* mmlu loose

* refactor

* msg

* fix tests

* move benchmark task def to file

* msg

* gen openapi

* openapi gen

* move dataset to hf llamastack repo

* remove todo

* refactor

* add register model to unit test

* rename

* register to client

* delete preregistered dataset/eval task

* comments

* huggingface -> remote adapter

* openapi gen

...

This commit is contained in:

Xi Yan 2024-11-11 14:49:50 -05:00 • committed by GitHub

parent b78ee3a0a5

commit 2b7d70ba86

No known key found for this signature in database

GPG key ID: B5690EEEBB952194

20 changed files with 1607 additions and 718 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1069

docs/resources/llama-stack-spec.html

View file

File diff suppressed because it is too large Load diff

754

docs/resources/llama-stack-spec.yaml

View file

File diff suppressed because it is too large Load diff