llama-stack-mirror

657 commits 61 branches 82 tags 77 MiB

Author	SHA1	Message	Date
Dinesh Yeduguru	c23363d561	Add ability to query and export spans to dataset (#574 ) This PR adds two new methods to the telemetry API: 1) Gives the ability to query spans directly instead of first querying traces and then using that to get spans 2) Another method save_spans_to_dataset, which builds on the query spans to save it on dataset. This give the ability to saves spans that are part of an agent session to a dataset. The unique aspect of this API is that we dont require each provider of telemetry to implement this method. Hence, its implemented in the protocol class itself. This required the protocol check to be slightly modified.	2024-12-05 21:07:30 -08:00
Xi Yan	58381dbe78	local persistence for eval tasks (#453 ) # What does this PR do? - add local persistence for eval tasks - follow https://github.com/meta-llama/llama-stack/pull/375 ## Test Plan 1. fresh llama stack run 2. kill server 3. restart server: llama stack run <img width="690" alt="image" src="https://github.com/user-attachments/assets/3d76e477-b91a-43a6-86ea-8e3ef2d04ed3"> Using run.yaml ```yaml eval_tasks: - eval_task_id: meta-reference-mmlu provider_id: meta-reference-0 dataset_id: mmlu scoring_functions: - basic::regex_parser_multiple_choice_answer ``` ## Before submitting - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). - [ ] Ran pre-commit to handle lint / formatting issues. - [ ] Read the [contributor guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md), Pull Request section? - [ ] Updated relevant documentation. - [ ] Wrote necessary unit or integration tests.	2024-11-14 10:36:23 -05:00
Xi Yan	b4416b72fd	Folder restructure for evals/datasets/scoring (#419 ) * rename evals related stuff * fix datasetio * fix scoring test * localfs -> LocalFS * refactor scoring * refactor scoring * remove 8b_correctness scoring_fn from tests * tests w/ eval params * scoring fn braintrust fixture * import	2024-11-11 17:35:40 -05:00

Author

SHA1

Message

Date

Dinesh Yeduguru

c23363d561

Add ability to query and export spans to dataset (#574 )

This PR adds two new methods to the telemetry API:
1) Gives the ability to query spans directly instead of first querying
traces and then using that to get spans
2) Another method save_spans_to_dataset, which builds on the query spans
to save it on dataset.

This give the ability to saves spans that are part of an agent session
to a dataset.

The unique aspect of this API is that we dont require each provider of
telemetry to implement this method. Hence, its implemented in the
protocol class itself. This required the protocol check to be slightly
modified.

2024-12-05 21:07:30 -08:00

Xi Yan

58381dbe78

local persistence for eval tasks (#453 )

# What does this PR do?

- add local persistence for eval tasks
- follow https://github.com/meta-llama/llama-stack/pull/375

## Test Plan

1. fresh llama stack run
2. kill server
3. restart server: llama stack run

<img width="690" alt="image"
src="https://github.com/user-attachments/assets/3d76e477-b91a-43a6-86ea-8e3ef2d04ed3">

Using run.yaml
```yaml
eval_tasks:
  - eval_task_id: meta-reference-mmlu
    provider_id: meta-reference-0
    dataset_id: mmlu
    scoring_functions:
      - basic::regex_parser_multiple_choice_answer
```

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the
other checks if that's the case).
- [ ] Ran pre-commit to handle lint / formatting issues.
- [ ] Read the [contributor
guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md),
      Pull Request section?
- [ ] Updated relevant documentation.
- [ ] Wrote necessary unit or integration tests.

2024-11-14 10:36:23 -05:00

Xi Yan

b4416b72fd

Folder restructure for evals/datasets/scoring (#419 )

* rename evals related stuff

* fix datasetio

* fix scoring test

* localfs -> LocalFS

* refactor scoring

* refactor scoring

* remove 8b_correctness scoring_fn from tests

* tests w/ eval params

* scoring fn braintrust fixture

* import

2024-11-11 17:35:40 -05:00

Renamed from llama_stack/providers/inline/meta_reference/eval/config.py (Browse further)

3 commits