llama-stack-mirror/llama_stack/providers/registry
Jash Gulabrai cc77f79f55
feat: Add NVIDIA Eval integration (#1890)
# What does this PR do?
This PR adds support for NVIDIA's NeMo Evaluator API to the Llama Stack
eval module. The integration enables users to evaluate models via the
Llama Stack interface.

## Test Plan
[Describe the tests you ran to verify your changes with result
summaries. *Provide clear instructions so the plan can be easily
re-executed.*]
1. Added unit tests and successfully ran from root of project:
`./scripts/unit-tests.sh tests/unit/providers/nvidia/test_eval.py`
```
tests/unit/providers/nvidia/test_eval.py::TestNVIDIAEvalImpl::test_job_cancel PASSED
tests/unit/providers/nvidia/test_eval.py::TestNVIDIAEvalImpl::test_job_result PASSED
tests/unit/providers/nvidia/test_eval.py::TestNVIDIAEvalImpl::test_job_status PASSED
tests/unit/providers/nvidia/test_eval.py::TestNVIDIAEvalImpl::test_register_benchmark PASSED
tests/unit/providers/nvidia/test_eval.py::TestNVIDIAEvalImpl::test_run_eval PASSED
```
2. Verified I could build the Llama Stack image: `LLAMA_STACK_DIR=$(pwd)
llama stack build --template nvidia --image-type venv`

Documentation added to
`llama_stack/providers/remote/eval/nvidia/README.md`

---------

Co-authored-by: Jash Gulabrai <jgulabrai@nvidia.com>
2025-04-24 17:12:42 -07:00
..
__init__.py API Updates (#73) 2024-09-17 19:51:35 -07:00
agents.py test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
datasetio.py [remove import *] clean up import *'s (#689) 2024-12-27 15:45:44 -08:00
eval.py feat: Add NVIDIA Eval integration (#1890) 2025-04-24 17:12:42 -07:00
files.py feat(api): don't return a payload on file delete (#1640) 2025-03-25 17:12:36 -07:00
inference.py fix: use torchao 0.8.0 for inference (#1925) 2025-04-10 13:39:20 -07:00
post_training.py feat: Add nemo customizer (#1448) 2025-03-25 11:01:10 -07:00
safety.py fix: Add 'accelerate' dependency to 'prompt-guard' (#1724) 2025-03-21 07:37:20 -07:00
scoring.py [remove import *] clean up import *'s (#689) 2024-12-27 15:45:44 -08:00
telemetry.py test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
tool_runtime.py chore: move embedding deps to RAG tool where they are needed (#1210) 2025-02-21 11:33:41 -08:00
vector_io.py feat: Qdrant inline provider (#1273) 2025-03-18 14:04:21 -07:00