feat: Add NVIDIA Eval integration

llama_stack/providers/remote/eval/nvidia/README.md

@@ -0,0 +1,120 @@
+# NVIDIA NeMo Evaluator Eval Provider
+
+
+## Overview
+
+For the first integration, Benchmarks are mapped to Evaluation Configs on in the NeMo Evaluator. The full evaluation config object is provided as part of the meta-data. The `dataset_id` and `scoring_functions` are not used.
+
+Below are a few examples of how to register a benchmark, which in turn will create an evaluation config in NeMo Evaluator and how to trigger an evaluation.
+
+### Example for register an academic benchmark
+
+```
+POST /eval/benchmarks
+```
+```json
+{
+  "benchmark_id": "mmlu",
+  "dataset_id": "",
+  "scoring_functions": [],
+  "metadata": {
+    "config": {
+      "type": "mmlu"
+    }
+  }
+}
+```
+
+### Example for register a custom evaluation
+
+```
+POST /eval/benchmarks
+```
+```json
+{
+  "benchmark_id": "my-custom-benchmark",
+  "dataset_id": "",
+  "scoring_functions": [],
+  "metadata": {
+    "config": {
+      "type": "custom",
+      "params": {
+        "parallelism": 8
+      },
+      "tasks": {
+        "qa": {
+          "type": "completion",
+          "params": {
+            "template": {
+              "prompt": "{{prompt}}",
+              "max_tokens": 200
+            }
+          },
+          "dataset": {
+            "files_url": "hf://datasets/default/sample-basic-test/testing/testing.jsonl"
+          },
+          "metrics": {
+            "bleu": {
+              "type": "bleu",
+              "params": {
+                "references": [
+                  "{{ideal_response}}"
+                ]
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+### Example for triggering a benchmark/custom evaluation
+
+```
+POST /eval/benchmarks/{benchmark_id}/jobs
+```
+```json
+{
+  "benchmark_id": "my-custom-benchmark",
+  "task_config": {
+    "eval_candidate": {
+      "type": "model",
+      "model": "meta/llama-3.1-8b-instruct"
+    },
+    "scoring_params": []
+  }
+}
+```
+
+Response example:
+```json
+{
+    "job_id": "1234"
+}
+```
+
+### Example for getting the status of a job
+```
+GET /eval/benchmarks/{benchmark_id}/jobs/{job_id}
+```
+
+### Example for getting the results
+```
+GET /eval/benchmarks/{benchmark_id}/result
+```
+```json
+{
+  "generations": [],
+  "scores": {
+    "{benchmark_id}": {
+      "score_rows": [],
+      "aggregated_results": {
+        "tasks": {},
+        "groups": {}
+      }
+    }
+  }
+}
+```