llama-stack/llama_stack/templates
yyymeta d117bfe597
feat: [new open benchmark] DocVQA (#1647)
# What does this PR do?
DocVQA asks model to look a a picture, then answer a question given in
text, with a text answer by text information in the picture. these
questions often require understanding of relative positions of texts
within the picture.

original dataset is defined in the "Task1" of
https://www.docvqa.org/datasets


## Test Plan
setup llama server with 

```
llama stack run ./llama_stack/templates/open-benchmark/run.yaml
```


then send traffic:

```
 llama-stack-client eval run-benchmark "meta-reference-docvqa"  --model-id   meta-llama/Llama-3.3-70B-Instruct     --output-dir /tmp/gpqa    --num-examples   200
```
2025-03-19 14:56:14 -07:00
..
bedrock test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
cerebras test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
ci-tests test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
dell test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
dev test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
experimental-post-training feat: [post training] support save hf safetensor format checkpoint (#845) 2025-02-25 23:29:08 -08:00
fireworks test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
groq test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
hf-endpoint test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
hf-serverless test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
meta-reference-gpu test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
meta-reference-quantized-gpu test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
nvidia feat: added nvidia as safety provider (#1248) 2025-03-17 14:39:23 -07:00
ollama test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
open-benchmark feat: [new open benchmark] DocVQA (#1647) 2025-03-19 14:56:14 -07:00
passthrough fix: passthrough provider template + fix (#1612) 2025-03-13 09:44:26 -07:00
remote-vllm fix: Add the option to not verify SSL at remote-vllm provider (#1585) 2025-03-18 09:33:35 -04:00
sambanova test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
tgi test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
together test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
vllm-gpu test: add unit test to ensure all config types are instantiable (#1601) 2025-03-12 22:29:58 -07:00
__init__.py Auto-generate distro yamls + docs (#468) 2024-11-18 14:57:06 -08:00
template.py feat(api): (1/n) datasets api clean up (#1573) 2025-03-17 16:55:45 -07:00