feat: add batch inference API to llama stack inference (#1945)

# What does this PR do?

This PR adds two methods to the Inference API:
- `batch_completion`
- `batch_chat_completion`

The motivation is for evaluations targeting a local inference engine
(like meta-reference or vllm) where batch APIs provide for a substantial
amount of acceleration.

Why did I not add this to `Api.batch_inference` though? That just
resulted in a _lot_ more book-keeping given the structure of Llama
Stack. Had I done that, I would have needed to create a notion of a
"batch model" resource, setup routing based on that, etc. This does not
sound ideal.

So what's the future of the batch inference API? I am not sure. Maybe we
can keep it for true _asynchronous_ execution. So you can submit
requests, and it can return a Job instance, etc.

## Test Plan

Run meta-reference-gpu using:
```bash
export INFERENCE_MODEL=meta-llama/Llama-4-Scout-17B-16E-Instruct
export INFERENCE_CHECKPOINT_DIR=../checkpoints/Llama-4-Scout-17B-16E-Instruct-20250331210000
export MODEL_PARALLEL_SIZE=4
export MAX_BATCH_SIZE=32
export MAX_SEQ_LEN=6144

LLAMA_MODELS_DEBUG=1 llama stack run meta-reference-gpu
```

Then run the batch inference test case.

llama_stack/templates/meta-reference-gpu/run-with-safety.yaml

@@ -16,11 +16,12 @@ providers:
     provider_type: inline::meta-reference
     config:
       model: ${env.INFERENCE_MODEL}
-      max_seq_len: 4096
       checkpoint_dir: ${env.INFERENCE_CHECKPOINT_DIR:null}
       quantization:
         type: ${env.QUANTIZATION_TYPE:bf16}
       model_parallel_size: ${env.MODEL_PARALLEL_SIZE:0}
+      max_batch_size: ${env.MAX_BATCH_SIZE:1}
+      max_seq_len: ${env.MAX_SEQ_LEN:4096}
   - provider_id: sentence-transformers
     provider_type: inline::sentence-transformers
     config: {}
@@ -28,11 +29,12 @@ providers:
     provider_type: inline::meta-reference
     config:
       model: ${env.SAFETY_MODEL}
-      max_seq_len: 4096
       checkpoint_dir: ${env.SAFETY_CHECKPOINT_DIR:null}
       quantization:
         type: ${env.QUANTIZATION_TYPE:bf16}
       model_parallel_size: ${env.MODEL_PARALLEL_SIZE:0}
+      max_batch_size: ${env.MAX_BATCH_SIZE:1}
+      max_seq_len: ${env.MAX_SEQ_LEN:4096}
   vector_io:
   - provider_id: faiss
     provider_type: inline::faiss

llama_stack/templates/meta-reference-gpu/run.yaml

@@ -16,11 +16,12 @@ providers:
     provider_type: inline::meta-reference
     config:
       model: ${env.INFERENCE_MODEL}
-      max_seq_len: 4096
       checkpoint_dir: ${env.INFERENCE_CHECKPOINT_DIR:null}
       quantization:
         type: ${env.QUANTIZATION_TYPE:bf16}
       model_parallel_size: ${env.MODEL_PARALLEL_SIZE:0}
+      max_batch_size: ${env.MAX_BATCH_SIZE:1}
+      max_seq_len: ${env.MAX_SEQ_LEN:4096}
   - provider_id: sentence-transformers
     provider_type: inline::sentence-transformers
     config: {}