mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-06-27 18:50:41 +00:00
f34f22f8c7
# What does this PR do? This PR adds two methods to the Inference API: - `batch_completion` - `batch_chat_completion` The motivation is for evaluations targeting a local inference engine (like meta-reference or vllm) where batch APIs provide for a substantial amount of acceleration. Why did I not add this to `Api.batch_inference` though? That just resulted in a _lot_ more book-keeping given the structure of Llama Stack. Had I done that, I would have needed to create a notion of a "batch model" resource, setup routing based on that, etc. This does not sound ideal. So what's the future of the batch inference API? I am not sure. Maybe we can keep it for true _asynchronous_ execution. So you can submit requests, and it can return a Job instance, etc. ## Test Plan Run meta-reference-gpu using: ```bash export INFERENCE_MODEL=meta-llama/Llama-4-Scout-17B-16E-Instruct export INFERENCE_CHECKPOINT_DIR=../checkpoints/Llama-4-Scout-17B-16E-Instruct-20250331210000 export MODEL_PARALLEL_SIZE=4 export MAX_BATCH_SIZE=32 export MAX_SEQ_LEN=6144 LLAMA_MODELS_DEBUG=1 llama stack run meta-reference-gpu ``` Then run the batch inference test case. |
||
|---|---|---|
| .. | ||
| bedrock | ||
| cerebras | ||
| ci-tests | ||
| dell | ||
| dev | ||
| experimental-post-training | ||
| fireworks | ||
| groq | ||
| hf-endpoint | ||
| hf-serverless | ||
| meta-reference-gpu | ||
| nvidia | ||
| ollama | ||
| open-benchmark | ||
| passthrough | ||
| remote-vllm | ||
| sambanova | ||
| tgi | ||
| together | ||
| verification | ||
| vllm-gpu | ||
| __init__.py | ||
| dependencies.json | ||
| template.py | ||