From f5a5c5d4591f0068e4b65380e47ad5004aed5b6c Mon Sep 17 00:00:00 2001 From: Yuan Tang Date: Thu, 20 Mar 2025 18:18:17 -0400 Subject: [PATCH] docs: Add instruction on enabling tool calling for remote vLLM (#1719) # What does this PR do? This PR adds a link to tool calling instructions in vLLM. Users have asked about this many times, e.g. https://github.com/meta-llama/llama-stack/issues/1648#issuecomment-2740642077 --------- Signed-off-by: Yuan Tang --- docs/source/distributions/self_hosted_distro/remote-vllm.md | 2 ++ llama_stack/templates/remote-vllm/doc_template.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/docs/source/distributions/self_hosted_distro/remote-vllm.md b/docs/source/distributions/self_hosted_distro/remote-vllm.md index b7e155385..643627fad 100644 --- a/docs/source/distributions/self_hosted_distro/remote-vllm.md +++ b/docs/source/distributions/self_hosted_distro/remote-vllm.md @@ -61,6 +61,8 @@ docker run \ --port $INFERENCE_PORT ``` +Note that you'll also need to set `--enable-auto-tool-choice` and `--tool-call-parser` to [enable tool calling in vLLM](https://docs.vllm.ai/en/latest/features/tool_calling.html). + If you are using Llama Stack Safety / Shield APIs, then you will need to also run another instance of a vLLM with a corresponding safety model like `meta-llama/Llama-Guard-3-1B` using a script like: ```bash diff --git a/llama_stack/templates/remote-vllm/doc_template.md b/llama_stack/templates/remote-vllm/doc_template.md index 0ca7279a7..8abef18fb 100644 --- a/llama_stack/templates/remote-vllm/doc_template.md +++ b/llama_stack/templates/remote-vllm/doc_template.md @@ -48,6 +48,8 @@ docker run \ --port $INFERENCE_PORT ``` +Note that you'll also need to set `--enable-auto-tool-choice` and `--tool-call-parser` to [enable tool calling in vLLM](https://docs.vllm.ai/en/latest/features/tool_calling.html). + If you are using Llama Stack Safety / Shield APIs, then you will need to also run another instance of a vLLM with a corresponding safety model like `meta-llama/Llama-Guard-3-1B` using a script like: ```bash