From 63cb0fbf547a2204e3c01da7fbdfcbbf69a9379f Mon Sep 17 00:00:00 2001 From: Yuan Tang Date: Thu, 20 Mar 2025 10:25:54 -0400 Subject: [PATCH] docs: Add instruction on enabling tool calling for remote vLLM --- docs/source/distributions/self_hosted_distro/remote-vllm.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/distributions/self_hosted_distro/remote-vllm.md b/docs/source/distributions/self_hosted_distro/remote-vllm.md index b7e155385..643627fad 100644 --- a/docs/source/distributions/self_hosted_distro/remote-vllm.md +++ b/docs/source/distributions/self_hosted_distro/remote-vllm.md @@ -61,6 +61,8 @@ docker run \ --port $INFERENCE_PORT ``` +Note that you'll also need to set `--enable-auto-tool-choice` and `--tool-call-parser` to [enable tool calling in vLLM](https://docs.vllm.ai/en/latest/features/tool_calling.html). + If you are using Llama Stack Safety / Shield APIs, then you will need to also run another instance of a vLLM with a corresponding safety model like `meta-llama/Llama-Guard-3-1B` using a script like: ```bash