remove llama configure

This commit is contained in:
Kai Wu 2024-11-05 17:22:44 -08:00
parent ec16d2cabb
commit c5663fea72
3 changed files with 48 additions and 34 deletions

View file

@ -6,14 +6,14 @@
"source": [
"## Safety API 101\n",
"\n",
"This document talks about the Safety APIs in Llama Stack.\n",
"This document talks about the Safety APIs in Llama Stack. Before you begin, please ensure Llama Stack is installed and set up by following the [Getting Started Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/index.html).\n",
"\n",
"As outlined in our [Responsible Use Guide](https://www.llama.com/docs/how-to-guides/responsible-use-guide-resources/), LLM apps should deploy appropriate system level safeguards to mitigate safety and security risks of LLM system, similar to the following diagram:\n",
"![Figure 1: Safety System](../_static/safety_system.webp)\n",
"\n",
"To that goal, Llama Stack uses **Prompt Guard** and **Llama Guard 3** to secure our system. Here are the quick introduction about them.\n",
"\n",
"Before you begin, please ensure Llama Stack is installed and set up by following the [Getting Started Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/index.html)."
"<div>\n",
"<img src=\"../_static/safety_system.webp\" alt=\"Figure 1: Safety System\" width=\"1000\"/>\n",
"</div>\n",
"To that goal, Llama Stack uses **Prompt Guard** and **Llama Guard 3** to secure our system. Here are the quick introduction about them.\n"
]
},
{
@ -41,30 +41,55 @@
"source": [
"### Configure Safety\n",
"\n",
"```bash\n",
"$ llama stack configure ~/.conda/envsllamastack-my-local-stack/my-local-stack-build.yaml\n",
"We can first take a look at our build yaml file for my-local-stack:\n",
"\n",
"....\n",
"> Configuring provider `(meta-reference)`\n",
"Do you want to configure llama_guard_shield? (y/n): y\n",
"Entering sub-configuration for llama_guard_shield:\n",
"Enter value for model (existing: Llama-Guard-3-1B) (required):\n",
"Enter value for excluded_categories (existing: []) (required):\n",
"Enter value for enable_prompt_guard (existing: False) (optional): True\n",
"```bash\n",
"cat /home/$USER/.llama/builds/conda/my-local-stack-run.yaml\n",
"\n",
"version: '2'\n",
"built_at: '2024-10-23T12:20:07.467045'\n",
"image_name: my-local-stack\n",
"docker_image: null\n",
"conda_env: my-local-stack\n",
"apis:\n",
"- inference\n",
"- safety\n",
"- agents\n",
"- memory\n",
"- telemetry\n",
"providers:\n",
" inference:\n",
" - provider_id: meta-reference\n",
" provider_type: meta-reference\n",
" config:\n",
" model: Llama3.1-8B-Instruct\n",
" torch_seed: 42\n",
" max_seq_len: 8192\n",
" max_batch_size: 1\n",
" create_distributed_process_group: true\n",
" checkpoint_dir: null\n",
" safety:\n",
" - provider_id: meta-reference\n",
" provider_type: meta-reference\n",
" config:\n",
" llama_guard_shield:\n",
" model: Llama-Guard-3-1B\n",
" excluded_categories: []\n",
" enable_prompt_guard: true\n",
"....\n",
"```\n",
"As you can see, we did basic configuration above and configured:\n",
"As you can see, we have the safety feature configured in the yaml:\n",
"- Llama Guard safety shield with model `Llama-Guard-3-1B`\n",
"- Prompt Guard safety shield, which by default will be `Prompt-Guard-86M` model.\n",
"\n",
"You will also need to manually change the yaml to support `Llama3.2-3B-Instruct` running along with `Llama-Guard-3-1B`. The yaml file for inference section should be like this:\n",
"However, you will also need to manually change the yaml to support `Llama3.1-8B-Instruct` running along with `Llama-Guard-3-1B`. The yaml file for inference section should be like this:\n",
"\n",
"```bash\n",
"inference:\n",
" - provider_id: meta0\n",
" - provider_id: meta-reference\n",
" provider_type: meta-reference\n",
" config:\n",
" model: Llama3.2-3B-Instruct\n",
" model: Llama3.1-8B-Instruct\n",
" torch_seed: null\n",
" max_seq_len: 4096\n",
" max_batch_size: 1\n",
@ -81,7 +106,7 @@
" checkpoint_dir: null\n",
"```\n",
"\n",
"Now, you can start the server by `llama stack run my-local-stack --port 5000`\n",
"Now, you can start the server by `llama stack run my-local-stack --port 5000`. \n",
"\n",
"After the server started, you can test safety example using the follow code:"
]
@ -95,7 +120,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
@ -105,7 +130,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 9,
"metadata": {},
"outputs": [
{