phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-07-29 15:23:51 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

kill llamastack-local-gpu/cpu

Browse source
This commit is contained in:
Xi Yan 2024-11-01 13:21:59 -07:00
parent 0bc087c81a
commit f1f8aa2029
2 changed files with 13 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

2
distributions/tgi/cpu/compose.yaml
View file

@ -17,7 +17,7 @@ services:
depends_on:
text-generation-inference:
condition: service_healthy
image: llamastack/llamastack-local-cpu
image: llamastack/llamastack-tgi
network_mode: "host"
volumes:
- ~/.llama:/root/.llama

24
docs/getting_started.ipynb
View file

@ -36,7 +36,7 @@
"1. Get Docker container\n",
"```\n",
"$ docker login\n",
"$ docker pull llamastack/llamastack-local-gpu\n",
"$ docker pull llamastack/llamastack-meta-reference-gpu\n",
"```\n",
"\n",
"2. pip install the llama stack client package \n",
@ -61,15 +61,15 @@
"```\n",
"For GPU inference, you need to set these environment variables for specifying local directory containing your model checkpoints, and enable GPU inference to start running docker container.\n",
"$ export LLAMA_CHECKPOINT_DIR=~/.llama\n",
"$ llama stack configure llamastack-local-gpu\n",
"$ llama stack configure llamastack-meta-reference-gpu\n",
"```\n",
"Follow the prompts as part of configure.\n",
"Here is a sample output \n",
"```\n",
"$ llama stack configure llamastack-local-gpu\n",
"$ llama stack configure llamastack-meta-reference-gpu\n",
"\n",
"Could not find /home/hjshah/.conda/envs/llamastack-llamastack-local-gpu/llamastack-local-gpu-build.yaml. Trying docker image name instead...\n",
"+ podman run --network host -it -v /home/hjshah/.llama/builds/docker:/app/builds llamastack-local-gpu llama stack configure ./llamastack-build.yaml --output-dir /app/builds\n",
"Could not find ~/.conda/envs/llamastack-llamastack-meta-reference-gpu/llamastack-meta-reference-gpu-build.yaml. Trying docker image name instead...\n",
"+ podman run --network host -it -v ~/.llama/builds/docker:/app/builds llamastack-meta-reference-gpu llama stack configure ./llamastack-build.yaml --output-dir /app/builds\n",
"\n",
"Configuring API `inference`...\n",
"=== Configuring provider `meta-reference` for API inference...\n",
@ -155,7 +155,7 @@
"metadata": {},
"outputs": [],
"source": [
"# For this notebook we will be working with the latest Llama3.2 vision models \n",
"# For this notebook we will be working with the latest Llama3.2 vision models\n",
"model = \"Llama3.2-11B-Vision-Instruct\""
]
},
@ -182,7 +182,7 @@
}
],
"source": [
"# Simple text example \n",
"# Simple text example\n",
"iterator = client.inference.chat_completion(\n",
" model=model,\n",
" messages=[\n",
@ -224,13 +224,13 @@
],
"source": [
"import base64\n",
"import mimetypes \n",
"import mimetypes\n",
"\n",
"from PIL import Image\n",
"\n",
"# We define a simple utility function to take a local image and \n",
"# convert it to as base64 encoded data url \n",
"# that can be passed to the server. \n",
"# We define a simple utility function to take a local image and\n",
"# convert it to as base64 encoded data url\n",
"# that can be passed to the server.\n",
"def data_url_from_image(file_path):\n",
" mime_type, _ = mimetypes.guess_type(file_path)\n",
" if mime_type is None:\n",
@ -273,7 +273,7 @@
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" { \"image\": { \"uri\": data_url } }, \n",
" { \"image\": { \"uri\": data_url } },\n",
" \"Write a haiku describing the image\"\n",
" ]\n",
" }\n",