forked from phoenix-oss/llama-stack-mirror
add image support to NVIDIA inference provider (#907)
# What does this PR do? add support to the NVIDIA Inference provider for image inputs ## Test Plan 1. Run local [Llama 3.2 11b vision instruct](https://build.nvidia.com/meta/llama-3.2-11b-vision-instruct?snippet_tab=Docker) NIM 2. Start a stack, e.g. `llama stack run llama_stack/templates/nvidia/run.yaml --env NVIDIA_BASE_URL=http://localhost:8000` 3. Run image tests, e.g. `LLAMA_STACK_BASE_URL=http://localhost:8321 pytest -v tests/client-sdk/inference/test_inference.py --vision-inference-model meta-llama/Llama-3.2-11B-Vision-Instruct -k image` ## Before submitting - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). - [x] Ran pre-commit to handle lint / formatting issues. - [x] Read the [contributor guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md), Pull Request section? - [ ] Updated relevant documentation. - [x] Wrote necessary unit or integration tests.
This commit is contained in:
parent
439d0da84c
commit
e21c8b6d80
3 changed files with 43 additions and 6 deletions
|
@ -186,6 +186,7 @@ async def localize_image_content(media: ImageContentItem) -> Tuple[bytes, str]:
|
|||
return content, format
|
||||
else:
|
||||
# data is a base64 encoded string, decode it to bytes first
|
||||
# TODO(mf): do this more efficiently, decode less
|
||||
data_bytes = base64.b64decode(image.data)
|
||||
pil_image = PIL_Image.open(io.BytesIO(data_bytes))
|
||||
return data_bytes, pil_image.format
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue