phoenix-oss/llama-stack
1
0
Fork 0
forked from phoenix-oss/llama-stack-mirror
Code Pull requests Releases Packages 2 Activity Actions

Update iOS inference instructions for new quantization

Browse source
This commit is contained in:
Dalton Flanagan 2024-10-24 14:47:27 -04:00 • committed by GitHub
parent 8aa8847b4a
commit 8eceebec98
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 12 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

13
llama_stack/providers/impls/ios/inference/README.md
View file

@ -56,9 +56,20 @@ We're working on making LocalInference easier to set up. For now, you'll need t
## Preparing a model
1. Prepare a `.pte` file [following the executorch docs](https://github.com/pytorch/executorch/blob/main/examples/models/llama2/README.md#step-2-prepare-model)
1. Prepare a `.pte` file [following the executorch docs](https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md#step-2-prepare-model)
2. Bundle the `.pte` and `tokenizer.model` file into your app
We now support models quantized using SpinQuant and QAT-LoRA which offer a significant performance boost (demo app on iPhone 13 Pro):
| Llama 3.2 1B | Tokens / Second (total) | | Time-to-First-Token (sec) | |
| :---- | :---- | :---- | :---- | :---- |
| | Haiku | Paragraph | Haiku | Paragraph |
| BF16 | 2.2 | 2.5 | 2.3 | 1.9 |
| QAT+LoRA | 7.1 | 3.3 | 0.37 | 0.24 |
| SpinQuant | 10.1 | 5.2 | 0.2 | 0.2 |
## Using LocalInference
1. Instantiate LocalInference with a DispatchQueue. Optionally, pass it into your agents service: