phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-13 22:17:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

make inference server load checkpoints for fp8 inference

Browse source 
- introduce quantization related args for inference config
- also kill GeneratorArgs
This commit is contained in:
Ashwin Bharambe 2024-07-20 21:10:17 -07:00
parent 7d2c0b14b8
commit ad62e2e1f3
10 changed files with 249 additions and 155 deletions
Download patch file Download diff file Expand all files Collapse all files

2
toolchain/configs/ashwin.yaml
View file

@ -7,3 +7,5 @@ model_inference_config:
model_parallel_size: 1
max_seq_len: 2048
max_batch_size: 1
quantization:
type: "fp8"