phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-04 04:04:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

use default_config file to configure inference

Browse source
This commit is contained in:
Hardik Shah 2024-07-21 19:26:11 -07:00
parent c64b8cba22
commit d95f5f863d
3 changed files with 25 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

9
toolchain/cli/inference/default_configuration.yaml Normal file
View file

@ -0,0 +1,9 @@
inference_config:
impl_type: "inline"
inline_config:
checkpoint_type: "pytorch"
checkpoint_dir: {checkpoint_dir}/
tokenizer_path: {checkpoint_dir}/tokenizer.model
model_parallel_size: {model_parallel_size}
max_seq_len: 2048
max_batch_size: 1