phoenix/litellm
1
0
Fork 0
forked from phoenix/litellm-mirror
Code Pull requests Activity

Merge pull request #3393 from Priva28/main

Browse source 
Add Llama3 tokenizer and allow custom tokenizers.
This commit is contained in:
Krish Dholakia 2024-05-02 16:32:41 -07:00 • committed by GitHub
commit 2200900ca2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 98 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

2
litellm/main.py
View file

@ -34,6 +34,8 @@ from litellm.utils import (
async_mock_completion_streaming_obj,
convert_to_model_response_object,
token_counter,
create_pretrained_tokenizer,
create_tokenizer,
Usage,
get_optional_params_embeddings,
get_optional_params_image_gen,