phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-25 10:44:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix(utils.py): fix select tokenizer for custom tokenizer (#7599)
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 36s
Details

Browse source 
* fix(utils.py): fix select tokenizer for custom tokenizer

* fix(router.py): fix 'utils/token_counter' endpoint
This commit is contained in:
Krish Dholakia 2025-01-07 22:37:09 -08:00 • committed by GitHub
parent 04eb718f7a
commit 07c5f136f1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 16 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

6
litellm/utils.py
View file

@ -1278,12 +1278,12 @@ def _select_tokenizer(
model: str, custom_tokenizer: Optional[CustomHuggingfaceTokenizer] = None
):
if custom_tokenizer is not None:
custom_tokenizer = Tokenizer.from_pretrained(
custom_tokenizer["identifier"],
_tokenizer = create_pretrained_tokenizer(
identifier=custom_tokenizer["identifier"],
revision=custom_tokenizer["revision"],
auth_token=custom_tokenizer["auth_token"],
)
return {"type": "huggingface_tokenizer", "tokenizer": custom_tokenizer}
return _tokenizer
return _select_tokenizer_helper(model=model)