phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-25 18:54:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #3393 from Priva28/main

Browse source 
Add Llama3 tokenizer and allow custom tokenizers.
This commit is contained in:
Krish Dholakia 2024-05-02 16:32:41 -07:00 • committed by GitHub
commit 2200900ca2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 98 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

2
litellm/main.py
View file

@ -34,6 +34,8 @@ from litellm.utils import (
async_mock_completion_streaming_obj,
convert_to_model_response_object,
token_counter,
create_pretrained_tokenizer,
create_tokenizer,
Usage,
get_optional_params_embeddings,
get_optional_params_image_gen,