phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-25 10:44:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Code Quality Improvement - remove tokenizers/ from /llms (#7163)

Browse source 
* move tokenizers out of /llms

* use updated tokenizers location

* fix test_google_secret_manager_read_in_memory
This commit is contained in:
Ishaan Jaff 2024-12-10 23:50:15 -08:00 • committed by GitHub
parent 350cfc36f7
commit 3055d9b81c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 4 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

2
litellm/utils.py
View file

@ -135,7 +135,7 @@ from litellm.types.utils import (
Usage,
)
with resources.open_text("litellm.llms.tokenizers", "anthropic_tokenizer.json") as f:
with resources.open_text("litellm.litellm_core_utils.tokenizers", "anthropic_tokenizer.json") as f:
json_data = json.load(f)
# Convert to str (if necessary)
claude_json_str = json.dumps(json_data)