16 commits

Author	SHA1	Message	Date
Krrish Dholakia	797a018bb0	fix(token_counter.py): New `get_modified_max_tokens' helper func ... Fixes https://github.com/BerriAI/litellm/issues/4439	2024-06-27 15:38:09 -07:00
Krrish Dholakia	74a27df9ba	feat(utils.py): bump tiktoken dependency to 0.7.0 ... adds support for gpt-4o token counting	2024-06-10 21:21:23 -07:00
Raymond Huang	dca10670ad	test	2024-06-06 10:11:27 -07:00
Krrish Dholakia	93dd54be6d	test: fix test	2024-05-15 08:51:40 -07:00
Krrish Dholakia	1b4836bd3d	test(test_token_counter.py): fix load test	2024-05-15 07:12:43 -07:00
Krrish Dholakia	7b52087dfb	test(test_token_counter.py): fix token counting test	2024-05-14 21:35:28 -07:00
Krrish Dholakia	7ab7e81aed	fix(utils.py): default claude-3 to tiktoken (0.8s faster than hf tokenizer)	2024-05-14 18:37:14 -07:00
Christian Privitelli	8b4bc4c832	include methods in init import, add test, fix encode/decode param ordering	2024-05-02 15:49:22 +10:00
ishaan-jaff	79e2ce870d	(fix) test_tokenizers	2024-02-23 10:13:38 -08:00
Krrish Dholakia	d4da63800e	fix(utils.py): support token counting for gpt-4-vision models	2024-01-02 14:41:42 +05:30
Krrish Dholakia	507b6bf96e	fix(utils.py): use local tiktoken copy	2023-12-28 11:22:33 +05:30
Krrish Dholakia	79978c44ba	refactor: add black formatting	2023-12-25 14:11:20 +05:30
Krrish Dholakia	3ffccf8804	test(test_token_counter.py): include testing for function calls	2023-11-30 18:24:21 -08:00
ishaan-jaff	5ab7470c84	(test) tokenizer worked test	2023-11-11 12:13:57 -08:00
Krrish Dholakia	2fa22dc91e	feat(utils.py): adding encode and decode functions	2023-10-20 11:59:47 -07:00
Krrish Dholakia	9876393355	adding support for cohere, anthropic, llama2 tokenizers	2023-09-22 14:03:52 -07:00