16 commits

Author	SHA1	Message	Date
Krrish Dholakia	d421486a45	fix(token_counter.py): New `get_modified_max_tokens' helper func ... Fixes https://github.com/BerriAI/litellm/issues/4439	2024-06-27 15:38:09 -07:00
Krrish Dholakia	b756374a4f	feat(utils.py): bump tiktoken dependency to 0.7.0 ... adds support for gpt-4o token counting	2024-06-10 21:21:23 -07:00
Raymond Huang	968f54a65c	test	2024-06-06 10:11:27 -07:00
Krrish Dholakia	f43da3597d	test: fix test	2024-05-15 08:51:40 -07:00
Krrish Dholakia	900bb9aba8	test(test_token_counter.py): fix load test	2024-05-15 07:12:43 -07:00
Krrish Dholakia	73b6b5e804	test(test_token_counter.py): fix token counting test	2024-05-14 21:35:28 -07:00
Krrish Dholakia	a1dd341ca1	fix(utils.py): default claude-3 to tiktoken (0.8s faster than hf tokenizer)	2024-05-14 18:37:14 -07:00
Christian Privitelli	2d43153efa	include methods in init import, add test, fix encode/decode param ordering	2024-05-02 15:49:22 +10:00
ishaan-jaff	ff815e93a8	(fix) test_tokenizers	2024-02-23 10:13:38 -08:00
Krrish Dholakia	0fffcc1579	fix(utils.py): support token counting for gpt-4-vision models	2024-01-02 14:41:42 +05:30
Krrish Dholakia	c4fc28ab0d	fix(utils.py): use local tiktoken copy	2023-12-28 11:22:33 +05:30
Krrish Dholakia	4905929de3	refactor: add black formatting	2023-12-25 14:11:20 +05:30
Krrish Dholakia	7b99cfe6b9	test(test_token_counter.py): include testing for function calls	2023-11-30 18:24:21 -08:00
ishaan-jaff	b74b051385	(test) tokenizer worked test	2023-11-11 12:13:57 -08:00
Krrish Dholakia	4eeadd284a	feat(utils.py): adding encode and decode functions	2023-10-20 11:59:47 -07:00
Krrish Dholakia	e62b584439	adding support for cohere, anthropic, llama2 tokenizers	2023-09-22 14:03:52 -07:00