11 commits

Author	SHA1	Message	Date
Krrish Dholakia	73b6b5e804	test(test_token_counter.py): fix token counting test	2024-05-14 21:35:28 -07:00
Krrish Dholakia	a1dd341ca1	fix(utils.py): default claude-3 to tiktoken (0.8s faster than hf tokenizer)	2024-05-14 18:37:14 -07:00
Christian Privitelli	2d43153efa	include methods in init import, add test, fix encode/decode param ordering	2024-05-02 15:49:22 +10:00
ishaan-jaff	ff815e93a8	(fix) test_tokenizers	2024-02-23 10:13:38 -08:00
Krrish Dholakia	0fffcc1579	fix(utils.py): support token counting for gpt-4-vision models	2024-01-02 14:41:42 +05:30
Krrish Dholakia	c4fc28ab0d	fix(utils.py): use local tiktoken copy	2023-12-28 11:22:33 +05:30
Krrish Dholakia	4905929de3	refactor: add black formatting	2023-12-25 14:11:20 +05:30
Krrish Dholakia	7b99cfe6b9	test(test_token_counter.py): include testing for function calls	2023-11-30 18:24:21 -08:00
ishaan-jaff	b74b051385	(test) tokenizer worked test	2023-11-11 12:13:57 -08:00
Krrish Dholakia	4eeadd284a	feat(utils.py): adding encode and decode functions	2023-10-20 11:59:47 -07:00
Krrish Dholakia	e62b584439	adding support for cohere, anthropic, llama2 tokenizers	2023-09-22 14:03:52 -07:00