8 commits

Author	SHA1	Message	Date
ishaan-jaff	ff815e93a8	(fix) test_tokenizers	2024-02-23 10:13:38 -08:00
Krrish Dholakia	0fffcc1579	fix(utils.py): support token counting for gpt-4-vision models	2024-01-02 14:41:42 +05:30
Krrish Dholakia	c4fc28ab0d	fix(utils.py): use local tiktoken copy	2023-12-28 11:22:33 +05:30
Krrish Dholakia	4905929de3	refactor: add black formatting	2023-12-25 14:11:20 +05:30
Krrish Dholakia	7b99cfe6b9	test(test_token_counter.py): include testing for function calls	2023-11-30 18:24:21 -08:00
ishaan-jaff	b74b051385	(test) tokenizer worked test	2023-11-11 12:13:57 -08:00
Krrish Dholakia	4eeadd284a	feat(utils.py): adding encode and decode functions	2023-10-20 11:59:47 -07:00
Krrish Dholakia	e62b584439	adding support for cohere, anthropic, llama2 tokenizers	2023-09-22 14:03:52 -07:00