8 commits

Author	SHA1	Message	Date
ishaan-jaff	79e2ce870d	(fix) test_tokenizers	2024-02-23 10:13:38 -08:00
Krrish Dholakia	d4da63800e	fix(utils.py): support token counting for gpt-4-vision models	2024-01-02 14:41:42 +05:30
Krrish Dholakia	507b6bf96e	fix(utils.py): use local tiktoken copy	2023-12-28 11:22:33 +05:30
Krrish Dholakia	79978c44ba	refactor: add black formatting	2023-12-25 14:11:20 +05:30
Krrish Dholakia	3ffccf8804	test(test_token_counter.py): include testing for function calls	2023-11-30 18:24:21 -08:00
ishaan-jaff	5ab7470c84	(test) tokenizer worked test	2023-11-11 12:13:57 -08:00
Krrish Dholakia	2fa22dc91e	feat(utils.py): adding encode and decode functions	2023-10-20 11:59:47 -07:00
Krrish Dholakia	9876393355	adding support for cohere, anthropic, llama2 tokenizers	2023-09-22 14:03:52 -07:00