Commit graph

5 commits

Author SHA1 Message Date
Krrish Dholakia
79978c44ba refactor: add black formatting 2023-12-25 14:11:20 +05:30
Krrish Dholakia
3ffccf8804 test(test_token_counter.py): include testing for function calls 2023-11-30 18:24:21 -08:00
ishaan-jaff
5ab7470c84 (test) tokenizer worked test 2023-11-11 12:13:57 -08:00
Krrish Dholakia
2fa22dc91e feat(utils.py): adding encode and decode functions 2023-10-20 11:59:47 -07:00
Krrish Dholakia
9876393355 adding support for cohere, anthropic, llama2 tokenizers 2023-09-22 14:03:52 -07:00