Commit graph

3 commits

Author SHA1 Message Date
ishaan-jaff
b74b051385 (test) tokenizer worked test 2023-11-11 12:13:57 -08:00
Krrish Dholakia
4eeadd284a feat(utils.py): adding encode and decode functions 2023-10-20 11:59:47 -07:00
Krrish Dholakia
e62b584439 adding support for cohere, anthropic, llama2 tokenizers 2023-09-22 14:03:52 -07:00