5 commits

Author	SHA1	Message	Date
Krrish Dholakia	74a27df9ba	feat(utils.py): bump tiktoken dependency to 0.7.0 ... adds support for gpt-4o token counting	2024-06-10 21:21:23 -07:00
Krrish Dholakia	ed46af19ec	fix(openai.py): return logprobs for text completion calls	2024-04-02 14:05:56 -07:00
Krrish Dholakia	695a4c1600	refactor(__init__.py): adds init.py file in tokenizers	2024-01-22 15:33:16 -08:00
Krrish Dholakia	507b6bf96e	fix(utils.py): use local tiktoken copy	2023-12-28 11:22:33 +05:30
Krrish Dholakia	9876393355	adding support for cohere, anthropic, llama2 tokenizers	2023-09-22 14:03:52 -07:00