Implement token counting utilities to determine prompt cacheability
(≥1024 tokens) with support for OpenAI, Llama, and multimodal content.
- Add count_tokens() function with model-specific tokenizers
- Support OpenAI models (GPT-4, GPT-4o, etc.) via tiktoken
- Support Llama models (3.x, 4.x) via transformers
- Fallback to character-based estimation for unknown models
- Handle multimodal content (text + images)
- LRU cache for tokenizer instances (max 10, <1ms cached calls)
- Comprehensive unit tests (34 tests, >95% coverage)
- Update tiktoken version constraint to >=0.8.0
This enables future PR to determine which prompts should be cached based on token count threshold.
Signed-off-by: William Caban <william.caban@gmail.com>
