mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-12-27 21:21:59 +00:00
b45cc42202
The `load_tiktoken_bpe()` function depends on blobfile to load tokenizer.model files. However, blobfile brings in pycryptodomex, which is primarily used for JWT signing in GCP - functionality we don’t require, as we always load tokenizers from local files. pycryptodomex implements its own cryptographic primitives, which are known to be problematic and insecure. While blobfile could potentially switch to the more secure PyCA cryptography library, the project appears inactive, so this transition may not happen soon. Fortunately, `load_tiktoken_bpe()` is a simple function that just reads a BPE file and returns a dictionary mapping byte sequences to their mergeable ranks. It’s straightforward enough for us to implement ourselves. Signed-off-by: Sébastien Han <seb@redhat.com> |
||
|---|---|---|
| .. | ||
| prompt_templates | ||
| quantization | ||
| vision | ||
| __init__.py | ||
| args.py | ||
| chat_format.py | ||
| datatypes.py | ||
| ffn.py | ||
| generation.py | ||
| model.py | ||
| moe.py | ||
| preprocess.py | ||
| prompt_format.md | ||
| prompts.py | ||
| tokenizer.model | ||
| tokenizer.py | ||