BiptokenLanguage and translationRepoBiptokenFast BPE tokenizer with guaranteed perfect text reconstruction. 2x faster than tiktoken at encoding/decoding while preserving all whitespace and formatting.bpe-tokenizerfast-tokenizerllm-tooling2PythonMIT18 mo agoJustVuggShare or keepTake it