TiktokenGitHubRepoTiktokenHigh-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.aibpecl100k-base93C#MIT5 d agotryAGIShare or keepTake it