bpe-tokenizer
Byte-Pair Encoding tokenizer for training large language models on huge datasets
- 6
- Python
- GPL-3.0
- 2 y ago
Byte-Pair Encoding tokenizer for training large language models on huge datasets
Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding.
GPT-2-style LLM built from scratch in C/CUDA with hand-written backprop, BPE tokenizer, FlashAttention, pretraining, and SFT.