Somali-Tokenizer
Somali Tokenizer is an open Somali subword tokenizer: a 48k ByteLevel BPE implemented in Rust with Python bindings, built for Natural Language Processing (NLP), Large Language Models (LLMs), and other AI applications for the Somali language.
- 8
- Rust
- Apache-2.0
- 1 d ago