chinese-tokenizer

Open repos and models their makers tagged chinese-tokenizer, most starred first. The same for everyone.

GitHubRepo

MicroTokenizer

一个轻量且功能全面的中文分词器,帮助学生了解分词器的工作原理。MicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..

  • 160
  • Python
  • MIT
  • 2 y ago
Search GitHub and Hugging Face for chinese-tokenizer

Get , open it once and sign in. Then come back here and press the button again.