
Robust and Fast tokenizations alignment library for Rust and Python https://tamuhey.github.io/tokenizations/
Repo explosion/tokenizations, read from GitHub 1 min ago. explosion has not joined Take My Repo.
explosion/tokenizations is a public repo on GitHub by explosion. Robust and Fast tokenizations alignment library for Rust and Python https://tamuhey.github.io/tokenizations/. It declares the MIT license on GitHub. It is written mainly in Rust, has 195 stars and last changed on 2023-10-04. It is archived: its maker no longer maintains it. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-18.
explosion/tokenizations is a public repo on GitHub by explosion. Robust and Fast tokenizations alignment library for Rust and Python https://tamuhey.github.io/tokenizations/.
It declares the MIT license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
No. It is archived on GitHub: its maker no longer maintains it.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
๐ฎ A refreshing functional take on deep learning, compatible with your favorite libraries
๐ธ Use pretrained transformers like BERT, XLNet and GPT-2 in spaCy