sentencepiece

Open repos and models their makers tagged sentencepiece, most starred first. The same for everyone.

Language and translationRepo

konoha

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.

  • 265
  • Python
  • MIT
  • 9 d ago
himktTake it
Images and videoRepo

FLUX-REALISM

FLUX-REALISM is an experimental, advanced image generation application designed to provide highly realistic image synthesis workflows. Powered by the state-of-the-art FLUX architecture, the suite offers a dual-model workspace featuring FLUX.1-Krea-dev and a specialized FLUX.1-dev.

  • 20
  • Python
  • Apache-2.0
  • 6 mo ago
Images and videoRepo

Z-Image-Turbo-LoRA-DLC

A Gradio-based demonstration for the Tongyi-MAI/Z-Image-Turbo diffusion pipeline, enhanced with a curated collection of LoRAs (Low-Rank Adaptations) for style transfer and creative image generation. Users can select from pre-listed LoRAs or add custom ones from Hugging Face repositories.

  • 5
  • Python
  • Apache-2.0
  • 11 mo ago
GitHubRepo

HunyuanOCR-Demo

A Gradio-based demonstration application for the Tencent HunyuanOCR model, focused on optical character recognition (OCR) tasks such as text detection, extraction, and coordinate formatting from images. Users can upload images, customize prompts (e.g., for Chinese/English text).

  • 4
  • Python
  • Apache-2.0
  • 12 mo ago
AI on my own machineRepo

Marcella-66M-SLM

A 66M parameter decoder-only transformer language model implemented from scratch in PyTorch. Features a custom SentencePiece tokenizer, RoPE positional embeddings, SwiGLU feed-forward network, per-layer KV cache for efficient autoregressive inference, and a Svelte-based streaming chat interface.

  • 2
  • Python
  • MIT
  • 6 mo ago
aidendorianTake it
AI on my own machineRepo

merge-barriers

Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization. Paper, 23 eval scripts, 86 result files, tokenizer definitions, 4 model checkpoints.

  • 0
  • Python
  • MIT
  • 5 mo ago
Search GitHub and Hugging Face for sentencepiece

Get , open it once and sign in. Then come back here and press the button again.

sentencepiece: open source repos and models on Take My Repo