tiktoken

Open repos and models their makers tagged tiktoken, most starred first. The same for everyone.

GitHubRepo

Tiktoken

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

  • 93
  • C#
  • MIT
  • 4 d ago
tryAGITake it
AI on my own machineRepo

NoRefund

Offline LLM cost calculator and token calculator with real tokenizers, for PDF, DOCX and PPTX files. Real context-window fit, API cost estimates. Nothing leaves your machine.

  • 3
  • TypeScript
  • MIT
  • 5 wk ago
Phantom-VKTake it
GitHubRepo

DocMark-Bench

DocMark Bench is an end-to-end Streamlit workbench for generating Markdown with several open-source document parsers, viewing the original beside every result, and ranking each conversion route for the current document.

  • 0
  • Python
  • MIT
  • 3 mo ago
Search GitHub and Hugging Face for tiktoken

Get , open it once and sign in. Then come back here and press the button again.