tokenization

Open repos and models their makers tagged tokenization, most starred first. The same for everyone.

AI on my own machineRepo

toon

🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

  • 26k
  • TypeScript
  • MIT
  • 8 d ago
toon-formatTake it
Security and privacyRepo

lunasec

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/

  • 1.5k
  • TypeScript
  • 3 y ago
lunasec-ioTake it
AI agentsRepo

asset-intel-orchestration-engine

Open-source platform for multi-agent AI orchestration, real-world asset tokenization, workflow automation. Includes OAuth 2.1 MCP agent management, a 44-field RWA schema, Stripe, designed to integrate with AetherNet QUAS & KVS sovereign, post quantum, architecture agnostic omni-substrate, semantic memory, & institutional settlement system.

  • 257
  • TypeScript
  • 8 wk ago
Language and translationRepo

Python_Natural_Language_Processing

This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.

  • 203
  • Jupyter Notebook
  • MIT
  • 4 y ago
milaan9Take it
AI on my own machineRepo

klmbr

klmbr - a prompt pre-processing technique to break through the barrier of entropy while generating text with LLMs

  • 89
  • TeX
  • AGPL-3.0
  • 2 y ago
avTake it
Websites and appsRepo

tokalator

Open-source context engineering for AI coding: context budget, Claude Code, GitHub Copilot and Codex usage in tokens or API-equivalent cost, tab relevance, current model pricing, code maps, and workspace security.

  • 18
  • TypeScript
  • AGPL-3.0
  • 6 d ago
vfaraji89Take it
Search GitHub and Hugging Face for tokenization

Get , open it once and sign in. Then come back here and press the button again.

tokenization: open source repos and models on Take My Repo