tokenizer-ratios
How many characters is a token? 951 samples measured on OpenAI o200k_base (GPT-4o): English, code, JSON, CSV, IDs, emoji, Chinese. chars/4 misses JSON by half. Free 26-line JS token estimator within ~2%, compared with tokenx.
- 0
- Python
- MIT
- 3 wk ago