README.md

---
title: Tokenization Playground
emoji: 📝
colorFrom: indigo
colorTo: purple
sdk: gradio
pinned: false
short_description: Compare different tokenizers
---

# tokenization_playground
Link to source code: https://github.com/athnlp/tokenization_playground