Tokens
Tokens are subword chunks the model learned from lots of text.
Loading tokenizer…
Token IDs
IDs are how the model “sees” tokens — just numbers.
Live view of tokens and token IDs. Powered by Transformers.js — all in your browser.
Tokens are subword chunks the model learned from lots of text.
IDs are how the model “sees” tokens — just numbers.