From where does BERT get the tokens it predicts?
There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here.
You can see that it contains one token per line, with a total of 30522 tokens. The softmax is computed over them.
The token granularity in the BERT vocabulary is subwords. This means that each token does not represent a complete word, but just a piece of word. Before feeding text as input to BERT, it is needed to segment it into subwords according to the subword vocabulary mentioned before. Having a subword vocabulary instead of a word-level vocabulary is what makes it possible for BERT (and any other text generation subword model) to only need a &quot;small&quot; vocabulary to be able to represent any string (within the character set seen in the training data).