How is GPT able to handle large vocabularies?
GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token.
BPE tries to find the pieces of words that are most reusable. BPE also keeps character subwords (e.g. &quot;a&quot;, &quot;W&quot;).
The subword vocabulary used by GPT-2 is 50k tokens big. You can take a look at the vocabulary here. There, you can see that token #12175 is &quot;retro&quot; and token #11147 is &quot;fit&quot;, so when tokenizing word &quot;retrofit&quot;, you would probably get the 2 subword tokens #68 and #12541. When tokenizing a low-frequency word (i.e. that did not appear in the BPE training data), you probably end up with small subwords tokenization, e.g. &quot;Kilimanjaro&quot; --&gt; &quot;Kil&quot;, &quot;iman&quot;, &quot;jar&quot;, &quot;o&quot;.
Using a word-level vocabulary is very troublesome, because:

the amount of existing word surface forms exceeds the size that it is manageable for a neural net, even for a morphologically simple language like English. For morphologically-rich fusional languages and, especially, agglutinative languages, using word-level vocabularies is much more inconvenient.
language is &quot;open&quot;, in the sense that new words can be created, either by stitching together existing words (e.g. manspreading), or by making up new words altogether (e.g. bazinga).
the less frequently a token appears in the training data, the less our net will learn to use it. If we have many many different tokens, the frequency of appearance of each token in the training data will be very low (i.e. data scarcity), so our net will not learn properly.

Therefore, subword vocabularies are the norm nowadays.