Why does GPT-2 Exclude the Transformer Encoder?
GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information from the prior words in the sentence. It works just like a traditional language model as it takes word vectors as input and produces estimates for the probability of the next word as outputs but it is auto-regressive as each token in the sentence has the context of the previous words. Thus GPT-2 works one token at a time.
BERT, by contrast, is not auto-regressive. It uses the entire surrounding context all-at-once. GPT-2 the context vector is zero-initialized for the first word embedding.