What is the difference between GPT blocks and Transformer Decoder blocks?
GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model:
 
For GPT-2, this is clarified by the authors in the paper:



There have been several lines of research studying the effects of having the layer normalization before or after the attention. For instance the &quot;sandwich transformer&quot; tries to study different combinations.
For GPT-3, there are further modifications on top of GPT-2, also explained in the paper: