The first is the encoder-decoder architecture that forms the basis of seq2seq and BART models. The second is the encoder-only architecture that formed the basis for the BERT model . The sole difference between these architectures is that the decoder architecture in the former is a transformer block with cross attention and the latter is a simple multilayer perceptron.