In order to assess the general effects of using alternative architectures, we investigated three different architectures, the baseline BART transformer model, the mixture-of-experts Switch Transformer and the linear scaling Perceiver model. Additionally, we studied the effects of halving the number of encoder and decoder layers and use KD on the various architectures. To isolate the effects of architecture and KD, all other hyper-parameters were kept identical to the baseline.