Alternative model architectures The initial findings indicated several important aspects. First, our investigation extended to alternative model architectures, including the Switch Transformer and the Perceiver model . The Switch Transformer showed similar accuracy as the baseline Transformer model, but was significantly slower. This was possibly due to a sub-optimal implementation and the usage of only a single GPU, making the sparsity effects of the original implementation less effective. The Perceiver model showed promise, having similar accuracy to the transformer trained from scratch, whilst being faster. However, we found that these alternative architectures under-performed when coupled with KD. The Switch transformer and the Perceiver models were chosen as alternative architectures as these architectures were relatively closely related to, though projected to be faster than the original Transformer architecture.