The fact that these model variants under-performed when trained with KD, indicates that the benefits of KD may not be applicable to all model architectures and emphasizes the need for careful evaluation. One notable aspect here is that the Perceiver model, which uses different dimensions, was unable to use the feature-based KD, and thus solely relied on the response-based KD. Future work could focus on implementing and evaluating the relation-based KD , which uses comparisons of in-between batch distances and angles to match between models, making them more impervious to dimensional changes.