Self-supervised learning using the Transformer has revolutionised Natural Language Processing (NLP) in recent years; large language models such as BERT , BART , GPT , UniLM and T5 have provided significant improvements on key benchmark NLP tasks. Pre-training these models -training on a large unlabelled dataset of text before fine-tuning on the dataset of interest -has been shown to improve results on downstream tasks, especially when the amount of data for fine-tuning is limited. Furthermore, pre-training can also significantly reduce the amount of time required for fine-tuning , thereby reducing computational costs and making state-of-the-art models more accessible to those with limited computational resources.