Taking inspiration from NLP, we aim to address the resource challenges within computational chemistry by exploiting transfer learning to provide a model which can be quickly applied to diverse tasks. The purpose of this work is therefore to use SMILES as a "language for Chemistry" to provide a common data format on which we then apply Transformer-based language models. We investigate the ability of self-supervised pre-training on a large dataset of unlabelled molecules to decrease convergence time for a number of sequence-to-sequence tasks, thereby improving the results on these tasks when training time is limited. We explore a number of self-supervised pre-training tasks and model architectures, and quantitatively compare their performance on both sequence-to-sequence and discriminative downstream tasks. We show that, with the help of transfer learning, our models can achieve state-of-the-art results on four downstream datasets. Additionally, we examine the ability of these models to fine-tune on multiple discriminative tasks simultaneously, further improving cheminformatics research efficiency.