In this work we introduced the Chemformer model which makes use of the SMILES language for application to diverse computational chemistry tasks. We investigated three different selfsupervised pre-training techniques and applied these on a large dataset of unlabelled SMILES. Finally, we fine-tuned the pre-trained Chemformer models on a selection of downstream tasks and compared their performance to randomly initialised models and existing benchmarks.