Transfer learning has also recently been shown to improve performance on reaction informatics tasks and, separately, on discriminative tasks . However, many of these approaches pre-train on a task-specific dataset, such as reaction informatics data. It is unclear how well these models would be able to transfer their knowledge to other domains. Other approaches make use of the encoder stack of the Transformer only, along with a fully-visible attention mask . This makes it difficult to apply these models to sequence-to-sequence tasks. In one study, embeddings from a self-supervised Augmented Transformer were used to build QSAR models , but the pre-trained weights were not subsequently fine-tuned.