• Augmentation The augmentation task is conducted similarly to the approach of the heteroencoder model ; the input to the model is modified by randomly generating another SMILES string which corresponds to the same molecule as the output. This is done following the SMILES enumeration technique -permuting the atom order before generating a noncanonical SMILES form. Unlike many corruption tasks used for pre-training NLP models , this task is specific to the SMILES language. • Combined Data for the combined task are created by first augmenting and then masking each SMILES string. This task can be seen as a method of combining pre-training techniques for both natural language and chemistry.