For the base Chemformer model, we investigated three SMILES modification techniques in this work: masking, augmentation and a combination of masking and augmentation. Due to resource constraints, however, the Chemformer-Large model was pre-trained only on the combined task. Figure illustrates example SMILES strings for all three pre-training tasks. Each of the tasks are implemented as follows: