Pre-training on Summit Following our previous work , we leveraged the Enamine REAL database as a starting point for language model training. We then augmented the data set using a previously trained language model to include approximately 3.6 • 10 10 molecules. A WordPiece tokenizer was then trained using the full data set. As described in , we used data parallelism with DeepSpeed's fused LAMB optimizer for mask prediction training on 3 • 10 9 molecules on the Summit supercomputer at the Oak Ridge Leadership Computing Facility using 1000 nodes (6 Nvidia 16 GB V100 GPUs per node). The data set was evenly partitioned with 5 • 10 5 molecules for each GPU. Pretraining was performed for 7 epochs using a batch size of 80 molecules with 3 gradient accumulation steps per GPU. The model is available for download at and can be used with the Hugging Face transformers library .