Large chemical language models. Strings are a simple representation for molecular graphs , thereby making sequence-based ML models a natural choice for working with chemical data. Following the demonstrated performance improvements of Transformer-based models with increasing model and dataset sizes , we designed a large generative language model for chemistry called ChemGPT to investigate the impact of dataset and model size on pre-training loss. ChemGPT is a Generative Pre-trained Transformer 3 (GPT3)-style model based on GPT-Neo with a tokenizer for Self-referencing embedded strings (SELFIES) representations of molecules. SELFIES enforce chemical validity and are straightforward to tokenize, but ChemGPT can easily be used with simplified molecular-input line-entry system (SMILES) strings as well . For chemical language modeling, a set of molecules (x 1 , x 2 , ..., x n ) is represented with each molecule as a sequence of symbols (s 1 , s 2 , ..., s n ). The probability of a sequence, p(x) is factorized as the product of conditional probabilities :