(2) ChemGPT uses the Transformer architecture with a self-attention mechanism to compute conditional probabilities, estimate p(x), and sample from it to generate new molecules. ChemGPT is pre-trained on molecules from Pub-Chem with a causal language modeling task, where the model must predict the next token in a sequence, given the previous tokens. ChemGPT models of up to one billion non-embedding parameters are trained on up to ten million molecules, whereas typical chemical generative models have less than one million parameters and are trained on less than one million samples .