Advances in language models can be directly applied to molecular structures by using the Simplified Molecular Input Line Entry System (SMILES) text representation . Using a SMILES string, atoms and bonds for a given molecule are converted to a sequence of characters. For example, benzene is given by c1ccccc1, where c represents an individual aromatic carbon atom and 1 represents the start and end of a ring. Similar to traditional text applications, tokenization can be used to split up a given molecule into commonly occurring subsequences . Mask prediction during pre-training then proceeds with the model learning to predict chemical structure based on context.