The pre-training procedure begins by converting each molecule in the batch to a non-canonical SMILES form which corresponds to the given molecule. SMILES strings are then randomly modified, tokenised and embedded into a sequence of vectors. Sinusoidal positional embeddings are added before the sequence is passed into the Transformer layers of the model. The modified sequence is passed to the bidirectional encoder, while the autoregressive decoder is asked to predict the original SMILES sequence, given the same sequence right-shifted. A fully-connected layer is applied to the output of the decoder to produce a distribution over the model's vocabulary and a cross-entropy loss function is used to train the model.