RoBERTa model builds on the BERT transformer, and modifies key hyperparameters of the BERT transformer. The choice of RoBERTa over the original BERT here was because of its use of BPE tokenisation approach, as discussed in the "ChemMLM tokeniser" Section, and dynamic mask method, as discussed in the "Mask method" Section. The other architecture of RoBERTa is close to BERT.