In conclusion, the analysis presented in this study showcases how both pre-training methods of the RoBERTa architecture, namely MLM and MTR, empower the models to grasp significant representations of molecular structures. Notably, the MLM model attains a performance level nearly on par with the current best results in predicting HIV activity in a downstream task.