All neural networks were trained using GSK's SynthCoder platform. The hyperparameters for the model pre-training with MLM were based on the published settings for the BERT-based BEE model. The networks were set to hidden size of 256, number of the attention heads was set to 4, the intermediate size was 512, the maximum number of position embeddings was set to 512, the maximum enrichment tokenization vocabulary was 64 and the pre-training learning rate was set to 2e-4. The remaining BERT configuration settings were set to default, as specified by HuggingFace. The pre-training with Pistachio-based reaction sets was allowed to run for up to 197370 training steps (equivalent to 30 epochs with Pistachio set with 1.68M reactions) using 16-bit mixed precision and the batch size of 64. Pre-training with the 15M Reaxys-Pistachio set and the artificial reaction sets was performed for 30 epochs. The pre-training models were evaluated based on the performance on the validation sets using the loss metrics, where the best models were further used for fine-tuning. Fine-tuning through the random split cross-validation scheme, was performed for a maximum of 300 epochs, and the early stopping criterium based on the performance of the models on the validation set was set to 100 epochs of no improvement. The training with out-of-sample splits was done for 300 epochs and no early stopping criterium was applied. The fine-tuning learning rate was set to 8e-5. The class imbalance in the data was accounted for by providing the pre-calculated class weights to the loss function. When applicable, 𝛼 and 𝜖 hyperparameters used for adversarial training were both set to 0.5 for pre-training and fine-tuning. Other hyperparameters remained constant with the settings used for the pre-training, unless specified otherwise for specific cases.