After determining the appropriate method of reaction representation and tokenization, we set to investigate the influence of the pre-training data size and source on the downstream yield prediction task considering random splits and out-of-sample splits separately. The pre-training of BERT and BERTbased models has proven as an excellent strategy for enhancing their effectiveness across various endtasks. In this study, the employed pre-training MLM forces the Encoder model to learn the "grammar" of chemical reactions, with the expectation that the model will use this pre-learned chemical knowledge to improve the yield and reaction success predictions. Chuang and Keiser suggested that a fully combinatorial layout of the Buchwald-Hartwig data set by Ahneman et al. (with random splits) provided the underlying structure to the data, resulting in pattern exploitation by classical ML models, without any reliance on the chemical knowledge. If the model can rely solely on the underlying data patterns, without referring to any molecular structure-based information, the different pre-training strategies would have a negligible effect on the results. To assert if this is true for the complex neural architecture of BERT, when used with the available HTE data sets, we fine-tuned BEE on the random splits using no information about the chemical structures, encoding each unique molecule in the data as an integer. Additionally, a model with no neuron pre-training and no pretrained tokenizer was trained with the SMILES-based data, where a new vocabulary dictionary was generated on the training data coinciding for each cross-validation split. This approach results in a nongeneralisable and very data specific vocabulary. As a sanity check, a supplemental y-scrambling was performed, with randomly shuffled labels for the corresponding data. These control experiments conclusively showed that the removal of the chemical information from the data resulted in no negative impact on the model performance with any of the four tested Buchwald and Suzuki HTE data sets, but y-scrambling was completely fatal for the predictions (R 2 of 0.00). The elimination of pretraining and the pre-trained tokenizer in case of random splits also did not show any decrease in the performance. Conversely, in a few cases a marginal but statistically significant improvement was shown for the simplified models, most likely due to the smaller number of tokens which needed to be processed in the input (Figures 6 and S32-S33, Tables ). This clearly indicates that even if complex pre-training is applied, the model can fully fall back onto the reliance on the chemistryagnostic data patterns. In light of these findings, we strongly advise against blindly employing random data splits for evaluating models intended to learn and operate on chemical knowledge. Random splits can enable models to exploit underlying data patterns that do not reflect authentic chemical comprehension, even when working with pre-trained models. Consequently, we abstain from using random splits in further model evaluations and instead prioritise out-of-sample splits in relation to the reactants.