In this study we evaluated the impact of tokenization and molecular notation on yield and reaction success prediction with BERT-based BEE model. We also investigated the importance of pre-training using reaction schemes on the downstream predictive tasks with HTE data sets, examining data sizes, sources and augmentation approaches. BERT proved to be insensitive to the specific type of molecular notation used and, thus, we recommend using the most convenient representation available for the data. However, we advise against pairing SELFIES or MFP-based notation with BERT's native WordPiece tokenizer, due to a large proportion of generated unknown [UNK] tokens, negatively impacting the information seen by the model and in effect decreasing the predictive performance. The results also showed that the popular tokenization through chemically informed string atomisation is not the most optimal approach. Based on the metrics collected for the random and out-of-sample data splits for different HTE reaction sets, BPE with vocabulary size of 500 and paired with SMILES was selected as the preferred approach in this study. It was found that utilising random data splits, wherein identical chemical structures are allocated to both the training and evaluation sets, results in models becoming dependent on inherent data patterns, thereby rendering pre-training unnecessary. In order to avoid drawing erroneous conclusions from conducted experiments, our recommendation is to abstain from using random splits for assessment of chemical understanding by the predictive models for synthetic reactions, and if that is not possible, appropriate control experiments should be employed. Evaluation of pre-training data sizes for BEE was conducted, showing that pre-training on data sets of relatively small sizes (<100K reactions) can achieve comparable downstream performance to the models pretrained on significantly larger collections with millions of reaction examples. We also showed that pretraining with completely artificially generated reaction schemes could be a viable approach in case real data cannot be obtained, is too costly or too time consuming to prepare. The optimal results were obtained when domain-specific, artificial data were combined with real reaction examples. Finally, a new approach to BERT-based adversarial training was demonstrated, improving the end-task performance of the yield and reaction success prediction models. Adversarial training does not require additional data curation or preparation by the practitioner and offers a convenient hands-off approach for potential improvement in model performance. A limitation of this study is that HTE data sets used herein are very narrow in scope and cover very limited chemical space. Thus, the conclusions presented here will be revalidated when larger, more diverse and chemically comprehensive finetuning sets become available publicly. Although the presented solutions focused on yield prediction, it is likely that they could translate to more general chemistry-based modelling tasks. To aid the community towards solving the yield prediction problem, GSK's modelling platform SynthCoder, used in this study, is being made publicly available with this work.