Despite the advances made with the YieldBERT, YieldBERT-DA and BEE models, the investigation of BERT pre-training, tailored for the predictive synthesis has been largely unexplored. To the best of our knowledge there is no systematic, comprehensive evaluation of tokenization and reaction representation approaches for Encoders. Thus, in this study, we first explore the most common tokenization methods for language models paired with common molecular notations, and then we perform an examination of the impact of pre-training, pre-training data sizes, data sources and data modifications on the downstream yield and reaction success predictions. The study focuses specifically on yield and reaction success predictions for publicly available Buchwald-Hartwig and Suzuki-Miyaura HTE data sets, using BEE as the base architecture.