Next, the models were trained on out-of-sample splits. Two scenarios were considered -the first was regression, where the models predicted the yield of the reaction, and the second was binary classification, where the models predicted the reaction success, defining success as >5% yield. Each data split was evaluated separately using the out-of-sample approach, as the model performance varied considerably across different compound combinations in each validation set. This very high variation rendered the traditional aggregation of results across splits, a common practice in standard cross-validation, impractical. To address this limitation, each model was trained and validated five times per split, using a distinct random seed for each iteration (see Figure for the screening results using SMILES notation with the data set by Ahneman et al. and SI sections 1.4 and 1.5 for the complementary data). Split number five for the data by Ahneman et al. proved particularly challenging for prediction, giving noticeably low R 2 , RMSE and Cohen's Kappa values in comparison to other validation sets. R 2 was typically lower by at least 0.2 than splits two through four, RMSE was higher by at least 0.05 and Cohen's Kappa dropped by ca. 0.15 or more. Split number one contained two aryl chlorides as the out-of-sample compounds, characterised by significantly lower average yields than the remaining bromides and iodides. The R 2 values for this compound set were typically low, normally below 0.5 (presumably due to the very limited yield range for the chlorides), but RMSE and Cohen's Kappa values were satisfactory and comparable to the splits two to four. In addition to the aforementioned drop in model performance when WordPiece was used with SELFIES or MFP-based representations, the combination of WordPiece with SMILES or DeepSMILES also resulted in subpar regression metrics for the data split number one. The regression performance with the set by Ahneman et al. was better for smaller vocabulary sizes (atomisation, 500 or 1000 tokens; see Figure ), but the trend was less apparent for classification. As expected, the overall out-of-sample model performance with the data set by Perera et al. was poor, where the average R 2 across splits was below 0.45 and Cohen's Kappa was below 0.32 for all methods. The metrics were the lowest for the splits where the validation quinolines were decorated with reactive chloride, iodide or triflate. The validation triflate, however, gave better results in regression than the chloride or iodide, and this can be ascribed to the triflate's yields being closer to the average for the whole set (Figure ). The results for Suzuki set are less conclusive, but the regression task generally favoured the largest used vocabulary of 20,000 tokens (Tables and). Comparison of the metrics for the best models for each representation indicated no or very little influence of the molecular representation type on the model performance (Tables S32, S33, S44 and Figures and).