To test the quality of IBM RXN's forward predictions, we queried 100 total reactions and collected results for up to five top predictions. Top-k accuracies were determined by whether the correct product was returned in the first k ranked prediction results. We adopted a lenient scoring policy with regards to enantioselectivity and racemization, to give the predictive model an advantage. Reactions where racemization is expected were counted as correct if one isomer was provided, or if an achiral representation of the product was predicted. For reactions where some mixture of constitutional isomers is expected (e.g., aromatic substitutions resulting in a mixture of ortho and para products), predictions were considered correct if the second isomer was present in the top-5 predictions collected. The top-k accuracies, shown in Table , show considerable variability across the reaction classes. Note that the bolded accuracies are the subsequent highest accuracies recorded as more predictive results are considered. Substitution reactions were among the most accurate, maintaining an accuracy of 90% regardless of the number of prediction results. Elimination reactions were characteristically poor performing with an accuracy of 30% for top-1 to top-4 predictions, increasing to 40% accuracy given a fifth prediction result. Most reaction classes peaked in accuracy with only one or two prediction outcomes, showing that the increasingly lower confidence predictions provide little in the way of increased accuracy. Table shows the cumulative accuracies for each top-k test across the entirety of the test set. We found that taking only top-1 predictions yielded an accuracy of 61%, which increased to 69% for top-2 predictions. This accuracy remained constant until a fifth prediction was considered, whereby we only saw a slight increase to 70%. These total accuracy scores are lower than those reported by Schwaller et al., who cited a top 1 accuracy between 71% and 91% depending on the specified data set. To understand why our accuracy scores are different than what was originally cited by Schwaller et al., we must think about the distributions these reaction data sets were sampled from. We constructed our test sets to be representative of undergraduate organic chemistry, similar to what a student might encounter on their final exam. The model itself was trained on patent data originally mined by Lowe, and is therefore different from the types of canonical reactions taught in an undergraduate curriculum. Thakkar et al. showed that patent data is sufficient for the prediction of medicinally important synthetic targets, however it is possible that these reactions do not completely characterize the types of transformations that are taught in the classroom. To investigate where the forward prediction model is the most uncertain, we began by looking at the reactions with the lowest prediction confidence. We defined a low confidence prediction as having a confidence below 0.50, and identified 19 reactions falling below this cutoff. Table shows the mean confidence scores across the reaction classes. Interestingly, the reaction classes with the most low confidence predictions were substitution, organocopper, acetylide, and amine, many of which have the lowest mean confidence scores overall. Six of these low confidence predictions were outright predictive failures, or cases where the model could not successfully provide a predicted product or simply returned the SMILES of the reactants and reagents as output. Eight of these low confidence reactions were actually correct predictions, albeit with low confidence. Other predictions were more troubling, and highlighted a major pitfall of sequence-based models. In reaction 39, shown in Figure , the number of carbons predicted to be added to the double bond does not match the number of carbons on the organocopper reagent itself. The model treats the reaction as if the organocopper reagent only adds a methyl group, and moreover adds the new carbon-carbon bond to the incorrect carbon belonging to the ring. Double elimination reactions are also problematic. In reactions 68 and 69, the model fails to return a prediction. In reaction 20, the model returns a prediction where only a single elimination occurs. To probe this partial success in the case of double eliminations, two alternative predictions were completed, shown in Figure . In test reaction 1 (TR1) two equivalents of base were provided explicitly in the SMILES string input to determine if this arose from stoichiometric considerations. Paradoxically, providing this additional information to the model changes the prediction: again, only a single elimination to an alkene is seen, but the product contains no halogens. Test reaction 2 (TR2) was constructed to test whether the model considered the double elimination as a two-step process: the singly eliminated alkene product from reaction 20 was reacted again with base. This results in the correct elimination to an alkyne. This supports our hypothesis that the model treats double elimination as a two-step process proceeding through a single elimination intermediate.