We evaluated forward and retrosynthetic predictive performance of the IBM RXN for Chemistry model on representative data sets of organic chemistry reactions. Top-k accuracies for forward prediction on our dataset were lower than the figure reported in the original Molecular Transformer paper, likely due to differences in the distributions from which training and test reactions were sampled. We identified representative cases of specific pathologies including failure to generate a prediction and incorrect predictions that fail to obey atom conservation or mechanistic considerations. These failures occur for both low confidence predictions (where they might be expected) and high confidence predictions (where they are particularly deleterious). Similarly for the retrosynthesis model, we identified failures associated with atom nonconservation and nonsensical chemical transformations, as well as a lack of reagent specificity with similar target molecules. We also showed cases where the predicted retrosynthesis completely failed when run through the forward prediction model. We compared differences in synthetic accessibility and molecular complexity for the predicted syntheses and retrosyntheses compared to the literature, and found that while the forward model reproduced similar distributions to the literature, the retrosynthesis model did not.