To interrogate these architectures with a more rigorous test of out-of-distribution performance, several case studies were performed where the EGAT and Chemprop architectures were trained on subsets of the training data split that excluded one or more classes of reactions (Fig. ). The gap between the within distribution and out-of-distribution performance is large in all of the cases studies (Fig. ). The accuracy drop for within distribution performance is small in most cases and generally mirrors the reduced amount of training data available. In contrast, the accuracy drop for the out-of-distribution reaction classes is much larger than can be explained from reduced training data and reflects the qualitative failure of these architectures to predict E a for reactions that differ from the training distribution in either the number and kind of bonds changes, molecularity, or size. We had hypothesized that the models would extrapolate better across some of these classes than others. This was incorrect-the transferability to unseen reaction populations is consistently poor, with accuracy often worse than the naïve mean-predicting model. We had also hypothesized that the multitask or transfer learning architectures might show better transferability, but this also wasn't the case (see Fig. for the combined results with all architectures). All the EGAT architectures perform poorly in the out-of-distribution prediction tasks with only a marginal benefit from the architectures with access to ∆H r . With the exception of large reactants, Chemprop generally shows significantly lower errors than EGAT in out-of-distribution tasks. But neither architecture achieves out-of-distribution performance that could be relied on for applications.