We studied the outliers of the MPNN to gain a further insight on the generalizability of the model. The compounds with the largest absolute errors are are those with particularly high solvation energies, with 15 of the top 25 errors being within the 1% most negative solvation energies. The relatively poor predictive performance on these materials is partly explained by the solvation energy being outliers compared to the rest of the data. Twelve of top 25 (48%) errors belong to a class of molecules, zwitterions, that is rare within the training data with only 396 (0.3%) entries within the training. We conclude that the model's predictions are most reliable for molecules with solvation energies of less negative than -25 kcal/mol and is, for example, inaccurate for zwitterions.