Machine learning (ML) models occasionally make wrong predictions and given their black-box nature it is not always obvious when that is the case. While there are several methods for assigning uncertainties to the predictions, these methods report (at best) on the likelihood of prediction errors and there is not a strong correlation between errors in the predictions and their uncertainties. There is therefore a great deal of interest in methods aimed at explaining the ML predictions, often referred to as explainable AI (XAI), which can help humans decide whether the predictions are reasonable. Within chemistry, in general, and drug discovery in particular, another motivation is to use the explanation to help guide the design of molecules with improved properties (for example: ).