Attribution methods, which aim at producing explanations by assigning a numerical value to each atom to create a so-called heatmap, are among the most popular XAI methods in chemistry. Some of these methods were originally developed for image classification (for example ) where it is often fairly obvious whether the heatmap highlights the correct part of the image. However, for chemical applications it is often less clear whether the atomic attributions are correct for a certain chemical property, which complicates the benchmarking of these methods. One solution is to use simple toy models such as classifying molecules with respect to the presence or absence of certain functional group. While methods that fail at such simple tasks can probably be discounted, it is not clear whether methods that succeed will also succeed for more complex classification and, especially, regression tasks.