Since the input to GCNN consists of a number of atomic feature vectors as well as how the atoms are connected, there is a more obvious link to getting atomic attributions compared to the fragment-based fingerprint used above. Here we use the "remove-an-atom" method used by Matveieva et al. . An atom is "removed" by deleting the row with atomic features corresponding to that atom as well as all connections (edges) to that atom. model is not reflected in the agreement of the ML-generated atomic attribution heatmaps with the ground truth. The average overlap with the ground truth is 0.47 for the GCNN model (down from 0.54 for the ECFP4/RF model). Interestingly, the distribution of overlaps is very different for the GCNN generated overlaps (labelled GCNN1 in Figure ) compared to that of the ECFP4/RF generated overlaps (RF in Figure ). While the ECFP4/RF overlaps show a single peak in the distribution centered roughly around the average overlap, we observe a double-peaked distribution for the GCNN overlaps with the ground truth vectors (labelled GCNN1). Thus, even though the GCNN overlaps produce a lot more atomic attribution vectors in bad agreement with the ground truth, it also produces many more atomic attribution vectors that are in close-to-perfect agreement (overlap ≈ 1) with the ground truth. The average overlap with the FPA vectors is 0.63, which is similar to the null-model where all atoms contribute the same. As the FPA vectors are designed specifically to reflect the ECFP4 fingerprint we also should not expect the GCNN generated contributions to reflect these. The average overlap with the ECFP4/RF generated atomic contributions is 0.53. While the heatmaps generated for the GCNN model are generally not closer to the ground truth than the ECFP4/RF atomic attribution heatmaps above, they might help assessing the quality of the ECFP4/RF heatmaps in the absence of the ground truth atomic attributions. We find, that the Pearson's linear correlation coefficient between the overlap of RF+ground truth and RF+GCNN is 0.56 (Figure ). Thus, a high overlap between atomic attributions generated by the two ML models indicates a higher probability of the ECFP4/RF attributions being in agreement with the ground truth. The correlation between GCNN+ground truth overlaps and RF+GCNN overlaps is less pronounced with a Pearson's r of 0.30 (Figure ). Heatmaps of the atomic attributions from the GCNN model for the four molecules analyzed above are shown in Figure (labelled GCNN1). Contrary to the heatmaps in Figure for the 150k RF/ECFP4 model, the worst overlap with the Crippen atomic contributions is observed for molecule 1 (overlap = 0.20). We observe problematic behavior around the amide bond (all negative contribution on oxygen and C+N are positive) as well as on the halogens, which are predicted to make a negative contribution to logP, while the opposite is true for the ground truth contributions (Figure ). For molecules 2, 3 and 4 the GCNN generated heatmaps have high overlaps (>0.75) with the ground truth. When analyzing atomic heatmaps generated by an XAI method, we can ascribe lack of agreement with the ground truth to being caused by either the model being unknowledgeable about the atomic contributions or the XAI method being incapable of extracting those atomic contributions. For example we saw that the XAI methods used on fragment-based fingerprints above would not actually extract atomic contributions since the XAI method "removes" more than just the atom. Similarly, we can use the Crippen logP benchmark with ground truth heatmaps to understand what causes the disagreement with the GCNN generated atomic heatmaps. Qualitative assessment of the atomic GCNN heatmaps shows the "halogen" problem observed for molecule 1 to be a common problem and choose to investigate this phenomenon more thoroughly. Figure shows a heatmap of the atomic contribution for the Crippen logP model of an example molecule where the three largest atomic contributions are from the chlorine atom and the two methyl carbon atoms. The GCNN prediction has a low error of 0.05 but the logP contribution from the chlorine atom as well as from the methyl carbon atoms are all predicted to be negative. Does that mean that the GCNN model actually assigns a negative contribution to these atoms or is it an artefact of the XAI method? If we, instead of masking the Cl atom with the XAI method, remove the Cl atom by changing it to a hydrogen atom the change in the logP predicted by the GCNN model (-0.68) is almost identical to the actual change in Crippen's logP (-0.65, Figure , upper arrow). This suggests that the GCNN model has learned about the positive contribution of a chlorine atom and gets its magnitude right. So why is the Cl atom given a negative contribution when we mask it with the XAI method where the logP predicted by the GCNN model increases 0.53 (Figure , bottom arrow). Since hydrogen atoms are not included explicitly in the graph-input, the only difference between the input where we change Cl to H and where we remove Cl with the XAI method is in the description of the carbon atom bound to Cl. Part of the atomic feature vector we use is a one hot encoding of the number of heavy-atom neighbors as well as the number of bound hydrogen atoms. The big difference in predicted logP then stems from a change from a one hot encoding of three heavy atoms to two heavy atoms and a change from zero to one bound hydrogen atoms. Thus, after masking the Cl atom with the XAI method, there still is some implicit information about it through the atomic feature vector of the bound carbon atom. To avoid this we tried training a GCNN model without such implicit information about each atom's neighbor- the Cl atom with the XAI method corresponds exactly to removing it by changing it to a hydrogen atom. The change in predicted logP from the GCNN2 model is -0.61 and the Cl atom (as well as the methyl groups) are not correctly assigned a positive logP contribution (Figures and). Note however that this does not result in a general improvement of the agreement between the ML generated atomic contributions and the ground truth, for which the average overlap decreases from 0.47 to 0.45 (Figure ). For the four example molecules we generally observe heatmaps that are similar to those generated by the GCNN1 model 9).