Clearly, the ML heatmap for 3 (Figure ) is not helpful in understanding the predicted logP values and we found that for 13% of the molecules the sum of the ML atomic attributions do not have the correct sign (Figure ). This sign problem is also found in the FPA heatmaps in 5% of the molecules, but only 2% of the molecules have a sign problem for both the ML and FPA heatmaps. About half of the sign problems in the FPA heatmaps occur for logP values that are ≤0.5 suggesting the sign problem is due to imperfect cancellation of nearly equal positive and negative contributions and the heatmaps still offer insights into the predicted logP value (see e.g. Figure ). The non-additive behavior observed for compound 3 is also observed for compound 2: at first sight the heat map seems to show that the model simply has erroneously learned that a protonated tertiary amine group makes a small contribution to a logP value. However, a similar bit-analysis shows that the NH + only makes small contributions when bits related to the distant sulfone group are on. Changing the sulfone group to a methylene group leads to an excellent logP prediction (0.53 vs 0.64) and a better overlap (0.60) where the NH + makes a sizable negative contribution to the logP value (Figure ). Figures show heat maps for the case of high error but average overlap. The relatively high overlap considering the difference in heatmaps reflect the fact that the overlap focuses more on the distribution of positive and negative contributions, rather than on their magnitudes. The ground truth and FPA vectors show that the ground truth logP value (0.02) is a result of the near perfect cancellation of positive and negative contributions of near equal magnitude. Comparing the ML-and FPA-vectors it is clear that while the colors, which are representative of the overlap, are matched reasonably well, the contours, which are representative of the magnitude of the contributions, are not. The error clearly comes from an underestimation of the effect of the N cation and an overestimation of some of the C atoms in the pyrimidine ring. Neutralising the cationic N decreases the error by ca 1 unit due the ground truth logP value being raised by roughly the same amount while the predicted value is essentially unchanged (Figure ). However, roughly the same error (2.26) can be obtained by removing the chlorophenyl group since the NH + 2 group now correctly makes a sizeable negative contribution to the predicted logP value (Figure ). So, just like for 2, the contribution of a cationic N is dependent on another, distant, functional group, which is clearly at odds with the additive nature of the ground truth model. In the absence of the proton or the chlorophenyl group, the remaining error is largely due to some of the C atoms in the pyrimidine ring. The reason seems to be a consistent overestimation of the effect of the corresponding bit (875, Figure ) for the pyridine ring and the overestimation can be removed simply by adding a methyl group (Figure ). Sheridan has developed an XAI approach in which the contribution of an atom is determined by replacing it with a Na atom and this approach has been adopted by others. The advantage if this approach is its ease of implementation and general applicability to all kinds of molecular representations such as FPs and graph neural networks. Figure shows the average overlaps as a function of training set size using this approach as implemented by Jimenez-Luna et al. The overlaps are significantly lower, especially using the FPA vector, where the average overlap is lower than the null-model for all training set sizes. The effects on the average overlaps with the ground truth vector is less pronounced but the average overlaps are similar to or lower than the null-model for training set sizes smaller than 5000 molecules.