How to find out what portions of an image is helping CNN to classify it
Yes, the typical approach is to obtain the saliency map of the input, which are &quot;heatmaps&quot; of the contribution of each pixel to the final classification.
In this free online book about Explainable ML, you can find the most relevant approaches to obtain saliency maps, like vanilla gradients, together with other pixel attribution techniques.
Here you can find an end-to-end tutorial on how to implement saliency maps with Keras.