How to get a heatmap of pixel influences from a Convolutional Neural Network
Yes, there are various proposals out there.  Many of them work by computing a saliency map.  If you search for &quot;saliency map&quot;, you should be able to find multiple schemes for that.
Personal opinion: these methods give heatmaps that seem superficially plausible; but my sense is that they have major limitations.  So don't set your expectations too high, or you may be disappointed.