title: Saliency maps of neural networks (using Keras)
body: <p>I have a fully connected multilayer perceptron trained in Keras. I feed it an N-dimensional feature vector and it predicts one out of M classes for the input vector. The training and prediction is working well. Now I want to analyze what part of the input feature vector is actually responsible for a particular class.<br>
  For example, lets say there are two classes <code>A</code> and <code>B</code> , and an input vector <code>f</code>. The vector <code>f</code> belongs to class <code>A</code> and the network predicts it correctly - the output of the network is <code>A=1 B=0</code>. Because I have some domain knowledge, I know that the entire <code>f</code> is actually not responsible for <code>f</code> belonging to <code>A</code>, only a certain part inside <code>f</code> is responsible for that. I want to know if the neural network has captured that. Drawing a correspondence to images, if an image <code>I</code> has a <code>cat</code> in it (with some grassy background) and a trained network predicts that correctly, then the network must know that the entire image is actually not a <code>cat</code>; the network internally knows the location of the <code>cat</code> in the image. Similarly, in my case, the network knows what part of <code>f</code> makes it belong to class <code>A</code>. I want to know what part that is.  </p>
  
  <p>I searched around, and believe what I want to do is called finding Saliency Maps for my network, for a given input. Is that correct?<br>
  If I've understood it correctly, Saliency Maps are simply <code>(change in output)/(change in input)</code>, and can be found by simply 1 backpropagation operation where I find the derivative of output with respect to the input.<br>
  I found the following code snippet for doing this in Keras, but I'm not really sure if it is correct:    </p>
  
  <pre><code>   inp = model.layers[0].get_input()
     outp = model.layers[-1].get_output()
     max_outp = T.max(outp, axis=1)
     saliency = theano.grad(max_outp.sum(), wrt=inp)
  </code></pre>
  
  <p>In the above code, when computing the gradient, is the backpropagation actually happening? The output is a non-linear function of the input, so the only way to find the gradient is to do backprop. But in the above code, there is nothing to connect <code>theano</code> and the network, how is theano "aware" of the network here?  As far as I know, when computing gradients with Theano, we first define the function in terms of input and output. So theano has to know what that non-linear function is. I don't think that is true in the above snippet..  </p>
  
  <p><strong>Update:</strong> The above code doesn't work because I have a fully connected MLP. It gives an error saying "Dense object doesn't have get_output()" . I have the following Keras function, which computes output of network given input. I want to now find gradient of this function wrt the input:  </p>
  
  <pre><code>    get_output = K.function([self.model.layers[0].input],[self.model.layers[-1].output])
  </code></pre>