title: Is there a way to reverse a graph in TensorFlow?
body: <p>As the title says, I'm looking for a way to reverse the flow of a TensorFlow graph. The reason for this, is that I want to visualize the hidden layers of the graph given a logit vector for the output of the trained graph.</p>
  
  <p>For example, say that I have a fully connected graph given as follows (inspired by MNIST):</p>
  
  <pre><code>inputs = tf.placeholder(dtype=tf.float32, shape=[None, 784])
  hidden_w1 = tf.get_variable('w1', [784,100], initializer=tf.random_normal_initializer)
  hidden_b1 = tf.get_variable('b1', [100], initializer=tf.random_normal_initializer)
  a1 = tf.matmul(inputs, hidden_w1) + hidden_b1
  z1 = tf.nn.relu(a1)
  
  hidden_w2 = tf.get_variable('w2', [100,100], initializer=tf.random_normal_initializer)
  hidden_b2 = tf.get_variable('b2', [100], initializer=tf.random_normal_initializer)
  a2 = tf.matmul(z1, hidden_w2) + hidden_b2
  z2 = tf.nn.relu(a2)
  
  output_w = tf.get_variable('w3', [100,10], initializer=tf.random_normal_initializer)
  output_b = tf.get_variable('b3', [10], initializer=tf.random_normal_initializer)
  a3 = tf.matmul(z2, output_w) + output_b
  output = tf.nn.relu(a3)
  
  loss = tf.nn.softmax_cross_entropy_with_logits_v2(labels=..., logits=output)
  train_op = tf.train.AdamOptimizer().minimize(loss)
  </code></pre>
  
  <p>Suppose I now train this graph and want to visualize hidden 1 when only the first output neuron is activated. The way I would do this would be to reverse the flow of the graph and feed a tensor <code>[1, 0, 0, 0, 0, 0, 0, 0, 0, 0]</code> from the output layer back through the reversed graph until I finally got the output of the hidden1 layer. I have tried to see if there is a way to do this in TensorFlow, but there seems to be little information about this. The way I would inutitively construct it is to add an operation sess.run_reverse() when running the graph as follows:</p>
  
  <pre><code>with tf.Session() as sess:
  while training:
      sess.run(train_op, feed_dict={inputs:...})
  
  # finished training, reverse graph
  category_to_visualize = tf.one_hot(indices=0, depth=10)
  sess.run_reverse(hidden1, feed_dict={output:category_to_visualize})
  </code></pre>
  
  <p>If this sort of operation doesn't exist or even is possible to get, however, I would instead construct separate operators for reversing the flow of the graph as follows:</p>
  
  <pre><code>output_reversed = tf.placeholder(dtype=tf.float32, shape=[1,10])
  z3_reversed = tf.nn.relu(output_reversed)
  a3_reversed = tf.matrix_inverse(output_w)*(z3_reversed - output_b)
  
  z2_reversed = tf.nn.relu(a3_reversed)
  a2_reversed = tf.matrix_inverse(hidden_w2)*(z2_reversed - hidden_b2)
  
  z1_reversed = tf.nn.relu(a2_reversed)
  a1_reversed = tf.matrix_inverse(hidden_w1)*(z1_reversed - hidden_b1)
  </code></pre>
  
  <p>I realize that there might be logical flaws to this method that wouldn't make it possible. A couple of things I've overlooked is singular matrices and undefined inversion of ReLu when input is below 0 (ReLu, though, can be replaced by sigmoid for theoretically defined inversion of the entire input space). The core idea, though, is to visualize a feature map given a category - something I believe should be possible if a few assumptions are allowed.</p>
  
  <p>Anyways, please tell me if I'm thinking wrongly here, and if there is a way to reverse the graph!</p>