title: Top Down Modulation Two Layer Network
body: <p>I have this simple two-layer feedforward network that is going to classify MNIST digits. With this network, I get satisfiable accuracies, but I want to modify it slightly. Following a paper by <a href="https://arxiv.org/abs/1612.06851" rel="nofollow noreferrer">Shrivastava et al.</a>, I want incorporate their top-down modulation without the lateral modules (see page 3). Also, I want to keep it limited to these two layers, so not create a deeper network. Basically, in the new network, as opposed to the simple feedforward network below, the second layer is connected again to the first layer to achieve this top-down feedback link. </p>
  
  <p>I have no idea how to go about it, and after considerable online digging I haven't found any other source code/examples on how to do this in TensorFlow (or any other library for that matter). Any help or a push in the right direction is much appreciated!</p>
  
  <pre><code># PREVENTS VANISHING GRADIENT PROBLEM
  stddev1=np.sqrt(2 / float(784))
  # second = 10 units
  stddev2=np.sqrt(2 / float(10))
  
  w1 = tf.get_variable('w1',[784,10],
                       initializer=tf.random_normal_initializer(mean=0, stddev=stddev1))
  b1 = tf.get_variable('b1',[10,],
                       initializer=tf.constant_initializer(0.0))
  # relu has vanishing gradient problem
  z1 = tf.matmul(x, w1) + b1
  y1 = tf.nn.relu(z1) # 
  
  # layer 2
  w2 = tf.get_variable('w2',[10,10],
                   initializer= tf.random_normal_initializer(mean=0,stddev=stddev2))
  b2 = tf.get_variable('b2',[10,],
                       initializer=tf.constant_initializer(0.0))
  y = tf.nn.softmax(tf.matmul(y1, w2) + b2, name="output")
  
  y_ = tf.placeholder(tf.float32, [None, 10])
  
  cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
  train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
  </code></pre>