title: How to propagate/fire recurrent neural networks(RNN)?
body: <p>I'm learning about  artificial neural networks and have implemented a standard feed-forward net with a couple hidden layers. Now, I'm trying to understand how a recurrent neural network(RNN) works in practice, and am having trouble with how activation/propagation flows through the network.  </p>
  
  <p>In my feed-forward, the activation is a simple layer by layer firing of the neurons. In a recurrent net, the neurons connect back to previous layers and sometimes themselves, so the way to propagate the network must be different. Trouble is, I can't seem to find an explanation of exactly how the propagation happens.</p>
  
  <p>How might it occur say for a network like this:</p>
  
  <pre><code>Input1 ---&gt;Neuron A1 ---------&gt;  Neuron B1 ---------------------&gt; Output
                  ^                   ^     ^      |
                  |                   |     --------
                  |                   |
  Input2 ---&gt;Neuron A2 ---------&gt;  Neuron B2
  </code></pre>
  
  <p>I imagined it would be a rolling activation with a gradual die down as the neuron's thresholds decrease the neuron firing to 0, much like in biology, but it appears there is a much more computational efficient way to do this through derivatives? </p>