title: Feed Forward ANN: calculating delta node from previous layer delta
body: <p>I am trying to implement a feed forward neural network in CUDA.
  So far, I've used <a href="https://www.youtube.com/user/HeatonResearch/videos" rel="nofollow">Jeff Heaton's YouTube videos</a> as a guide to infer the algorithms and implement them.
  I'm not clear on one thing:</p>
  
  <p>Heaton in his <a href="https://www.youtube.com/watch?v=p1-FiWjThs8" rel="nofollow">Gradient Calculation</a> video, explains how to get the node delta δ[i] for output neurons. 
  Then I have to reuse it as δ[k] in my previous (hidden layer) in order to calculate δ[i]. </p>
  
  <p>However, there is no mention of what happens when <strong>I have more than one node delta</strong>, or more than one outgoing weights for layer <em>i</em> to layer <em>k</em>. Similarly its confusing on how I calculate the gradient for a specific weight; do I use a node delta, or a layer's delta?</p>
  
  <p>For example, what happens if my output layer has 2 neurons? Do I sum the δ[k] for all nodes in layer <em>k</em>? </p>
  
  <p>The formula provided by Heaton:</p>
  
  <pre><code>f'( Sum (weight[i] * input[i] ) ) * Σ w[ki]  * δ[k]
  </code></pre>
  
  <p>Appears to suggest that δ[k] is representative of the previous layer (and not just the output node) as shown in the video @ 9:25'. </p>
  
  <p>In fact, two of the comments in that YouTube video also ask the same thing, but no satisfactory reply is given.</p>
  
  <p>As far as I understand, δ[k] represents the error from the previous/following layer <em>k</em> to which layer <em>i</em> connects to, and not the <strong>node</strong> to which current node in layer <em>i</em> connects to?</p>
  
  <p><strong>EDIT</strong> </p>
  
  <p>I've read a few papers and tutorials/lessons online, but the one that seems to somewhat answer my question, can be <a href="http://arduinobasics.blogspot.co.uk/2011/08/neural-network-part-5-back-propagation.html" rel="nofollow">found here</a>.
  In specific, the formula the blog's author uses, is the same as the one used by Heaton, but he explains:</p>
  
  <pre><code>HLN = Hidden Layer Neuron, 
  LLN = Last Layer Neuron,
  aO=actualOUTPUT,
  dE=deltaError
  
  
  HLN.dE = (HLN.aO) x (1-HLN.aO) x (Sum of   [LLN.dE   x   LLN to HLN connection weight])
  </code></pre>
  
  <p>This seems to imply that the formula actually is:</p>
  
  <pre><code>Si = Σ (w[ji] * a[j])
  δ[i] = f'( Si ) * Σ( w[ki]  * δ[k] )
  </code></pre>
  
  <p>In words:</p>
  
  <blockquote>
    <p>The Sum of the previous activation outputs multiplied by their edges
    (weights), is put through the sigmoid derivative.</p>
    
    <p>Then for each out-going weight <em>w[ki]</em> is multiplied by the respective
    <em>δ[k]</em> (e.g., the <em>w[ki]</em> * <em>δ[κ]</em>).  Those values are summed, and multiplied with the result of the sigmoid derivative.</p>
  </blockquote>
  
  <p>I'd still like to hear from someone who has implemented a feed-forward neural network, if that is what happens.</p>