title: Backpropagation example
body: <p>I'm trying to implement my own neural network, but I'm not very confident that my math is correct.</p>
  
  <p>I'm doing the MNIST digit recognition, so I have a softmaxed output of 10 probabilities. as output. I then compute my delta output thus:</p>
  
  <pre><code>delta_output = vector of outputs - one-hot encoded actual label
  </code></pre>
  
  <p>delta_output is a matrix of dimension 10 x 1.</p>
  
  <p>Then I compute the delta for the weights of the last hidden layer thus:</p>
  
  <pre><code>delta_hidden = weight_hidden.transpose * delta_output * der_hidden_activation(output_hidden)
  </code></pre>
  
  <p>Assuming there are N nodes in the last hidden layer, weight_hidden is a matrix of dimension of N by 10, delta_output from above is 10x1, and the result of der_hidden_activation(output_hidden) is  N x 1.</p>
  
  <p>Now, my first question here is, should the multiplication of delta_output and der_hidden_activation(output_hidden) return a 10 x N matrix using outer product? I think the I need to do a Hadamard product of this resulting matrix with the untouched weights to get delta_hidden to be N x 10 still.</p>
  
  <p>Finally I multiply this delta_hidden by my learning rate and subtract it from the original weight of the last hidden layer to get my new weights.</p>
  
  <p>My second and final question here is, did I miss anything?</p>
  
  <p>Thanks in advance.</p>