title: Neural Net gradient calculation with Batch Normalization C++
body: <p>I was trying to change my activation function of my neural net from sigmoid to RELU (or more specifically SELU). Since I got a lot of exploding gradients with that change, I tried to use the batch normalization. I calculated the gradients of my error function w.r.t the learning parameters \beta and \gamma, but it seems that they are a bit different from the ones I saw in several (sadly only Python) examples.</p>
  
  <p><a href="https://kratzert.github.io/2016/02/12/understanding-the-gradient-flow-through-the-batch-normalization-layer.html" rel="nofollow noreferrer">Here</a>, for example, the code example on the bottom of the page says <code>dbeta = np.sum(dout, axis=0)</code> and I wonder what exactly this <code>dout</code> is. </p>
  
  <p>My derivatives look like this:</p>
  
  <p><a href="https://i.stack.imgur.com/Izgfk.png" rel="nofollow noreferrer">Derivation of error function w.r.t \beta</a></p>
  
  <p>What am I doing wrong in this derivation?</p>
  
  <p>Thank you a lot for your help.</p>