title: Neural network backpropagation with RELU
body: <p>I am trying to implement neural network with RELU.</p>
  
  <p>input layer -> 1 hidden layer -> relu -> output layer -> softmax layer</p>
  
  <p>Above is the architecture of my neural network. 
  I am confused about backpropagation of this relu. 
  For derivative of RELU, if x &lt;= 0, output is 0.
  if x > 0, output is 1. 
  So when you calculate the gradient, does that mean I kill gradient decent if x&lt;=0?</p>
  
  <p>Can someone explain the backpropagation of my neural network architecture 'step by step'?</p>