Why doesn't relu cause back propegation to get stuck Say you have a neural net that is being trained using back propagation and you are using relu activation. Say the input to a node is a weighted sum of the previous layer with a bias term and say for a particular data point, this weighted sum plus bias is negative. Then relu returns 0. Notice the change in the loss as a function of the change in one of these weights or the bias is 0. Therefore the network won't improve the bias as the network does back propagation. Why is this not a problem?
Normally, any ReLU unit will return 0 for some samples and not for other samples (in particular, it'll probably return 0 for about half of the samples).  Because of how SGD works, it won't get stuck: the former samples will yield a contribution to the gradient of 0, but the latter samples will have a non-zero gradient, so the weights and bias will be updated for the latter samples.  So it's not a problem; the weights and bias still get updated for many of the samples.
That said, there can be cases where this problem can occur.  If the ReLU unit outputs 0 for all (or most) samples, then SGD won't update the weights, and learning for that neuron will get stuck.  This is sometimes called the &quot;dying ReLU&quot; problem.  See, e.g., https://datascience.stackexchange.com/q/5706/8560 and https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Potential_problems.   Typically we're able to avoid this problem by appropriate initialization of the weights and by suitable choice of learning rate for SGD, so it's usually not a big deal in practice if you choose those appropriately, but it can happen.