Why do CNNs with ReLU learn that well?
ignoring that is not differentiable at 00, as I think it is done in practice


yes see  ReLUs are not differentiable at zero


  If a neuron outputs 0 for every sample of the training data, it is basically lost, correct? Its weights will never be adjusted again?


yes see What is the &quot;dying ReLU&quot; problem in neural networks?