Intuition: why are ReLu activations boundary lines linear?
As far as i understand, relu gives linear boundary because it is linear for X&gt;0. If you expand it to write it in the form of an equation like Z=f1(x1)+f2(x2)....fn(xn), we simply get a linear equation even if we consider the dead neurons. The part where x&lt;0 makes sure that no -ve signals are carried forward but dosent add any non linearity to the final equation. The effect of dead neurons probaly the the final coefficients of the equaion will have coefficients with slightly less value than had the activation been just a normal fuction since there is no negative component which is added to a neuron.