If ReLU is so close to being linear, why does it perform much better than a linear function? ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers.

I'm a beginner to deep learning research and methodologies but I've already seen several examples that claim that using ReLU as activation functions on a network will be superior than other common functions, including simply linear ones, of course.

ReLU has 1 point of "drastic" change, and is "otherwise linear". 

My questions are:


Is there research or opinions on why this change significantly affects the performance of a network?
Is there something unique about $0$, the point of change? What if we move the point of change to $1$ instead for instance?


My intuition is that, as $0$ separates the line, it separates the sign function and that may have high affect on certain types of data that rely on positive/negative classification, perhaps.


How relevant is the dataset being learned? Do we see that ReLU is "better" on most examples of datasets?


Why ReLU is better than the other activation functions

The above question is relevant. Mine is more focused on the difference between ReLU and a linear function.
A neural network is a succession of layers (dense/linear, convolutional). After each layer, you need a non-linear activation function. Why do you need a non-linear activation function? Because composing two linear transformations is equivalent to having a single linear transformation:

$ (x \cdot W_1 + b_1) \cdot W_2 + b_2 = x (W_1 \cdot W_2) + (b_1 \cdot W_2 + b_2)$

Therefore, stacking linear transformations would be pointless, because the whole network would be equivalent to a linear transformation.

A ReLU may seem similar to a linear function, but its non-linear nature gives the network the ability to model non-linear functions.
No, there is nothing unique about $0$ being the point of change. Most neural network layers have a bias term that makes the point of change irrelevant, as during the training the bias will be adjusted to the proper value.
While in deep learning everything is data-dependent, the fact that the ReLU's changing point is irrelevant is independent from the data (see previous answer).


The advantage of ReLU is against other non-linearities, like sigmoid and tanh, which suffer the vanishing gradient problem. ReLU itself has its own problem, namely the "dying ReLU problem".