Is there a ReLU-like activation function that concatenates positive and negative values?
It seems I have found it. It is called concatenated ReLU (CReLU).


  Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.



Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (the original paper that introduced CReLU).
A TensorFlow implementation


There is also Negative CReLU. It seems that the difference is only the sign.

$$\text{NCReLU}(x) = (\rho(x) , −\rho(−x) )$$