Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning
ReLU and all its variants ( except ReLU-6 ) are linear i.e $ y = x $ for values greater than or equal to 0.

This gives ReLU and specifically ELU an advantage like:


Linearity means that the slope does not plateau or saturate when $x$ becomes larger. Hence, the vanishing gradient problem is solved.




Now, the graph $ y = log( 1 + e^x ) $ isn't linear for values > 0.

For larger negative values, the graph produces values which are very close to zero. This is also found in sigmoid where larger values produce a fully saturated activation. Hence, $ y = log( 1 + e^x ) $ can raise problems which sigmoid and tanh suffer.

About ELU:



ELU has a log curve for all negative values which is $ y = \alpha( e^x - 1 )$. It does not produce a saturated firing for some extent but saturates for larger negative values.

See here for more information.


  Hence, $ y = log( 1 + e^x ) $ is not used because of early saturation
  for negative values and also non linearity for values > 0. This may
  produce problems and even bring down some features which ReLU and
  variants exhibit.