Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the activation functions instead of ELU?

In other words what is the advantage of ELU over $log(1+e^x)$?
ReLU and all its variants ( except ReLU-6 ) are linear i.e $ y = x $ for values greater than or equal to 0.

This gives ReLU and specifically ELU an advantage like:


Linearity means that the slope does not plateau or saturate when $x$ becomes larger. Hence, the vanishing gradient problem is solved.




Now, the graph $ y = log( 1 + e^x ) $ isn't linear for values > 0.

For larger negative values, the graph produces values which are very close to zero. This is also found in sigmoid where larger values produce a fully saturated activation. Hence, $ y = log( 1 + e^x ) $ can raise problems which sigmoid and tanh suffer.

About ELU:



ELU has a log curve for all negative values which is $ y = \alpha( e^x - 1 )$. It does not produce a saturated firing for some extent but saturates for larger negative values.

See here for more information.


  Hence, $ y = log( 1 + e^x ) $ is not used because of early saturation
  for negative values and also non linearity for values > 0. This may
  produce problems and even bring down some features which ReLU and
  variants exhibit.