feed-forward neural networks figure - rectified linear unit there are few disadvantages with reluhowever when inputs approach near zerothe gradient of the function becomes zero and thus gets stuck within the training steps with no progress in the training this is commonly known as the dying relu problem yperbolic tangent the hyperbolic tangent unit transforms the input (used in conjunction with linear transformationas followsy tanh wx the underlying activation function (figure - is given by tanh the hyperbolic tangent unit is also commonly used as hidden unit figure - covers only handful of the available options in activation functions for deep learning