feed-forward neural networks figure - softmax layer rectified linear unit rectified linear unit (reluused in conjunction with linear transformation transforms the input as max ,wx the underlying activation function is (xmax ( xrecentlythe relu is more commonly used as hidden unit results show that relus lead to large and consistent gradientswhich helps gradient-based learning (figure - although relu looks like linear unitit has derivative function and thus allows for computing the gradient of the losses in recent timesthe relu has been the most popular choice for hidden network activation in most casesa relu can be default choice that would lead into desirable results within timely manner