can be efficiently implemented using dedicated computing hardware such as graphical processor units (gpusand other parallel computing architecture note also that many matrix computations that run in quadratic time can be replaced with linear-time componentwise multiplication specificallymultiplication of vector with diagonal matrix is equivalent to componentwise multiplicationa = |{zb diag(aconsequentlywe can write dl- dl- > dl asdl- (zl- > dl we now summarize the back-propagation algorithm for the computation of typical /th in the following algorithmlines to are the feed-forward part of the algorithmand lines to are the back-propagation part of the algorithm algorithm computing the gradient of typical (thl inputtraining example (xy)weight matrices and bias vectors {wl bl } = =thl activation functions {sl } = outputthe derivatives with respect to all weight matrices and bias vectors for do /feed-forward zl wl al- bl al sl (zl dl zl for do dl bl dl > - wl dl- (zl- saturation /arbitrary assignment needed to finish the loop /back-propagation > dl return and for all and the value (xal (if neededl note that for the gradient of (thto exist at every pointwe need the activation functions to be differentiable everywhere this is the casefor examplefor the logistic activation function in figure it is not the case for the relu functionwhich is differentiable everywhereexcept at howeverin practicethe kink of the relu function at is unlikely to trip the back-propagation algorithmbecause rounding errors and the finiteprecision computer arithmetic make it extremely unlikely that we will need to evaluate the relu at precisely this is the reason why in theorem we merely required that (this almost-everywhere differentiable in spite of its kink at the originthe relu has an important advantage over the logistic function while the derivative of the logistic function decays exponentially fast to zero as we move away from the origina phenomenon referred to as saturationthe derivative of the relu function is always unity for positive thusfor large positive zthe derivative of the logistic function does not carry any useful informationbut the derivative of the relu can help guide gradient optimization algorithm the situation for the heaviside function in figure is even worsebecause its derivative is completely noninformative for any in this respectthe lack of saturation of the relu function for makes it desirable activation function for training network via back-propagation