methods for training neural networks have been studied for long timeyet it is only recently that there have been sufficient computational resources to train them effectively the training of neural networks requires minimization of training loss` (th) ni= ci (th)which is typically difficult high-dimensional optimization problem with multiple local minima we next consider number of simple training methods in this sectionthe vectors dt and use the notation of section and should not be confused with the derivative and the prediction function grespectively steepest descent if we can compute the gradient of ` ( (th)via back-propagationthen we can apply the steepest descent algorithmwhich reads as follows starting from guess th we iterate the following step until convergencetht+ tht at ut learning rate ( where ut :(tht and at is the learning rate th observe thatrather than operating directly on the weights and biaseswe operate inpl stead on th :{wl bl } = - column vector of length = (pl- pl pl that stores all the weight and bias parameters the advantage of organizing the computations in this way is that we can easily compute the learning rate at for examplevia the barzilai-borwein formula in ( algorithm training via steepest descent inputtraining set {(xi yi )}ni= initial weight matrices and bias vectors {wl bl } = =th activation functions {sl } = outputthe parameters of the trained learner ut- /initialization while stopping condition is not met do compute the gradient ut (tht using algorithm th ut ut- if then /check if hessian is positive-definite /barzilai-borwein else - xa /failing positivitydo something heuristic - ut tht+ tht - + return tht as the minimizer of the training loss typicallywe initialize the algorithm with small random values for th while being careful to avoid saturating the activation function for examplein the case of the relu