activation functionwe will use small positive values to ensure that its derivative is not zero zero derivative of the activation function prevents the propagation of information useful for computing good search direction recall that computation of the gradient of the training loss via algorithm requires averaging over all training examples when the size of the training set tn is too largecomputation of the gradient `tn /th via algorithm may be too costly in such caseswe may employ the stochastic gradient descent algorithm in this algorithmwe view the training loss as an expectation that can be approximated via monte carlo sampling in particularif is random variable with distribution [ / for nthen we can write loss(yk (xk th) loss(yk (xk th)` (th) = stochastic gradient descent we can thus approximate ` (th)via monte carlo estimator using iid copies of kn ` ( (th):loss(yki (xki th) = the iid monte carlo sample kn is called minibatch (see also exercise typicallyn so that the probability of observing ties in minibatch of size is negligible finallynote that if the learning rate of the stochastic gradient descent algorithm satisfies the conditions in ( )then the stochastic gradient descent algorithm is simply version of the stochastic approximation algorithm levenberg-marquardt method since neural network with squared-error loss is special type of nonlinear regression modelit is possible to train it using classical nonlinear least-squares minimization methodssuch as the levenberg-marquardt algorithm for simplicity of notationsuppose that the output of the net for an input is scalar (xfor given input parameter th of dimension dim(th)the levenberg-marquardt algorithm requires computation of the following vector of outputsg( th:[ ( th) (xn th)]as well as the matrix of jacobigof at th to compute these quantitieswe can again use the back-propagation algorithm as follows algorithm output for training via levenberg-marquardt inputtraining set {(xi yi )}ni= parameter th outputvector ( thand matrix of jacobi for use in algorithm for do /loop over all training examples run algorithm with input (xi yi (using in line to compute (xi thand (xthi thg( th[ ( th) (xn th)] ( thi (xn th th th return ( thand minibatch