applied to neural networkthat means finding analytically the combination of weight values that yields the smallest possible loss function this can be done by solving the equation gradient( )( for this is polynomial equation of variableswhere is the number of coefficients in the network although it would be possible to solve such an equation for or doing so is intractable for real neural networkswhere the number of parameters is never less than few thousand and can often be several tens of millions insteadyou can use the four-step algorithm outlined at the beginning of this sectionmodify the parameters little by little based on the current loss value on random batch of data because you're dealing with differentiable functionyou can compute its gradientwhich gives you an efficient way to implement step if you update the weights in the opposite direction from the gradientthe loss will be little less every time draw batch of training samples and corresponding targets run the network on to obtain predictions y_pred compute the loss of the network on the batcha measure of the mismatch between y_pred and compute the gradient of the loss with regard to the network' parameters ( backward passmove the parameters little in the opposite direction from the gradient--for example -step gradient--thus reducing the loss on the batch bit easy enoughwhat just described is called mini-batch stochastic gradient descent (minibatch sgdthe term stochastic refers to the fact that each batch of data is drawn at random (stochastic is scientific synonym of randomfigure illustrates what happens in dwhen the network has only one parameter and you have only one training sample loss value stepalso called learning rate starting point ( = = = = parameter value figure sgd down loss curve (one learnable parameterlicensed to