loss value local minimum global minimum parameter value figure local minimum and global minimum as you can seearound certain parameter valuethere is local minimumaround that pointmoving left would result in the loss increasingbut so would moving right if the parameter under consideration were being optimized via sgd with small learning ratethen the optimization process would get stuck at the local minimum instead of making its way to the global minimum you can avoid such issues by using momentumwhich draws inspiration from physics useful mental image here is to think of the optimization process as small ball rolling down the loss curve if it has enough momentumthe ball won' get stuck in ravine and will end up at the global minimum momentum is implemented by moving the ball at each step based not only on the current slope value (current accelerationbut also on the current velocity (resulting from past accelerationin practicethis means updating the parameter based not only on the current gradient value but also on the previous parameter updatesuch as in this naive implementationpast_velocity constant momentum factor momentum optimization loop while loss wlossgradient get_current_parameters(velocity past_velocity momentum learning_rate gradient momentum velocity learning_rate gradient past_velocity velocity update_parameter(wchaining derivativesthe backpropagation algorithm in the previous algorithmwe casually assumed that because function is differentiablewe can explicitly compute its derivative in practicea neural network function consists of many tensor operations chained togethereach of which has simpleknown derivative for instancethis is network composed of three tensor operationsaband cwith weight matrices and ( ( ( ( ))calculus tells us that such chain of functions can be derived using the following identitycalled the chain rulef( ( ) '( ( ) '(xapplying the chain rule to the computation of the gradient values of neural network gives rise to an algorithm licensed to