would contribute to minimizing the loss this would have to be repeated for all coefficients in the network but such an approach would be horribly inefficientbecause you' need to compute two forward passes (which are expensivefor every individual coefficient (of which there are manyusually thousands and sometimes up to millionsa much better approach is to take advantage of the fact that all operations used in the network are differentiableand compute the gradient of the loss with regard to the network' coefficients you can then move the coefficients in the opposite direction from the gradientthus decreasing the loss if you already know what differentiable means and what gradient isyou can skip to section otherwisethe following two sections will help you understand these concepts what' derivativeconsider continuoussmooth function (xymapping real number to new real number because the function is continuousa small change in can only result in small change in --that' the intuition behind continuity let' say you increase by small factor epsilon_xthis results in small epsilon_y change to yf( epsilon_xy epsilon_y in additionbecause the function is smooth (its curve doesn' have any abrupt angles)when epsilon_x is small enougharound certain point pit' possible to approximate as linear function of slope aso that epsilon_y becomes epsilon_xf( epsilon_xy epsilon_x obviouslythis linear approximation is valid only when is close enough to the slope is called the derivative of in if is negativeit means small change of around will result in decrease of ( (as shown in figure )and if is positivea small change in will result in an increase of (xfurtherthe absolute value of (the magnitude of the derivativetells you how quickly this increase or decrease will happen local linear approximation of fwith slope figure derivat ive of in for every differentiable function ( (differentiable means "can be derived"for examplesmoothcontinuous functions can be derived)there exists derivative function '(xthat maps values of to the slope of the local linear approximation of in those licensed to