before we beginthe mathematical building blocks of neural networks points for instancethe derivative of cos(xis -sin( )the derivative of (xa is '(xaand so on if you're trying to update by factor epsilon_x in order to minimize ( )and you know the derivative of fthen your job is donethe derivative completely describes how (xevolves as you change if you want to reduce the value of ( )you just need to move little in the opposite direction from the derivative derivative of tensor operationthe gradient gradient is the derivative of tensor operation it' the generalization of the concept of derivatives to functions of multidimensional inputsthat isto functions that take tensors as inputs consider an input vector xa matrix wa target yand loss function loss you can use to compute target candidate y_predand compute the lossor mismatchbetween the target candidate y_pred and the target yy_pred dot(wxloss_value loss(y_predyif the data inputs and are frozenthen this can be interpreted as function mapping values of to loss valuesloss_value (wlet' say the current value of is then the derivative of in the point is tensor gradient( )( with the same shape as wwhere each coefficient gradient( ( )[ijindicates the direction and magnitude of the change in loss_value you observe when modifying [ijthat tensor gradient( )( is the gradient of the function (wloss_value in you saw earlier that the derivative of function (xof single coefficient can be interpreted as the slope of the curve of likewisegradient( )( can be interpreted as the tensor describing the curvature of (waround for this reasonin much the same way thatfor function ( )you can reduce the value of (xby moving little in the opposite direction from the derivativewith function (wof tensoryou can reduce (wby moving in the opposite direction from the gradientfor examplew step gradient( )( (where step is small scaling factorthat means going against the curvaturewhich intuitively should put you lower on the curve note that the scaling factor step is needed because gradient( )( only approximates the curvature when you're close to so you don' want to get too far from stochastic gradient descent given differentiable functionit' theoretically possible to find its minimum analyticallyit' known that function' minimum is point where the derivative is so all you have to do is find all the points where the derivative goes to and check for which of these points the function has the lowest value licensed to