However, if is in an arbitrary inner layer of the network, finding the derivative with respect to is less obvious. Considering as a function with the inputs being all neurons receiving input from neuron , and taking the total derivative with respect to , a recursive expression for the derivative is obtained: Therefore, the derivative with respect to can be calculated if all the derivatives with respect to the outputs of the next layer – the ones closer to the output neuron – are known. [Note, if any of the neurons in set were not connected to neuron , they would be independent of and the corresponding partial derivative under the summation would vanish to 0.] Substituting , and in we obtain: with if is the logistic function, and the error is the square error: To update the weight using gradient descent, one must choose a learning rate, . The change in weight needs to reflect the impact on of an increase or decrease in . If , an increase in increases ; conversely, if , an increase in decreases . The new is added to the old weight, and the product of the learning rate and the gradient, multiplied by guarantees that changes in a way that always decreases . In other words, in the equation immediately below, always changes in such a way that is decreased: Second-order gradient descent Using a Hessian matrix of second-order derivatives of the error function, the Levenberg-Marquardt algorithm often converges faster than first-order gradient descent, especially when the topology of the error function is complicated. It may also find solutions in smaller node counts for which other methods might not converge. The Hessian can be approximated by the Fisher information matrix. Loss function The loss function is a function that maps values of one or more variables onto a real number intuitively representing some "cost" associated with those values. For backpropagation, the loss function calculates the difference between the network output and its expected output, after a training example has propagated through the network. Assumptions The mathematical expression of the loss function must fulfill two conditions in order for it to be possibly used in backpropagation. The first is that it can be written as an average over error functions , for individual training examples, . The reason for this assumption is that the backpropagation algorithm calculates the gradient of the error function