for a single training example, which needs to be generalized to the overall error function. The second assumption is that it can be written as a function of the outputs from the neural network. Example loss function Let be vectors in . Select an error function measuring the difference between two outputs. The standard choice is the square of the Euclidean distance between the vectors and :The error function over training examples can then be written as an average of losses over individual examples: Limitations Gradient descent with backpropagation is not guaranteed to find the global minimum of the error function, but only a local minimum; also, it has trouble crossing plateaus in the error function landscape. This issue, caused by the non-convexity of error functions in neural networks, was long thought to be a major drawback, but Yann LeCun et al. argue that in many practical problems, it is not. Backpropagation learning does not require normalization of input vectors; however, normalization could improve performance. Backpropagation requires the derivatives of activation functions to be known at network design time. History The term backpropagation and its general use in neural networks was announced in , then elaborated and popularized in , but the technique was independently rediscovered many times, and had many predecessors dating to the 1960s. The basics of continuous backpropagation were derived in the context of control theory by Henry J. Kelley in 1960, and by Arthur E. Bryson in 1961. They used principles of dynamic programming. In 1962, Stuart Dreyfus published a simpler derivation based only on the chain rule. Bryson and Ho described it as a multi-stage dynamic system optimization method in 1969. Backpropagation was derived by multiple researchers in the early 60's and implemented to run on computers as early as 1970 by Seppo Linnainmaa. Paul Werbos was first in the US to propose that it could be used for neural nets after analyzing it in depth in his 1974 dissertation. While not applied to neural networks, in 1970 Linnainmaa published the general method for automatic differentiation (AD). Although very controversial, some scientists believe this was actually the first step toward developing a back-propagation algorithm. In 1973 Dreyfus adapts parameters of controllers in proportion to error gradients. In 1974 Werbos mentioned the possibility of applying this principle to artificial neural networks, and in 1982 he applied Linnainmaa's AD method to non-linear functions. Later the Werbos method