solution. Additional constraints could either be generated by setting specific conditions to the weights, or by injecting additional training data. One commonly used algorithm to find the set of weights that minimizes the error is gradient descent. By backpropagation, the steepest descent direction is calculated of the loss function versus the present synaptic weights. Then, the weights can be modified along the steepest descent direction, and the error is minimized in an efficient way. Derivation The gradient descent method involves calculating the derivative of the loss function with respect to the weights of the network. This is normally done using backpropagation. Assuming one output neuron, the squared error function is where is the loss for the output and target value , is the target output for a training sample, and is the actual output of the output neuron. For each neuron , its output is defined as where the activation function is non-linear and differentiable over the activation region (the ReLU is not differentiable at one point). A historically used activation function is the logistic function: which has a convenient derivative of: The input to a neuron is the weighted sum of outputs of previous neurons. If the neuron is in the first layer after the input layer, the of the input layer are simply the inputs to the network. The number of input units to the neuron is . The variable denotes the weight between neuron of the previous layer and neuron of the current layer. Finding the derivative of the error Calculating the partial derivative of the error with respect to a weight is done using the chain rule twice: In the last factor of the right-hand side of the above, only one term in the sum depends on , so that If the neuron is in the first layer after the input layer, is just . The derivative of the output of neuron with respect to its input is simply the partial derivative of the activation function: which for the logistic activation function This is the reason why backpropagation requires the activation function to be differentiable. (Nevertheless, the ReLU activation function, which is non-differentiable at 0, has become quite popular, e.g. in AlexNet) The first factor is straightforward to evaluate if the neuron is in the output layer, because then and If half of the square error is used as loss function we can rewrite it as