feed-forward neural networks disagreement would be the least (ideallyzerothis is where we interface with backpropagationan intuitive algorithm that enables the computation of gradients of the loss with respect to the weights using chain rule in the forward passthe network computes the prediction for given input sampleand the loss function measures the disagreement between the actual target value and network' prediction backpropagation computes the gradient of the loss with respect to the weights and biases and thus provides us with fair overall picture of how small change in the weight impacts the overall loss we would then need to update the weights iteratively and with small increments (in the opposite direction of the gradientto reach the local minima this process is called the gradient descent-- reducing the loss function to reach the minimum the network therefore learns (iterative and incremental updates on weightsthe patterns that can correctly predict for given input sample with the least disagreement there are several variants to update the weights in gradient descent for neural networks the next section explores few of them in the next we will take brief look at automatic differentiation that enables the idea of backpropagation programmatically gradient descent variants there are primarily three variants of gradient descent techniques each of them differs in its approach by the amount of data used to compute the gradient of the loss depending on the amount of data usedwe make trade-off between the accuracy of the parameter update and the time it takes to perform an update belowwe discuss three different variants used in training deep learning networks and later (in the following sectionwe study few popular gradient descent optimization algorithms