feed-forward neural networks batch gradient descent the original gradient descent is referred to as the batch gradient descent (bgdtechnique the name is derived from the amount of data used to compute the gradient--in this casethe entire batch the bgd technique essentially leverages the entire dataset available to compute the gradient of the cost function with respect to the parameters (weightsthis results in inherently slow andin most casesa non-viable optionas we might run out of memory to load the entire batch in most common scenarioswe would mostly tend to avoid the bgd approachsparring small datasets (which is rare phenomenon in deep learningstochastic gradient descent to overcome the issues from bgdwe have stochastic gradient descent (sgdwith sgdwe compute the gradient and update the weights for each sample in the dataset this process results in far less use of memory in the deep learning hardware and achieves results faster howeverthe updates are far more frequent than desired with more frequent updates to the weightsthe cost function fluctuates heavily sgdhoweverresults in bigger problems when the goal is to converge the updates towards the exact minima given the far more frequent updatesthe possibility of overshooting an update is very high to overcome these tradeoffswe might need to slowly reduce the learning rate over period of time in order to help the network converge to local or global minima mini-batch gradient descent mini-batch gradient descent (mbgdcombines the best of sgd and bgd instead of using the entire dataset (batchor just single sample from the dataset to compute the gradient of the cost function with respect to the parametersmbgd leverages smaller batchwhich