feed-forward neural networks is greater than but smaller than the entire dataset common batch sizes are etc number in the range of powers of is recommended (but not necessary)as it suits best from computation perspective with mbgdthe updates are less frequent than sgd but more frequent than bgdand leverage small batch instead of individual samples or the entire dataset in this waythe variance reduces to greater extent and we achieve better trade-off on the speed gradient-based optimization techniques in the following sectionwe will discuss in brief few popular optimization techniques commonly used in deep learning the details of the math used in each technique are beyond the scope of this book gradient descent with momentum the problems we discussed earlier between sgd and bgd are fairly smoothed using mbgd howevereven with the use of mbgdthe direction of the update still fluctuates (though less than with sgd but more than with mgdgradient descent with momentum leverages the past gradients to calculate an exponentially weighted average of the gradients to further smoothen the parameter updates figure - illustrates the update process figure - gradient descent with momentum