before we beginthe mathematical building blocks of neural networks as you can seeintuitively it' important to pick reasonable value for the step factor if it' too smallthe descent down the curve will take many iterationsand it could get stuck in local minimum if step is too largeyour updates may end up taking you to completely random locations on the curve note that variant of the mini-batch sgd algorithm would be to draw single sample and target at each iterationrather than drawing batch of data this would be true sgd (as opposed to mini-batch sgdalternativelygoing to the opposite extremeyou could run every step on all data availablewhich is called batch sgd each update would then be more accuratebut far more expensive the efficient compromise between these two extremes is to use mini-batches of reasonable size although figure illustrates gradient descent in parameter spacein practice you'll use gradient descent in highly dimensional spacesevery weight coefficient in neural network is free dimension in the spaceand there may be tens of thousands or even millions of them to help you build intuition about loss surfacesyou can also visualize gradient descent along loss surfaceas shown in figure but you can' possibly visualize what the actual process of training neural network looks like--you can' represent , , -dimensional space in way that makes sense to humans as suchit' good to keep in mind that the intuitions you develop through these low-dimensional representations may not always be accurate in practice this has historically been source of issues in the world of deep-learning research starting point figure gradient descent down loss surface (two learnable paramet ersfinal point additionallythere exist multiple variants of sgd that differ by taking into account previous weight updates when computing the next weight updaterather than just looking at the current value of the gradients there isfor instancesgd with momentumas well as adagradrmspropand several others such variants are known as optimization methods or optimizers in particularthe concept of momentumwhich is used in many of these variantsdeserves your attention momentum addresses two issues with sgdconvergence speed and local minima consider figure which shows the curve of loss as function of network parameter licensed to