feed-forward neural networks where is hyperparameter that defines the initial learning rateand gt is the gradient at time for parameter/weight in th we add to the denominator to avoid divide by zero situations adam simplified name for adaptive moment estimationadam is the most popular choice recently for optimizers in deep learning in simple wayadam combines the best of rmsprop and stochastic gradient descent with momentum from rmspropit borrows the idea of using squared gradients to scale the learning rateand it takes the idea of moving averages of the gradient instead of directly using the gradient when compared to sgd with momentum herefor each weight in thwe have   gt and st st gt which then is used to compute  gt st andfinallythe weight is updated as wt wt the preceding three types of optimization algorithms represent just few from the breadth of available options for different types of use cases within deep learning we have definitely not covered the detailed depths and math in each of these topicsso readers are highly recommended to