feed-forward neural networks the update process can be simplified using the following equations firstwe compute an exponentially weighted average of the past gradients as ntwhere nt gnt ethj(thand th th nt the here is hyperparameter that takes values between and nextwe use this exponentially weighted average in the updates of weights instead of the gradients directly by leveraging the exponentially weighted averages of the gradientsinstead of directly using the gradientsthe incremental steps are smoother and faster and thus overcome the problems with oscillating around the minima rmsprop rmsprop is an unpublished optimization algorithm proposed by geoffry hinton in lecture of the online course "neural networks for machine learningon coursera at the corermsprop computes the moving average of the squared gradients for each weight and divides the gradient by the square root of the mean square this complex process should help in decoding the name root mean square prop leveraging exponential average here helps in giving recent updates more preferences than less recent ones the rmsprop can be represented as followsfor each weight in thwe have   gt and gt  to update the weight wt wt