Difference between RMSProp and Momentum?
Optimizers evolved with small Fix/Improvement on the previous one. So, if you will read in sequence, you will have a better understanding. In this context, RMSProp was a fix on Adagrad and it was an improvement on Momentum.
Let's see this Loss surface which is like a Valley (Imagine a River)
$\hspace{2cm}$
$\hspace{5cm}$Image source - http://d2l.ai/
Momentum -
Let's start from the red-circled point. We have a very large Gradient in X2 direction and very little in X1 and global minima is towards X1.
In Momentum, we accumulated the resultant Gradient which will obviously point more towards X2.
As a result, we will move very fast towards the other side of the river and very little towards X1. As we cross the river and start moving up, counter Gradient of X2 will start minimizing the Aggregate. Remember, it's leaky aggregation i.e. recent ones have more say. At a point, it will stop and reverse.
In the whole process, we had a little movement in X1 and a lot of oscillations in X2
 This was one of the points of the Author
What AdaGrad did -
 - Manage Gradient for each coordinate separately
 - Added a scaling factor in the denominator which will act as a brake. This scaling is based on the square of past Gradients.
Now X2 will have a large brake, so it will not move so fast with momentum to cross the river. Since X1 is having a very small Gradient, its scaling will be positive(if &lt; 1) Or almost constant(if ~ 1). So, movement in X1 will be same Or even faster.
That's why the author said: &quot;RMSProp impedes our search in direction of oscillations&quot;
Problem with Adagrad was that it aggregates all the past Gradients for the scaling factor which causes the brake to become larger for any case after a good number of iterations even if it has not reached Global optimum.
 Let's say, if Gradient is small 0.5 then also it will start dividing by 2.5 after 10 iterations. if it is large e.g. 10, then it will start dividing by 1000 after 10 iterations. Even this large gradient will become small in the subsequent iteration.
 What RMSProp changed - It made the aggregation leaky i.e. recent one will be considered more (Just like Momentum does for Gradient). With this change, the aggregation will be almost constant or at least not die as fast as in Adagrad.
$\hspace{2cm}$
$\hspace{5cm}$Image source - http://d2l.ai/