Do adaptive learning optimizers follow the steepest decent?
Adam and RMSprop are both optimization algorithms which make the vanilla gradient descent more robust. You still need to calculate the gradient however, these algorithms will facilitate the descent in certain directions and punish the descent in others based on some criteria. 

Before delving into the details consider this analogy; imagine you are on a mountain and you want to get back to the village. To do this you will follow the downhill slope until you get to the bottom of the mountain. However imagine a mountain that looks like the following



Evidently the best path down the mountain should just be straight down the snowy path. However, if we carefully calculate the gradient we will notice that it will always have a slight left and right component. This will cause us to oscilate from left to right while going down the mountain. This is a huge waste of energy, but it's not catastrophic. However, when we start reaching the base of the mountain the downward slope will start to subside and the oscillations will become more prominent. This may cause us to never truly reach the minimum due to the oscillations becoming dominant, or cause the learning process to be dragged on excessively thus wasting resources. 

The optimization techniques such as Adam and RMSprop simply make adjustments to the gradient descent step such that we can more directly find the minimum of the objective function without distraction from oscillating gradients. This is achieved in RMSprop and Adam by maintaining statistics relevant to the past computed gradients. RMSprop keeps track of the mean and Adam keeps track of both the mean and the moment of the past gradients.



An amazing reference for the inner-workings of the gradient descent variants can be found here. 

Gradient descent

Vanilla gradient descent is defined as

$\theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta)$ 

where $\theta$ is the set of parameters which define our objective function $J(\theta)$. The learning rate is $\eta$ and $\nabla$ defines the gradient operator. We will extend this function.

Adding Momentum

Firstly, if we throw a ball down our cliff it will get some momentum as it falls. This will cause it to tend towards the steepest part of the gradient and left to right oscillations would be minimized. We can do the same to the gradient descent algorithm by introducing momentum $\gamma$ as

$\nu_{t} = \gamma \nu_{t-1} + \eta \nabla_\theta J(\theta)$ 

$\theta_{t+1} = \theta_{t} - \nu_t$

This is a good result but we can still do better.

Adagrad

It is preferable to have a distinct learning rate for each parameter such that we can adjust their impact on the gradient based on their past values. This is achieved by weighing the gradients by a sum of squares term of their past gradients up until the current time $t$. The matrix $G_t$ is a diagonal matrix where its entries are precisely this sum of squares of past gradients for each parameter. The Adagrad update is thus,

$\theta_{t+1,i} = \theta_{t,i} - \frac{\eta}{\sqrt{G_{t,ii} + \epsilon}}\nabla_{\theta_t}J(\theta_{t,i})$.

However, this method suffers from diminishing gradients when the sum of squares matrix makes it such that the past values are all too small and cause no more updates to be possible. Or for parameters which may be impactfull in the future to lose all relevance due to their low gradients at the start.

RMSprop

This is a solution to the diminishing gradients problem. We will use a running average of the past gradients until a time $t$ as defined by $E[g^2]_t$. The update rule is thus

$E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g^2_t$

$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}}g_t$

where $g$ is the gradient for a single parameter.

Adam

In addition to keeping track of the running average of the past gradients we will also keep track of their second order moment. $m$ is the mean and $\nu$ is the moment. Thus we keep track of them as follows

$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$ 

$\nu_t = \beta_2 \nu_{t-1} + (1-\beta_2)g^2_t$ 

We will correct these terms to remove their bias that tends towards a zero vector as

$\hat{m}_t = \frac{m_t}{1-\beta^t_1}$

$\hat{\nu}_t = \frac{\nu_t}{1-\beta^t_2}$

Finally, the update rule is

$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{\nu}} + \epsilon}\hat{m}_t$