v_t      \leftarrow v_{t-1} \rho + g^2_t (1 - \rho)
      \      &\hspace{5mm}\Delta x_t    \leftarrow
      \frac{\sqrt{u_{t-1} +          \epsilon }}{ \sqrt{v_t +
      \epsilon}  }g_t \hspace{21mm}                           \
      &\hspace{5mm} u_t  \leftarrow   u_{t-1}  \rho +           \Delta
      x^2_t  (1 - \rho)
      \      &\hspace{5mm}\theta_t      \leftarrow   \theta_{t-1} -
      \gamma  \Delta x_t            \      &\rule{110mm}{0.4pt}
      \[-1.ex]      &\bf{return} \:  \theta_t
      \[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex] \end{aligned}
For further details regarding the algorithm we refer to ADADELTA:
   An Adaptive Learning Rate Method.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **rho** (*float**, **optional*) -- coefficient used for
    computing a running average of squared gradients (default:
    0.9)
