\hspace{3mm}              \      &\rule{110mm}{0.4pt}
      \[-1.ex]      &\bf{return} \:  \theta_t
      \[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex] \end{aligned}
For further details regarding the algorithm we refer to lecture
   notes by G. Hinton. and centered version Generating Sequences With
   Recurrent Neural Networks. The implementation here takes the square
   root of the gradient average before adding epsilon (note that
   TensorFlow interchanges these two operations). The effective
   learning rate is thus \gamma/(\sqrt{v} + \epsilon) where \gamma is
   the scheduled learning rate and v is the weighted moving average of
   the squared gradient.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **lr** (*float**, **optional*) -- learning rate (default:
    1e-2)

  * **momentum** (*float**, **optional*) -- momentum factor
    (default: 0)
