\eta)                  \      &\hspace{5mm} \textbf{if} \:
      \lambda \neq 0                                          \
      &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1}
      \      &\hspace{5mm}state_sum_t  \leftarrow  state_sum_{t-1}
      + g^2_t                      \      &\hspace{5mm}\theta_t
      \leftarrow          \theta_{t-1}- \tilde{\gamma}
      \frac{g_t}{\sqrt{state_sum_t}+\epsilon}            \
      &\rule{110mm}{0.4pt}
      \[-1.ex]      &\bf{return} \:  \theta_t
      \[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex] \end{aligned}
For further details regarding the algorithm we refer to Adaptive
   Subgradient Methods for Online Learning and Stochastic
   Optimization.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **lr** (*float**, **optional*) -- learning rate (default:
    1e-2)

  * **lr_decay** (*float**, **optional*) -- learning rate decay
