\       &\hspace{12mm}\theta_t \leftarrow \theta_{t-1} - \gamma
      \widehat{m_t}                \       &\rule{110mm}{0.4pt}
      \[-1.ex]       &\bf{return} \:  \theta_t
      \[-1.ex]       &\rule{110mm}{0.4pt}
      \[-1.ex]  \end{aligned}
For further details regarding the algorithm we refer to On the
   variance of the adaptive learning rate and beyond.
This implementation uses the same weight_decay implementation as
   Adam (were the weight_decay is applied to the gradient) and not the
   one from AdamW (were weight_decay is applied to the update). This
   is different from the author's implementation.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **lr** (*float**, **optional*) -- learning rate (default:
    1e-3)

  * **betas** (*Tuple**[**float**, **float**]**, **optional*) --
    coefficients used for computing running averages of gradient
