RAdam
class torch.optim.RAdam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, *, foreach=None, differentiable=False)
Implements RAdam algorithm.
  \begin{aligned}       &\rule{110mm}{0.4pt}
  \\       &\textbf{input}      : \gamma \text{ (lr)}, \: \beta_1,
  \beta_2           \text{ (betas)}, \: \theta_0 \text{ (params)},
  \:f(\theta) \text{ (objective)}, \:           \lambda \text{
  (weightdecay)},
  \\       &\hspace{13mm} \epsilon \text{ (epsilon)}
  \\       &\textbf{initialize} :  m_0 \leftarrow 0 \text{ ( first
  moment)},           v_0 \leftarrow 0 \text{ ( second moment)},
  \\       &\hspace{18mm} \rho_{\infty} \leftarrow 2/(1-\beta_2)
  -1                      \\[-1.ex]       &\rule{110mm}{0.4pt}  \\
  &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do}
  \\       &\hspace{6mm}g_t           \leftarrow   \nabla_{\theta}
  f_t (\theta_{t-1})           \\       &\hspace{5mm} \textbf{if}
