AdamW
class torch.optim.AdamW(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01, amsgrad=False, *, maximize=False, foreach=None, capturable=False, differentiable=False)
Implements AdamW algorithm.
  \begin{aligned}      &\rule{110mm}{0.4pt}
  \\      &\textbf{input}      : \gamma \text{(lr)}, \: \beta_1,
  \beta_2          \text{(betas)}, \: \theta_0 \text{(params)}, \:
  f(\theta) \text{(objective)},          \: \epsilon \text{
  (epsilon)}                                                    \\
  &\hspace{13mm}      \lambda \text{(weight decay)},  \:
  \textit{amsgrad},          \: \textit{maximize}
  \\      &\textbf{initialize} : m_0 \leftarrow 0 \text{ (first
  moment)}, v_0 \leftarrow 0          \text{ ( second moment)}, \:
  \widehat{v_0}^{max}\leftarrow 0              \\[-1.ex]
  &\rule{110mm}{0.4pt}
  \\      &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \:
  \textbf{do}                         \\
