CosineAnnealingWarmRestarts
class torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0, T_mult=1, eta_min=0, last_epoch=- 1, verbose=False)
Set the learning rate of each parameter group using a cosine
   annealing schedule, where \eta_{max} is set to the initial lr,
   T_{cur} is the number of epochs since the last restart and T_{i} is
   the number of epochs between two warm restarts in SGDR:
  \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} -
  \eta_{min})\left(1 +
  \cos\left(\frac{T_{cur}}{T_{i}}\pi\right)\right)

When T_{cur}=T_{i}, set \eta_t = \eta_{min}. When T_{cur}=0 after
   restart, set \eta_t=\eta_{max}.
It has been proposed in SGDR: Stochastic Gradient Descent with Warm
   Restarts.
Parameters:
      * optimizer (Optimizer) -- Wrapped optimizer.
  * **T_0** (*int*) -- Number of iterations for the first restart.

  * **T_mult** (*int**, **optional*) -- A factor increases T_{i}
    after a restart. Default: 1.
