CyclicLR
class torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr, max_lr, step_size_up=2000, step_size_down=None, mode='triangular', gamma=1.0, scale_fn=None, scale_mode='cycle', cycle_momentum=True, base_momentum=0.8, max_momentum=0.9, last_epoch=- 1, verbose=False)
Sets the learning rate of each parameter group according to
   cyclical learning rate policy (CLR). The policy cycles the learning
   rate between two boundaries with a constant frequency, as detailed
   in the paper Cyclical Learning Rates for Training Neural Networks.
   The distance between the two boundaries can be scaled on a per-
   iteration or per-cycle basis.
Cyclical learning rate policy changes the learning rate after every
   batch. step should be called after a batch has been used for
   training.
This class has three built-in policies, as put forth in the paper:


"triangular": A basic triangular cycle without amplitude scaling.


"triangular2": A basic triangular cycle that scales initial

