class_docstr: Sets the learning rate of each parameter group according to
  1Cycle learning rate policy (1CLR). 1CLR is a variation of the
  Cyclical Learning Rate (CLR) policy that involves one cycle followed by
  decay. The policy simultaneously cycles the learning rate (and momentum)
  between two boundaries with a constant frequency, as detailed in
  the paper `A disciplined approach to neural network hyper-parameters`_.
  
  1CLR policy changes the learning rate after every batch.
  `step` should be called after a batch has been used for training.
  
  This implementation was adapted from the github repo: `pytorch/pytorch`_
  
  Args:
      optimizer (Optimizer): Wrapped optimizer.
      cycle_min_lr (float or list): Initial learning rate which is the
          lower boundary in the cycle for each parameter group.
      cycle_max_lr (float or list): Upper learning rate boundaries in the cycle
          for each parameter group. Functionally,
          it defines the cycle amplitude (cycle_max_lr - cycle_min_lr).
          The lr at any cycle is the sum of cycle_min_lr
          and some scaling of the amplitude; therefore
          cycle_max_lr may not actually be reached depending on
          scaling function.
      decay_lr_rate(float): Decay rate for learning rate. Default: 0.
      cycle_first_step_size (int): Number of training iterations in the
          increasing half of a cycle. Default: 2000
      cycle_second_step_size (int): Number of training iterations in the
          decreasing half of a cycle. If cycle_second_step_size is None,
          it is set to cycle_first_step_size. Default: None
      cycle_first_stair_count(int): Number of stairs in first half of cycle phase. This means
      lr/mom are changed in staircase fashion. Default 0, means staircase disabled.
      cycle_second_stair_count(int): Number of stairs in second half of cycle phase. This means
      lr/mom are changed in staircase fashion. Default 0, means staircase disabled.
      decay_step_size (int): Intervals for applying decay in decay phase. Default: 0, means no decay.
      cycle_momentum (bool): If ``True``, momentum is cycled inversely
          to learning rate between 'cycle_min_mom' and 'cycle_max_mom'.
          Default: True
      cycle_min_mom (float or list): Initial momentum which is the
          lower boundary in the cycle for each parameter group.
          Default: 0.8
      cycle_max_mom (float or list): Upper momentum boundaries in the cycle
          for each parameter group. Functionally,
          it defines the cycle amplitude (cycle_max_mom - cycle_min_mom).
          The momentum at any cycle is the difference of cycle_max_mom
          and some scaling of the amplitude; therefore
          cycle_min_mom may not actually be reached depending on
          scaling function. Default: 0.9
      decay_mom_rate (float): Decay rate for momentum. Default: 0.
      last_batch_iteration (int): The index of the last batch. This parameter is used when
          resuming a training job. Since `step()` should be invoked after each
          batch instead of after each epoch, this number represents the total
          number of *batches* computed, not the total number of epochs computed.
          When last_batch_iteration=-1, the schedule is started from the beginning.
          Default: -1
  
  Example:
      >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
      >>> scheduler = OneCycle(optimizer, 0.0001, 0.0010)
      >>> data_loader = torch.utils.data.DataLoader(...)
      >>> for epoch in range(10):
      >>>     for batch in data_loader:
      >>>         train_batch(...)
      >>>         scheduler.step()
  
  
  .. _A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay: https://arxiv.org/abs/1803.09820
class_docstr_tok: ["Sets","the","learning","rate","of","each","parameter","group","according","to","1Cycle","learning","rate","policy","(","1CLR",")",".","1CLR","is","a","variation","of","the","Cyclical","Learning","Rate","(","CLR",")","policy","that","involves","one","cycle","followed","by","decay",".","The","policy","simultaneously","cycles","the","learning","rate","(","and","momentum",")","between","two","boundaries","with","a","constant","frequency",",","as","detailed","in","the","paper","`","A","disciplined","approach","to","neural","network","hyper-parameters","`","_",".","1CLR","policy","changes","the","learning","rate","after","every","batch",".","`","step","`","should","be","called","after","a","batch","has","been","used","for","training",".","This","implementation","was","adapted","from","the","github","repo",":","`","pytorch\/pytorch","`","_","Args",":","optimizer","(","Optimizer",")",":","Wrapped","optimizer",".","cycle_min_lr","(","float","or","list",")",":","Initial","learning","rate","which","is","the","lower","boundary","in","the","cycle","for","each","parameter","group",".","cycle_max_lr","(","float","or","list",")",":","Upper","learning","rate","boundaries","in","the","cycle","for","each","parameter","group",".","Functionally",",","it","defines","the","cycle","amplitude","(","cycle_max_lr","-","cycle_min_lr",")",".","The","lr","at","any","cycle","is","the","sum","of","cycle_min_lr","and","some","scaling","of","the","amplitude",";","therefore","cycle_max_lr","may","not","actually","be","reached","depending","on","scaling","function",".","decay_lr_rate","(","float",")",":","Decay","rate","for","learning","rate",".","Default",":","0.","cycle_first_step_size","(","int",")",":","Number","of","training","iterations","in","the","increasing","half","of","a","cycle",".","Default",":","2000","cycle_second_step_size","(","int",")",":","Number","of","training","iterations","in","the","decreasing","half","of","a","cycle",".","If","cycle_second_step_size","is","None",",","it","is","set","to","cycle_first_step_size",".","Default",":","None","cycle_first_stair_count","(","int",")",":","Number","of","stairs","in","first","half","of","cycle","phase",".","This","means","lr\/mom","are","changed","in","staircase","fashion",".","Default","0",",","means","staircase","disabled",".","cycle_second_stair_count","(","int",")",":","Number","of","stairs","in","second","half","of","cycle","phase",".","This","means","lr\/mom","are","changed","in","staircase","fashion",".","Default","0",",","means","staircase","disabled",".","decay_step_size","(","int",")",":","Intervals","for","applying","decay","in","decay","phase",".","Default",":","0",",","means","no","decay",".","cycle_momentum","(","bool",")",":","If","``","True","``",",","momentum","is","cycled","inversely","to","learning","rate","between","'cycle_min_mom","'","and","'cycle_max_mom","'",".","Default",":","True","cycle_min_mom","(","float","or","list",")",":","Initial","momentum","which","is","the","lower","boundary","in","the","cycle","for","each","parameter","group",".","Default",":","0.8","cycle_max_mom","(","float","or","list",")",":","Upper","momentum","boundaries","in","the","cycle","for","each","parameter","group",".","Functionally",",","it","defines","the","cycle","amplitude","(","cycle_max_mom","-","cycle_min_mom",")",".","The","momentum","at","any","cycle","is","the","difference","of","cycle_max_mom","and","some","scaling","of","the","amplitude",";","therefore","cycle_min_mom","may","not","actually","be","reached","depending","on","scaling","function",".","Default",":","0.9","decay_mom_rate","(","float",")",":","Decay","rate","for","momentum",".","Default",":","0.","last_batch_iteration","(","int",")",":","The","index","of","the","last","batch",".","This","parameter","is","used","when","resuming","a","training","job",".","Since","`","step","(",")","`","should","be","invoked","after","each","batch","instead","of","after","each","epoch",",","this","number","represents","the","total","number","of","*","batches","*","computed",",","not","the","total","number","of","epochs","computed",".","When","last_batch_iteration=-1",",","the","schedule","is","started","from","the","beginning",".","Default",":","-1","Example",":",">",">",">","optimizer","=","torch.optim.SGD","(","model.parameters","(",")",",","lr=0.1",",","momentum=0.9",")",">",">",">","scheduler","=","OneCycle","(","optimizer",",","0.0001",",","0.0010",")",">",">",">","data_loader","=","torch.utils.data.DataLoader","(","...",")",">",">",">","for","epoch","in","range","(","10",")",":",">",">",">","for","batch","in","data_loader",":",">",">",">","train_batch","(","...",")",">",">",">","scheduler.step","(",")","..","_A","disciplined","approach","to","neural","network","hyper-parameters",":","Part","1","--","learning","rate",",","batch","size",",","momentum",",","and","weight","decay",":","https",":","\/\/arxiv.org\/abs\/1803.09820"]
code: def _initialize_lr(
      self,
      optimizer,
      cycle_min_lr,
      cycle_max_lr,
      decay_lr_rate,
      last_batch_iteration,
  ):
      self.min_lrs = [cycle_min_lr] * len(optimizer.param_groups)
      if last_batch_iteration == -1:
          for lr, group in zip(self.min_lrs, optimizer.param_groups):
              group["lr"] = lr
  
      self.max_lrs = [cycle_max_lr] * len(optimizer.param_groups)
      self.decay_lr_rate = decay_lr_rate
  
      if math.isclose(self.decay_lr_rate, 0):
          self.skip_lr_decay = True
  
imports: import argparse
  from torch.optim import Optimizer
  import math
  from deepspeed.utils import logger
label_desc: Use image node_id 3 for calling the OneCycle obj's underlying member method code with example usage: obj._initialize_lr(optimizer, cycle_min_lr, cycle_max_lr, decay_lr_rate, last_batch_iteration) without return types