class_docstr: Sets the learning rate of each parameter group according to
  learning rate range test (LRRT) policy. The policy increases learning
  rate starting from a base value with a constant frequency, as detailed in
  the paper `A disciplined approach to neural network hyper-parameters: Part1`_.
  
  LRRT policy is used for finding maximum LR that trains a model without divergence, and can be used to
  configure the LR boundaries for Cyclic LR schedules.
  
  LRRT changes the learning rate after every batch.
  `step` should be called after a batch has been used for training.
  
  Args:
      optimizer (Optimizer): Wrapped optimizer.
      lr_range_test_min_lr (float or list): Initial learning rate which is the
          lower boundary in the range test for each parameter group.
      lr_range_test_step_size (int): Interval of training steps to increase learning rate. Default: 2000
      lr_range_test_step_rate (float): Scaling rate for range test. Default: 1.0
      lr_range_test_staircase (bool): Scale in staircase fashion, rather than continuous. Default: False.
      last_batch_iteration (int): The index of the last batch. This parameter is used when
          resuming a training job. Since `step()` should be invoked after each
          batch instead of after each epoch, this number represents the total
          number of *batches* computed, not the total number of epochs computed.
          When last_batch_iteration=-1, the schedule is started from the beginning.
          Default: -1
  
  Example:
      >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
      >>> scheduler = LRRangeTest(optimizer)
      >>> data_loader = torch.utils.data.DataLoader(...)
      >>> for epoch in range(10):
      >>>     for batch in data_loader:
      >>>         train_batch(...)
      >>>         scheduler.step()
  
      _A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay:
      https://arxiv.org/abs/1803.09820
class_docstr_tok: ["Sets","the","learning","rate","of","each","parameter","group","according","to","learning","rate","range","test","(","LRRT",")","policy",".","The","policy","increases","learning","rate","starting","from","a","base","value","with","a","constant","frequency",",","as","detailed","in","the","paper","`","A","disciplined","approach","to","neural","network","hyper-parameters",":","Part1","`","_",".","LRRT","policy","is","used","for","finding","maximum","LR","that","trains","a","model","without","divergence",",","and","can","be","used","to","configure","the","LR","boundaries","for","Cyclic","LR","schedules",".","LRRT","changes","the","learning","rate","after","every","batch",".","`","step","`","should","be","called","after","a","batch","has","been","used","for","training",".","Args",":","optimizer","(","Optimizer",")",":","Wrapped","optimizer",".","lr_range_test_min_lr","(","float","or","list",")",":","Initial","learning","rate","which","is","the","lower","boundary","in","the","range","test","for","each","parameter","group",".","lr_range_test_step_size","(","int",")",":","Interval","of","training","steps","to","increase","learning","rate",".","Default",":","2000","lr_range_test_step_rate","(","float",")",":","Scaling","rate","for","range","test",".","Default",":","1.0","lr_range_test_staircase","(","bool",")",":","Scale","in","staircase","fashion",",","rather","than","continuous",".","Default",":","False",".","last_batch_iteration","(","int",")",":","The","index","of","the","last","batch",".","This","parameter","is","used","when","resuming","a","training","job",".","Since","`","step","(",")","`","should","be","invoked","after","each","batch","instead","of","after","each","epoch",",","this","number","represents","the","total","number","of","*","batches","*","computed",",","not","the","total","number","of","epochs","computed",".","When","last_batch_iteration=-1",",","the","schedule","is","started","from","the","beginning",".","Default",":","-1","Example",":",">",">",">","optimizer","=","torch.optim.SGD","(","model.parameters","(",")",",","lr=0.1",",","momentum=0.9",")",">",">",">","scheduler","=","LRRangeTest","(","optimizer",")",">",">",">","data_loader","=","torch.utils.data.DataLoader","(","...",")",">",">",">","for","epoch","in","range","(","10",")",":",">",">",">","for","batch","in","data_loader",":",">",">",">","train_batch","(","...",")",">",">",">","scheduler.step","(",")","_A","disciplined","approach","to","neural","network","hyper-parameters",":","Part","1","--","learning","rate",",","batch","size",",","momentum",",","and","weight","decay",":","https",":","\/\/arxiv.org\/abs\/1803.09820"]
code: def load_state_dict(self, sd):
      self.last_batch_iteration = sd["last_batch_iteration"]
  
imports: import argparse
  from torch.optim import Optimizer
  import math
  from deepspeed.utils import logger
label_desc: Use image node_id 10 for calling the LRRangeTest obj's underlying member method code with example usage: obj.load_state_dict(sd) without return types