class_docstr: this class is adapted from https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py to
   include ignoring LARS application specific parameters (e.g. 1D params)
  
  Args:
      optimizer: Pytorch optimizer to wrap and modify learning rate for.
      trust_coefficient: Trust coefficient for calculating the lr. See https://arxiv.org/abs/1708.03888
      clip: Decides between clipping or scaling mode of LARS. If `clip=True` the learning rate is set to `min(optimizer_lr, local_lr)` for each parameter. If `clip=False` the learning rate is set to `local_lr*optimizer_lr`.
      eps: epsilon kludge to help with numerical stability while calculating adaptive_lr
class_docstr_tok: ["this","class","is","adapted","from","https",":","\/\/github.com\/NVIDIA\/apex\/blob\/master\/apex\/parallel\/LARC.py","to","include","ignoring","LARS","application","specific","parameters","(","e.g",".","1D","params",")","Args",":","optimizer",":","Pytorch","optimizer","to","wrap","and","modify","learning","rate","for",".","trust_coefficient",":","Trust","coefficient","for","calculating","the","lr",".","See","https",":","\/\/arxiv.org\/abs\/1708.03888","clip",":","Decides","between","clipping","or","scaling","mode","of","LARS",".","If","`","clip=True","`","the","learning","rate","is","set","to","`","min","(","optimizer_lr",",","local_lr",")","`","for","each","parameter",".","If","`","clip=False","`","the","learning","rate","is","set","to","`","local_lr","*","optimizer_lr","`",".","eps",":","epsilon","kludge","to","help","with","numerical","stability","while","calculating","adaptive_lr"]
code: def step(self):
      with torch.no_grad():
          weight_decays = []
          for group in self.optim.param_groups:
              # absorb weight decay control from optimizer
              weight_decay = (
                  group["weight_decay"]
                  if "weight_decay" in group
                  else 0
              )
              weight_decays.append(weight_decay)
              apply_LARS = (
                  group["apply_LARS"] if "apply_LARS" in group else True
              )
              if not apply_LARS:
                  continue
              group["weight_decay"] = 0
              for p in group["params"]:
                  if p.grad is None:
                      continue
                  if (
                      self.ignore_1d_param and p.ndim == 1
                  ):  # ignore bias
                      continue
                  param_norm = torch.norm(p.data)
                  grad_norm = torch.norm(p.grad.data)
  
                  if param_norm != 0 and grad_norm != 0:
                      # calculate adaptive lr + weight decay
                      adaptive_lr = (
                          self.trust_coefficient
                          * (param_norm)
                          / (
                              grad_norm
                              + param_norm * weight_decay
                              + self.eps
                          )
                      )
  
                      # clip learning rate for LARS
                      if self.clip:
                          # calculation of adaptive_lr so that when multiplied by lr it equals `min(adaptive_lr, lr)`
                          adaptive_lr = min(
                              adaptive_lr / group["lr"], 1
                          )
  
                      p.grad.data += weight_decay * p.data
                      p.grad.data *= adaptive_lr
  
      self.optim.step()
      # return weight decay control to optimizer
      for i, group in enumerate(self.optim.param_groups):
          group["weight_decay"] = weight_decays[i]
  
file_path: SlowFast/slowfast/models/optimizer.py
imports: import torch
  import slowfast.utils.lr_policy
label_desc: Use image node_id 12 for calling the LARS obj's underlying member method code with example usage: obj.step() without return types