class_docstr: Implements AdaBound algorithm with Decoupled Weight Decay (arxiv.org/abs/1711.05101)
  It has been proposed in `Adaptive Gradient Methods with Dynamic Bound of Learning Rate`_.
  Arguments:
      params (iterable): iterable of parameters to optimize or dicts defining
          parameter groups
      lr (float, optional): Adam learning rate (default: 1e-3)
      betas (Tuple[float, float], optional): coefficients used for computing
          running averages of gradient and its square (default: (0.9, 0.999))
      final_lr (float, optional): final (SGD) learning rate (default: 0.1)
      gamma (float, optional): convergence speed of the bound functions (default: 1e-3)
      eps (float, optional): term added to the denominator to improve
          numerical stability (default: 1e-8)
      weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
      amsbound (boolean, optional): whether to use the AMSBound variant of this algorithm
  .. Adaptive Gradient Methods with Dynamic Bound of Learning Rate:
      https://openreview.net/forum?id=Bkg3g2R9FX
class_docstr_tok: ["Implements","AdaBound","algorithm","with","Decoupled","Weight","Decay","(","arxiv.org\/abs\/1711.05101",")","It","has","been","proposed","in","`","Adaptive","Gradient","Methods","with","Dynamic","Bound","of","Learning","Rate","`","_",".","Arguments",":","params","(","iterable",")",":","iterable","of","parameters","to","optimize","or","dicts","defining","parameter","groups","lr","(","float",",","optional",")",":","Adam","learning","rate","(","default",":","1e-3",")","betas","(","Tuple","[","float",",","float","]",",","optional",")",":","coefficients","used","for","computing","running","averages","of","gradient","and","its","square","(","default",":","(","0.9",",","0.999",")",")","final_lr","(","float",",","optional",")",":","final","(","SGD",")","learning","rate","(","default",":","0.1",")","gamma","(","float",",","optional",")",":","convergence","speed","of","the","bound","functions","(","default",":","1e-3",")","eps","(","float",",","optional",")",":","term","added","to","the","denominator","to","improve","numerical","stability","(","default",":","1e-8",")","weight_decay","(","float",",","optional",")",":","weight","decay","(","L2","penalty",")","(","default",":","0",")","amsbound","(","boolean",",","optional",")",":","whether","to","use","the","AMSBound","variant","of","this","algorithm","..","Adaptive","Gradient","Methods","with","Dynamic","Bound","of","Learning","Rate",":","https",":","\/\/openreview.net\/forum","?","id=Bkg3g2R9FX"]
docstr: Performs a single optimization step.
  Arguments:
      closure (callable, optional): A closure that reevaluates the model
          and returns the loss.
docstr_tok: ["Performs","a","single","optimization","step",".","Arguments",":","closure","(","callable",",","optional",")",":","A","closure","that","reevaluates","the","model","and","returns","the","loss","."]
code: def step(self, closure=None):
      """Performs a single optimization step.
      Arguments:
          closure (callable, optional): A closure that reevaluates the model
              and returns the loss.
      """
      loss = None
      if closure is not None:
          loss = closure()
  
      for group, base_lr in zip(self.param_groups, self.base_lrs):
          for p in group["params"]:
              if p.grad is None:
                  continue
              grad = p.grad.data
              if grad.is_sparse:
                  raise RuntimeError(
                      "Adam does not support sparse gradients, please consider SparseAdam instead"
                  )
              amsbound = group["amsbound"]
  
              state = self.state[p]
  
              # State initialization
              if len(state) == 0:
                  state["step"] = 0
                  # Exponential moving average of gradient values
                  state["exp_avg"] = torch.zeros_like(p.data)
                  # Exponential moving average of squared gradient values
                  state["exp_avg_sq"] = torch.zeros_like(p.data)
                  if amsbound:
                      # Maintains max of all exp. moving avg. of sq. grad. values
                      state["max_exp_avg_sq"] = torch.zeros_like(p.data)
  
              exp_avg, exp_avg_sq = (
                  state["exp_avg"],
                  state["exp_avg_sq"],
              )
              if amsbound:
                  max_exp_avg_sq = state["max_exp_avg_sq"]
              beta1, beta2 = group["betas"]
  
              state["step"] += 1
  
              # Decay the first and second moment running average coefficient
              exp_avg.mul_(beta1).add_(1 - beta1, grad)
              exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
              if amsbound:
                  # Maintains the maximum of all 2nd moment running avg. till now
                  torch.max(
                      max_exp_avg_sq, exp_avg_sq, out=max_exp_avg_sq
                  )
                  # Use the max. for normalizing running avg. of gradient
                  denom = max_exp_avg_sq.sqrt().add_(group["eps"])
              else:
                  denom = exp_avg_sq.sqrt().add_(group["eps"])
  
              bias_correction1 = 1 - beta1 ** state["step"]
              bias_correction2 = 1 - beta2 ** state["step"]
              step_size = (
                  group["lr"]
                  * math.sqrt(bias_correction2)
                  / bias_correction1
              )
  
              # Applies bounds on actual learning rate
              # lr_scheduler cannot affect final_lr, this is a workaround to apply lr decay
              final_lr = group["final_lr"] * group["lr"] / base_lr
              lower_bound = final_lr * (
                  1 - 1 / (group["gamma"] * state["step"] + 1)
              )
              upper_bound = final_lr * (
                  1 + 1 / (group["gamma"] * state["step"])
              )
              step_size = torch.full_like(denom, step_size)
              step_size.div_(denom).clamp_(
                  lower_bound, upper_bound
              ).mul_(exp_avg)
  
              if group["weight_decay"] != 0:
                  decayed_weights = torch.mul(
                      p.data, group["weight_decay"]
                  )
                  p.data.add_(-step_size)
                  p.data.sub_(decayed_weights)
              else:
                  p.data.add_(-step_size)
  
      return loss
  
file_path: AdaBound/adabound/adabound.py
imports: import math
  import torch
  from torch.optim import Optimizer
label_desc: Use image node_id 3 for calling the AdaBoundW obj's underlying member method code with example usage: obj.step(closure) and returns: loss