class_docstr: Implements the AdaHessian algorithm from "ADAHESSIAN: An Adaptive Second OrderOptimizer for Machine Learning"
  
  Arguments:
      params (iterable): iterable of parameters to optimize or dicts defining parameter groups
      lr (float, optional): learning rate (default: 0.1)
      betas ((float, float), optional): coefficients used for computing running averages of gradient and the
          squared hessian trace (default: (0.9, 0.999))
      eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8)
      weight_decay (float, optional): weight decay (L2 penalty) (default: 0.0)
      hessian_power (float, optional): exponent of the hessian trace (default: 1.0)
      update_each (int, optional): compute the hessian trace approximation only after *this* number of steps
          (to save time) (default: 1)
      n_samples (int, optional): how many times to sample `z` for the approximation of the hessian trace (default: 1)
code: def __init__(
      self,
      params,
      lr=0.1,
      betas=(0.9, 0.999),
      eps=1e-8,
      weight_decay=0.0,
      hessian_power=1.0,
      update_each=1,
      n_samples=1,
      avg_conv_kernel=False,
  ):
      if 0.0 > lr:
          raise ValueError(f"Invalid learning rate: {lr}")
      if 0.0 > eps:
          raise ValueError(f"Invalid epsilon value: {eps}")
      if not 0.0 <= betas[0] < 1.0:
          raise ValueError(
              f"Invalid beta parameter at index 0: {betas[0]}"
          )
      if not 0.0 <= betas[1] < 1.0:
          raise ValueError(
              f"Invalid beta parameter at index 1: {betas[1]}"
          )
      if not 0.0 <= hessian_power <= 1.0:
          raise ValueError(
              f"Invalid Hessian power value: {hessian_power}"
          )
  
      self.n_samples = n_samples
      self.update_each = update_each
      self.avg_conv_kernel = avg_conv_kernel
  
      # use a separate generator that deterministically generates the same `z`s across all GPUs in case of distributed training
      self.seed = 2147483647
      self.generator = torch.Generator().manual_seed(self.seed)
  
      defaults = dict(
          lr=lr,
          betas=betas,
          eps=eps,
          weight_decay=weight_decay,
          hessian_power=hessian_power,
      )
      super(Adahessian, self).__init__(params, defaults)
  
      for p in self.get_params():
          p.hess = 0.0
          self.state[p]["hessian step"] = 0
  
file_path: pywick/pywick/optimizers/adahessian.py
imports: import torch
label_desc: Use image node_id 1 to create a new Adahessian object from inherited base classes: torch.optim with example: obj = Adahessian(params, lr, betas, eps, weight_decay, hessian_power, update_each, n_samples, avg_conv_kernel)