class_docstr: Optimizer that implements NovoGrad.
  
  The NovoGrad Optimizer was first proposed in [Stochastic Gradient
  Methods with Layerwise Adaptive Moments for training of Deep
  Networks](https://arxiv.org/pdf/1905.11286.pdf) NovoGrad is a
  first-order SGD-based algorithm, which computes second moments per
  layer instead of per weight as in Adam. Compared to Adam, NovoGrad
  takes less memory, and has been found to be more numerically stable.
  (For more information on the computation please refer to this
  [link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html))
  
  Second order moment = exponential moving average of Layer-wise square
  of grads:
      v_t <-- beta_2 * v_{t-1} + (1-beta_2) * (g_t)^2
  First order moment in one of four modes:
      1. moment of grads normalized by v_t:
          m_t <- beta_1 * m_{t-1} + [ g_t / (sqrt(v_t)+epsilon)]
      2. moment similar to Adam: exponential moving average of grads
      normalized by v_t (set grad_averaging = True to use this):
          m_t <- beta_1 * m_{t-1} +
                 [(1 - beta_1) * (g_t / (sqrt(v_t) + epsilon))]
      3. weight decay adds a w_d term after grads are rescaled by
      1/sqrt(v_t) (set weight_decay > 0 to use this0:
          m_t <- beta_1 * m_{t-1} +
                 [(g_t / (sqrt(v_t) + epsilon)) + (w_d * w_{t-1})]
      4. weight decay + exponential moving average from Adam:
          m_t <- beta_1 * m_{t-1} +
                 [(1 - beta_1) * ((g_t / (sqrt(v_t + epsilon)) +
                 (w_d * w_{t-1}))]
  Weight update:
      w_t <- w_{t-1} - lr_t * m_t
  
  Example of usage:
  ```python
  opt = tfa.optimizers.NovoGrad(
      lr=1e-3,
      beta_1=0.9,
      beta_2=0.999,
      weight_decay=0.001,
      grad_averaging=False
  )
  ```
docstr: Construct a new NovoGrad optimizer.
  
  Args:
      learning_rate: A `Tensor` or a floating point value. or a schedule
          that is a `tf.keras.optimizers.schedules.LearningRateSchedule`
          The learning rate.
      beta_1: A float value or a constant float tensor.
          The exponential decay rate for the 1st moment estimates.
      beta_2: A float value or a constant float tensor.
          The exponential decay rate for the 2nd moment estimates.
      epsilon: A small constant for numerical stability.
      weight_decay: A floating point value. Weight decay for each param.
      grad_averaging: determines whether to use Adam style exponential
          moving averaging for the first order moments.
      **kwargs: keyword arguments. Allowed to be {`clipnorm`,
          `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients
          by norm; `clipvalue` is clip gradients by value, `decay` is
          included for backward compatibility to allow time inverse
          decay of learning rate. `lr` is included for backward
          compatibility, recommended to use `learning_rate` instead.
docstr_tok: ["Construct","a","new","NovoGrad","optimizer",".","Args",":","learning_rate",":","A","`","Tensor","`","or","a","floating","point","value",".","or","a","schedule","that","is","a","`","tf.keras.optimizers.schedules.LearningRateSchedule","`","The","learning","rate",".","beta_1",":","A","float","value","or","a","constant","float","tensor",".","The","exponential","decay","rate","for","the","1st","moment","estimates",".","beta_2",":","A","float","value","or","a","constant","float","tensor",".","The","exponential","decay","rate","for","the","2nd","moment","estimates",".","epsilon",":","A","small","constant","for","numerical","stability",".","weight_decay",":","A","floating","point","value",".","Weight","decay","for","each","param",".","grad_averaging",":","determines","whether","to","use","Adam","style","exponential","moving","averaging","for","the","first","order","moments",".","*","*","kwargs",":","keyword","arguments",".","Allowed","to","be","{","`","clipnorm","`",",","`","clipvalue","`",",","`","lr","`",",","`","decay","`","}",".","`","clipnorm","`","is","clip","gradients","by","norm",";","`","clipvalue","`","is","clip","gradients","by","value",",","`","decay","`","is","included","for","backward","compatibility","to","allow","time","inverse","decay","of","learning","rate",".","`","lr","`","is","included","for","backward","compatibility",",","recommended","to","use","`","learning_rate","`","instead","."]
code: def __init__(
      self,
      learning_rate: Union[FloatTensorLike, Callable] = 0.001,
      beta_1: FloatTensorLike = 0.9,
      beta_2: FloatTensorLike = 0.999,
      epsilon: FloatTensorLike = 1e-7,
      weight_decay: FloatTensorLike = 0.0,
      grad_averaging: bool = False,
      amsgrad: bool = False,
      name: str = "NovoGrad",
      **kwargs,
  ):
      r"""Construct a new NovoGrad optimizer.
  
      Args:
          learning_rate: A `Tensor` or a floating point value. or a schedule
              that is a `tf.keras.optimizers.schedules.LearningRateSchedule`
              The learning rate.
          beta_1: A float value or a constant float tensor.
              The exponential decay rate for the 1st moment estimates.
          beta_2: A float value or a constant float tensor.
              The exponential decay rate for the 2nd moment estimates.
          epsilon: A small constant for numerical stability.
          weight_decay: A floating point value. Weight decay for each param.
          grad_averaging: determines whether to use Adam style exponential
              moving averaging for the first order moments.
          **kwargs: keyword arguments. Allowed to be {`clipnorm`,
              `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients
              by norm; `clipvalue` is clip gradients by value, `decay` is
              included for backward compatibility to allow time inverse
              decay of learning rate. `lr` is included for backward
              compatibility, recommended to use `learning_rate` instead.
      """
      super().__init__(name, **kwargs)
      if weight_decay < 0.0:
          raise ValueError("Weight decay rate cannot be negative")
      self._set_hyper("learning_rate", kwargs.get("lr", learning_rate))
      self._set_hyper("decay", self._initial_decay)
      self._set_hyper("beta_1", beta_1)
      self._set_hyper("beta_2", beta_2)
      self._set_hyper("weight_decay", weight_decay)
      self._set_hyper("grad_averaging", grad_averaging)
      self.amsgrad = amsgrad
      self.epsilon = epsilon or tf.keras.backend.epsilon()
  
file_path: addons/tensorflow_addons/optimizers/novograd.py
imports: import tensorflow
  from tensorflow_addons.utils.types import FloatTensorLike
  from tensorflow_addons.optimizers import KerasLegacyOptimizer
  from typing import Union, Callable
  from typeguard import typechecked
label_desc: Use image node_id 1 to create a new NovoGrad object from inherited base classes: KerasLegacyOptimizer with example: obj = NovoGrad(learning_rate, beta_1, beta_2, epsilon, weight_decay, grad_averaging, amsgrad, name)