class_docstr: Optimizer that implements NovoGrad.
  
  The NovoGrad Optimizer was first proposed in [Stochastic Gradient
  Methods with Layerwise Adaptive Moments for training of Deep
  Networks](https://arxiv.org/pdf/1905.11286.pdf) NovoGrad is a
  first-order SGD-based algorithm, which computes second moments per
  layer instead of per weight as in Adam. Compared to Adam, NovoGrad
  takes less memory, and has been found to be more numerically stable.
  (For more information on the computation please refer to this
  [link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html))
  
  Second order moment = exponential moving average of Layer-wise square
  of grads:
      v_t <-- beta_2 * v_{t-1} + (1-beta_2) * (g_t)^2
  First order moment in one of four modes:
      1. moment of grads normalized by v_t:
          m_t <- beta_1 * m_{t-1} + [ g_t / (sqrt(v_t)+epsilon)]
      2. moment similar to Adam: exponential moving average of grads
      normalized by v_t (set grad_averaging = True to use this):
          m_t <- beta_1 * m_{t-1} +
                 [(1 - beta_1) * (g_t / (sqrt(v_t) + epsilon))]
      3. weight decay adds a w_d term after grads are rescaled by
      1/sqrt(v_t) (set weight_decay > 0 to use this0:
          m_t <- beta_1 * m_{t-1} +
                 [(g_t / (sqrt(v_t) + epsilon)) + (w_d * w_{t-1})]
      4. weight decay + exponential moving average from Adam:
          m_t <- beta_1 * m_{t-1} +
                 [(1 - beta_1) * ((g_t / (sqrt(v_t + epsilon)) +
                 (w_d * w_{t-1}))]
  Weight update:
      w_t <- w_{t-1} - lr_t * m_t
  
  Example of usage:
  ```python
  opt = tfa.optimizers.NovoGrad(
      lr=1e-3,
      beta_1=0.9,
      beta_2=0.999,
      weight_decay=0.001,
      grad_averaging=False
  )
  ```
code: def set_weights(self, weights):
      params = self.weights
      # If the weights are generated by Keras V1 optimizer, it includes vhats
      # even without amsgrad, i.e, V1 optimizer has 3x + 1 variables, while V2
      # optimizer has 2x + 1 variables. Filter vhats out for compatibility.
      num_vars = int((len(params) - 1) / 2)
      if len(weights) == 3 * num_vars + 1:
          weights = weights[: len(params)]
      super().set_weights(weights)
  
file_path: addons/tensorflow_addons/optimizers/novograd.py
imports: import tensorflow
  from tensorflow_addons.utils.types import FloatTensorLike
  from tensorflow_addons.optimizers import KerasLegacyOptimizer
  from typing import Union, Callable
  from typeguard import typechecked
label_desc: Use image node_id 4 for calling the NovoGrad obj's underlying member method code with example usage: obj.set_weights(weights) without return types