class_docstr: RMSProp module.
  
  See: http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
  
  Maintain a moving (discounted) average of the square of updates. Divides each
  update by the root of this average.
  
      ms <- decay * ms + (1-decay) * update^2
      mom <- momentum * mom + learning_rate * update / sqrt(ms + epsilon)
      parameter <- parameter - mom
  
  This implementation of `RMSprop` uses plain momentum, not Nesterov momentum.
  
  The centered version additionally maintains a moving average of the
  gradients, and uses that average to estimate the variance:
  
      mg <- decay * mg + (1-decay) * update
      ms <- decay * ms + (1-decay) * update^2
      mom <- momentum * mom + learning_rate * update / sqrt(ms - mg^2 + epsilon)
      parameter <- parameter - mom
  
  Attributes:
    learning_rate: Learning rate.
    decay: Learning rate decay over each update.
    momentum: Momentum scalar.
    epsilon: Small value to avoid zero denominator.
    centered: `True` if centered.
    mom: Accumulated mom for each parameter.
    ms: Accumulated ms for each parameter.
    mg: Accumulated mg for each parameter.
docstr: Constructs an `RMSProp` module.
  
  Args:
    learning_rate: Learning rate.
    decay: Learning rate decay over each update.
    momentum: Momentum scalar.
    epsilon: Small value to avoid zero denominator.
    centered: If True, gradients are normalized by the estimated variance of
      the gradient; if False, by the uncentered second moment. Setting this to
      True may help with training, but is slightly more expensive in terms of
      computation and memory. Defaults to False.
    name: Name for this module.
docstr_tok: ["Constructs","an","`","RMSProp","`","module",".","Args",":","learning_rate",":","Learning","rate",".","decay",":","Learning","rate","decay","over","each","update",".","momentum",":","Momentum","scalar",".","epsilon",":","Small","value","to","avoid","zero","denominator",".","centered",":","If","True",",","gradients","are","normalized","by","the","estimated","variance","of","the","gradient",";","if","False",",","by","the","uncentered","second","moment",".","Setting","this","to","True","may","help","with","training",",","but","is","slightly","more","expensive","in","terms","of","computation","and","memory",".","Defaults","to","False",".","name",":","Name","for","this","module","."]
code: def __init__(
      self,
      learning_rate: Union[types.FloatLike, tf.Variable],
      decay: Union[types.FloatLike, tf.Variable] = 0.9,
      momentum: Union[types.FloatLike, tf.Variable] = 0.0,
      epsilon: Union[types.FloatLike, tf.Variable] = 1e-10,
      centered: bool = False,
      name: Optional[str] = None,
  ):
      """Constructs an `RMSProp` module.
  
      Args:
        learning_rate: Learning rate.
        decay: Learning rate decay over each update.
        momentum: Momentum scalar.
        epsilon: Small value to avoid zero denominator.
        centered: If True, gradients are normalized by the estimated variance of
          the gradient; if False, by the uncentered second moment. Setting this to
          True may help with training, but is slightly more expensive in terms of
          computation and memory. Defaults to False.
        name: Name for this module.
      """
      super().__init__(name)
      self.learning_rate = learning_rate
      self.decay = decay
      self.momentum = momentum
      self.epsilon = epsilon
      self.centered = centered
      self.mom = []
      self.ms = []
      self.mg = []
  
file_path: sonnet/sonnet/src/optimizers/rmsprop.py
imports: import itertools
  from typing import Optional, Sequence, Union
  from sonnet.src import base
  from sonnet.src import once
  from sonnet.src import types
  from sonnet.src import utils
  from sonnet.src.optimizers import optimizer_utils
  import tensorflow
label_desc: Use image node_id 1 to create a new RMSProp object from inherited base classes: base with example: obj = RMSProp(learning_rate, decay, momentum, epsilon, centered, name)