class_docstr: Variant of the Adam optimizer whose adaptive learning rate is rectified
  so as to have a consistent variance.
  
  It implements the Rectified Adam (a.k.a. RAdam) proposed by
  Liyuan Liu et al. in [On The Variance Of The Adaptive Learning Rate
  And Beyond](https://arxiv.org/pdf/1908.03265v1.pdf).
  
  Example of usage:
  
  ```python
  opt = tfa.optimizers.RectifiedAdam(lr=1e-3)
  ```
  
  Note: `amsgrad` is not described in the original paper. Use it with
        caution.
  
  RAdam is not a placement of the heuristic warmup, the settings should be
  kept if warmup has already been employed and tuned in the baseline method.
  You can enable warmup by setting `total_steps` and `warmup_proportion`:
  
  ```python
  opt = tfa.optimizers.RectifiedAdam(
      lr=1e-3,
      total_steps=10000,
      warmup_proportion=0.1,
      min_lr=1e-5,
  )
  ```
  
  In the above example, the learning rate will increase linearly
  from 0 to `lr` in 1000 steps, then decrease linearly from `lr` to `min_lr`
  in 9000 steps.
  
  Lookahead, proposed by Michael R. Zhang et.al in the paper
  [Lookahead Optimizer: k steps forward, 1 step back]
  (https://arxiv.org/abs/1907.08610v1), can be integrated with RAdam,
  which is announced by Less Wright and the new combined optimizer can also
  be called "Ranger". The mechanism can be enabled by using the lookahead
  wrapper. For example:
  
  ```python
  radam = tfa.optimizers.RectifiedAdam()
  ranger = tfa.optimizers.Lookahead(radam, sync_period=6, slow_step_size=0.5)
  ```
class_docstr_tok: ["Variant","of","the","Adam","optimizer","whose","adaptive","learning","rate","is","rectified","so","as","to","have","a","consistent","variance",".","It","implements","the","Rectified","Adam","(","a.k.a",".","RAdam",")","proposed","by","Liyuan","Liu","et","al",".","in","[","On","The","Variance","Of","The","Adaptive","Learning","Rate","And","Beyond","]","(","https",":","\/\/arxiv.org\/pdf\/1908.03265v1.pdf",")",".","Example","of","usage",":","``","`","python","opt","=","tfa.optimizers.RectifiedAdam","(","lr=1e-3",")","``","`","Note",":","`","amsgrad","`","is","not","described","in","the","original","paper",".","Use","it","with","caution",".","RAdam","is","not","a","placement","of","the","heuristic","warmup",",","the","settings","should","be","kept","if","warmup","has","already","been","employed","and","tuned","in","the","baseline","method",".","You","can","enable","warmup","by","setting","`","total_steps","`","and","`","warmup_proportion","`",":","``","`","python","opt","=","tfa.optimizers.RectifiedAdam","(","lr=1e-3",",","total_steps=10000",",","warmup_proportion=0.1",",","min_lr=1e-5",",",")","``","`","In","the","above","example",",","the","learning","rate","will","increase","linearly","from","0","to","`","lr","`","in","1000","steps",",","then","decrease","linearly","from","`","lr","`","to","`","min_lr","`","in","9000","steps",".","Lookahead",",","proposed","by","Michael","R.","Zhang","et.al","in","the","paper","[","Lookahead","Optimizer",":","k","steps","forward",",","1","step","back","]","(","https",":","\/\/arxiv.org\/abs\/1907.08610v1",")",",","can","be","integrated","with","RAdam",",","which","is","announced","by","Less","Wright","and","the","new","combined","optimizer","can","also","be","called","``","Ranger","''",".","The","mechanism","can","be","enabled","by","using","the","lookahead","wrapper",".","For","example",":","``","`","python","radam","=","tfa.optimizers.RectifiedAdam","(",")","ranger","=","tfa.optimizers.Lookahead","(","radam",",","sync_period=6",",","slow_step_size=0.5",")","``","`"]
docstr: Construct a new RAdam optimizer.
  
  Args:
      learning_rate: A `Tensor` or a floating point value, or a schedule
          that is a `tf.keras.optimizers.schedules.LearningRateSchedule`.
          The learning rate.
      beta_1: A float value or a constant float tensor.
          The exponential decay rate for the 1st moment estimates.
      beta_2: A float value or a constant float tensor.
          The exponential decay rate for the 2nd moment estimates.
      epsilon: A small constant for numerical stability.
      weight_decay: A `Tensor` or a floating point value, or a schedule
          that is a `tf.keras.optimizers.schedules.LearningRateSchedule`.
          Weight decay for each parameter.
      amsgrad: boolean. Whether to apply AMSGrad variant of this
          algorithm from the paper "On the Convergence of Adam and
          beyond".
      sma_threshold. A float value.
          The threshold for simple mean average.
      total_steps: An integer value. Total number of training steps.
          Enable warmup by setting a positive value.
      warmup_proportion: A floating point value.
          The proportion of increasing steps.
      min_lr: A floating point value. Minimum learning rate after warmup.
      name: Optional name for the operations created when applying
          gradients. Defaults to "RectifiedAdam".
      **kwargs: keyword arguments. Allowed to be {`clipnorm`,
          `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients
          by norm; `clipvalue` is clip gradients by value, `decay` is
          included for backward compatibility to allow time inverse
          decay of learning rate. `lr` is included for backward
          compatibility, recommended to use `learning_rate` instead.
docstr_tok: ["Construct","a","new","RAdam","optimizer",".","Args",":","learning_rate",":","A","`","Tensor","`","or","a","floating","point","value",",","or","a","schedule","that","is","a","`","tf.keras.optimizers.schedules.LearningRateSchedule","`",".","The","learning","rate",".","beta_1",":","A","float","value","or","a","constant","float","tensor",".","The","exponential","decay","rate","for","the","1st","moment","estimates",".","beta_2",":","A","float","value","or","a","constant","float","tensor",".","The","exponential","decay","rate","for","the","2nd","moment","estimates",".","epsilon",":","A","small","constant","for","numerical","stability",".","weight_decay",":","A","`","Tensor","`","or","a","floating","point","value",",","or","a","schedule","that","is","a","`","tf.keras.optimizers.schedules.LearningRateSchedule","`",".","Weight","decay","for","each","parameter",".","amsgrad",":","boolean",".","Whether","to","apply","AMSGrad","variant","of","this","algorithm","from","the","paper","``","On","the","Convergence","of","Adam","and","beyond","''",".","sma_threshold",".","A","float","value",".","The","threshold","for","simple","mean","average",".","total_steps",":","An","integer","value",".","Total","number","of","training","steps",".","Enable","warmup","by","setting","a","positive","value",".","warmup_proportion",":","A","floating","point","value",".","The","proportion","of","increasing","steps",".","min_lr",":","A","floating","point","value",".","Minimum","learning","rate","after","warmup",".","name",":","Optional","name","for","the","operations","created","when","applying","gradients",".","Defaults","to","``","RectifiedAdam","''",".","*","*","kwargs",":","keyword","arguments",".","Allowed","to","be","{","`","clipnorm","`",",","`","clipvalue","`",",","`","lr","`",",","`","decay","`","}",".","`","clipnorm","`","is","clip","gradients","by","norm",";","`","clipvalue","`","is","clip","gradients","by","value",",","`","decay","`","is","included","for","backward","compatibility","to","allow","time","inverse","decay","of","learning","rate",".","`","lr","`","is","included","for","backward","compatibility",",","recommended","to","use","`","learning_rate","`","instead","."]
code: def __init__(
      self,
      learning_rate: Union[FloatTensorLike, Callable, Dict] = 0.001,
      beta_1: FloatTensorLike = 0.9,
      beta_2: FloatTensorLike = 0.999,
      epsilon: FloatTensorLike = 1e-7,
      weight_decay: Union[FloatTensorLike, Callable, Dict] = 0.0,
      amsgrad: bool = False,
      sma_threshold: FloatTensorLike = 5.0,
      total_steps: int = 0,
      warmup_proportion: FloatTensorLike = 0.1,
      min_lr: FloatTensorLike = 0.0,
      name: str = "RectifiedAdam",
      **kwargs,
  ):
      r"""Construct a new RAdam optimizer.
  
      Args:
          learning_rate: A `Tensor` or a floating point value, or a schedule
              that is a `tf.keras.optimizers.schedules.LearningRateSchedule`.
              The learning rate.
          beta_1: A float value or a constant float tensor.
              The exponential decay rate for the 1st moment estimates.
          beta_2: A float value or a constant float tensor.
              The exponential decay rate for the 2nd moment estimates.
          epsilon: A small constant for numerical stability.
          weight_decay: A `Tensor` or a floating point value, or a schedule
              that is a `tf.keras.optimizers.schedules.LearningRateSchedule`.
              Weight decay for each parameter.
          amsgrad: boolean. Whether to apply AMSGrad variant of this
              algorithm from the paper "On the Convergence of Adam and
              beyond".
          sma_threshold. A float value.
              The threshold for simple mean average.
          total_steps: An integer value. Total number of training steps.
              Enable warmup by setting a positive value.
          warmup_proportion: A floating point value.
              The proportion of increasing steps.
          min_lr: A floating point value. Minimum learning rate after warmup.
          name: Optional name for the operations created when applying
              gradients. Defaults to "RectifiedAdam".
          **kwargs: keyword arguments. Allowed to be {`clipnorm`,
              `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients
              by norm; `clipvalue` is clip gradients by value, `decay` is
              included for backward compatibility to allow time inverse
              decay of learning rate. `lr` is included for backward
              compatibility, recommended to use `learning_rate` instead.
      """
      super().__init__(name, **kwargs)
  
      if isinstance(learning_rate, Dict):
          learning_rate = tf.keras.optimizers.schedules.deserialize(
              learning_rate
          )
  
      if isinstance(weight_decay, Dict):
          weight_decay = tf.keras.optimizers.schedules.deserialize(
              weight_decay
          )
  
      self._set_hyper("learning_rate", kwargs.get("lr", learning_rate))
      self._set_hyper("beta_1", beta_1)
      self._set_hyper("beta_2", beta_2)
      self._set_hyper("decay", self._initial_decay)
      self._set_hyper("weight_decay", weight_decay)
      self._set_hyper("sma_threshold", sma_threshold)
      self._set_hyper("total_steps", float(total_steps))
      self._set_hyper("warmup_proportion", warmup_proportion)
      self._set_hyper("min_lr", min_lr)
      self.epsilon = epsilon or tf.keras.backend.epsilon()
      self.amsgrad = amsgrad
      self._has_weight_decay = weight_decay != 0.0
      self._initial_total_steps = total_steps
  
file_path: addons/tensorflow_addons/optimizers/rectified_adam.py
imports: import tensorflow
  from tensorflow_addons.utils.types import FloatTensorLike
  from tensorflow_addons.optimizers import KerasLegacyOptimizer
  from typing import Union, Callable, Dict
  from typeguard import typechecked
label_desc: Use image node_id 1 to create a new RectifiedAdam object from inherited base classes: KerasLegacyOptimizer with example: obj = RectifiedAdam(learning_rate, beta_1, beta_2, epsilon, weight_decay, amsgrad, sma_threshold, total_steps, warmup_proportion, min_lr, name)