class_docstr: Variant of the Adam optimizer.
  
  It achieves fast convergence as Adam and generalization comparable to SGD.
  It adapts the step size depending on its "belief" in the gradient direction
  — the optimizer adaptively scales step size by the difference between the
  predicted and observed gradients.
  
  It implements the AdaBelief proposed by
  Juntang Zhuang et al. in [AdaBelief Optimizer: Adapting stepsizes by the
  belief in observed gradients](https://arxiv.org/abs/2010.07468).
  
  Example of usage:
  
  ```python
  opt = tfa.optimizers.AdaBelief(lr=1e-3)
  ```
  
  Note: `amsgrad` is not described in the original paper. Use it with
        caution.
  
  You can enable enable warmup by setting `total_steps` and
  `warmup_proportion`,
  and enable recitifcation as in RAdam by setting 'rectify':
  ```python
  opt = tfa.optimizers.AdaBelief(
      lr=1e-3,
      total_steps=10000,
      warmup_proportion=0.1,
      min_lr=1e-5,
      rectify=True,
  )
  ```
  
  In the above example, the learning rate will increase linearly
  from 0 to `lr` in 1000 steps, then decrease linearly from `lr` to `min_lr`
  in 9000 steps.
  
  Note 'rectify' is independent of 'warmup', you can choose any combinations.
  
  Lookahead, proposed by Michael R. Zhang et.al in the paper
  [Lookahead Optimizer: k steps forward, 1 step back]
  (https://arxiv.org/abs/1907.08610v1), can be integrated with AdaBelief,
  which is called 'ranger_adabelief' in the author's implementation
  https://github.com/juntang-zhuang/Adabelief-Optimizer.
  The mechanism can be enabled by using the lookahead wrapper. For example:
  
  ```python
  adabelief = tfa.optimizers.AdaBelief()
  ranger = tfa.optimizers.Lookahead(adabelief, sync_period=6, slow_step_size=0.5)
  ```
class_docstr_tok: ["Variant","of","the","Adam","optimizer",".","It","achieves","fast","convergence","as","Adam","and","generalization","comparable","to","SGD",".","It","adapts","the","step","size","depending","on","its","``","belief","''","in","the","gradient","direction","\u2014","the","optimizer","adaptively","scales","step","size","by","the","difference","between","the","predicted","and","observed","gradients",".","It","implements","the","AdaBelief","proposed","by","Juntang","Zhuang","et","al",".","in","[","AdaBelief","Optimizer",":","Adapting","stepsizes","by","the","belief","in","observed","gradients","]","(","https",":","\/\/arxiv.org\/abs\/2010.07468",")",".","Example","of","usage",":","``","`","python","opt","=","tfa.optimizers.AdaBelief","(","lr=1e-3",")","``","`","Note",":","`","amsgrad","`","is","not","described","in","the","original","paper",".","Use","it","with","caution",".","You","can","enable","enable","warmup","by","setting","`","total_steps","`","and","`","warmup_proportion","`",",","and","enable","recitifcation","as","in","RAdam","by","setting","'rectify","'",":","``","`","python","opt","=","tfa.optimizers.AdaBelief","(","lr=1e-3",",","total_steps=10000",",","warmup_proportion=0.1",",","min_lr=1e-5",",","rectify=True",",",")","``","`","In","the","above","example",",","the","learning","rate","will","increase","linearly","from","0","to","`","lr","`","in","1000","steps",",","then","decrease","linearly","from","`","lr","`","to","`","min_lr","`","in","9000","steps",".","Note","'rectify","'","is","independent","of","'warmup","'",",","you","can","choose","any","combinations",".","Lookahead",",","proposed","by","Michael","R.","Zhang","et.al","in","the","paper","[","Lookahead","Optimizer",":","k","steps","forward",",","1","step","back","]","(","https",":","\/\/arxiv.org\/abs\/1907.08610v1",")",",","can","be","integrated","with","AdaBelief",",","which","is","called","'ranger_adabelief","'","in","the","author","'s","implementation","https",":","\/\/github.com\/juntang-zhuang\/Adabelief-Optimizer",".","The","mechanism","can","be","enabled","by","using","the","lookahead","wrapper",".","For","example",":","``","`","python","adabelief","=","tfa.optimizers.AdaBelief","(",")","ranger","=","tfa.optimizers.Lookahead","(","adabelief",",","sync_period=6",",","slow_step_size=0.5",")","``","`"]
code: def _create_slots(self, var_list):
      for var in var_list:
          self.add_slot(var, "m")
      for var in var_list:
          self.add_slot(var, "v")
      if self.amsgrad:
          for var in var_list:
              self.add_slot(var, "vhat")
  
file_path: addons/tensorflow_addons/optimizers/adabelief.py
imports: import tensorflow
  from tensorflow_addons.utils.types import FloatTensorLike
  from tensorflow_addons.optimizers import KerasLegacyOptimizer
  from typing import Union, Callable, Dict
label_desc: Use image node_id 2 for calling the AdaBelief obj's underlying member method code with example usage: obj._create_slots(var_list) without return types