class_docstr: Implements Adafactor algorithm.
  
  This implementation is based on:
  `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost`
  (see https://arxiv.org/abs/1804.04235)
  
  Taken from the fairseq implementation, which can be found here:
  <https://github.com/pytorch/fairseq/blob/main/fairseq/optim/adafactor.py>.
  
  :param params (iterable):
      iterable of parameters to optimize or dicts defining parameter groups
  :param lr (float, optional):
      external learning rate (default: None)
  :param eps (tuple[float, float]):
      regularization constans for square gradient and parameter scale
      respectively (default: (1e-30, 1e-3))
  :param clip_threshold (float):
      threshold of root mean square of final gradient update
      (default: 1.0)
  :param decay_rate (float):
      coefficient used to compute running averages of square gradient
      (default: -0.8)
  :param beta1 (float):
      coefficient used for computing running averages of gradient
      (default: None)
  :param weight_decay (float, optional):
      weight decay (L2 penalty) (default: 0)
  :param scale_parameter (bool):
      if true, learning rate is scaled by root mean square of parameter
      (default: True)
  :param relative_step (bool):
      if true, time-dependent learning rate is computed instead of external
      learning rate (default: True)
  :param warmup_init (bool):
      time-dependent learning rate computation depends on whether warm-up
      initialization is being used (default: False)
class_docstr_tok: ["Implements","Adafactor","algorithm",".","This","implementation","is","based","on",":","`","Adafactor",":","Adaptive","Learning","Rates","with","Sublinear","Memory","Cost","`","(","see","https",":","\/\/arxiv.org\/abs\/1804.04235",")","Taken","from","the","fairseq","implementation",",","which","can","be","found","here",":","<","https",":","\/\/github.com\/pytorch\/fairseq\/blob\/main\/fairseq\/optim\/adafactor.py",">",".",":","param","params","(","iterable",")",":","iterable","of","parameters","to","optimize","or","dicts","defining","parameter","groups",":","param","lr","(","float",",","optional",")",":","external","learning","rate","(","default",":","None",")",":","param","eps","(","tuple","[","float",",","float","]",")",":","regularization","constans","for","square","gradient","and","parameter","scale","respectively","(","default",":","(","1e-30",",","1e-3",")",")",":","param","clip_threshold","(","float",")",":","threshold","of","root","mean","square","of","final","gradient","update","(","default",":","1.0",")",":","param","decay_rate","(","float",")",":","coefficient","used","to","compute","running","averages","of","square","gradient","(","default",":","-0.8",")",":","param","beta1","(","float",")",":","coefficient","used","for","computing","running","averages","of","gradient","(","default",":","None",")",":","param","weight_decay","(","float",",","optional",")",":","weight","decay","(","L2","penalty",")","(","default",":","0",")",":","param","scale_parameter","(","bool",")",":","if","true",",","learning","rate","is","scaled","by","root","mean","square","of","parameter","(","default",":","True",")",":","param","relative_step","(","bool",")",":","if","true",",","time-dependent","learning","rate","is","computed","instead","of","external","learning","rate","(","default",":","True",")",":","param","warmup_init","(","bool",")",":","time-dependent","learning","rate","computation","depends","on","whether","warm-up","initialization","is","being","used","(","default",":","False",")"]
docstr: Root mean square of a tensor.
code: def _rms(self, tensor):
      """
      Root mean square of a tensor.
      """
      return tensor.norm(2) / (tensor.numel() ** 0.5)
  
imports: import math
  from itertools import chain
  from typing import Optional
  import parlai.utils.logging
label_desc: Use image node_id 4 for calling the Adafactor obj's underlying member method code with example usage: obj._rms(tensor) and returns: unknown