class_docstr: Adam optimizer for fairseq.
  
  Important note: this optimizer corresponds to the "AdamW" variant of
  Adam in its weight decay behavior. As such, it is most closely
  analogous to torch.optim.AdamW from PyTorch.
class_docstr_tok: ["Adam","optimizer","for","fairseq",".","Important","note",":","this","optimizer","corresponds","to","the","``","AdamW","''","variant","of","Adam","in","its","weight","decay","behavior",".","As","such",",","it","is","most","closely","analogous","to","torch.optim.AdamW","from","PyTorch","."]
docstr: Reduce Params is only used during BMUF distributed training.
docstr_tok: ["Reduce","Params","is","only","used","during","BMUF","distributed","training","."]
code: def average_params(self):
      """Reduce Params is only used during BMUF distributed training."""
      state_dict = self.optimizer.state_dict()
      total_gpus = float(dist.get_world_size())
  
      for _, value in state_dict["state"].items():
          value["exp_avg"] /= total_gpus
          value["exp_avg_sq"] /= total_gpus
          dist.all_reduce(value["exp_avg"], op=dist.ReduceOp.SUM)
          dist.all_reduce(value["exp_avg_sq"], op=dist.ReduceOp.SUM)
  
imports: import logging
  import math
  from dataclasses import dataclass, field
  from typing import List
  import torch
  import torch.distributed
  import torch.optim
  from fairseq.dataclass import FairseqDataclass
  from fairseq.optim import FairseqOptimizer, register_optimizer
  from omegaconf import II
  from lightseq.training.ops.pytorch.adam import LSAdam
label_desc: Use image node_id 3 for calling the LSFSAdam obj's underlying member method code with example usage: obj.average_params() without return types