class_docstr: Adam optimizer for fairseq.
  
  Important note: this optimizer corresponds to the "AdamW" variant of
  Adam in its weight decay behavior. As such, it is most closely
  analogous to torch.optim.AdamW from PyTorch.
docstr: Reduce Params is only used during BMUF distributed training.
docstr_tok: ["Reduce","Params","is","only","used","during","BMUF","distributed","training","."]
code: def average_params(self):
      """Reduce Params is only used during BMUF distributed training."""
      state_dict = self.optimizer.state_dict()
      total_gpus = float(dist.get_world_size())
  
      for _, value in state_dict["state"].items():
          value["exp_avg"] /= total_gpus
          value["exp_avg_sq"] /= total_gpus
          dist.all_reduce(value["exp_avg"], op=dist.ReduceOp.SUM)
          dist.all_reduce(value["exp_avg_sq"], op=dist.ReduceOp.SUM)
  
imports: import logging
  import math
  from collections.abc import Collection
  from dataclasses import dataclass, field
  from typing import Any, List
  import torch
  import torch.distributed
  import torch.optim
  from fairseq.dataclass import FairseqDataclass
  from fairseq.optim import FairseqOptimizer, register_optimizer
  from fairseq.optim.fused_adam import get_fused_adam_class
  from omegaconf import II, OmegaConf
label_desc: Use image node_id 3 for calling the FairseqAdam obj's underlying member method code with example usage: obj.average_params() without return types