class_docstr: :class:`apex.parallel.Reducer` is a simple class that helps allreduce a module's parameters
  across processes.  :class:`Reducer` is intended to give the user additional control:
  Unlike :class:`DistributedDataParallel`, :class:`Reducer` will not automatically allreduce
  parameters during ``backward()``.
  Instead, :class:`Reducer` waits for the user to call ``<reducer_instance>.reduce()`` manually.
  This enables, for example, delaying the allreduce to be carried out every
  several iterations instead of every single iteration.
  
  Like :class:`DistributedDataParallel`, :class:`Reducer` averages any tensors it allreduces
  over the number of participating processes.
  
  :class:`Reducer` is designed to work with the upstream launch utility script
  ``torch.distributed.launch`` with ``--nproc_per_node <= number of gpus per node``.
  When used with this launcher, :class:`Reducer` assumes 1:1 mapping of processes to GPUs.
  It also assumes that your script calls ``torch.cuda.set_device(args.rank)`` before creating the model.
  
  Args:
      module_or_grads_list: Either a network definition (module) being run in multi-gpu/distributed mode, or an iterable of gradients to be reduced.  If a module is passed in, the Reducer constructor will sync the parameters across processes (broadcasting from rank 0) to make sure they're all initialized with the same values.  If a list of gradients (that came from some module) is passed in, the user is responsible for manually syncing that module's parameters at the beginning of training.
code: def __init__(self, module_or_grads_list):
      if isinstance(module_or_grads_list, Module):
          self.module = module_or_grads_list
          flat_dist_call(
              [param.data for param in self.module.parameters()],
              dist.broadcast,
              (0,),
          )
  
      else:
          self.module = None
          self.grads = []
          extract_tensors(module_or_grads_list, self.grads)
  
imports: from collections import OrderedDict
  import copy
  import importlib
  from itertools import chain
  import torch
  import torch.distributed
  from torch.nn.modules import Module
  from torch.autograd import Variable
  from ..multi_tensor_apply import multi_tensor_applier
label_desc: Use image node_id 1 to create a new Reducer object from inherited base classes: object with example: obj = Reducer(module_or_grads_list)