class_docstr: :class:`apex.parallel.Reducer` is a simple class that helps allreduce a module's parameters
  across processes.  :class:`Reducer` is intended to give the user additional control:
  Unlike :class:`DistributedDataParallel`, :class:`Reducer` will not automatically allreduce
  parameters during ``backward()``.
  Instead, :class:`Reducer` waits for the user to call ``<reducer_instance>.reduce()`` manually.
  This enables, for example, delaying the allreduce to be carried out every
  several iterations instead of every single iteration.
  
  Like :class:`DistributedDataParallel`, :class:`Reducer` averages any tensors it allreduces
  over the number of participating processes.
  
  :class:`Reducer` is designed to work with the upstream launch utility script
  ``torch.distributed.launch`` with ``--nproc_per_node <= number of gpus per node``.
  When used with this launcher, :class:`Reducer` assumes 1:1 mapping of processes to GPUs.
  It also assumes that your script calls ``torch.cuda.set_device(args.rank)`` before creating the model.
  
  Args:
      module_or_grads_list: Either a network definition (module) being run in multi-gpu/distributed mode, or an iterable of gradients to be reduced.  If a module is passed in, the Reducer constructor will sync the parameters across processes (broadcasting from rank 0) to make sure they're all initialized with the same values.  If a list of gradients (that came from some module) is passed in, the user is responsible for manually syncing that module's parameters at the beginning of training.
code: def reduce(self):
      if self.module:
          grads = [
              param.grad.data
              for param in self.module.parameters()
              if param.grad is not None
          ]
          flat_dist_call(grads, dist.all_reduce)
      else:
          flat_dist_call(self.grads, dist.all_reduce)
  
imports: from collections import OrderedDict
  import copy
  import importlib
  from itertools import chain
  import torch
  import torch.distributed
  from torch.nn.modules import Module
  from torch.autograd import Variable
  from ..multi_tensor_apply import multi_tensor_applier
label_desc: Use image node_id 2 for calling the Reducer obj's underlying member method code with example usage: obj.reduce() without return types