class_docstr: Implements distributed data parallelism that is based on
  byteps push-pull.
  
  This container parallelizes the application of the given module by splitting
  the input across multiple devices, and each device handles a portion of the
  input. During the backwards pass, gradients from each node are averaged.
  
  ``DistributedDataParallel`` can be used in the following way:
  
  Single-Process Single-GPU
  
  This is currently the only way to use ``DistributedDataParallel``, with
  multiple processes, each of which operates on a single GPU.
  
  Here is how to use it: on each host with N GPUs, you should spawn up N
  processes, while ensuring that each process individually works on a single
  GPU from 0 to N-1. Therefore, it is your job to ensure that your training
  script operates on a single given GPU by calling:
  
      >>> torch.cuda.set_device(i)
  
  where i is from 0 to N-1. In each process, you should refer the following
  to construct this module:
  
      >>> model = DistributedDataParallel(model, device_ids=[i])
  
  In order to spawn up multiple processes per node, you can use either
  ``torch.distributed.launch`` or ``torch.multiprocessing.spawn``
  
  .. note:: If you use ``torch.save`` on one process to checkpoint the module,
      and ``torch.load`` on some other processes to recover it, make sure that
      ``map_location`` is configured properly for every process. Without
      ``map_location``, ``torch.load`` would recover the module to devices
      where the module was saved from.
  
  .. warning::
      This module works only with the ``device_ids`` containing one entry.
  
  .. warning::
      Constructor, forward method, and differentiation of the output (or a
      function of the output of this module) is a distributed synchronization
      point. Take that into account in case different processes might be
      executing different code.
  
  .. warning::
      This module assumes all parameters are registered in the model by the
      time it is created. No parameters should be added nor removed later.
      Same applies to buffers.
  
  .. warning::
      This module assumes all buffers and gradients are dense.
  
  .. warning::
      This module doesn't work with :func:`torch.autograd.grad` (i.e. it will
      only work if gradients are to be accumulated in ``.grad`` attributes of
      parameters).
  
  .. warning::
      Forward and backward hooks defined on :attr:`module` and its submodules
      won't be invoked anymore, unless the hooks are initialized in the
      :meth:`forward` method.
  
  .. warning::
      You should never try to change your model's parameters after wrapping
      up your model with DistributedDataParallel. In other words, when
      wrapping up your model with DistributedDataParallel, the constructor of
      DistributedDataParallel will register the additional gradient
      reduction functions on all the parameters of the model itself at the
      time of construction. If you change the model's parameters after
      the DistributedDataParallel construction, this is not supported and
      unexpected behaviors can happen, since some parameters' gradient
      reduction functions might not get called.
  
  .. note::
      Parameters are never broadcast between processes. The module performs
      an all-reduce step on gradients and assumes that they will be modified
      by the optimizer in all processes in the same way. Buffers
      (e.g. BatchNorm stats) are broadcast from the module in rank 0 to all
      other replicas in the system in every iteration.
  
  .. note::
      Some models have branches, part of the model is skipped during the
      forward pass. In that case it's required to call the
      DistributedDataParallel.synchronize() after loss.backward(), e.g:
  
          >>> model = DistributedDataParallel(model, device_ids=[i])
          >>> output = model(data)
          >>> loss = F.nll_loss(output, target)
          >>> loss.backward()
          >>> model.synchronize()
          >>> optimizer.step()
  
  Args:
      module (Module): module to be parallelized
      device_ids (list of int or torch.device): CUDA devices. This should
                 contain only one entry. The `module` replica is placed on
                 ``device_ids[0]``.
      broadcast_buffers (bool): flag that enables syncing (broadcasting) buffers of
                        the module at beginning of the forward function.
                        (default: ``True``)
  
  Attributes:
      module (Module): the module to be parallelized
  
  Example::
  
      >>> net = torch.nn.DistributedDataParallel(model, device_ids=[2])
code: def _push_pull_grad_async(self, p):
      if self._is_tensor_instance:
          name = self._parameter_names.get(p.__hash__())
      else:
          name = self._parameter_names.get(p)
      if self._enable_async:
          # the real handle will be created in step()
          handle, ctx = None, None
      else:
          tensor = p.grad
          tensor_compressed, ctx = self._compression.compress(tensor)
          handle = byteps_push_pull(
              tensor_compressed, average=True, name="Gradient." + name
          )
      return handle, ctx
  
imports: import torch
  from torch.nn.modules import Module
  from byteps.torch.ops import push_pull_group_sync_inplace
  from byteps.torch.ops import push_pull_async_inplace
  from byteps.torch.ops import poll, synchronize, declare, byteps_torch_set_num_grads
  from byteps.torch.ops import size, local_size, rank, local_rank
  from contextlib import contextmanager
  import byteps
  from byteps.torch.compression import Compression
  from torch.cuda._utils import _get_device_index
  import os
label_desc: Use image node_id 7 for calling the DistributedDataParallel obj's underlying member method code with example usage: obj._push_pull_grad_async(p) and returns: handle,  ctx