created on each node as needed.
This context serves the following purpose:


Multiple nodes running distributed backward passes might accumulate
   gradients on the same tensor and as a result the ".grad" field of
   the tensor would have gradients from a variety of distributed
   backward passes before we have the opportunity to run the
   optimizer. This is similar to calling "torch.autograd.backward()"
   multiple times locally. In order to provide a way of separating out
   the gradients for each backward pass, the gradients are accumulated
   in the "torch.distributed.autograd.context" for each backward pass.


During the forward pass we store the "send" and "recv" functions
   for each autograd pass in this context. This ensures we hold
   references to the appropriate nodes in the autograd graph to keep
   it alive. In addition to this, it is easy to look up the
   appropriate "send" and "recv" functions during the backward pass.

