
In general we also use this context to store some metadata for each
   distributed autograd pass.

From the user's perspective the autograd context is setup as follows:
import torch.distributed.autograd as dist_autograd
   with dist_autograd.context() as context_id:
     loss = model.forward()
     dist_autograd.backward(context_id, loss)
It is important to note that your model's forward pass must be invoked
within the distributed autograd context manager, as a valid context is
needed in order to ensure that all "send" and "recv" functions are
stored properly to run the backward pass across all participating
nodes.
Distributed Backward Pass
In this section we outline the challenge of computing dependencies
accurately during a distributed backward pass and describe a couple of
algorithms (with tradeoffs) on how we can execute a distributed
backward pass.
Computing dependencies
Consider the following piece of code being run on a single machine