details see Distributed Autograd Design.
torch.distributed.autograd.backward(context_id: int, roots: List[Tensor], retain_graph=False) -> None
Kicks off the distributed backward pass using the provided roots.
   This currently implements the FAST mode algorithm which assumes all
   RPC messages sent in the same distributed autograd context across
   workers would be part of the autograd graph during the backward
   pass.
We use the provided roots to discover the autograd graph and
   compute appropriate dependencies. This method blocks until the
   entire autograd computation is done.
We accumulate the gradients in the appropriate
   "torch.distributed.autograd.context" on each of the nodes. The
   autograd context to be used is looked up given the "context_id"
   that is passed in when "torch.distributed.autograd.backward()" is
   called. If there is no valid autograd context corresponding to the
   given ID, we throw an error. You can retrieve the accumulated