on the local autograd engine for that worker.

Finally, instead of accumulating the gradients on the ".grad" field
   of the Tensor, we accumulate the gradients separately per
   Distributed Autograd Context. The gradients are stored in a
   "Dict[Tensor, Tensor]", which is basically a map from Tensor to its
   associated gradient and this map can be retrieved using the
   "get_gradients()" API.

As an example the complete code with distributed autograd would be as
follows:
import torch
   import torch.distributed.autograd as dist_autograd
   import torch.distributed.rpc as rpc
def my_add(t1, t2):
     return torch.add(t1, t2)
# On worker 0:
# Setup the autograd context. Computations that take
   # part in the distributed backward pass must be within
   # the distributed autograd context manager.
   with dist_autograd.context() as context_id:
     t1 = torch.rand((3, 3), requires_grad=True)
     t2 = torch.rand((3, 3), requires_grad=True)
 # Perform some computation remotely.
