Perform some computation remotely.
 t3 = rpc.rpc_sync("worker1", my_add, args=(t1, t2))

 # Perform some computation locally based on remote result.
 t4 = torch.rand((3, 3), requires_grad=True)
 t5 = torch.mul(t3, t4)

 # Compute some loss.
 loss = t5.sum()

 # Run the backward pass.
 dist_autograd.backward(context_id, [loss])

 # Retrieve the gradients from the context.
 dist_autograd.get_gradients(context_id)

The distributed autograd graph with dependencies would be as follows
(t5.sum() excluded for simplicity):
[image]
The FAST mode algorithm applied to the above example would be as
follows:


On "Worker 0" we start from the roots "loss" and "send1" to compute
   dependencies. As a result "send1" is marked with a dependency of 1
   and "mul" on "Worker 0" is marked with a dependency of 1.


Now, we kickoff the local autograd engine on "Worker 0". We first
   execute the "mul" function, accumulate its output in the autograd

