autograd. Consider this piece of code which uses RPC.
import torch
   import torch.distributed.rpc as rpc
a = torch.rand((3, 3), requires_grad=True)
   b = torch.rand((3, 3), requires_grad=True)
   c = torch.rand((3, 3), requires_grad=True)
d = rpc.rpc_sync("worker1", torch.add, args=(a, b))
   e = rpc.rpc_sync("worker1", torch.mul, args=(b, c))
   loss = d.sum()
The associated autograd graph for the code above would be:
[image]
Computing dependencies of this distributed autograd graph is much more
challenging and requires some overhead (either in terms of computation
or network communication).
For performance sensitive applications we can avoid a lot of overhead
by assuming every "send" and "recv" function are valid as part of the
backward pass (most applications don't perform RPCs that aren't used).
This simplifies the distributed autograd algorithm and is much more
efficient, but at the cost that the application needs to be aware of