its inputs are retrieved from operators executed on the destination
  using the input tensors. The output gradients of this function are
  sent to the source node to the appropriate "send" function during
  the backward pass.


Each "send-recv" pair is assigned a globally unique
  "autograd_message_id" to uniquely identify the pair. This is useful
  to look up the corresponding function on a remote node during the
  backward pass.


For RRef, whenever we call "torch.distributed.rpc.RRef.to_here()" we
  attach an appropriate "send-recv" pair for the tensors involved.


As an example, this is what the autograd graph for our example above
would look like (t5.sum() excluded for simplicity):
[image]
Distributed Autograd Context
Each forward and backward pass that uses distributed autograd is
assigned a unique "torch.distributed.autograd.context" and this
context has a globally unique "autograd_context_id". This context is
created on each node as needed.