backward pass on such distributed models with the "loss" that we've
computed and record appropriate gradients for all tensors that require
gradients.
Autograd recording during the forward pass
PyTorch builds the autograd graph during the forward pass and this
graph is used to execute the backward pass. For more details see How
autograd encodes the history.
For distributed autograd, we need to keep track of all RPCs during the
forward pass to ensure the backward pass is executed appropriately.
For this purpose, we attach "send" and "recv" functions to the
autograd graph when we perform an RPC.


The "send" function is attached to the source of the RPC and its
  output edges point to the autograd function for the input tensors of
  the RPC. The input for this function during the backward pass is
  received from the destination as the output of the appropriate
  "recv" function.


The "recv" function is attached to the destination of the RPC and

