the limitations. This algorithm is called the FAST mode algorithm and
is described in detail below.
In the general case it might not be necessary that every "send" and
"recv" function is valid as part of the backward pass. To address
this, we have proposed a SMART mode algorithm which is described in a
later section. Please note that currently, only the FAST mode
algorithm is implemented.
FAST mode algorithm
The key assumption of this algorithm is that each "send" function has
a dependency of 1 when we run a backward pass. In other words, we
assume we'll receive a gradient over RPC from another node.
The algorithm is as follows:


We start from the worker which has the roots for the backward pass
   (all roots must be local).


Lookup all the "send" functions for the current Distributed
   Autograd Context.


Compute dependencies locally starting from the provided roots and
   all the "send" functions we retrieved.

