context as the gradient for "t4". Then, we execute "recv2" which
   sends the gradients to "Worker 1".


Since this is the first time "Worker 1" has heard about this
   backward pass, it starts dependency computation and marks the
   dependencies for "send2", "add" and "recv1" appropriately.


Next, we enqueue "send2" on the local autograd engine of "Worker
   1", which in turn executes "add" and "recv1".


When "recv1" is executed it sends the gradients over to "Worker 0".


Since "Worker 0" has already computed dependencies for this
   backward pass, it just enqueues and executes "send1" locally.


Finally, gradients for "t1", "t2" and "t4" are accumulated in the
   Distributed Autograd Context.


SMART mode algorithm
Full details of this algorithm are still in the works, but for the
general idea you can refer to Distributed Autograd Algorithm Smart
mode section in the RFC.
Distributed Optimizer
The "DistributedOptimizer" operates as follows: