The "DistributedOptimizer" operates as follows:


Takes a list of remote parameters ("RRef") to optimize. These could
   also be local parameters wrapped within a local "RRef".


Takes a "Optimizer" class as the local optimizer to run on all
   distinct "RRef" owners.


The distributed optimizer creates an instance of the local
   "Optimizer" on each of the worker nodes and holds an "RRef" to
   them.


When "torch.distributed.optim.DistributedOptimizer.step()" is
   invoked, the distributed optimizer uses RPC to remotely execute all
   the local optimizers on the appropriate remote workers. A
   distributed autograd "context_id" must be provided as input to
   "torch.distributed.optim.DistributedOptimizer.step()". This is used
   by local optimizers to apply gradients stored in the corresponding
   context.


If multiple concurrent distributed optimizers are updating the same
   parameters on a worker, these updates are serialized via a lock.


Simple end to end example