


 lr=0.05,
  >>>   )
  >>>   dist_optim.step(context_id)




step(context_id)
  Performs a single optimization step.

  This will call "torch.optim.Optimizer.step()" on each worker
  containing parameters to be optimized, and will block until all
  workers return. The provided "context_id" will be used to
  retrieve the corresponding "context" that contains the gradients
  that should be applied to the parameters.

  Parameters:
     **context_id** -- the autograd context id for which we should
     run the optimizer step.

class torch.distributed.optim.PostLocalSGDOptimizer(optim, averager)
Wraps an arbitrary "torch.optim.Optimizer" and runs post-local SGD,
   This optimizer runs local optimizer at every step. After the warm-
   up stage, it averages parameters periodically afer the local
   optimizer is applied.
Parameters:
      * optim (Optimizer) -- The local optimizer.