


Note that warmup_steps used in PostLocalSGDOptimizer must be the same as
  >>> # ``start_localSGD_iter`` used in ``PostLocalSGDState``.
  >>> local_optim = torch.optim.SGD(params=model.parameters(), lr=0.01)
  >>> opt = PostLocalSGDOptimizer(
  >>>     optim=local_optim,
  >>>     averager=averagers.PeriodicModelAverager(period=4, warmup_steps=100)
  >>> )
  >>>
  >>> # In the first 100 steps, DDP runs global gradient averaging at every step.
  >>> # After 100 steps, DDP runs gradient averaging within each subgroup (intra-node by default),
  >>> # and post-localSGD optimizer runs global model averaging every 4 steps after applying the local optimizer.
  >>> for step in range(0, 200):
  >>>    opt.zero_grad()
  >>>    loss = loss_fn(output, labels)
  >>>    loss.backward()
  >>>    opt.step()




load_state_dict(state_dict)
  This is the same as "torch.optim.Optimizer" "load_state_dict()",
