but also restores model averager's step value to the one saved
      in the provided "state_dict".
  If there is no ""step"" entry in "state_dict", it will raise a
  warning and initialize the model averager's step to 0.

state_dict()
  This is the same as "torch.optim.Optimizer" "state_dict()", but
  adds an extra entry to record model averager's step to the
  checkpoint to ensure reload does not cause unnecessary warm up
  again.

step()
  Performs a single optimization step (parameter update).

class torch.distributed.optim.ZeroRedundancyOptimizer(params, optimizer_class, process_group=None, parameters_as_bucket_view=False, overlap_with_ddp=False, **defaults)
This class wraps an arbitrary "optim.Optimizer" and shards its
   states across ranks in the group as described by ZeRO. The local
   optimizer instance in each rank is only responsible for updating
   approximately "1 / world_size" parameters and hence only needs to