Warning:
 If you pass "overlap_with_ddp=True", be wary of the following:
 Given the way that overlapping "DistributedDataParallel" with
 "ZeroRedundancyOptimizer" is currently implemented, the first two
 or three training iterations do not perform parameter updates in
 the optimizer step, depending on if "static_graph=False" or
 "static_graph=True", respectively. This is because it needs
 information about the gradient bucketing strategy used by
 "DistributedDataParallel", which is not finalized until the
 second forward pass if "static_graph=False" or until the third
 forward pass if "static_graph=True". To adjust for this, one
 option is to prepend dummy inputs.

Warning:
 ZeroRedundancyOptimizer is experimental and subject to change.

add_param_group(param_group)
  Add a parameter group to the "Optimizer" 's "param_groups".

  This can be useful when fine tuning a pre-trained network, as
