training by "FullyShardedDataParallel".


"FULL_SHARD": Parameters, gradients, and optimizer states are
     sharded. For the parameters, this strategy unshards (via all-
     gather) before the forward, reshards after the forward, unshards
     before the backward computation, and reshards after the backward
     computation. For gradients, it synchronizes and shards them (via
     reduce-scatter) after the backward computation. The sharded
     optimizer states are updated locally per rank.


"SHARD_GRAD_OP": Gradients and optimizer states are sharded
     during computation, and additionally, parameters are sharded
     outside computation. For the parameters, this strategy unshards
     before the forward, does not reshard them after the forward, and
     only reshards them after the backward computation. The sharded
     optimizer states are updated locally per rank. Inside
     "no_sync()", the parameters are not resharded after the backward
     computation.

