then "model[1]" would incorrectly see "float16" activations
     instead of "bfloat16" ones.
class torch.distributed.fsdp.CPUOffload(offload_params=False)
This configures CPU offloading.
Variables:
      offload_params (bool) -- This specifies whether to offload
      parameters to CPU when not involved in computation. If enabled,
      this implicitly offloads gradients to CPU as well. This is to
      support the optimizer step, which requires parameters and
      gradients to be on the same device.