class_docstr: Hook used to call torch.cuda.synchronize() between layer computations,
  slowing the performance down but decreasing the memory usage when using FSDP
docstr: Only sync the FSDP modules, so that two successive FSDP module
  do not overlap each other, and other modules should run as fast
  as they can.
  
  We ignore the SyncBN for each of them is wrapped in an FSDP
  module to switch them back to fp32, and not because they contain
  a lot of parameters
code: def _should_sync_module(self, module_name: str, m: nn.Module) -> bool:
      """
      Only sync the FSDP modules, so that two successive FSDP module
      do not overlap each other, and other modules should run as fast
      as they can.
  
      We ignore the SyncBN for each of them is wrapped in an FSDP
      module to switch them back to fp32, and not because they contain
      a lot of parameters
      """
      if isinstance(m, FullyShardedDataParallel):
          wrapped = m._fsdp_wrapped_module
          if isinstance(wrapped, SyncBatchNorm):
              return False
          if isinstance(wrapped, FlattenParamsWrapper):
              if isinstance(wrapped._fpw_module, SyncBatchNorm):
                  return False
          return True
      else:
          return False
  
imports: import logging
  import os
  import torch
  import torch.nn
  from classy_vision import tasks
  from classy_vision.hooks.classy_hook import ClassyHook
  from fairscale.nn import FullyShardedDataParallel
  from fairscale.nn.misc import FlattenParamsWrapper
  from torch.nn import SyncBatchNorm
  from vissl.config.attr_dict import AttrDict
  from vissl.utils.env import get_machine_local_and_dist_rank
  from vissl.utils.layer_memory_tracking import LayerwiseMemoryTracker
  from vissl.utils.profiler import create_runtime_profiler
label_desc: Use image node_id 4 for calling the CudaSynchronizeHook obj's underlying member method code with example usage: obj._should_sync_module(module_name, m) and returns: True, False, False, False