class_docstr: Gradient scaler for model-parallel inf check. The inf in gradients are checked across tensor-parallel
  ranks in (1) executing optimizer step and (2) gradient scaler update.
code: def __init__(
      self,
      init_scale=2.0**16,
      growth_factor=2.0,
      backoff_factor=0.5,
      growth_interval=2000,
      enabled=True,
  ):
      super().__init__(
          init_scale=init_scale,
          growth_factor=growth_factor,
          backoff_factor=backoff_factor,
          growth_interval=growth_interval,
          enabled=enabled,
      )
  
file_path: apex/apex/transformer/amp/grad_scaler.py
imports: from collections import defaultdict
  import torch
  from apex.transformer import parallel_state
label_desc: Use image node_id 1 to create a new GradScaler object from inherited base classes: torch.cuda.amp with example: obj = GradScaler(init_scale, growth_factor, backoff_factor, growth_interval, enabled)