class_docstr: Dynamically adjusts the loss scaling factor.
  
  Dynamic loss scalers are important in mixed-precision training. They help
  us avoid underflows and overflows in low-precision gradients.
  
  See here for information:
  <https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#lossscaling>
  
  Shamelessly stolen and adapted from Fairseq.
  <https://github.com/pytorch/fairseq/blob/main/fairseq/optim/fp16_optimizer.py>
class_docstr_tok: ["Dynamically","adjusts","the","loss","scaling","factor",".","Dynamic","loss","scalers","are","important","in","mixed-precision","training",".","They","help","us","avoid","underflows","and","overflows","in","low-precision","gradients",".","See","here","for","information",":","<","https",":","\/\/docs.nvidia.com\/deeplearning\/performance\/mixed-precision-training\/index.html","#","lossscaling",">","Shamelessly","stolen","and","adapted","from","Fairseq",".","<","https",":","\/\/github.com\/pytorch\/fairseq\/blob\/main\/fairseq\/optim\/fp16_optimizer.py",">"]
docstr: :param init_scale:
      Initial loss scale.
  :param scale_factor:
      Factor by which to increase or decrease loss scale.
  :param scale_window:
      If we do not experience overflow in scale_window iterations,
      loss scale will increase by scale_factor.
  :param tolerance:
      Pct of iterations that have overflowed after which we must
      decrease the loss scale
  :param threshold:
      If not None, loss scale will decrease below this threshold
code: def __init__(
      self,
      init_scale: float = 2.0**15,
      scale_factor: float = 2.0,
      scale_window: int = 2000,
      tolerance: float = 0.00,
      threshold: float = None,
  ):
      """
      :param init_scale:
          Initial loss scale.
      :param scale_factor:
          Factor by which to increase or decrease loss scale.
      :param scale_window:
          If we do not experience overflow in scale_window iterations,
          loss scale will increase by scale_factor.
      :param tolerance:
          Pct of iterations that have overflowed after which we must
          decrease the loss scale
      :param threshold:
          If not None, loss scale will decrease below this threshold
      """
      self.loss_scale = init_scale
      self.scale_factor = scale_factor
      self.scale_window = scale_window
      self.tolerance = tolerance
      self.threshold = threshold
  
      self._iter = 0
      self._last_overflow_iter = -1
      self._last_rescale_iter = -1
      self._overflows_since_rescale = 0
  
imports: import math
  from itertools import chain
  from typing import Optional
  import parlai.utils.logging
label_desc: Use image node_id 1 to create a new DynamicLossScaler object from inherited base classes: object with example: obj = DynamicLossScaler(init_scale, scale_factor, scale_window, tolerance, threshold)