class_docstr: An instance ``scaler`` of :class:`GradScaler`.
  
  Helps perform the steps of gradient scaling
  conveniently.
  
  * ``scaler.scale(loss)`` multiplies a given loss by ``scaler``'s current scale factor.
  * ``scaler.step(optimizer)`` safely unscales gradients and calls ``optimizer.step()``.
  * ``scaler.update()`` updates ``scaler``'s scale factor.
  
  Example::
  
      # Creates a GradScaler once at the beginning of training.
      scaler = GradScaler()
  
      for epoch in epochs:
          for input, target in data:
              optimizer.zero_grad()
              output = model(input)
              loss = loss_fn(output, target)
  
              # Scales loss.  Calls backward() on scaled loss to create scaled gradients.
              scaler.scale(loss).backward()
  
              # scaler.step() first unscales gradients of the optimizer's params.
              # If gradients don't contain infs/NaNs, optimizer.step() is then called,
              # otherwise, optimizer.step() is skipped.
              scaler.step(optimizer)
  
              # Updates the scale for next iteration.
              scaler.update()
  
  See the :ref:`Automatic Mixed Precision examples<amp-examples>` for usage
  (along with autocasting) in more complex cases like gradient clipping, gradient accumulation, gradient penalty,
  and multiple losses/optimizers.
  
  ``scaler`` dynamically estimates the scale factor each iteration.  To minimize gradient underflow,
  a large scale factor should be used.  However, ``float16`` values can "overflow" (become inf or NaN) if
  the scale factor is too large.  Therefore, the optimal scale factor is the largest factor that can be used
  without incurring inf or NaN gradient values.
  ``scaler`` approximates the optimal scale factor over time by checking the gradients for infs and NaNs during every
  ``scaler.step(optimizer)`` (or optional separate ``scaler.unscale_(optimizer)``, see :meth:`unscale_`).
  
  * If infs/NaNs are found, ``scaler.step(optimizer)`` skips the underlying ``optimizer.step()`` (so the params
    themselves remain uncorrupted) and ``update()`` multiplies the scale by ``backoff_factor``.
  
  * If no infs/NaNs are found, ``scaler.step(optimizer)`` runs the underlying ``optimizer.step()`` as usual.
    If ``growth_interval`` unskipped iterations occur consecutively, ``update()`` multiplies the scale by
    ``growth_factor``.
  
  The scale factor often causes infs/NaNs to appear in gradients for the first few iterations as its
  value calibrates.  ``scaler.step`` will skip the underlying ``optimizer.step()`` for these
  iterations.  After that, step skipping should occur rarely (once every few hundred or thousand iterations).
  
  Args:
      init_scale (float, optional, default=2.**16):  Initial scale factor.
      growth_factor (float, optional, default=2.0):  Factor by which the scale is multiplied during
          :meth:`update` if no inf/NaN gradients occur for ``growth_interval`` consecutive iterations.
      backoff_factor (float, optional, default=0.5):  Factor by which the scale is multiplied during
          :meth:`update` if inf/NaN gradients occur in an iteration.
      growth_interval (int, optional, default=2000):  Number of consecutive iterations without inf/NaN gradients
          that must occur for the scale to be multiplied by ``growth_factor``.
      enabled (bool, optional):  If ``False``, disables gradient scaling. :meth:`step` simply
          invokes the underlying ``optimizer.step()``, and other methods become no-ops.
          Default: ``True``
class_docstr_tok: ["An","instance","``","scaler","``","of",":","class",":","`","GradScaler","`",".","Helps","perform","the","steps","of","gradient","scaling","conveniently",".","*","``","scaler.scale","(","loss",")","``","multiplies","a","given","loss","by","``","scaler","``","'s","current","scale","factor",".","*","``","scaler.step","(","optimizer",")","``","safely","unscales","gradients","and","calls","``","optimizer.step","(",")","``",".","*","``","scaler.update","(",")","``","updates","``","scaler","``","'s","scale","factor",".","Example",":",":","#","Creates","a","GradScaler","once","at","the","beginning","of","training",".","scaler","=","GradScaler","(",")","for","epoch","in","epochs",":","for","input",",","target","in","data",":","optimizer.zero_grad","(",")","output","=","model","(","input",")","loss","=","loss_fn","(","output",",","target",")","#","Scales","loss",".","Calls","backward","(",")","on","scaled","loss","to","create","scaled","gradients",".","scaler.scale","(","loss",")",".backward","(",")","#","scaler.step","(",")","first","unscales","gradients","of","the","optimizer","'s","params",".","#","If","gradients","do","n't","contain","infs\/NaNs",",","optimizer.step","(",")","is","then","called",",","#","otherwise",",","optimizer.step","(",")","is","skipped",".","scaler.step","(","optimizer",")","#","Updates","the","scale","for","next","iteration",".","scaler.update","(",")","See","the",":","ref",":","`","Automatic","Mixed","Precision","examples","<","amp-examples",">","`","for","usage","(","along","with","autocasting",")","in","more","complex","cases","like","gradient","clipping",",","gradient","accumulation",",","gradient","penalty",",","and","multiple","losses\/optimizers",".","``","scaler","``","dynamically","estimates","the","scale","factor","each","iteration",".","To","minimize","gradient","underflow",",","a","large","scale","factor","should","be","used",".","However",",","``","float16","``","values","can","``","overflow","''","(","become","inf","or","NaN",")","if","the","scale","factor","is","too","large",".","Therefore",",","the","optimal","scale","factor","is","the","largest","factor","that","can","be","used","without","incurring","inf","or","NaN","gradient","values",".","``","scaler","``","approximates","the","optimal","scale","factor","over","time","by","checking","the","gradients","for","infs","and","NaNs","during","every","``","scaler.step","(","optimizer",")","``","(","or","optional","separate","``","scaler.unscale_","(","optimizer",")","``",",","see",":","meth",":","`","unscale_","`",")",".","*","If","infs\/NaNs","are","found",",","``","scaler.step","(","optimizer",")","``","skips","the","underlying","``","optimizer.step","(",")","``","(","so","the","params","themselves","remain","uncorrupted",")","and","``","update","(",")","``","multiplies","the","scale","by","``","backoff_factor","``",".","*","If","no","infs\/NaNs","are","found",",","``","scaler.step","(","optimizer",")","``","runs","the","underlying","``","optimizer.step","(",")","``","as","usual",".","If","``","growth_interval","``","unskipped","iterations","occur","consecutively",",","``","update","(",")","``","multiplies","the","scale","by","``","growth_factor","``",".","The","scale","factor","often","causes","infs\/NaNs","to","appear","in","gradients","for","the","first","few","iterations","as","its","value","calibrates",".","``","scaler.step","``","will","skip","the","underlying","``","optimizer.step","(",")","``","for","these","iterations",".","After","that",",","step","skipping","should","occur","rarely","(","once","every","few","hundred","or","thousand","iterations",")",".","Args",":","init_scale","(","float",",","optional",",","default=2",".","*","*","16",")",":","Initial","scale","factor",".","growth_factor","(","float",",","optional",",","default=2.0",")",":","Factor","by","which","the","scale","is","multiplied","during",":","meth",":","`","update","`","if","no","inf\/NaN","gradients","occur","for","``","growth_interval","``","consecutive","iterations",".","backoff_factor","(","float",",","optional",",","default=0.5",")",":","Factor","by","which","the","scale","is","multiplied","during",":","meth",":","`","update","`","if","inf\/NaN","gradients","occur","in","an","iteration",".","growth_interval","(","int",",","optional",",","default=2000",")",":","Number","of","consecutive","iterations","without","inf\/NaN","gradients","that","must","occur","for","the","scale","to","be","multiplied","by","``","growth_factor","``",".","enabled","(","bool",",","optional",")",":","If","``","False","``",",","disables","gradient","scaling",".",":","meth",":","`","step","`","simply","invokes","the","underlying","``","optimizer.step","(",")","``",",","and","other","methods","become","no-ops",".","Default",":","``","True","``"]
code: def _unscale_grads_(
      self,
      optimizer: torch.optim.Optimizer,
      inv_scale: torch.Tensor,
      found_inf: torch.Tensor,
      allow_fp16: bool,
  ) -> Dict[torch.device, torch.Tensor]:
      per_device_inv_scale = _MultiDeviceReplicator(inv_scale)
      per_device_found_inf = _MultiDeviceReplicator(found_inf)
  
      # To set up _amp_foreach_non_finite_check_and_unscale_, split grads by device and dtype.
      # There could be hundreds of grads, so we'd like to iterate through them just once.
      # However, we don't know their devices or dtypes in advance.
  
      # https://stackoverflow.com/questions/5029934/defaultdict-of-defaultdict
      # Google says mypy struggles with defaultdicts type annotations.
      per_device_and_dtype_grads: Dict[
          torch.device, Dict[torch.dtype, List[torch.Tensor]]
      ] = defaultdict(lambda: defaultdict(list))
      with torch.no_grad():
          for group in optimizer.param_groups:
              for param in group["params"]:
                  assert isinstance(param, torch.Tensor)
                  if param.grad is None:
                      continue
                  if (
                      not allow_fp16
                  ) and param.grad.dtype == torch.float16:
                      raise ValueError(
                          "Attempting to unscale FP16 gradients."
                      )
                  if param.grad.is_sparse:
                      # is_coalesced() == False means the sparse grad has values with duplicate indices.
                      # coalesce() deduplicates indices and adds all values that have the same index.
                      # For scaled fp16 values, there's a good chance coalescing will cause overflow,
                      # so we should check the coalesced _values().
                      if param.grad.dtype is torch.float16:
                          param.grad = param.grad.coalesce()
                      to_unscale = param.grad._values()
                  else:
                      to_unscale = param.grad
  
                  # TODO: is there a way to split by device and dtype without appending in the inner loop?
                  per_device_and_dtype_grads[to_unscale.device][
                      to_unscale.dtype
                  ].append(to_unscale)
  
          for (
              device,
              per_dtype_grads,
          ) in per_device_and_dtype_grads.items():
              for grads in per_dtype_grads.values():
                  torch._amp_foreach_non_finite_check_and_unscale_(
                      grads,
                      per_device_found_inf.get(device),
                      per_device_inv_scale.get(device),
                  )
  
      return per_device_found_inf._per_device_tensors
  
imports: from __future__ import annotations
  import inspect
  import warnings
  from collections import abc, defaultdict
  from enum import Enum
  from typing import Any, cast, Dict, Iterable, List, Optional, overload, Tuple, Union
  import torch
  from .common import amp_definitely_not_available
label_desc: Use image node_id 9 for calling the GradScaler obj's underlying member method code with example usage: obj._unscale_grads_(optimizer, inv_scale, found_inf, allow_fp16) and returns: per_device_found_inf