class_docstr: A wrapper over pytorch GradScaler made specifically for training hivemind.Optimizer with reuse_grad_buffers=True.
  
  :note: if not using reuse_grad_buffers=True, one can and *should* train normally without this class, e.g. using
    standard PyTorch AMP or Apex. This custom GradScaler is more memory-efficient, but requires custom training code.
  
  hivemind.GradScaler makes 3 modifications to the regular PyTorch AMP:
  
  - bypass .unscale_ and .update calls in order to accumulate gradients over several steps
  - limit increasing gradient scale to only immediately after global optimizer steps
  - allow training with some or master parameters in float16
  
  :note: The above modiffications will be enabled automatically. One can (and should) use hivemind.GradScaler exactly
    as regular ``torch.amp.GradScaler``.
code: def __init__(self, *args, **kwargs):
      super().__init__(*args, **kwargs)
      self._is_running_global_step = False
      self._is_ready_to_update = False
      self._inner_optimizer_states = {}
      self._optimizer_states_to_reset = set()
      self._lock = threading.RLock()
  
imports: import contextlib
  import threading
  from copy import deepcopy
  from typing import Dict, Optional
  import torch
  from torch.cuda.amp import GradScaler
  from torch.cuda.amp.grad_scaler import OptState, _refresh_per_optimizer_state
  from torch.optim import Optimizer
  import hivemind
  from hivemind.utils.logging import get_logger
label_desc: Use image node_id 1 to create a new GradScaler object from inherited base classes: TorchGradScaler with example: obj = GradScaler()