class_docstr: Optimizer that implements the Adafactor algorithm.
  
  Adafactor is described in https://arxiv.org/abs/1804.04235.
  
  Adafactor is most similar to Adam (Kingma and Ba), the major differences are:
  
  1. For a two-dimensional AxB weight matrix, Adafactor uses only A+B auxiliary
     parameters to maintain the second-moment estimator, instead of AB.
     This is advantageous on memory-limited systems.  In addition, beta1
     (momentum) is set to zero by default, saving an additional auxiliary
     parameter per weight.  Variables with >=3 dimensions are treated as
     collections of two-dimensional matrices - factorization is over the final
     two dimensions.
  
  2. Adafactor incorporates "update-clipping" - a scale-invariant analog of
     gradient clipping.  This adds stability
  
  3. Adafactor does not require an external "learning rate".  By default, it
     incorporates a relative-update-scale schedule, corresponding to
     inverse-square-root learning-rate-decay in ADAM.  We hope this works well
     for most applications.
  
  ALGORITHM:
  
  parameter -= absolute_update_scale * clip(grad / grad_scale)
  
  where:
  
    absolute_update_scale := relative_update_scale * parameter_scale
    relative_update_scale := min((step_num + 1)**-0.5, 1e-2)
    parameter_scale := max(rms(var)), epsilon2)
    clip(x) := x / max(1.0, rms(x))
    grad_scale := tf.sqrt(v)   (v is the second-moment estimator)
  
  The second-moment estimator v is maintained in a manner similar to Adam:
  We initialize
  ```
  if var is 2-dimensional:
    v_r <- zeros([num_rows])
    v_c <- zeros([num_cols])
  if var is 0-dimensional or 1-dimensional:
    v <- zeros(shape(var))
  ```
  
  The update rule is as follows:
  ```
  decay_rate = 1 - (step_num + 1) ^ -0.8
  grad_squared = tf.square(grad) + epsilon1
  if var is 2-dimensional:
    v_r <- decay_rate * v_r + (1 - decay_rate) * reduce_mean(grad_squared, 1)
    v_c <- decay_rate * v_c + (1 - decay_rate) * reduce_mean(grad_squared, 0)
    v = outer_prod(v_r, v_c) / reduce_mean(v_r)
  if var is 0-dimensional or 1-dimensional:
    v <- decay_rate * v + (1 - decay_rate) * grad_squared
  ```
  
  For variables with >=3 dimensions, we factorize the second-moment accumulator
  over the final 2 dimensions.  See the code for details.
  
  
  Several parts of this algorithm are configurable from the initializer.
  
    multiply_by_parameter_scale:  If True, then compute absolute_update_scale
      as described above.  If False, let absolute_update_scale be the externally
      supplied learning_rate.
    learning_rate: represents relative_update_scale if
      multiply_by_parameter_scale==True, or absolute_update_scale if
      multiply_by_parameter_scale==False.
    decay_rate: Decay rate of the second moment estimator (varies by step_num).
      This should be set to a function such that:
      1-1/(step_num + 1) <= decay_rate(step_num) < 1.0
    beta1: enables momentum, as in Adam.  Uses extra memory if nonzero.
    clipping_threshold: should be >=1.0 or None for no update clipping
    factored: whether to factor the second-moment estimator.  True means
      less memory usage.
code: def _resource_apply_dense(self, grad, handle):
      var = handle
      grad = tf.to_float(grad)
      grad_squared = tf.square(grad) + self._epsilon1
      grad_squared_mean = tf.reduce_mean(grad_squared)
      decay_rate = self._call_if_callable(self._decay_rate)
      update_scale = self._call_if_callable(self._learning_rate)
      update_scale = tf.convert_to_tensor(
          update_scale, name="update_scale"
      )
      update_scale = tf.cast(
          update_scale, grad_squared_mean.dtype.base_dtype
      )
      old_val = var
      if var.dtype.base_dtype == tf.bfloat16:
          old_val = tf.to_float(
              self._parameter_encoding.decode(old_val)
          )
      if self._multiply_by_parameter_scale:
          update_scale *= tf.to_float(self._parameter_scale(old_val))
      # HACK: Make things dependent on grad.
      # This confounds the XLA rewriter and keeps it from fusing computations
      # across different variables.  This fusion is a bad for HBM usage, since
      # it causes the gradients to persist in memory.
      decay_rate += grad_squared_mean * 1e-30
      update_scale += grad_squared_mean * 1e-30
      # END HACK
      mixing_rate = 1.0 - decay_rate
      shape = var.get_shape().as_list()
      updates = []
      if self._should_use_factored_second_moment_estimate(shape):
          grad_squared_row_mean = tf.reduce_mean(grad_squared, -1)
          grad_squared_col_mean = tf.reduce_mean(grad_squared, -2)
          vr = self.get_slot(var, "vr")
          new_vr = decay_rate * vr + mixing_rate * grad_squared_row_mean
          vc = self.get_slot(var, "vc")
          new_vc = decay_rate * vc + mixing_rate * grad_squared_col_mean
          vr_update = tf.assign(
              vr, new_vr, use_locking=self._use_locking
          )
          vc_update = tf.assign(
              vc, new_vc, use_locking=self._use_locking
          )
          updates = [vr_update, vc_update]
          long_term_mean = tf.reduce_mean(new_vr, -1, keepdims=True)
          r_factor = tf.rsqrt(new_vr / long_term_mean)
          c_factor = tf.rsqrt(new_vc)
          x = (
              grad
              * tf.expand_dims(r_factor, -1)
              * tf.expand_dims(c_factor, -2)
          )
      else:
          v = self.get_slot(var, "v")
          new_v = decay_rate * v + mixing_rate * grad_squared
          v_update = tf.assign(v, new_v, use_locking=self._use_locking)
          updates = [v_update]
          x = grad * tf.rsqrt(new_v)
      if self._clipping_threshold is not None:
          clipping_denom = tf.maximum(
              1.0, reduce_rms(x) / self._clipping_threshold
          )
          x /= clipping_denom
      subtrahend = update_scale * x
      if self._beta1:
          m = self.get_slot(var, "m")
          new_m = (
              self._beta1 * tf.to_float(m)
              + (1.0 - self._beta1) * subtrahend
          )
          subtrahend = new_m
          new_m = common_layers.cast_like(new_m, var)
          updates.append(
              tf.assign(m, new_m, use_locking=self._use_locking)
          )
      new_val = tf.to_float(old_val) - subtrahend
      if var.dtype.base_dtype == tf.bfloat16:
          new_val = self._parameter_encoding.encode(
              new_val, self._quantization_noise
          )
      if self._simulated_quantize_bits:
          new_val = quantization.simulated_quantize(
              var - subtrahend,
              self._simulated_quantize_bits,
              self._quantization_noise,
          )
      new_val = tf.cast(new_val, var.dtype)
      var_update = tf.assign(
          var, new_val, use_locking=self._use_locking
      )
      updates = [var_update] + updates
      return tf.group(*updates)
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensor2tensor.layers import common_layers
  from tensor2tensor.utils import quantization
  import tensorflow.compat.v1
label_desc: Use image node_id 8 for calling the AdafactorOptimizer obj's underlying member method code with example usage: obj._resource_apply_dense(grad, handle) and returns: tf